We used six benchmarks to test our implementation. Five of these are from Coumeri and Thomas . This suite consists of small benchmarks that vary in hierarchy depth, partitioning, and operators used. The benchmarks include a series of test vectors and the vectors are run up to 1,000,000 times to produce measurable run times. The arms_counterbenchmark is particularly interesting because it contains unclocked feedback. This suite also contains five other benchmarks that we are not using. They are all variations on a 64-bit bit-wise adder, and our prototype does not yet support efficient 64-bit operations. The final benchmark, MIPS-Lite, is a simple behavioral description of a MIPS-compatible processor.
We compared the performance of our simulator with VCS 2.3 from Chronologic Simulation, a state-of-the-art commercial simulator. Both VCS and VeriSUIF generate C code. In all cases, the generated code was compiled with the MIPS C compiler v3.19 with -O2. The simulations were executed on a Silicon Graphics Indigo with 64MB of memory and a 100 MHz R4000 processor. Scheduling overhead was measured using the pixie tool. The results are shown in Table 1.
The overhead we show for VCS indicates the amount of time spent in the run-time library manipulating the event queue. VCS does some optimizations to reduce run-time overhead by bypassing the event queue in some cases, such as simple signal propagation. In these cases it uses function calls to activate events. The time spent performing these function calls is not included in our measurements. The overhead for VeriSUIF shows the amount of time spent in the code that sets and tests triggers.
The absolute run time is shown as well as the time spent performing scheduling tasks. When looking at the absolute times, it is important to realize that VCS has a highly tuned code generator, and in some cases generates better code than VeriSUIF. This is apparent on the gcd benchmarks where our poorer computational code is the sole reason for the reduced performance. However, even with this disadvantage, we still run almost two times faster on average. More interesting is the time spent performing scheduling tasks. Here we can see that the time spent performing event queue management in VCS can be substantial. On average, we spend only 4%as much time as VCS in scheduling overhead.
It is also interesting to measure the benefit of tracking variable values during static simulation. A comparison is presented in Table 2. The percentage of scheduling states that do not require a run-time test nearly double with variable tracking, from 40%to 76%, with several requiring no tests for the entire simulation (the 40%of states that are marked even without variable tracking are due to unconditional control flow and delays). As a result of eliminating these run-time tests, variable tracking was able to reduce scheduling overhead on all the benchmarks by an average of 1.9 times. In many cases this is because the benchmark used a series of constant test vectors and the compiler was able to determine which events must trigger during each test. Also, in the arms_counterand MIPS-Lite benchmarks there is a top level clock and the compiler can eliminate run-time tests for events that depend on the clock edges. However, there is a disadvantage to using variable tracking: the number of static states can increase significantly. This is because it takes longer to detect a cycle when variable values are taken into account.