The Method for Reporting Results

Composite results are summarized for the 3-D wireframe (PLBwire93) and the 3-D surface (PLBsurf93) files. The PLB_lit and PLB_opt numbers for "oceantopo," currently the only file in the "other" category, are also included in the summaries. In addition, a report page listing results for all standard benchmark files is provided for each system tested.

Performance results for each standard benchmark file are reported using two numbers separated by a colon. The number on the left is designated as "PLB_lit." It is a literal number aimed at graphics users who want to measure performance for the same graphics entities from one platform to another and who are unable or unwilling to tune their applications to a particular hardware system or graphics interface. The PLB_lit number reflects an application file that is run without optimizations. This number might be better suited to some hardware architectures than others; that is why the PLB subcommittee decided on using two numbers to measure performance.

The number on the right is designed as "PLB_opt." It is an optimized number for those users who are willing to tune their applications to achieve better performance on specific hardware platforms with specialized graphics interfaces. The PLB_opt number shows the best possible performance for a vendor's specialized hardware configuration and application programmer's interface (API).

A typical performance number for a standard benchmark file, such as "sys_chassis," looks like this:

The left number represents the literal (PLB_lit) number, the right the optimized (PLB_opt) number. "NP" on the left side of the colon indicates that is not possible for the platform hardware or software to execute the benchmark file's graphics entities under the PLB_lit guidelines. In this case, the vendor has chosen not to emulate the functionality in its PLB port. "NA" on the right side of the colon means an optimized number is not available at this time.

If the optimized number, for example, is not reported, the number in the report would look like this:

If no number is reported for a particular benchmark file, this is represented by "NR."

When optimizations are made, they are listed at the bottom of the PLB Report page. A PLB Report is published for each hardware configuration for which the vendor submits performance numbers.

Performance numbers are derived from a normalized constant that is divided by the elapsed time in seconds required to perform the test. A high number represents better performance (i.e., fewer seconds to display the picture under test).

Some of the PLB Report pages might include a note regarding conversion time ratio in the "Comments/Notes" section. If a ratio of "2" or more is reported, it indicates that it takes a significant amount of time to generate optimizations used to obtain the PLB_opt number the vendor reported for a particular standard benchmark file.

Although PLB subcommittee members agree that no single figure of merit can adequately describe the performance characteristics of graphics systems across all applications, the subcommittee currently provides composite numbers in two categories. The subcommittee believes that graphics characteristics within these categories are similar enough to make a single figure of merit credible.

The single figure of merit number is computed by taking the geometric mean for both the PLB_lit and PLB_opt numbers for standard benchmark files in a given category. If an "NR" is reported for a given benchmark file within a category, the submitter must report an "NR" for the single figure of merit; numbers can still be reported, however, for individual benchmark files within that category. Where an "NP" is listed for a PLB_lit number, it is replaced with a value that is one-third that of the optimized number in order to calculate the single figure of merit for the category. Where "NA" (not available) is listed for a PLB_opt number, the PLB_opt value is considered the same as the PLB_lit value for the particular standard benchmark file.

The reporting format used in this report is a summary of the detailed report generated by the Benchmark Reporting Format (BRF) component of the PLB. This standardized report lists data about the hardware and software configuration of the system tested and summarizes key characteristics of the test images. Included in the summary is data about the average number of primitives displayed, the number of times called per frame, data pertaining to graphics attributes and matrix operations, and a global exceptions table. Test loop timing information lists the number of frames displayed, elapsed time, and average time per frame.

The BRF provides enough details about the test to enable users to accurately compare systems. Should a vendor implement a test substantially different from a competitor's, the detailed data is likely to show how the discrepancy occurred.