SPECweb2005 Release 1.30 User's Guide

Version 1.30, Last modified 2009-11-20



1.0 Requirements

These instructions assume that you are familiar with the following:

Web server Software configuration
Server and Network configuration
Installing and Using Open Source Software
Installing a JVM and running Java programs

The SPECweb2005 kit contains the following components:

The SPECweb2005 kit contains the following components:

Additional Software Requirements:

Hardware Requirements:

Note: For debugging purposes, you could run everything on 1 or 2 systems but these instructions assume you have at least 3.


2.0 Setting Up The Test Environment

2.1 Running the SPECweb2005 installer

Place a copy of the provided setup.jar or setup.exe on all systems that are part of your test setup. You will need to have a JVM installed; a JDK or JRE version 1.4.1 or later is recommended.  Invoke the Java installer as follows:

Follow the menu prompts and select the type of installation appropriate for the role of the system being installed.  Sample output from the Java installer is shown below:

SPECweb2005 - InstallShield Wizard
Choose the installation type that best suits your needs.
[X] 1 - Client / Prime Client Installation
The benchmark Java source code and class files for running the prime
client and client harness. Requires a Java Virtual Machine (JVM).
Benchmark documentation is included as well in this installation type.
[ ] 2 - Web server Installation
The Web server php, jsp and aspx script implementations that run on the System
Under Test (SUT), file set generator tool for script paddings, images,
and static content (requires a Java Virtual Machine), and benchmark
documentation are included in this installation type.

[ ] 3 - Backend Simulator (BeSim) Installation
Backend Simulator (BeSim); a Web server API that simulates an application
or database server. Must be installed on a separate machine than the SUT.
Benchmark documentation is included as well in this installation type.
[ ] 4 - Full Installation
The program will be installed with the suggested configuration.
Recommended for most users.
[ ] 5 - Custom
The program will be installed with the features you choose.
Recommended for advanced users.
 
 

If the install of the SPECweb2005 components fail with the following message:

Errors occurred during the installation.

  - JVM not found

Please follow the directions below:

If installing with Java 1.5 ( JVM 5), set the JAVA_HOME variable to the root path of your JVM engine.

        Ex: export JAVA_HOME=/usr/path_to_java

There are known capability issues with SPECweb2005 installer and Java 1.6 (JVM 6), please use Java 1.5 or earlier to complete the install.

 

2.2 Web Server Setup

General Directions:

  1. Set localization on the server system to US.
  2. Install and configure your favorite Web server with SSL and PHP, JSP or ASP.NET support configured in.  If using JSP, a separate Java application server may be needed. Check your Web server software documentation for assistance.
  3. Run the SPECweb2005 installer (see above) and set your current working directory to the wafgen directory.
  4. If you plan to use the JSP dynamic scripts provided:
  5. If you plan to use the PHP dynamic scripts provided:
  6. If you plan to use the ASP.NET dynamic scripts provided:
  7. Review the Wafgen README which contains the instructions for tailoring the resource files (.rc) to create the workload specific file sets for the Web server.
  8. Edit the .rc files for the workload(s) that you wish build the file sets for.  Ensure that at the very least you set SIMULTANEOUS_SESSIONS and DOCROOT to the correct values.
  9. Run wafgen according to the directions provided.
  10. Check that the files generated are correct and under your Web server’s document root.

2.3 BeSim Setup

  1. Configure the BeSim system with Web server software installed on port 81 or another port (avoid port 80 to avoid confusion with the primary Web server).
  2. Run the SPECweb2005 installer, do a "BeSim" or "Full" installation, and change your current working directory to the <specweb_install_directory>/besim directory.
  3. Review the BeSim Make_Readme.
    NOTE: For Windows/IIS, a precompiled ISAPI is provided (BeSim.dll); copy this to a directory with ISAPI execute permissions, and skip to step 6.
  4. You may choose from FastCGI* for Zeus or Apache or ISAPI for Zeus or IIS or NSAPI for SUN One and edit the appropriate MakeIncl.<type>.<OS> file to reflect your compile and linker options and select one or more Defines.  Then you can build the BeSim code for your selected API.
  5. Install this code to your BeSim Web server.
  6. Start your BeSim Web server.
  7. Use the test_besim_bank, test_besim_ecom, and test_besim_support scripts (included in the "besim" subdirectory of the installation) to test whether you're getting valid BeSim responses.  Both Perl scripts and Linux bash scripts are provided.  The Perl scripts require a Perl interpreter as well as the following Perl modules: URI, HTML-Tagset, HTML-Parser, and libwww-perl.  The bash scripts require cURL to be in the path.

    Invoke these scripts with the URL to your compiled BeSim API, i.e.
    perl test_besim_bank.pl http://192.2.1.132:81/fcgi-bin/besim_fcgi.fcgi
    bash test_besim_bank.sh http://bsim614:81/fcgi-bin/besim_fcgi.fcgi

*Note: For more FastCGI instructions (particularly for the Apache HTTP Server), see Appendix G.

If you notice that one BeSim system has a bottleneck (CPU, disk, network, etc.), there is now (as of 1.10 and above) an option to use multiple physical BeSim systems.  Follow the same steps above to install BeSim on another system, and specify multiple BESIM_SERVERs (delimited by spaces) in Test.config.  See the example Test.config files for more information.

2.4 Client Setup

  1. Install a 1.4.n JVM (or later) on all client systems.
  2. Run the SPECweb2005 installer on each client and set your current directory to the SPECweb2005 directory created by the installation.
  3. Select one of the client systems to serve as the "Prime Client" (can also be a load generating client).
  4. On the Prime Client, decide on an appropriate set of configuration files for your particular system.  For a Red Hat Linux server running Apache and PHP, you should use the *.Unix-PHP.config files.  Copy or rename these files to SPECweb_Banking.config, SPECweb_Ecommerce.config, SPECweb_ Support .config, and Test.config, i.e.

    # cp SPECweb_Banking.Unix-PHP.config SPECweb_Banking.config
    # cp SPECweb_Ecommerce.Unix-PHP.config SPECweb_Ecommerce.config
    # cp SPECweb_ Support .Unix-PHP.config SPECweb_ Support .config
    # cp Test.Unix-PHP.config Test.config
     
  5. Review the SPECweb_<workload>.<OS/Web server>-<Script Engine>.config, Test.<OS/Web server>-<Script Engine>.config, and Testbed.config to understand the parameters in these files, with attention to those in the beginning (configurable) section of the workload and Test configuration files which may need to be updated to match your test bed.
  6. Edit configuration files to match your Web server, BeSim configuration, Client, and network configuration.  Change only those parameters listed above the section marked "# Fixed workload properties" or in Test.config above the line marked " # SPECweb2005 fixed test properties":

2.5 Network Setup

Configure all the test bed systems so that they can communicate with one another; make sure to update your /etc/hosts file (or equivalent).  The Prime must be able to communicate with each Client in the test environment.  The Web server and the BeSim system must be able to communicate with each other.   Finally, the Prime Client and BeSim system(s) must be able to communicate with each other. 


3.0 Running SPECweb2005

Once the Web server, clients, and BeSim system have been configured and networked, the next step is to try running a test strobe for one of the workloads.  To start with a short test, edit Test.config again and put these lines at the end of the file to override the defaults:

KILL_CLIENT = 1
DEBUG_LEVEL = 2
RAMPUP_SECS = 60
WARMUP_SECONDS = 60
RUN_SECONDS = 300
SIMULTANEOUS_SESSIONS = 1

Also set TEST_TYPE to the desired workload (SPECweb_Banking, SPECweb_Ecommerce, or SPECweb_ Support ).

Next start the test using the commands below.  Note, you may want to use 2 windows or redirect the output of the two Java commands to files or create a script run the harness and to save the output:

After test has completed check the redirected output and the contents of the results.

Note: you may wish to set other Java parameters besides the heap parameters to optimize the specwebclient processes as you increase the test load.  Check the documentation for your JVM.

Did that test work? 

Yes? Great, so did these instructions “work” or were there missing details or do you have additional questions that you are now writing up to send to web2005support@spec.org?

No? Well, there should be enough information in the debug output to start debugging. Please send any corrections or requests for clarification to these instructions to the email address above.  Continue reading and reviewing the Web server,  BeSim, and Client Checks below, along with the related README files for the various components in the attached appendixes, particularly Appendix E for common errors and Appendix F for the Support FAQ.

Now, that the basics are working,  you can try tests using more Sessions and adding more clients.

3.1 Web Server Checks

1.      Have you configured your Web server to support the HTTP or HTTPS based on the workload you are running?  Banking is all HTTPS (SSL), Ecommerce requires both HTTP and HTTPS, and Support uses only HTTP.  Note: that you can temporarily override the use of HTTPS in Banking and Ecommerce by setting USE_SSL = 0 in the .config file.  Results will be noncompliant but this may be useful when debugging. 

2.      Have you configured you Web server to support PHP, JSP or ASP.NET according to the directions provided with your Web server?  If using JSP and a separate Java application server such as Tomcat also review its documentation and configure appropriately.  Try running the PHP, JSP or ASP.NET "Hello World" equivalent to make sure your Web server's script engine is working. 

3.      Have you configured the wafgen .rc files for the workload and run Wafgen to create the fixed and scaling file sets? Did you set the number of SIMULTANEOUS_SESSIONS to a value equal or greater than the value you plan to use in your testing? 

4.      Can the Web server system ping the BeSim system?  Connect to the BeSim Web server (telnet [BESIM_IP] [BESIM_port])?

3.2 BeSim Checks

  1. Have you installed and configured a Web server on you BeSim system to run on port 81 (or other non-standard port number)?
  2. Have you built and installed the BeSim application (FastCGI, NSAPI, ISAPI) in the appropriate directory under the Web server's document root?
  3. Have you run the test_besim_* scripts provided with BeSim to check that the BeSim setup is working?  If not, see section 2.3.

3.3 Prime Client Checks

  1. Have you checked the SPECweb_<test type>.config?  Verify that BESIM_INIT_SCRIPT matches the relative path of the BeSim API installed on the BeSim Web server.
  2. Have you checked Test.config? Verify the following parameters match your test environment:

TEST_TYPE=SPECweb_<test type>
WEB_SERVER = <SUT>
BESIM_SERVER = <BeSim system>
BESIM_PORT = 81 [or the non-80 value set up on the BeSim Web server]
 

  1. Verified that the Prime Client can “ping” the other clients, the SUT and BeSim system?
  2. Made a quick check that the prime and other clients can successfully do a “telnet [WEB_SERVER_IP] [80 and/or 443]” and that you can successfully perform a "telnet [BESIM_IP] [BESIM_port]" from the prime?
  1. Checked that the prime (and other clients) have set per process limits for threads (i.e. max_thread_proc) and file descriptors (i.e. maxfiles) adequately for the number of simultaneous sessions you plan to test?  For example, if you are running a test for 1500 simultaneous sessions and using 2 clients, each client is responsible for 750 sessions.  Each session uses ~3 connections to the server, so each client process (i.e. java -jar specwebclient) will need just over 750 * 3 threads or 2100+ threads and 750 * 3 file descriptors (socket = FD) or 2100 file descriptors.  If your clients’ settings aren’t adequate, then your test will fail and you’ll get a Java error telling you it couldn’t start a thread or establish a connection.

3.4 Collecting Statistics (optional)

Starting with version 1.10, the prime client has the ability to poll clients for data at specified intervals during a run and display it as CSV or in a GUI.  Data is collected and displayed only during the warm-up and benchmark run phases.  The following configuration parameters in Test.config control the collection and display of this information:

3.5 Optimizing Performance

Once the SPECweb2005 benchmark is running the workload(s) without errors, you will want to optimize your testbed for optimal performance.  Here are some tips:


Appendix A - README file from SPECweb2005 client kit

Here are the basics for setting up SPECweb2005 Clients:
 
First, you should at minimum have version 1.4.1 of the java jdk and/or jre 
installed on the client machines running this code to take advantage of garbage 
collection optimizations in the more recent jvms. JVM v1.4.n is strongly 
recommended for best performance. However, older jvm versions back to 1.3.1 
may work.
 
Next, there are five different .config files used for SPECweb2005. There is a 
SPECweb_<workload>.config for each of the three workloads (Banking, Ecommerce, 
and Support) that contain both configurable and fixed parameters specific to 
the workload. Additionally, Test.config contains parameters that are common 
to all workloads and includes configurable and fixed parameters. Lastly, 
Testbed.config contains the testbed hardware/software configuration properties.
 
Note that Test.config and the workload-specific configuration files mentioned 
above do not exist in a new installation. Instead, example workload and Test 
configuration files are provided, and you will need to copy the sample file 
most suited to your environment. Ex:
 
cp Test.Unix-PHP.config Test.config
cp SPECweb_Banking.Unix-PHP.config SPECweb_Banking.config
cp SPECweb_Ecommerce.Unix-PHP.config SPECweb_Ecommerce.config
cp SPECweb_Support.Unix-PHP.config SPECweb_Support.config
 
"specwebclient" is the SPECweb2005 client load generator. After editing the 
.config files to match your test environment, you need to start this process on 
all client machines. From the directory where specwebclient.jar is installed, 
and at a command prompt type:
   
         java -jar specwebclient.jar 
 
Starting "specwebclient" with tuned heap sizes and garbage collection 
parameters is also recommended. Specifically, for v1.4.1 of the jdk on a 
multi-processor client, using parallel and CMS garbage collection is strongly 
recommended. Example: 
 
        java -cp c:\sw200x -Xms512m -Xmx512m -XX:+UseParNewGC \
               -XX:+UseConcMarkSweepGC -jar specwebclient.jar
 
The usage for specwebclient is:
 
java -jar specwebclient.jar  -h
Usage: java -jar specwebclient.jar [-v] [-p port] [-n hostname] 
        [-s servername] [-le errorlogfilename] [-lo stdoutlogfilename]
 
Command line flags:
   -v  Print version string and exit
   -p <port> Override default port number 
               (allows multiple instances on same machine)
   -n <hostname>  Override hostname (use Alias)
   -s <servername> Overrides WEB_SERVER
   -le <errorlogfilename>  redirects error messages
   -lo <stdoutlogfilename> redirects other logging messages
 
 
"specweb" is the SPECweb2005 test manager and is run on the Prime Client.  It
is started after all the "specwebclient" processes have be started.  Starting 
specweb should begin the test.  The "specweb" process exits at the end of the 
test, and the "specwebclient" process can be configured to terminate as well 
(see Test.config).  To start the "specweb" process, from the directory where 
specweb.jar is installed, at a command prompt type:
 
         java -jar specweb.jar
 
The usage for specweb is:
 
java -jar specweb.jar  -h
Usage: java -jar specweb.jar [-C] [-v] [-w workloadname] [-le errorlogfilename] 
       [-lo stdoutlogfilename]
 
Command line flags:
   -h  Print this message and exit
   -v  Print version string and exit
   -C  Overrides any non-compliant settings in the .config files with the
       default compliant values
   -w  Overides the TEST_TYPE in the config file (SPECweb_Banking, 
       SPECweb_Ecommerce, SPECweb_Support)
   -le <errorlogfilename>  redirects error messages
   -lo <stdoutlogfilename> redirects other logging messages
 
 
"reporter" is the SPECweb2005 report generator and is invoked by "specweb" at
the end of a test to produce .TXT and .HTML reports.  It can also be invoked to
create a merged .raw file from results from running the 3 workloads on a given
testbed for a submission to SPEC; as well as generating  combined .TXT and 
.HTML report files. Here is the usage for the reporter:
 
java -jar reporter.jar -h
SPECweb2005 Reporter 1.10
Copyright (c) 2005 Standard Performance Evaluation Corporation
 
Command line flags:
   -h  Print this message and exit
   -v  Print version string and exit
   -c  Combine 3 separate workload files into one submittable .raw file
   -R  Regenerate HTML and ASCII reports from combined rawfile
 
Example Usage:
--------------
Regenerate reports from a single raw file:
java -jar reporter.jar results\SPECweb_Support.20050323-102204.raw
 
Combine three compliant workload raw files into one submittable raw file:
java -jar reporter.jar -c bank.raw ecom.raw support.raw submit.raw
 
Regenerate reports from a previously combined raw file:
java -jar reporter.jar -R submit.raw
 

Appendix B - Examples of common changes to configurable parameters


SPECweb_Banking.config:

36c36

< PADDING_DIR = "/www/webroot/bank/dynamic_padding/"

---

> PADDING_DIR = "/www/bank/dynamic_padding/"

47,48c47,48

< CHECK_IMAGE_DIR = "c:/www/webroot/bank/images"

< CHECK_IMAGE_SUBDIRS = 10

---

> CHECK_IMAGE_DIR = "/www/bank/images"

> CHECK_IMAGE_SUBDIRS = 100


SPECweb_Ecommerce.config:
36c36
< PADDING_DIR = "/www/webroot/ecommerce/dynamic_padding/"
---
> PADDING_DIR = "/www/ecommerce/dynamic_padding/"

SPECweb_ Support .config:

40c40
< PADDING_DIR = "c:/www/webroot/support/dynamic_padding/"
---
> PADDING_DIR = "/www/support/dynamic_padding/"
50c50
< DOWNLOAD_SUBDIRS = 0
---
> DOWNLOAD_SUBDIRS = 2


Test.config:

30c30

< CLIENTS = "localhost"

---

> CLIENTS = "client50:1095 client40:1094 client30:1093 client20:1092"

32c32

< SIMULTANEOUS_SESSIONS = 5

---

> SIMULTANEOUS_SESSIONS = 1100

35,36c35,36

< TEST_TYPE=SPECweb_Banking

< # TEST_TYPE=SPECweb_Ecommerce

---

> # TEST_TYPE=SPECweb_Banking

> TEST_TYPE=SPECweb_Ecommerce

41c41

< WEB_SERVER = localhost

---

> WEB_SERVER = sut2005

54c54

< BESIM_SERVER = "localhost"

---

> BESIM_SERVER = "BeSimbox"

56c56

< BESIM_PORT = 81

---

> BESIM_PORT = 82

63a64

> BESIM_INIT_SCRIPT = "/FastCGI/BESIM_fcgi.fcgi"

67,70c68,71

< SMARTY_DIR = "c:/www/php/Smarty-2.6.3/"

< SMARTY_BANK_DIR = "c:/www/php/Smarty-2.6.3/banking/"

< SMARTY_ECOMMERCE_DIR = "c:/www/php/Smarty-2.6.3/ecommerce/"

< SMARTY_SUPPORT_DIR = "c:/www/php/Smarty-2.6.3/support/"

---

> SMARTY_DIR = "/www/Smarty-2.6.9/"

> SMARTY_BANK_DIR = "./"

> SMARTY_ECOMMERCE_DIR = "./"

> SMARTY_SUPPORT_DIR = "./"

107c108

< KILL_CLIENT = 0

---

> KILL_CLIENT = 1


Appendix C - Wafgen README file

Invoking Wafgen:
================
 
    Unix: Wafgen <workload_specific>.rc
 
    Windows: Wafgen.bat <workload_specific>.rc
 
    or alternately: 
      java -Xms384m -Xmx384m -jar Wafgen.jar <workload>.rc
    or:
      java -Xms384m -Xmx384m -classpath Wafgen.jar org.spec.wafgen.Wafgen <workload>.rc
 
 
Wafgen Overview 
===============
 
Wafgen is used to create the file sets used by SPECweb2005.  Each workload
has a "fixed" and a "scaling" file set. The fixed file set includes a set of
static elements that are commonly referenced by the pages that make up a given
workload.  The scaling file set includes a set of directories each containing 
additional static elements for the workload.  The number of directories created
(or referenced by a test) is calculated by:
        (SIMULTANEOUS_SESSIONS * DIRSCALING) = Total #directories 
 
The total amount of storage required for a benchmark is also be dependent on 
the total number of directories in the scaling file set and the size of the 
contents of each directory (see Storage Requirement below).  Since the total
number of directories needed could exceed system limits for the number of files
per directory (usually ~32,000), an additional level of subdirectories can be
added using the SUBDIR* parameters.  By default, banking and ecommerce use
SUBDIR's.  The scaling directories are round-robined across the specified
number of subdirectories.
 
This kit contains the resource files (.rc) for each workload that are used as 
input to Wafgen. There is a set tailored for both Windows and Unix/Linux in 
the windows and unix subdirectories.  To use these files copy the appropriate
set into the upper directory and edit them as needed to match your test 
environment. For conforming results - ONLY parameters listed under Section A 
of the .rc file may be modified.  Double check all section A parameters, 
especially:
        DOCROOT (if not set will default to current directory)
        BASEDIRNAME (see examples in <workload_specific>.rc files)
        BASESUBDIRNAME (if applicable)
        SIMULTANEOUS_SESSIONS (may be larger than the corresponding value in
                              value in SPECweb_<workload>.config)
 
Also verify that the SPECweb_<workload>.config file parameters in the test 
harness on the Prime Client are in agreement.
 
The .rc files for fixed file sets include:
        bank_image_props.rc
        ecommerce_image_props.rc
        support_image_props.rc
 
The .rc files for scaling file sets include:
        bank_usercheck_props.rc
        ecommerce_productline_props.rc
        support_downloads_props.rc
 
The .rc files are annotated to provide additional guidance.
 
Reminder: Altering .rc file parameters listed in section B will result
in non-conforming tests, so only modify parameters in section A.
 
 
Storage Requirements
====================
 
Since the benchmark requires that the System Under Test (SUT) contain the file 
sets for all three workloads, it is important to understand the storage
requirements for each workload.  Below are the data and formulas to help 
estimate storage requirements for SPECweb2005.
 
Banking
=======
 
The fixed file set created by bank_image_props.rc includes: 
 
o bank/images -
        44 files; Average size: 3570 bytes; Total size: 157096 bytes
 
o bank/dynamic_padding - 
        15 files; Average size: 21800 bytes; Total size: 327000 bytes
 
The scaling file set created by bank_usercheck_props.rc creates a number
of user0000nnnnnn directories based on the SIMULTANEOUS_SESSIONS value.
 
o Each bank/images/{subdir00mm}/user0000nnnnnn contains - 
        20 files; Average size: 10431 bytes; Total size: 208625 bytes
 
To estimate the storage requirement for the user check directories use
the following formula:
 
        (SIMULTANEOUS_SESSIONS * CLIENT_SESSION_USER_IDS) * 208625
 
For Conforming results, CLIENT_SESSION_USER_IDS (alias for DIRSCALING)
must be 50.
 
Example:
    ((1000 SIMULTANEOUS_SESSIONS * 50 CLIENT_SESSION_USER_IDS) * 208625) =
        10,431,250,000 bytes
 
 
Ecommerce
=========
 
The fixed file set created by ecommerce_image_props.rc includes: 
 
o ecommerce/images -
        172 files; Average size: 2032 bytes; Total size: 349557 bytes
 
o ecommerce/dynamic_padding - 
        11 files; Average size: 73000 bytes; Total size: 803000 bytes
 
The scaling file set created by ecommerce_productline_props.rc creates a number
of productline0000nnnnnn directories based on the SIMULTANEOUS_SESSIONS value.
 
o Each ecommerce/images/{subdir00mm}/productline0000nnnnnn contains - 
        30 files; Average size: 18816 bytes; Total size: 564495 bytes
 
To estimate the storage requirement for the product line directories use
the following formula:
 
        (SIMULTANEOUS_SESSIONS * DIRSCALING) * 564495
 
For Conforming results, DIRSCALING must be 5.
 
Example:
    ((1000 SIMULTANEOUS_SESSIONS * 5 DIRSCALING) * 564495) =
        2,822,475,000 bytes
 
 
Support
=======
 
The fixed file set created by support_image_props.rc includes: 
 
o support/images -
        31 files; Average size: 728 bytes; Total size: 22580 bytes
 
  Warning: this file set contains several very small files (30 bytes)
  so the path name must not be longer than 14 characters. So, 
  "support/images" is OK but "supporting/image" is NOT.  
 
o support/dynamic_padding - 
        6 files; Average size: 59616 bytes; Total size: 357700 bytes
 
The scaling file set created by support_downloads_props.rc creates a number
of user0000nnnnnn directories based on the SIMULTANEOUS_SESSIONS value.
 
o Each support/downloads/{subdir00mm}/dir0000nnnnnn contains - 
        16 files; Average size: 4279501 bytes; Total size: 68472021 bytes
 
To estimate the storage requirement for the download directories use
the following formula:
 
        (SIMULTANEOUS_SESSIONS * DIRSCALING) * 68472021
 
For Conforming results, DIRSCALING must be 0.25.
 
Example:
    ((1000 SIMULTANEOUS_SESSIONS * .25 DIRSCALING) * 68472021) =
        17,118,005,250 bytes
 
Note: The heap parameters used for Wafgen are required to support the
      large files in the scaling file set for Support workload.
        
 
Tips for Building Large Filesets
================================
 
o To increase the size of an existing file set to support a larger number of 
  Simultaneous Sessions than used initially, the steps are:
 
        Edit the <workload>.rc file for the scaling fileset create previously
        and change these parameters:
 
        SIMULTANEOUS_SESSIONS=<new higher value>
        FIRSTDIRECTORY=<1+highest directory number previously create>
 
 
        For example:
 
        Originally a support_downloads_props.rc file used: 
               SIMULTANEOUS_SESSIONS=1000
               FIRSTDIRECTORY=1
 
        When a file set to support 2000 Simultaneous Sessions is needed,
        change:
 
               SIMULTANEOUS_SESSIONS=2000
               FIRSTDIRECTORY=501
 
        Then run: Wafgen <updated_workload>.rc
 
 
o To run several Wafgen's in parallel to create a large file_set, copies of i
  the <workload>.rc file can be for each parallel wafgen to be run and make the
  changes to the SIMULTANEOUS_SESSIONS and FIRSTDIRECTORY directories.  To 
  calculate the correct values for FIRSTDIRECTORY in each copy use the formula:
 
        Number of Directories = SIMULTANEOUS_SESSIONS * DIRSCALING 
        (or for banking the DIRSCALING alias CLIENT_SESSION_USER_IDS is used). 
 
  Use the directory scaling formula for the workload (shown above) to
  calculate the correct values for FIRSTDIRECTORY in each copy.
 
        For example, to create a support downloads file set for a total of
        4000 Simultaneous Sessions:
 
        support_downloads_props.rc_1:
               SIMULTANEOUS_SESSIONS=1000
               FIRSTDIRECTORY=1
        
        support_downloads_props.rc_2:
               SIMULTANEOUS_SESSIONS=2000
               FIRSTDIRECTORY=501
        
        support_downloads_props.rc_3:
               SIMULTANEOUS_SESSIONS=3000
               FIRSTDIRECTORY=1001
        
        support_downloads_props.rc_4:
               SIMULTANEOUS_SESSIONS=4000
               FIRSTDIRECTORY=1501
 
        Then run 4 parallel wafgens:
 
               Wafgen support_downloads_props.rc_1 &
               Wafgen support_downloads_props.rc_2 &
               Wafgen support_downloads_props.rc_3 &
               Wafgen support_downloads_props.rc_4 &
 
        (Note: depending on the number of CPUs and the I/O subsystem being 
         used this sequence may not take significantly less time than just 
         running a single Wafgen for the whole file set.)   
 
        
o To convert from a file set that was recreated without using *SUBDIR* 
  parameters or to add additional subdirectories, the original file set will
  need to be re-created from scratch. The <workload>.rc file will need to 
  be edited to configure these parameters.  The parmeters are preceeded by 
  the comment string "#subdirs#" if use of SUBDIRS is not the default 
  (i.e Support).  
 
o Multiple disks or mount points may be used to hold subsets of a workload's
  file sets, for example: /<docroot>/bank may have the images directory as
  as separate mount point or /<docroot>/bank/images may have separate mount 
  points for each of 10 subdirectories (subdir0001 thru subdir0010).
  
        
Warning:
 
        The path names have been standarize in the wafgen <workload>.rc files
        and the SPECweb_<workload>.config files in the harness, so that
        in the DOCROOT there would be these direcories:
 
               bank/images
               bank/dynamic_padding
 
               ecommerce/images
               ecommerce/dynamic_padding
 
               support/images
               support/dynamic_padding
               support/downloads
 
        While it is possible to change these names in the .rc and .config i
        files with due care (and making sure not to exceed path length limits 
        for some very small files); it is recommended that users retain this 
        naming to avoid errors due to typos and to simplify debugging.
 

Appendix D - BeSim Make_Readme file

How to Build BESIM:
 
UNIX/LINUX:
 
Type: uname -s to get name of the operating system (i.e. HP-UX)
In the Makefiles directory use one of the MakeIncl.<type>.<OS>
files as a template for your  MakeIncl.<type>.<your_OS> and
update to match you environment.
 
To Build Fast-cgi version of BESIM:   make fcgi
 
To Build NSAPI version of BESIM:      make nsapi
 
To Build Zeus ISAPI version of BESIM: make zisapi
 
Optionally you can specify other make targets by adding TARGET='list of targets'
For example:
 
        make zisapi TARGET='clean'
       make nsapi TARGET='clean all install'
 
The install target will use the path specified by DEST to install the besim
exeutatble or library, so it should be set to the appropriate place on your
Besim webserver directory tree (i.e. /www/fast-cgi).
 
If you plan to use Fast-cgi you must build and install the Fast-CGI development
kit which provides the fast-cgi library and include files before build BESIM. 
If using Apache for the BESIM webserver, that will need to include mod_fastcgi.
See: http://www.fastcgi.com/
 
Note: Please send any new or updated  MakeIncl.* files you want included in 
a future release to web2005(at)spec.org.
 
 
Windows:
 
Project files are included as an example: BeSim.def, BeSim.dsp, BeSim.dsw

Appendix E - Common Errors and Debugging Tips

  1. By default the DEBUG parameter is set to 1 in Test.config, if  a validation error occurs this value is sufficient to print out the ERROR message related to the invalid response.  The invalid response is also printed out and that is followed by a printout of the Request that produced the invalid response.  Reading  the returned HTML response and the associated HTTP Request is key to debugging problems.
  2. Status 404 (file not found) errors usually mean that there is a path problem either in the .config files on on the associated Web server.  In some cases it may indicate that not a large enough file set was created by Wafgen.
  3. Errors Returned by BeSim are found in the body of the returned HTML

Appendix F - Support FAQ

NOTE: For the latest version of this document, please visit http://www.spec.org/web2005/docs/support-faq.html.

Q: It's not working, what should I do?

A: Reread this User's Guide and make sure that you've followed all the steps and have run through the check lists in section 3. If you still haven't resolved your problem, review this FAQ to see if there is anything similar to the condition you are seeing.

If you have reviewed the User's Guide, various README files, and have tried higher DEBUG_LEVEL values and are still unable to resolve your problem, then write up all the details of the problem along with any logs your .config files and send them to web2005support@spec.org.

 

Q: How long will it take to complete a full compliant 3 iteration run for one of the 3 workloads?

A: The Minimum required time is:  ~144 minutes (2.4hrs), with each iteration at 48 minutes. If WARMUP_SECONDS or THREAD_RAMUP_SECONDS have been increased, the test will run somewhat longer.

 

Q: My clients are printing "Java.lang.OutOfMemoryError: Java heap space" errors. What can I do to eliminate these?

A: Invoke the client JVMs with more heap space; for example, assuming the clients have sufficient RAM, the initial and maximum heap sizes can be forced to 512MB (these values are only suggestions, and will vary depending upon system):
java -Xms512m -Xmx512m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC specwebclient

More information about heap size and garbage collection can be found here:
http://java.sun.com/docs/hotspot/gc1.4.2/faq.html
http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html

 

Q: When trying to manually access PHP pages (i.e. from a browser), the PHP error log or the response displays "failed to open stream: No such file or directory in [PATH]\inc.common.php"

A: You must first initialize the benchmark by running 'java specweb' with all the variables in the .config files set properly. The init.php script for each workload will write these variables to init_vars.php, so be sure user the Web server runs as has write permissions for the SPECweb2005 directories.

 

Q: I'm running the PHP scripts on Apache (either as the primary Web server or as the BeSim server) and I'm seeing errors.

A: If you have BESIM_PERSISTENT=1, try changing it to 0. During testing, Apache was observed to have issues with attempting to use persistent sockets to BeSim. It is suspected that this is due to the Apache architecture which forks multiple child processes to respond to requests; this makes it difficult to reuse a connection across script executions.

More information about the issue can be found here (search on Apache):
http://www.php.net/pfsockopen

 

Q: When starting the benchmark, the prime client attempts to run the INIT_SCRIPT on the Web server but fails.

A: Ensure that the user the Web server process runs as has sufficient permissions to write to the appropriate SPECweb2005 directories. For PHP (all workloads), init.php attempts to write init_vars.php, which contains server-specific variables required to run the benchmark.

 

Q: Why do I get "access to globals failed" in BeSim responses?

A: For Unix, BeSim attempts to write a globals file to the /tmp directory; for Win32, the root of the C: drive. There are two possible solutions:

·         Change the permissions on the appropriate directory so the user the Web server runs as can read and write this globals file.

·         Modify BESIM_banking.h, BESIM_ecommerce.h, and BESIM_supportsite.h to specify different BESIM_BANKING_GLOBALS_PATH, BESIM_ECOMMERCE_GLOBALS_PATH, and BESIM_SUPPORT_GLOBALS_PATH variables respectively. You must then recompile the API.

 

Q: The PHP scripts don't appear to be running properly, and I don't get any output from a browser when requesting the pages manually. How do I troubleshoot this?

A: PHP ships with two ini files, php.ini-dist and php.ini-recommended. While php.ini-recommended is optimized for performance and security, for initial troubleshooting, php.ini-dist may be a better choice. This will configure PHP to display errors in the browser. You may also want to set "error_reporting = E_ALL", which will display all errors and warnings.

 

Q: How can I test whether I've compiled and installed BeSim correctly?

A: The test_besim_bank, test_besim_ecom, and test_besim_support scripts (included in the BeSim directory) good ways to test whether you're getting valid BeSim responses. Invoke these scripts with the URL to your compiled BeSim API, i.e.

 
perl test_besim_bank.pl http://192.2.1.132:81/fcgi-bin/besim_fcgi.fcgi
bash test_besim_bank.sh http://bsim614:81/fcgi-bin/besim_fcgi.fcgi

.

 

Q: I'm getting invalid account values in client logs and it used to work fine.

A: BeSim may need to be to restarted cleanly along with the rest of the testbed software. Try:

·         stopping the webserver on the SUT

·         stopping the BeSim webserver

·         remove /tmp/besim_*.globals

·         remove old besim access logs

·         check that none of your clients are running "java specwebclient" processes (kill these processes if present)

·         start the webserver on the SUT and on BeSim box

·         re-run your test

 

Q: My clients are dying with "Exceeded max allowed overthink time" errors, how do I prevent these?

A: Check the following items:

·         Ensure your clients are not overloaded.

·         Ensure THINK_TIME to a reasonable level.

·         Increase THREAD_RAMPUP_SECONDS in Test.config.

 

Q: My client logs show sequences of errors that begin with:
2005-05-14 07:27:44:909 SPECweb_ Support : [ERROR] SocketTimeoutException encountered during run! followed by a couple of other header and response errors.
Occasionally my log also has error sequences that begin with:
2005-05-14 07:37:02:613 Connection: [ERROR] Bad status: 503

A: Your server is overloaded or under-tuned for the number of Simultaneous Sessions the test is requesting. The SocketTimeoutException error occurs when the server was unable to send a response back before the PROTOCOL_TIMEOUT_SECONDS (60 sec) has expired.The 503 status is returned by some webservers to indicate that the service is temporarily unavailable. See if reducing the load on the server eliminates these messages, if so you may wish to investigate if there is addition tuning that could be done to the SUT.

 

Q: My clients reported errors during the test but they don't show up in my run results. Why?

A: If the errors occur before the line that reads "WorkloadScheduler: Clearing statistics...", then they occurred during warmup and thus are not counted as errors.

 

Q: When navigating the ecommerce pages in a browser, why does "Session does not exist" appear when I click the "Add to Cart" button?

A: Make sure that you are accessing the Web server with the same host name as you specified for the WEB_SERVER variable in Test.config. For example, if WEB_SERVER=192.168.0.1 in your Test.config, you should use a URI such as http://192.168.0.1/ecommerce/index.php (and not a hostname that resolves to that IP address). This is due to the way the Location redirects are done in the scripts.

 

Q: Why do I get a segmentation fault when using the Sun 1.4.x JVM on the x86_64 architecture?

A: This problem is not specific to SPECweb2005. It has to do with the processor having NX (No-Execute, also known as Execute Disabled) on. There are two known workarounds:

·         Upgrade to a 1.5.x JVM

·         Disable NX in your operating system. Under Linux, this can be done by passing the "noexec=off" command-line option to the kernel. If the kernel is 64-bit, "noexec32=off" should also be passed.

 

Q: How do I get the PHP scripts running on Windows Server 2003 / IIS 6.0?

A: There is a helpful tutorial available here: http://www.peterguy.com/php/install_IIS6.html

You must also configure IIS to serve all files regardless of file name extension by adding a wildcard character (*) MIME type. NOTE: This should not be performed on production Web servers. More information can be found here:
http://www.microsoft.com/resources/documentation/iis/6/all/proddocs/en-us/wsa_mimemapcfg.mspx

 

Q: I changed BASEDIRNAME in support_image_props_rc file from the default "support" to "supp1data". The wafgen was failing to create the image files under the image directory with an error message as: Error: path length too long for requested file size - what's wrong?

A: The message: Error: path length too long for requested file size has to do with very small files (like ccc which is 30 bytes). To be a specweb file it has to have the byte count, relative file name and at least 1 char. of data. So to handle support's smallest image file, the path can be <7chars>/<6chars>/<name>. While you can change the path to something else (sup/sup_images or sup1data/img01 or other as long as it fits into this footprint and the changes are reflected in the SPECweb_<workload>.config. The Wafgen README does include the following warning under the section on Support : Warning: this file set contains several very small files (30 bytes) so the path name must not be longer than 14 characters. So, "support/images" is OK but "supporting/image" is NOT.

Q: I am trying to initialize multiple physical web server systems, or multiple web server instances on one physical system, how do I do this?

A: List all web server instances (hostnames or IP addresses) in SERVER_LIST, and all ports in SERVER_PORT_LIST. For example, if you have two web server instances (192.2.1.1:80 and 192.2.2.1:81) on one machine, your entries should read:

 

SERVER_LIST = "192.2.1.1 192.2.2.1"
SERVER_PORT_LIST = "80 81" 

Q: I am trying to run the ecommerce workload and get the following error message, how can I fix it?

 

2006-03-26 15:16:54:183 SPECweb_Ecommerce[1]: [ERROR] state = 8 response.indexOf fail to find unit price: NN.NN, scaled_load =1  

A: One possible reason for this failure is a localization issue. Try changing the localization settings on the clients to US-format.


Appendix G - Installing BeSim as a FastCGI for the Apache HTTP Server

There are two pieces necessary to get BeSim for FastCGI working properly under Apache:

  1. Compiling the FastCGI source code and BeSim
  2. Compiling/configuring mod_fastcgi, an Apache module to invoke FastCGI for specific directories

G.1 FastCGI source compilation

  1. Compile the FastCGI source code.  The source tree is uded on a BeSim (or full) installation of SPECweb2005. Here are steps to compile it:
cd <path_to_SPECweb2005>/besim/fcgi-2.4.0/
./configure --libdir=/lib
make
make install

NOTE: --libdir=/lib was added above due to the default FastCGI Makefile installing libraries to /usr/local/lib, which is not a default library path on Linux and could cause this error upon execution of the FastCGI:

besim_fcgi.fcgi: error while loading shared libraries: libfcgi.so.0: cannot open shared object file: No such file or directory

NOTE #2: For x86_64 versions of Fedora Core 3 and 4, it was observed that --libdir=/lib64 should be specified instead of /lib.
 

  1. Create a directory for the BeSim FastCGI within your web server directory hierarchy (but preferably outside the document root).  Here are some recommended locations for different setups:

SuSE: /srv/www/fcgi-bin
Red Hat: /var/www/fcgi-bin
Apache 2.x compiled from source: /usr/local/apache2/fcgi-bin
 

  1. Compile BeSim as a FastCGI (replace /var/www/fcgi-bin/ with the directory created in the previous step):
cd <SPECweb2005>/besim/

make fcgi TARGET='clean all install' DEST=/var/www/fcgi-bin/

This should create besim_fcgi.fcgi and install it into DEST.  If not, check the make output for errors.

G.2 Apache FastCGI Web server module: mod_fastcgi

These instructions show how to compile the FastCGI web server module for Apache (mod_fastcgi). For other web servers, please refer to their documentation to see whether FastCGI support is available.

NOTE for SuSE: SUSE Linux (8.1 and above) ships with a precompiled mod_fastcgi for Apache 2.x.  To check if it is installed, do a rpm -q apache2-mod_fastcgi; if it's not, install this package from the CDs (see your OS documentation for more instructions).  If this package is installed, you can skip the rest of this section.

NOTE for Red Hat: If using the httpd package that ships with the distribution, the httpd-devel RPM must also be installed, as it contains the necessary utilities and headers to compile Apache modules.  To check, do a rpm -q httpd-devel.

  1. Download the latest mod_fastcgi tarball from http://www.fastcgi.com/dist/ and follow the directions in the INSTALL document (INSTALL.AP2 for Apache 2.x).

    NOTE: As INSTALL.AP2 states, unless you are compiling from source,  you will need to modify top_dir in your make command.  For example, Red Hat users should use the following 'make install' command:

    make top_dir=/usr/lib/httpd install

    You should now have a mod_fastcgi.so in your Apache modules directory:
2.           Red Hat:
# ls -l /usr/lib/httpd/mod_fastcgi.so

-rwxr-xr-x 1 root root 161408 Sep 16 13:47 /usr/lib/httpd/modules/mod_fastcgi.so
Compiled from source:
# ls -l /usr/local/apache2/modules/mod_fastcgi.so
-rwxr-xr-x 1 root root 156434 Sep 21 10:03 /usr/local/apache2/modules/mod_fastcgi.so
  1. Create a directory for FastCGI to store Unix socket files, and change the permissions so the Web server can have full access to it:
    Red Hat:
    # mkdir -p /etc/httpd/fastcgi
    # chmod 777 /etc/httpd/fastcgi

    Apache 2.x from source:
    # mkdir -p /usr/local/apache2/fastcgi
    # chmod 777 /usr/local/apache2/fastcgi
     
  2. Add support for handling FastCGIs by making the following changes to httpd.conf (for Red Hat, this is in /etc/httpd/conf/):
     
    1. Add this line in the "Dynamic Shared Object (DSO) Support " section:
      LoadModule fastcgi_module modules/mod_fastcgi.so
       
    2. Add the following lines just after the ScriptAlias /cgi-bin/ line:
      ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/" # <fcgi-bin directory from Section 1, Step 2>
      FastCgiIpcDir /etc/httpd/fastcgi # <FastCGI directory from previous step>

      NOTE: Replace /var/www and /etc/httpd with /usr/local/apache2 if compiling from source.
       
    3. Add the following section, preferably after the <Directory "<CGIDIR>/cgi-bin"> section:

      <Directory "<CGIDIR>/fcgi-bin"> # <fcgi-bin directory from Section 1, Step 2>
          AllowOverride None
          Options +ExecCGI -Includes
          SetHandler fastcgi-script
          Order allow,deny
          Allow from all
      </Directory>
      AddHandler fastcgi-script fcgi

G.3 Testing the BeSim FastCGI

  1. Start (or restart) your web server.
  2. At the console, monitor your Apache error log to view any errors or warnings during FCGI execution, i.e.
# tail -f <APACHE_ROOT>/logs/error_log
You should initially see some FastCGI initialization messages such as
[Wed Sep 21 10:36:39 2005] [notice] FastCGI: process manager initialized (pid 25700)
  1. Test the BeSim FastCGI, using the scripts in the subdirectory of the SPECweb2005 installation.  See BeSim Setup for examples.

See BeSim sections above for additional information.


The SPECweb2005 benchmark utilizes the JFreeChart library, which is licensed under the Lesser General Public License (LGPL).  Source code and a copy of the license are included with the benchmark kit.


Copyright © 2005-2007 Standard Performance Evaluation Corporation.  All rights reserved.

Java® is a registered trademark of Sun Microsystems.