Skip navigation.

Rectal Probes and Benchmarking with WAST

Having a problem with your bowels? Hurts like hell? You can check the pulse, use a thermometer, use a stethoscope, but sometimes you need to get messy and have rectal probe to find out the the real cause of your problem. Similarly, when benchmarking, we can use simplistic tools such as ApacheBench, but sometimes we need more sophisticated probes to find out the "truth of the matter".

ApacheBench (ab) is a nice tool for sending large number of HTTP requests and measuring the response times. But it has its limitations. There are serious scalability issues when you want to test anything that involves sessions. ApacheBench limits you to one hard-coded PHPSESSID. If you run 10 concurrent threads, the threads will be serialised because the session file will be constantly locked. This means your benchmark figures with ApacheBench will be wrong if you are using sessions - the figures will be too low. One other symptom of this behaviour is no matter how many requests you pump to the server, the server continues to idle 70-80% of the time, and the throughput (pages/second) remains static.

That's when you have to throw away these simplistic tools and bring out the rectal probe. My favorite web server probe when ApacheBench is insufficient is Web Application Stress Tool (WAST), a free download from Microsoft.

WAST allows you to run multiple threads (similar to the -c parameter in ab) in a benchmarking test. You can assign virtual users to each thread, and each user can be assigned a unique cookie. This allows you to assign different PHPSESSID's to each thread.

Benchmarks

Here is the "truth of the matter" using ApacheBench and WAST. Higher throughput (Pages/Sec) is better:

ThreadsApacheBench
Pages/Sec
(CPU%)
WAST
Pages/Sec (CPU%)
83.95 (20%)14.38 (65%)
124.08 (20%)16.34 (71%)

You can see the throughput is much higher when testing with WAST because sessions are no longer a bottleneck. Also the CPU utilization on the web server is more realistic. Good shit indeed!

Setup

The rest of this article details how to setup WAST for multiple sessions/cookies. The WAST screen is divided into 2 panes. The tree view on the left and the edit pane on the right. First you create a script, in this case my-test, and determine which web pages you are going to visit. To create a script, right click on the tree view and select Add.

Then you define the users. Select Users from the tree view. To create a new set of users, double-clicking on "Default" below:

The following screen will appear:

If you right click on the left pane of the above screen, you can create a new set of users, in this case juris-users. Then you decide how many users you want to add. Click on any user and press the delete button to remove a user.

Next you configure the threads. Click on Settings in the tree view. In this example, I set it to 12 threads. WAST will automatically assign a user to each thread. Note that I set the benchmark to run for 60 seconds, with a 10 second warmup period.

Lastly you need to assign cookies to each user. First go to your sessions directory and duplicate the session files multiple times. I like to backup the session files so I can reproduce the benchmark later. Once you have enough session files (12 in this case), go to Cookies in the tree view and enter the session id's:

Now you will be able to run the WAST benchmark and probe the web server. Click on my-test in the tree view, and then the arrow button in the toolbar above to run a test. To view the results, click on the reports icon in the toolbar. The results look like this:

Overview
================================================================================
Report name:                  12t-14/03/2005 2:34:05 PM
Run on:                       14/03/2005 2:34:05 PM
Run length:                   00:01:10

Web Application Stress Tool Version:1.1.293.1

Number of test clients:       1

Number of hits:               982
Requests per Second:          16.34

Socket Statistics
--------------------------------------------------------------------------------
Socket Connects:              982
Total Bytes Sent (in KB):     491.00
Bytes Sent Rate (in KB/s):    8.17
Total Bytes Recv (in KB):     36234.34
Bytes Recv Rate (in KB/s):    603.04

Socket Errors
--------------------------------------------------------------------------------
Connect:                      0
Send:                         0
Recv:                         0
Timeouts:                     0

Script Settings
================================================================================
Server:                       10.1.2.21
Number of threads:            12

Test length:                  00:01:00
Warmup:                       00:00:10
Cooldown:                     00:00:00

  ....  TRUNCATED ....

Also see: