Best Practices
PureLoad 3.5 November 2007	http://www.minq.se support@minq.se

• Runtime configuration
• Worker Threads
• Realistic Testing
  • Scenario Design
  • Data
  • Bandwidth
  • IP Addresses
• Long/Large Test Execution
• Massive Session Testing
• PureLoad and Firewalls
  • Example
  • NAT

Runtime configuration

The PureLoad runtime can be configured in many ways. A common and recommended configuration for larger load tests are as follows:

• Install and run the naming and taskspace servers on one host
• Install and run worker managers on as many hosts that you find are required
• Install and run the Console on one (any) host
  

Worker Threads

How many worker threads you can effectively use per worker, depends on your hardware's capabilities and OS. In general modern server OS'es (for example Linux and Windows Server editions) have better threading capabilities compared to a desktop OS (such as Windows XP).

But in addition it also depends on:

• how you scenarios are designed
  If scenarios includes "wait time", simulated using sleep tasks, then a thread will not work all the time, but instead sleep and give other threads the possibility to work
• how fast is the server(s) you are testing
  A faster server makes each worker thread work harder since it returns request quicker. The more work a thread does, the more each thread has to wait to get access to the CPU, and the more inflated the timing information gets, etc.
  

So, to find out how many worker treads you can run in your environment, the best is to start with fewer treads an monitor the workers during execution. From this point of view you should check the CPU usage and make sure it stays below 90% most of the time.

Realistic Testing

It's important to understand that using a load test tool you always simulate load. It's not real users, doing real work accessing your servers under test. So what is a realistic test? This depend on several factors, but our suggestion in general is that you should look at the goals and requirements you have on your system.

Depending on the system under test, it might be a useful simulation to focus on request per second, instead of trying to simulate real clients. In other cases you really have o simulate real clients a close as possible. The following gives some hits on what you should consider in these cases.

Scenario Design

If you want to simulate "real" users, the scenarios executed by each worker thread, must simulate how a real user would access the server application. This means using realistic data (see below) and include "wait (or think) times" in the scenario, using SleepTask and RandomSleepTask. When you have a scenario like this one worker thread will correspond to one simulated virtual client.

You probably also want to use several scenarios simulating different users doing different things.

Data

Trying to simulate real clients includes using realistic data. It's not enough using a recorded (using the HTTP recorder) scenario. Repeating the same requests, with the same values could produce unrealistic response values.

Instead you have to use dynamically generated values, using parameter generators, generating data for different user accounts etc.

Bandwidth

A user connecting at 100Mbps through a local network and a user connecting with DSL connection might do not load the server in the same way. In PureLoad the simulated bandwidth may be limited, by using the HttpInitTask.

IP Addresses

When testing a web application, using different IP addresses is normally not required. But for example if load balancers is used, they might use the client IP address to balance the load over several servers. In this case you have to make sure you simulate access from different IP addresses.

PureLoad doesn't support "spoofing" of IP addresses, but for example the HTTP tasks supports setting the client IP address. This means that by defining several virtual IP addresses on the worker hosts you can simulate access from different IP addresses using the same worker host.

Long/Large Test Execution

When running a load test in PureLoad over a long time (say a weekend) there are several issues that you should be aware of:

• PureLoad keeps all results in memory, meaning that there is a limited about of information that can be stored.
• Polling for information often uses more memory.
• Logging does not take much memory, but might influence the test results.
• Allowing workers to report information per worker thread uses resources if many worker threads are used.

With this in mind, we suggest that you check the following in the Console Tool Properties before you start the execution:

General / Update Interval

Set this to something like 10 minutes (600 seconds).

Workers / Report Worker Thread Information

De-select this setting.

Workers / Logging, Log Size

Set this to Error.

Workers / Persistence

Enable this might be useful if the worker still (for some reason) run out of resources.

Massive Session Testing

Massive testing, using many thousands of simulated clients, where the scenario initiates and keeps a session alive (for examle an HTTP session, using cookies), the default model where each worker thread corresponds to one native thread sets a limit. Read more about Worker Threads above.

For certain scenarios where the main focus is to test a massive amount of sessions performing infrequent operations, the thread limitation can be avoided by using the Worker Threading mode Multi. Running in this mode, the worker will execute multiple worker threads for each native thread.

Consider the following scenario showing a simple usecase of some web based application:

Let us say that we want to simulate how 100,000 users use the application during the course of a day. The average user logs in in the morning between 8:00 AM and 9:00 AM and then makes a search every other hour and finally logs out at the end of the day. The sleep task prior to searching is set to sleep randomly for zero to two hours. The second sleep task simulates think time and is set to 5 - 10 seconds.  This means that the server will receive 100,000 login requests randomly distributed over one hour - about 30 logins per second. The rest of the day it will receive about 30 searches per second and 30 view product requests per second. In this example, the server can easily handle these requests and the response times are near zero.

Using native threads would require somewhere around 30 to 100 machines to execute this scenario. Since the amount of actual requests per second is never more than about 60 per second, it should be enough with a single worker machine if it could handle the amount of threads. This is where the Worker Threading mode Multi comes in. We set up a worker with 100,000 threads configured to use Threading Multi with a ratio of 1000 and a delay of 3600:

This means that the worker will use 100 native threads. The delay is set to 3600 to match that the users log in during one hour. Each worker thread will start with going to sleep for 0 to 3600 seconds. This gives the 100 native threads enough time to execute the 30 logins per second. It is important that the search/view loop contains a substantial amount of sleep task(s) (one hour in average in this case) to allow the Threading Multi mode enough time to schedule the worker threads properly.

Using Threading Multi requires careful thought. Resources are easily exhausted when the numbers are hundreds of thousands. The worker will log a warning message if it detects that it fails to execute the tasks in the correct rate:

In the above example, the server response times were low. Had the Search Product request instead had a response time of 0.5 seconds, it would be difficult to generate 60 requests per second with 100 native threads and it would be necessary to increase the number of machines somewhat, but still no way near the 30 - 100 machines required using native threads.

PureLoad and Firewalls

PureLoad uses RMI for internal communication between servers and the console and by default there is no control over the ports being used by RMI, which makes it hard to open up communication when firewalls are used.

There are however System Properties to control all port being used, and hence allow communication over firewalls.

Example

As an example, let us say that we have a network where the Console is being used from another network that the servers and the network are separated by a firewall:

In this case we need to open up the firewall to allow communication from the console to the servers. We do this by first defining the system properties for each server:

Naming and Taskspace

naming.port=1099 naming.service.port=60002 taskspace.port=60003 taskspace.service.port=60004

Worker Manager 1

manager.port=60005 manager.service.port=60008

Worker Manager 2

manager.port=60007 manager.service.port=60008

The actual ports used can be any ports, with the exception of the naming.port that must match the PureLoad license.

Now you can open the ports 1099 and 60002-60008 in the firewall and communication will work as expected.

NAT

If NAT (Network Address Translation) is used, you also have to make sure the servers expose their remote IP address (i.e the address used outside the firewall) used to contact the servers. Use the external.host property for this.

If we in the example above must use IP address 192.168.100.126 to reach Worker Manager 1, we simply specify:

in the pureload.properties file for the manager.