Scenario Editor
November 2007
support@minq.se
Scenario Editor |
|
| PureTest
3.5 November 2007 |
http://www.minq.se support@minq.se |
This chapter describes how scenarios and tasks are defined and also
the various features related to scenario management in the scenario
editor.
Scenarios of tasks is the common structure used throughout
all applications in the Pure Performance suite. A task is the actual
function that will be performed, i.e get a web page, insert data into a
database, etc. A series of tasks are defined in a scenario. An
extensive set of features is available to make tasks modular and
dynamic.
A scenario in the Pure Performance suite consists of task sequences
and tasks. The purpose with the scenario is to assemble all tasks as
they are going to be executed in any of the run time environments. I.e
PureLoad, PureAgent or PureTest.
Task sequence is basically a container that groups other task sequences and tasks. An example is a web application where a scenario shall include requests to many web pages with their specific HTML pages and graphics. All requests for a complete HTML page is in this example grouped in a task sequence:
Here we have a Scenario (Test Login) that consist of tree task
sequences, one per web-page (Start Page, Login and Logut). Each of the
task sequences consist of tasks that represent one HTTP request.
Once the PureLoad Console or PureTest application has been started a
window similar to the following appear. It consists of one main tab, Scenarios which have tree sub tabs
at the bottom of the screen:
The left pane shows the managed scenarios in tree structure. The tree
can maintain several scenarios concurrently. The menu items and tool
bar buttons are used to manage the content of the scenario tree.
Selecting an object in the tree displays the characteristics of the
object in the properties panel to the right.
The scenario tree presents objects by the value of the Name
parameter. The number within parentheses indicates the number of
iterations that will be executed. The Sequences/Tasks information below
the tree indicates the following:
Note: There is basically no difference between a scenario and a task
sequence object. Scenarios can however only be defined on the top level
and may also define a distribution. Scenarios are also the object type
that is executed in the runtime environment.
The Name parameter that can be specified in the property panel for
scenario, task sequence and task objects needs some extra attention.
Name is primarily used in the scenario editor and in the result
presentation. Its default value is the same as the selected task class
but can be specified to anything meaningful. Name is also used when
PureLoad visualizes the tasks in the the result presentation. All tasks
with the same name per scenario or task sequence will be presented as
one task in the result presentation.
An example is a scenario for load testing of a web site. The
scenario to the left will give a result tree as shown to the right:
If we are dot interested in individual resuts for the image URLs, we
simply name these tasks "image" only the results will be displayed as:
Create a new scenario by selecting the Root node in tree and select
Edit->Create Scenario or use the 
button in the tool bar:
Selecting the newly created scenario in the scenario tree shows the
properties for the scenario in the right area. The properties are:
A task sequence is a container in which tasks and other task
sequences can be grouped. There is no limit on the depth of a scenario
tree.
Select the object in the scenario tree into which the task sequence
shall be added and select the Edit->Create
Task Sequence menu option. If the currently selected object in
the tree is a scenario or task sequence then the newly created task
sequence will be added as a child object. If a task object is currently
selected then the new task sequence will be created below the selected
task object:
Selecting the newly created task sequence in the scenario tree shows the properties for the task sequence in the right area. The properties are:
Tasks are added either to a scenario or a task sequence. Select the Edit->Create menu choice to
create a new task. If the currently selected object in the tree is a
scenario or task sequence then the newly created task will be added as
a child object. If a task object is currently selected then the new
task will be created below the selected task object.
Note: The tool bar button creates an object in the tree
according to the current selection. If the selection is either a
scenario, task sequence or task object then the new object is always a
task.
The following dialog lists the Task Types to choose from. Select a task
in the list and press Add or double click to add the task type.
The added task will now appear in the scenario editor:
All tasks have task parameters. These are grouped into generic task
parameters and task specific parameters. All tasks have the following
generic parameters:
All tasks are using the task parameter mechanism. Task parameters
defines the input interface for the task and the actual values are
specified in the scenario editor. Task parameters might be of different
types and are displayed different depending on type.
The value for all task parameters can automatically be defined using
task parameters generators and/or variables.
Scenarios of tasks are normally executed in any of the Pure
Performance suite applications. This can be PureLoad for load testing,
PureAgent for performance monitoring or the PureTest application
itself. This section describes how to execute tasks from within the
Scenario Debugger in the PureTest application.
The main purpose with the Scenario Debugger is to quickly test and
verify each individual task in a scenario. This is performed by single
stepping through each task in a scenario. The result in terms of
execution result and execution log is presented.
The debugger mini tool bar is displayed by selecting the View->Show Debug Toolbar menu
choice (if not already displayed) and its control buttons will appear
in the Scenario Editor sub tab:
The following operations are available in the debugger tool bar:
Toggle debug/edit mode
Start debug session
) in from of the current task
in the tree.
Stop execution
Continue/Start execution
Single step
Toggle breakpointMake sure debug mode is enabled (use the
button in the debug tool bar).
Now press the button to start the debug session.
This operation moves the execution pointer to the first task in the
scenario which is indicated by the task pointer icon (). This pointer indicates
that the task is going to be executed during the next Single Step () or Continue () operation.
The task execution result is presented to right of the scenario tree:
Now you can continue to step trough the scenario analyzing the results from each task executed.
Scenarios are often designed so that they can be parameterized and
repeated many times with different data. The parameter generator
mechanism makes this possible. Each task parameter value can refer to
parameter generators to have the actual value generated during the
test. The following parameter generator types are available:
Parameter generators have five operations that they can perform:
Before assigning a parameter generator to a task parameter it needs
to be created and configured. This is done in the Parameter Generators
sub tab.
Selecting View->Create or the button in the tool bar shows a dialog with
the available data sources.
Double click on selected data source or press Add to create it. All
parameter generators have a Name parameter that must be unique. It is
recommended to set a name that match the purpose with the parameter
generator. For example User ids, Cities, Models, etc.
The counter generates numbers within specified boundaries.
Start and stop defines the range. The special constant INF is used
for infinity.
The file data source read values from a file. The file can either
contain one value per row or several values per row. Several values per
row must be delimited with a character that is used to split the values
into Elements. The file below contains 6 rows, each with 3 elements
separated with a "%" character.
| A road to heaven%Clark Moore%287 Somewhere in the sky%Vanessa Hearst%99 Man in the moon%Dirk Dustin%129 Java Server Pages%Hans Bergsten%199 Gagarin: a myth%Roland Petersson%19 Sun, Moon, Earth%Laura Craft%78 |
The following figure shows the properties for the file data source.
The books.txt file has been
loaded and the Preview Data button has be pressed. Since the specified Field Separator matches the
separator in the file will three columns appear in the grid each
representing the values in the file. The name of each element can be
changed by clicking on the table header and it is those names that are
used to access the parameter generator elements in a task parameter.
The following syntax is used in a task parameter to access a element in
a file data source:
| $Books.Prices.getNext $Books.Authors.getNext |
If having several elements defined for a file parameter generator
then each of these are managed separated from another. I.e doing a $Books.Prices.getNext will only
forward the row pointer for that specific element and not for any other.
The list parameter generator is handy when a small list of values shall be read. The management of the list is done in the List parameter generator options pane.
The calendar parameter generator is useful when generating timestamps values. The generator can be configured to increase parts of the timestamp between each getNext operation.
Having a Date Format set to yyyy-MM-dd HH:mm and Start Time set to 2007-10-12 13:37 and Increment set to 1 Hours will cause each use of the getNext operation to return the following values:
| 2007-10-12 13:37 2007-10-12 13:38 2007-10-12 13:39 etc. |
The Date Format lists a few standard formats but it can be manually edited to accommodate custom patterns.
All generators can be tested using the Test button at the bottom of the pane. The following dialog is displayed for all but File generators.
The main difference between the test dialog for file data sources is that an element (column) may be selected.
A powerful feature when testing a dynamic application is to extract information from a task, save his information to a scenario variable and use defined variables in subsequent tasks.
Variables can be used in any task parameter. The syntax of a
variable is:
| ${variable-name} |
where the value, if any, of variable-name is substituted before the
task is executed. A task parameter value can mix static text with
variables and also contain more than one variable. Assume that we have
two variables defined itemno and itemcount, then we could for example
have a URL parameter to a HTTP tasks as follows:
| http://some.where.com/getItem?item=${itemno}&count=${itemcount} |
Variables values are set by Tasks. To assign a value to a variable a task must be used. Normally this is done by extracting part of data received from a previous task, using any of the extract tasks available.
The most common use of variables is when testing a dynamic web application. Please read more about testing web applications in the Testing Web Applications guide.
But note that using the generic extract task it is possible to extract data and assign a value to a variable from any task that returns data. This includes Telnet, FTP, Mail and JDBC tasks.
Task sequences can have a condition which determines if the sequence
should be executed or not depending on previous errors or an execute
condition expression.
A scenario is normally executed task by task from beginning to end. The default behavior is to ignore errors in individual tasks and continue execution. By using the RuntimeSettingsTask the runtime settings flag "Stop On Error" can be set to false, to force the whole scenario execution to end in case of a task error.
However, sometimes it is important to execute certain tasks to perform cleanup even if an error has occurred. For this purpose, the "Stop On Error" can be overridden for a task sequence by setting the task sequence execute condition to "Always". All tasks and sub-sequences under a task sequence with condition "Always" will be executed regardless of previous errors.
Given the following scenario:
Now imagine that the CreateEntryPage results in an error. With "Stop
On Error" set to false, the execution will continue after the error and
cause the UpdatePage and the StoreEntryPage to fail as well, since no
entry was created.
If we want only the initial error to be reported as an error this can
be accomplished by setting "Stop On Error" to false. The scenario
execution will then halt right after the task 'create' has been
executed. In this case the user never logged out and the server will be
left with a user session that must time out by itself.
In order to both report only the initial error and perform logout, the
"Stop On Error" flag should be set to true and the LogoutPage task
sequence should be set to "Always". This will halt the execution after
the task 'create', but will be forced to resume for the LogoutPage task
sequence.
Task sequences can be executed conditionally by setting an execute
condition. An execute condition can be set to one of the following:
Expressions are written using standard Java language syntax with the
addition of PureLoad variables. They must evaluate to the primitive
value true or false.
PureLoad variables are automatically typed. Variables holding numerical
values will be typed as primitive double values. Variables holding
non-numerical values will be typed as strings (java.lang.String).
For example, let us say that the PureLoad variable ${foo} has a value
of "minq" and that ${bar} has a value of 17. Now the following
expressions will be evaluated as:
| Expression |
Value |
|---|---|
| ${bar} == 17 |
true |
| ${foo}.equals(“minq”) |
true |
| ${foo}.length() == ${bar} |
false |
| ${bar} > 0 && ${bar}
< 42 |
true |
| ${foo}.startsWith("m") |
true |
| ${foo}.indexOf("q") |
Error (not a boolean result) |
The Replace Task Parameters dialog is used to replace task parameter
values for a collection of tasks. The replace functionality can search
for existing task parameter values either matched by a part of a value
or complete values. It can also replace complete values as well as
parts of values.
A selection in the scenario tree is needed in order to to launch the
replace task parameter dialog. Select the desired objects in the tree
and select Edit->Replace Task
Parameters. All tasks that are included in these scenarios will
be subject for replacement in the replace task parameter dialog:
The dialog consists of two areas from top:
All task parameters are of a specific data type. Normally the data
type is of minor interest for a load tester but the replace task
parameter dialog relies on it since it is used to assist the user to
search and replace task parameter values. Example: it is not possible
to search on parts of existing Boolean values as it is with all other
data types. Also, it is not possible to replace parts of Integer, Long,
Float and Double as it is with String data types.
The following figure illustrates an example where all selected task parameter values shall be replaced independent of any existing value.
Set the Old Value selection
box to Any Match. This means
that any existing value shall be ignored when finding the actual task
parameters to replace.
New Value is simply set to the
new value that all task parameters in the selection will get. Press Ok
when ready and a confirm dialog is displayed to confirm that the
replace shall be performed.
The result of this example is that all selected task parameters
(URL) will get "http://www.some.com/index.html" as value independent of
any previous value.
The following shows how to change task parameter values that have a
specific value.
Set the Old Value selection
box to Exact Match and enter
the string to search for in the Old
Value text field. New Value
is set to the new value that all task parameters that match the Old Value will have.
The result of this example is that all task parameters in the selection
(URL) that have "http://localhost:8080/index.html" as its current value
will be replaced with "http://www.some.com/index.html".
The following shows how to replace only parts of an existing value
based on parts of the current value.
Set the Old Value selection
box to Substring Match and
enter the string to search for in the Old
Value text field. Set the New
Value selection box to Substring
replace and New Value
to the value that all task parameters that match the substring of Old Value will have.
The result of this example is that any "http://www.come.com" in the
value of the selected task parameters will be replaced with
"https://www.come.com". All other data in the task parameter value
(URL) are left as is. In other words: we have changed the URLs to use
HTTPS/SSL.
Note: Scenario Distribution and
the Function parameter for scenario
objects is only applicable when using PureLoad.
Rather than just executing all tasks as fast as possible in a burst,
there is a possibility to make the amount of scenarios per second
change over time. This is done by setting the Function parameter for a
scenario.
By default no distribution of scenarios are used. I.e. the scenarios
are fetched by worker threads from the task space and executed as fast
as possible. When the task space is empty the execution stops.
If you want to distribute the execution of the scenarios over a period
of time, select a scenario and change the function parameter.
The function dialog contains various parameters to configure the
selected distribution function. I.e. the start and stop time of the
scenario and additional function specific parameters. All times in the
dialog is given in seconds relative to load test start (time = 0).
Note: Defining a load test session using the time distribution does not
guarantee that the execution result will be exactly as defined in the
scenarios. The test result is affected by parameters such as: Are there
enough workers or worker threads. Is the tested system capable of
handling the amount of load defined in the scenarios. Due to the
inertia in most tested systems and in PureLoad itself, it is also
important to run tests over a long period of time (at least 15 minutes
up to many hours) for the distribution to be accurate.
The following sections explains the distribution functions that are
available in PureLoad.
The default distribution function (actually the same as Burst, with
start time set to zero).
The scenario instances will be executed as fast as possible starting
at Start Time.
The scenarios will be distributed in a linear way over time. For
example starting with 2 scenarios per second and ending with 5
scenarios per second. Note that it is not possible to set the total
number of scenarios executed by editing the Iterations parameter. The iterations
parameter is calculated by the system using the defined parameters. The
calculated value is showed as "view only" in the iterations field.
The example above shows:
The specified number of scenarios will be distributed quadratically
starting at Start Time.
This distribution function is used to distribute the scenarios with
a Normal function.
The parameters to set for this function are:
If you are using distributions for several scenarios it is possible
to view the summary graph of all the time distributions. This is done
by selecting the Root object in the scenarios Tab. The example below
models the load on a system where there is a base load and an event
occurs which causes a peak in the load (this could be users arriving at
the office after lunch and performing some task on a server).
