Thursday, 5 March 2015

Best Practices for your Jmeter Tests

Guidelines to overcome limitations of JMeter in distributed environment:
  1. Limit the Number of Threads
  2. Using a proxy server
  3. Using variables
  4. Reduce resource requirement
  5. Check the JMeter logs
  6. Erase the local path from CSV Data Set Config
  7. Follow file naming convention
JMeter has some limitations especially when it is run in a distributed environment. To use JMeter efficiently for testing, you should use following guidelines:

Limit the Number of Threads

The maximum number of threads you can effectively run with JMeter is 300. This limit is because of hardware's capabilities. If JMeter is made to run with more number of threads, the accuracy of timing information will decrease.

Using a proxy server

The Proxy server helps you to abstract out certain common elements from the recorded samples. Moreover, it is useful features to record your testing.

Using variables

Some test plans need to use different values for different users/threads. For example, you might want to test   a sequence that requires a unique login for each user. This is easy to achieve using variables.

Reduce resource requirement

The GUI mode consumes a lot of computer memory under heavy load. It causes performance issues.
There're some tips to reduce resource requirement:
  • Use non-GUI mode
  • Disable the "View Result Tree" listener during the Load test. Because it consumes more memory and causes JMeter running to run out of memory.
  • Disable all JMeter graphs results
  • Use CSV test result format.
  • Only save the needed test result. JMeter could take a long time to save very detailed test results.

Check the JMeter logs

Any errors in the test plan or test execution will be recorded in the log files. Monitoring the log file help you   to find the error early

Erase the local path from CSV Data Set Config

If you are using an existing CSV data file that you created on your local computer, you should delete the existing local path (Current path of CSV file). If you don't delete the local path, JMeter cannot find the CSV data file on your local PC.

Follow file naming convention

Don't save test plan under complex file name, use only alphanumeric characters.

How to perform Distributed Testing in JMeter

Overview

Distributed testing is a kind of testing which use multiple systems to perform stress testing. Distributed testing is applied for testing web sites and server applications when they are working with multiple clients simultaneously.
Distributes testing uses client-server model as figure below:
  • Master: the system running JMeter GUI, control each slave.
  • Slave: the system running jmeter-server, receive command from the master and send a request to server under test.
  • Target: the web server under test, get request from slaves.

Start your test

Precondition:
  • The firewalls on the systems are turned off. In some cases, the firewall may still be blocking the traffic. You should disable the Window firewall or Linux firewall.
  • All the machines should be on same subnet. If machines are not on same subnet, maybe they will not recognize each other in the network.
  • Use the same version of JMeter to avoid unanticipated errors/issues.
Here is the roadmap of this testing:

Step 1) System configuration

On the slave systems, go to jmeter/bin directory and execute file "jmeter-server.bat".
Assume that a slave machine has IP address: 192.168.0.10. On windows, you should see a window appear like following figure:
On the master systems, go to /bin directory and edit file jmeter.properites, add IP slave machine as below

Step 2) Run the test

On the master machine, run JMeter GUI and open the test plan.
Click Run on the menu bar; select Remote start -> select the IP address of slave machine

Step 3) Troubleshooting

If you are unable to run test form the above machine and see below error, simply ask owner of slave machine to run the jmeter-server.bat File.

Limitation:

There are some basic limitations for distributed testing. Here's list of the known items:
  • Server and all clients must be on the same subnet.
  • Distributed testing required target server to have large processing power. The target Server could be easilyoverloaded in case it gets too many requests by distributed JMeter tests.
  • A single JMeter can only handle a limited number of threads (100- 300 threads).
  • The distributed JMeter tests are complex, difficult for beginner to build.

How to use Processor in JMeter

Processor is used to modify the Samplers in their scope.
There are 2 Types of processors:
  1. Pre-processor
  2. Post-processor

Pre-processor:

Pre-processor executes some action before making Sampler Request.
Consider a simple example: let's say you wanted JMeter to "spider" through website under test, parse link(check all links on the page) and return the HTML. You would add some action such as "HTML link parser" to your controller before creating an HTTP request.

Post-processor:

Post-processor executes some action after making a Sampler Request.
Consider a simple example: JMeter send HTTP request to the web server under test (etc www.google.com) and get the response. You want JMeter to stop the testif the server response is error. You can use the post-processor to do above task as following:

Processor-Hands on

This tutorial will show you step-by-step instructions how to use Post-processor in JMeter. Let start with simple test script.
  1.  JMeter sends HTTP request to the web server under test www.google.com.
  2.  JMeter gets response from the Google server.
  3.  If server response is error, JMeter will stop the test.
  4.  If server response OK (no error), JMeter will continue the test.
Here is the roadmap of this example:
Pre-condition:
We re-use the Step 1 and Step 2 in article  JMeter Performance Testing.

Step 1) Add Thread Group

Right click on the Test Plan and add a new thread group: Add -> Threads (Users) -> Thread Group
But in Thread Group control panel, enter Thread Properties as following:
This setting lets JMeter create 10 user request to http://www.google.com  10 times.

Step 2) Add JMeter elements

  • Add HTTP request default
  • Add HTTP request
We still make JMeter send request http://www.google.com to Google server.

Step 3) Add Post-Processor Element

Right Click Thread Group -> Add -> Post Processor -> Result Status Action Handler
Result Status Action Handler allows the user to stop the thread or the whole test if the user request failed.
In Result Status Action Handle Pane, choose Stop Test Now. This selection will stop the test if JMeter get the error from server response.

Step 4) Config the HTTP Request

Open the HTTP Request Panel. Enter "abc" to the Path field.
When you enter "abc" to the path, JMeter will create URL request to Google server:http://www.google.com/abc. This URL doesn't exist on Google server. It is wrong URL request so Google server will return error.

Step 5) Add View Result Tree

Right Click Thread Group  -> Add  -> Listener  -> View Result Tree

Step 6) Run Test

Select View Result Tree, press Run button on Menu bar. You will see the error response from Google server and the test will stop with out completing 100 threads.
Now return to step 4, open the HTTP Request pane, enter "calendar" to the pane. It makes JMeter create URL request http://www.google.com/calendar to the Google server. This is correct URL request so Google server will return OK (no error).
Select View Result Tree, press Run button on Menu bar. You will see the OK response from Google server and the test will continue until all 100 threads are complete.

Troubleshooting:

If you face the issue while running the above scenario ... do the following:
  1. Check whether you are connecting to internet via a proxy. If yes, remove the proxy.
  2. Open a new instance of Jmeter
  3. Open the ProcessorTestPlan.jmx in Jmeter
  4. Double-click on Thread Group -> View Results Tree
  5. Run the Test

How to use Controllers in JMeter

If you want to control "when" to send a user request to a web server under test, what would you do?
JMeter gives us a feature to do that. It's Logic Controllers.

Logic Controllers

Logic Controllers let you define the order of processing request in a Thread. For example, you can use Random Controllers to send HTTP requests to the server randomly.
Logic Controllers determine the order in which user request are executed.
Some commonly used Logic controllers are below:

Recording Controller:

JMeter can record your testing steps; recording controller is a place holder to store these recording steps.

Simple Controller:

Simple Controller is just a container for user request.

Loop Controller:

Loop Controller makes the user request run specified number of times or run forever as shown in figure:

Random Controller:

Random Controller makes all the user requests run in random order in each loop period.
For example, you have 3 user requests to website http://www.google.com in following order:
  1. HTTP request
  2. FTP request
  3. JDBC request
These 3 requests should run 5 times; Total 15 (5*3) user requests will be sent to Google server by JMeter.
In sequential order, requests are sent sequentially in following order:
HTTP request ->FTP request->JDBC request
for each loop.
In random order, requests are sent as randomly,
FTP request ->HTTP request->JDBC request
Or
JDBC request ->FTP request->HTTP request
For each loop.

Module Controller:

The goal of Module Controller is to add modularity to JMeter.
The general idea is that web applications consist of small units of functionality (i.e. Logon, Create Account, Logoff...). This functionality can be stored in Simple Controller as "modules".  Module Controller will choose which module needs to run.
Consider the following scenario -
You want to simulate:

  • 50 users logging out,
  • 100 users logging in
  • 30 users  search www.google.com
You can use JMeter to create 3 modules. Each module simulates each user activity: Login, Logout, and Search.
The Module controller chooses which module needs to run.

Other Important Controllers:

  • Interleave Controller:  picks up and makes one of user request run in each loop of the thread.
  • Runtime Controller: controls how long its children are allowed to run.
For example, if you specified Runtime Controller 10 seconds, JMeter will run your test for 10 seconds.
  • Transaction Controller: measures the overall time taken to finish a test execution
  • Include Controller: is designed to use an external test plan. This controller allows you to use multiple test plans in JMeter. See detail in JMeter Performance Testing.

Handons with Loop Controller

This section shows you step-by-steps instruction to add Loop Controller setting to your current performance test plan.
The Loop Controller makes the samplers run as a certain number of times, in addition to the loop value you specified for the Thread Group. For example, if you
  • Add one HTTP Request to a Loop Controller with a loop count 50
  • Configure the Thread Group loop count to 2
  • Then, JMeter will send a total of 50 * 2 = 100 HTTP Requests.
This is the roadmap of this example:

Step 1) Configuring Thread Group

We re-use the Step 1, 2 in tutorial JMeter Performance Testing.
  1. Add Thread Group

          Right click on the Test Plan and add a new thread group: Add-> Threads (Users) ->Thread Group
          But in Thread Group control panel, enter Thread Properties as following:
          It will make one user request to the web server google.com and  run it 2 times.
  1. Add JMeter elements

          Add HTTP request default to www.google.com.
  1. Adding Loop Controller

          Right Click Thread Group -> Logic Controller -> Loop Controller

Step 2) Configuring Loop Controller

Add value 50 to Loop Count field as below figure. It will make one user request to the web server google.comrun it 50 times, in addition to the loop value =2 , you specified for the Thread Group above.So JMeter will send a total of 2 * 50 = 100 HTTP Requests.
Right click Loop Controller, Add -> Sampler -> HTTP request

Step 3) Add View Results in Table

We re-use Step 2 in Timer to add View Results in Table
So the test plan is shown in below figure

Step 4) Run your test

Now return View Results in Table, click Start button on Menu bar (Ctrl+R) to run test
As shown in the figure below, JMeter simulates one user request, which is sent 100 times, to the web serverhttp://www.google.com/. The Test is stopped after user request was sent in 100 times.

Troubleshooting:

  1. If you face the issue while running the above scenario ... do the following
  2. Check whether you are connecting to internet via a proxy. If yes, remove the proxy.
  3. Open a new instance of Jmeter
  4. Open the ControllerTestPlan.jmx in Jmeter
  5. Click on Thread Group -> View Result in Table
  6. Run the Test

How to use Assertions in JMeter

Assertion help verify that your server under test returns the expected results.
Following are some commonly used Assertion in JMeter:

Response Assertion

The response assertion lets you add pattern strings to be compared against various fields of the server response.
For example, you send a user request to the website http://www.google.com and get the server response. You can use Response Assertion to verify if the server response contains expected pattern string (e.g. "OK").

Duration Assertion

The Duration Assertion tests that each server response was received within a given amount of time. Any response that takes longer than the given number of milliseconds (specified by the user) is marked as a failed response.
For example, a user request is sent to www.google.com by JMeter and get a response within expected time 5 ms then test case pass, else, test case failed.

Size Assertion

The Size Assertion tests that each server response contains the expected number of byte in it. You can specify that the size be equal to, greater than, less than, or not equal to a given number of bytes.
JMeter sends a user request to www.google.com and gets response packet with size less than expected byte 5000 bytes àtest case pass. If else, test case failed.

XML Assertion

The XML Assertion tests that the response data consists of a formally correct XML document.

HTML Assertion

The HTML Assertion allows the user to check the HTML syntax of the response data. It means the response data must be met the HTML syntax.

Handson - Assertion

We will continue on the script we developed in the  earlier tutorial.
In this test, we are using Response Assertion to compare the response packet from www.google.com matches your expected string.
Here is the roadmap of this test:
The response assertion control panel lets you add pattern strings to be compared against various fields of the response.

Step 1) Add Response Assertion

Right-Click Thread Group -> Add -> Assertions -> Response Assertion
Response Assertion Pane displays as below figure:

Step 2) Add Pattern to test

When you send a request to Google server, it may return some response code as below:
  • 404: Server error
  • 200: Server OK
  • 302: Web server redirect to other page. This usually happens when you access google.com from outside USA. Google re-directs to country specific website. As shown below, google.com redirects to google.co.in for Indian Users.


Assume that you want to verify that the web server google.com responses code contains pattern 302,
On Response Field To Test, choose Response Code,
On Response Assertion Panel, click Add -> a new blank entry display -> enter 302 in Pattern to Test.

Step 3) Add Assertion Results

Right click Thread Group, Add -> Listener -> Assertion Results

Step 4) Run your test

Click on Thread Group -> Assertion Result
When you ready to run test, click Run button on the menu bar, or short key Ctrl+R.
The test result will display on Assertion Results pane. If Google server response code contains the pattern 302, the test case is passed. You will see the message displayed as follows:
Now back to Response Assertion Panel, you change the Pattern to test to from 302 to 500.
Because Google server response code doesn't contain this pattern, you will see the test case Failed as following:

Troubleshooting:

If you face the issue while running the above scenarios ... do the following:
  1. Check whether you are connecting to internet via a proxy. If yes, remove the proxy.
  2. Open a new instance of JMeter
  3. Open the AssertionTestPlan.jmx in JMeter
  4. Click on Thread Group -> Assertion Result
  5. Run the Test