Amazon cloud instances (EC2) as performance test load generators

Recently I was given a task of executing a performance test for one of our customer's applications.

Test was simple. It was primarily to test a "Maps download" scenario (each file size to be around 30 MB). A dedicated Maps generating/downloading server is there to cater to the need and our task was to find out how well the server responds to the user need.

JMeter (2.9) was used for this test and it was ideal for the scenario.

Creating valid/ realistic user load

Doing some calculations/statistical analysis, we estimated a figure for "Downloads/Hour". Our test was around assessing the performance for the above.

Load Generation

For the purpose of load generation, we decided to go for a cloud solution, and we choose Amazon AWS services, after doing some researches using its "Free tier" provision.

Some reasons why we go for a Cloud solution

1. The hosting Map server is on the Internet - We, as test team is located far away geographically as opposed to the real uses who are using it. So we needed to perform the test as close as possible . Amazon gives the flexibility of choosing a closer location because their data centers are located all over. (Or at least in our case they had regions which we wanted)

2. We wanted to test the application at "last mile" as opposed to "First mile" because, no body knows what's beyong your firewall. You need to try out your entire delivery chain including your ISPs, 3rd party applications like CDN, Ad servers, other web services that your application consumes which are basically beyond your control.

3. We were afraid to do it using our internal network since it could generate heavy traffic inside which could have an impact on the rest. (Still we do not have a separated test lab for this purpose)

4. Since we do not have an internal test lab in place, there was an issue in finding right hardware to generate the anticipated load for the test.

Support from Amazon  

I was impressed by the support given by Amazon team (Support service) for some account related issues I had initially. It was prompt and useful.

Using EC2 Instances

For the test we used "On-demand" extra-large standard instances having 8 compute units (4 cores) and 15 Gig RAM. The IO performance relatively OK, having intermittent lagging.

Standard Instances comes with 30 Gig Hard disk and the operating system consumes 75% if it (It was Windows OS) and left over with 7GB usable. So had to add a new volume using the Management console and attach it to the instance. After restarting the instance, you will see a new partition being added in My computer.

Distributed Load testing using JMeter - Problem using EC2 instances

JMeter can make use of several load generators to empower its load generating capability. But, it is required that all the load generators need to hook to the same network in order to communicate among them. You might experience that your started instances belong to different subnets and thus  cannot ping to each other.

To solve this issue, we can create- VPC or Virtual Private Could in the Amazon cloud. I used to have a VPC having "VPC with a single public subnet only" configuration, and it serves the purpose.

Once you use a VPC, you can start-up instances inside this VPC so that they will be assigned IP addresses in the same subnet fulfilling our requirement. Also in order to communicate these instances with the Internet, you will have to allocate some ELASTIC IPs, and associate them to the instances.

Performance Testing challenges

#1 Create a valid load (Load validity)

The first challenge is the selection of a subset
of different transactions from the set of all
transactions that can be executed by clients.
The spread of transactions, and the percentage
of each transaction in relation to each other is
of the essence here. Whenever there are backoffice
transactions that have to be taken into
account, it might be necessary to verify the
impact of their execution upon the response
time experienced by users (more on this in the
second article in this series).

#2 Create realistic users (User Realism)

The second challenge is to have the users behaving
as realistically as possible, and thus
generating as realistic a load as possible in
terms of spread of data within a single transaction
(i.e. not always looking for the same set of
www.testingexperience.com The Magazine for Professional Testers 25
data in a table).


#3 Validate the responses (Response validity)

A standard practice for testing is to test the
actual results against expected results. In performance
tests, what often happens is that the
transaction responses are timed. This relies on
the responses being of adequate quality (functionally).
This is contrary to a basic tenet of
software testing that implies that you cannot
rely on an answer to remain the same if conditions
change. When one is stressing a system,
such a reliance is unwarranted, and the
response validity is to be verified.

#4 Reduce the number of injectors 1(Injector
reduction)

Simulating a large number of clients is the
fourth challenge. Here a fine balance exists
between adding more machines and simulating
more clients per machine. The first option
has an impact in terms of hardware (adding
injectors, connecting and synchronizing them
during campaigns), while the second has an
impact in terms of load on the client (injector)
machine, as simulating a large number of
transactions and validating the responses obtained
creates a load on the client hardware.

#5 Transactions coherence (Transaction realism)

A fifth challenge comes in terms of the validity
of the stream of transactions coming from
a client. This is the case when the transactions
involve different screens linked to each other,
such as when purchasing an item from an online
store, or processing banking transactions.
This requires the server to keep data linked to
the opened session through means other than
cookies on the client side or inclusion of the
full information in the request 2.

#6 Encryption (Encryption impact)

The use of virtual private network (VPN) and
dynamic encryption methodology to ascertain
that the transactions are secure can have an impact
on the ability of capture replay to create
transactions that will be correctly interpreted
by the server and not rejected due to invalid
encryption keys (that would happen in case
of encryption based on keys synchronized between
server and client).

Extracted from testingexperience magazine "Agile testing" Sept,2009 http://www.testingexperience.com/

Generate and Personalize a Html Report for JMeter with Ant

One of the easiest ways to generate a report out of JMeter test run is to make use of "extra" folder comes with JMeter and build with Ant

Steps to create an Ant report

1. Download Apache Ant and update user variable 'Path" pointing to the Ant installation 
2. Go to %jMeter_Installation%/extras folder, and then copy your .jmx (JMeter script) file there 
3. Using CMD prompt, run following command : jmeter -n -t test.jmx -l test.jmx.jtl to run the Jmeter script and then create the .jtl results file
4. Then run ant -Dtest=test.jmx to create the Ant report
with –Dtest switch, you point to your Jmeter script file 


If it is successful, you will get a html file created with the result (ex: test.jmx.html)
%jMeter_Installation%/extras contains necessary artifacts to run ant and generate the report. 
In build.xml, you can change the source and destination file locations if you want

make sure test.jmx.jtl file is there. Run jmeter in command line: jmeter -n –t test.jmx -l test.jmx.jtl to generate the .jtl file

If you get an error "Fatal Error! Content is not allowed in prolog." it could probbly be due to CSV format of your .jtl file, and for successful XSLT transformation it needs to be XML. 
To fix this add the following line to user.properties file (lives under /bin folder of your ${jmeter-home}

 jmeter.save.saveservice.output_format=xml

To automate the process , create a .bat file and schedule it in Task scheduler.Add following to the batch file.
del test.jmx.jtl (always delete the existing .jtl file before a new test run)
jmeter -n -t sod.jmx -l test.jmx.jtl
ant -Dtest=test.jmx

host the test.jmx.html in web server

Web page rendering- steps

Just happened to see this blog post which simply describes the steps on how a web page being rendered in the browser.

http://friendlybit.com/css/rendering-a-web-page-step-by-step/

Introduction to Performance Testing

Introduction to Performance Testing from Tharinda Liyanage

Basic JMeter functions useful in scripting

This is a small presentation to illustrate some of the basic JMeter functions/ features that can be used to accurately construct JMeter scripts

Useful JMeter functions for scripting from Tharinda Liyanage

Effetc of GZip compression in Web Performance

Following is a test I did for one of my web sites and statistics prove how compression can improve performance of a web site 

http://www.port80software.com/tools/compresscheck.asp

 

 

Useful PerfMon counter KPIs in Web Performance testing

Object\Counter	Preferred or Ideal Value
Memory\Pages/sec	0–20. (Unhealthy if greater than 80; probably indicates not enough RAM.)
Memory\Available Bytes	10% of physical memory.
Memory\Committed Bytes	No more than 75 percent of physical memory.
Memory\Pool Nonpaged Bytes	A steady value. (A slow rise might indicate a memory leak.)
Processor\% Processor Time	Less than 75 percent.
Processor\Interrupts/sec	Depends on the processor, and on network hardware and drivers. Up to 3,500 for a 90–megahertz (MHz) Pentium; more than 19,000 for a 500–MHz Pentium or more than 58,000 for a 1.5–gigahertz (GHz) Pentium. Lower is better. If the value is too high, try moving some hardware devices to a different server.
Processor\System Processor Queue Length	4 or less.
LogicalDisk\% Disk Time PhysicalDisk\% Disk Time	As low as possible.
LogicalDisk\Avg. Disk Queue Length PhysicalDisk\Avg. Disk Queue Length	Less than 4.
LogicalDisk\Avg. Disk Bytes/Transfer PhysicalDisk\Avg. Disk Bytes/Transfer	As high as possible.
System\Context Switches/sec	Compare this value with the value of Web Service\Total Method Requests/sec. Context switches per request (Context Switches/sec divided by Total Method Requests/sec) should be low.
System\System Calls/sec	As low as possible.
Web Service\Bytes Total/sec	As high as possible.
Web Service\Total Method Requests/sec	As high as possible.
Web Service\Current Connections	As high as possible.
Web Service Cache\File Cache Hits %	As high as possible for static content. Note: This value might be low if the Kernel URI cache hits percentage is high.
Web Service Cache\Kernel:URI Cache Flushes	As low as possible, relative to the number of requests. Note: This number increases every time a file is flushed from the HTTP.sys response cache (HTTP.sys is the kernel-mode device driver in IIS 6.0), which means that the content has not been accessed in the past 2–4 minutes. The only way to decrease this number is to flush the cache less often, although frequent flushing can cause HTTP.sys to use more memory for content that is not being accessed.
Web Service Cache\Kernel:URI Cache Misses	As low as possible. (Each request for dynamic content increases the value of the counter by one.)
Web Service Cache\Kernel:URI Cache Hits %	As high as possible. (Applies to static unauthenticated content and dynamic content that is marked as cacheable.)
Active Server Pages\Request Wait Time	As low as possible.
Active Server Pages\Requests Queued	As low as possible.
Active Server Pages\Transactions/sec	As high as possible. Note: ASP transactions degrade overall server performance because each transaction requires interaction with a database. If you are concerned about server performance, use ASP transactions sparingly.