Performance
testing is a general term used to describe testing activities where a
simulated user load is directed at a system and measurements are
gathered. It is designed to meet or exceed a set of system performance
goals while maintaining a particular user load profile. It places
variable load – from a minimum to a maximum – on the system. The
variable load demonstrates what the system can sustain without running
out of resource or having transactions perform in a less-than-optimal
fashion.
Performance testing is a suite of testing that can consists of, but
is not limited to, Benchmark Testing, Load Testing, Durability Testing,
Volume Testing, Stress Testing, and Scalability Testing. In some cases a
separate Performance Test Plan will be created to explain the details
of the Performance Testing effort.
Benchmark Testing: Benchmark testing is a performance test which
subjects the system to varying workloads to measure and evaluate the
performance behaviors and ability of the system to continue to function
properly under these different workloads.
Durability Testing: Durability testing is designed to determine the
characteristics of a system under load conditions, over time. This is an
excellent test to identify memory leaks in a system.
Load Testing: Load testing is a performance test which subjects the
system to varying workloads to measure and evaluate the performance
behaviors and ability of the system to continue to function properly
under these different workloads. The goal of load testing is to
determine and ensure that the system functions properly beyond the
expected maximum workload. Additionally, load testing evaluates the
performance characteristics (response times, transaction rates, and
other time sensitive issues).
Scalability Testing: Scalability Testing determines the ability of a
system to expand as the usage increases. It measures individual
components of the system to determine their performance characteristics
under increasing user loads. It also examines inter-component and
multiple component behavior.
Stress Testing: Stress Testing determines the ability of the
application to perform under low and/or excessive loads to ensure that
critical information and services are available when and how the
end-user expects it. In addition, it is also a way to identify and
document conditions under which the system fails to continue functioning
properly.
Volume Testing: Volume Testing subjects the target-of-test to large
amounts of data to determine if limits are reached that cause the
software to fail. Volume testing also identifies the continuous maximum
load or volume the target-of-test can handle for a given period.
NOTE: Transactions above refer to “logical
business transactions.” These transactions are defined as specific
functions that an end user of the system is expected to perform using
the application, such as add or modify a given contract.
Below are some open source/free tools that can help you with
performance testing the application under test (AUT).
Please keep in mind that every AUT is different so the tool you pick
for one application may not be the same tool that you pick for another.
My advice would be to pick a tool that can meet most of your current and
near future needs.
Allmon:
A generic system collecting and storing metrics used for performance
and availability monitoring
Apache JMeter:
A 100% pure Java desktop application designed to load test functional
behavior and measure performance. It was originally designed for
testing Web Applications but has since expanded to other test
functions. Apache JMeter may be used to test performance both on static
and dynamic resources (files, Servlets, Perl scripts, Java Objects,
Data Bases and Queries, FTP Servers and more). It can be used to
simulate a heavy load on a server, network or object to test its
strength or to analyze overall performance under different load types.
You can use it to make a graphical analysis of performance or to test
your server/script/object behavior under heavy concurrent load. Requirement:
Solaris, Linux, Windows (98, NT, 2000). JDK1.4 (or higher).
benerator:
A framework for creating realistic and valid high-volume test data,
used for (unit/integration/load) testing and showcase setup. Metadata
constraints are imported from systems and/or configuration files. Data
can be imported from and exported to files and systems, anonymized or
generated from scratch. Domain packages provide reusable generators for
creating domain-specific data as names and addresses
internationalizable in language and region. It is strongly customizable
with plugins and configuration options. Requirement:
Platform Independent
CLIF is a
Load Injection Framework: A modular and flexible distributed load
testing platform. It may address any target system that is reachable
from a Java program (HTTP, DNS, TCP/IP…) CLIF provides 3 user
interfaces (Swing or Eclipse GUI, command line) to deploy, control and
monitor a set of distributed load injectors and resource consumption
probes (CPU, memory…) An Eclipse wizard helps programming support for
new protocols. Load scenarios are defined through XML-editing, using a
GUI, or using a capture tool. The scenario execution engine allows the
execution of up to millions of virtual users per load injector. Requirement:
Java 1.5 or greater, with enhanced support for Linux, Windows XP,
MacOSX/PPC
curl-loader:
A C-written web application testing and load generating tool. The goal
of the project is to provide a powerful open-source alternative to
Spirent Avalanche and IXIA IxLoad. The loader uses real HTTP, FTP and
TLS/SSL protocol stacks, simulating tens of thousand and hundred
users/clients each with own IP-address. The tool supports user
authentication, login and a range of statistics. Requirement:
linux
Database
Opensource Test Suite (DOTS): A set of test cases designed for the
purpose of stress-testing database server systems in order to measure
database server performance and reliability. Requirement:
Linux, POSIX
DBMonster:
Application to generate random data for testing SQL database driven
applications under heavy load. Requirement: OS
Independent
Deluge:
Simulates multiple user types and counts. Includes proxy server for
recording playback scripts, and log evaluator for generating result
statistics. Note: this tool is no longer under active development
although it is still available on Sourceforge. Requirement:
OS independent
Dieseltest:
Contains the high-end features common to packages costing $50,000 or
more. Dieseltest is a Windows application that simulates hundreds or
thousands of users hitting a website. To run a load test, you first
create a test script using our script editor. The script contains all
of the requests that a real-world user would make of a website. You
then load the script and run the test. The system will show you
real-time results while the script is running, and produce a report
analyzing the results at the conclusion. Requirement:
Windows
Eclipse
Test & Performance Tools Platform Project (TPTP): Supplies
powerful frameworks and services that allow software developers to build
unique test and performance tools, both open source and commercial,
that can be easily integrated with the platform and with other tools
Faban: A facility for developing and running
benchmarks, developed by Sun. It has two major components, the Faban
harness and the Faban driver framework. The Faban harness is a harness
to automate running of server benchmarks as well as a container to host
benchmarks allowing new benchmarks to be deployed in a rapid manner.
Faban provides a web interface to launch & queue runs, and
extensive functionality to view, compare and graph run outputs. Requirement:
OS independent; JVM 1.5 or later.
FunkLoad: A functional and load web tester, written
in Python, whose main use cases are functional and regression testing
of web projects, performance testing by loading the web application and
monitoring your servers, load testing to expose bugs that do not
surface in cursory testing, and stress testing to overwhelm the web
application resources and test the application recoverability, and
writing web agents by scripting any web repetitive task, like checking
if a site is alive. Requirement: OS independent –
except for the monitoring which is Linux specific.
FWPTT:
Web application testing program for load testing web applications. It
can record normal and AJAX requests. It has been tested on ASP.Net
applications, but it should work with JSP, PHP or other. Requirement:
windows
Grinder: A Java-based load-testing framework freely
available under a BSD-style open-source license. Orchestrate
activities of a test script in many processes across many machines,
using a graphical console application. Test scripts make use of client
code embodied in Java plug-ins. Most users do not write plug-ins
themselves, instead using one of the supplied plug-ins. Comes with a
mature plug-in for testing HTTP services, as well as a tool which
allows HTTP scripts to be automatically recorded. Requirement:
OS Independent
Hammerhead
2: A stress testing tool designed to test out your web server and
web site. It can initiate multiple connections from IP aliases and
simulated numerous (256+) users at any given time. The rate at which
Hammerhead 2 attempts to pound your site is fully configurable, there
are numerous other options for trying to create problems with a web
site (so you can fix them). Requirement:Hammerhead has
been used with Linux, Solaris and FreeBSD.
Hammerora: A load generation tool for the Oracle
Database and Web Applications. Hammerora includes pre-built schema
creation and load tests based on the industry standard TPC-C and TPC-H
benchmarks to deploy against the Oracle database with multiple users.
Hammerora also converts and replays Oracle trace files and enables
Web-tier testing to build bespoke load tests for your entire Oracle
application environment. Requirement: Platform
Independent (Binaries for Linux and Windows)
httperf:
A tool for measuring web server performance. It provides a flexible
facility for generating various HTTP workloads and for measuring server
performance. The focus is not on implementing one particular
benchmark but on providing a robust, high-performance tool that
facilitates the construction of both micro and macro level benchmarks.
The three distinguishing characteristics of httperf are its robustness,
which includes the ability to generate and sustain server overload,
support for the HTTP/1.1 and SSL protocols, and its extensibility. Requirement:
linux (Debian package available), HP-UX, perhaps other Unix
http_load:
Runs multiple HTTP fetches in parallel, to test the throughput of a
Web server. However, unlike most such test clients, it runs in a single
process, to avoid bogging the client machine down. It can also be
configured to do HTTPS fetches. Requirement: tbc
IxoraRMS:
A tool for gathering and visualizing monitoring data. Requirement:
Windows, Unix
JChav: A way to see the change in performance of
your web application over time, by running a benchmark test for each
build you produce. JChav reads all the JMeter logs from each of your
runs (one per build), and produces a set of charts for each test in
each run. Requirement: JMeter
JCrawler: Stress-Testing Tool for web-applications.
It comes with the crawling/exploratory feature. You can give JCrawler a
set of starting URLs and it will begin crawling from that point
onwards, going through any URLs it can find on its way and generating
load on the web application. The load parameters (hits/sec) are
configurable. Requirement: OS Independent
Load
Impact : Online load testing service from Gatorhole/loadimpact.com
for load- and stress- testing of your website over the Internet;
access to our distributed network of load generator nodes – server
clusters with very fast connections to enable simulation of tens of
thousands of users accessing your website concurrently. Free low level
load tests for 1-50 simulated users; higher levels have monthly fees.
Lobo,
Continuous Tuning: A tool for performance testing and monitoring
that allows you to monitor the evolution of performance along the
time-line of the project. It was specially designed to be used in
agile-iterative and evolutionary approaches. Requirement:
Java
MessAdmin: A light-weight and non-intrusive
notification system and HttpSession administration for J2EE Web
Applications, giving detailed statistics and informations on the
application. It installs as a plug-in to any Java EE WebApp, and
requires zero-code modification. Requirement: OS
Independant
Multi
Router Traffic Grapher (MRTG): Written in perl and works on
Unix/Linux as well as Windows and even Netware systems. MRTG is free
software licensed under the Gnu GPL.
NTime:
Very similar to NUnit tool to perform repeatable tasks that help
managers, architects, developers and testers to test an application
against its performance. Requirement: Windows 98 or
above, .Net framework 1.1 or 2.0
OpenSTA:
A distributed software testing architecture based on CORBA. Using
OpenSTA (Open System Testing Architecture) a user can generate
realistic heavy loads simulating the activity of hundreds to thousands
of virtual users. OpenSTA graphs both virtual user response times and
resource utilization information from all Web Servers, Application
Servers, Database Servers and Operating Platforms under test, so that
precise performance measurements can be gathered during load tests and
analysis on these measurements can be performed. Requirement:
Windows 2000, NT4 and XP
OpenWebLoad:
A tool for load testing web applications. It aims to be easy to use
and providing near real-time performance measurements of the
application under test. Requirement: Linux, Windows
p-unit: Framework for unit test and performance
benchmark, which was initiated by Andrew Zhang, under GPL license.
p-unit supports to run the same tests with single thread or
multi-threads, tracks memory and time consumption, and generates the
result in the form of plain text, image or pdf file. Requirement:
OS Independent
PandoraFMS: A monitoring software. It watches your
systems and applications, and allows you to know the status of any
element of those systems. Pandora FMS could detect a network interface
down, a defacement in your website, a memory leak in one of your server
application, or the movement of any value of the NASDAQ new
technology market. If you want, Pandora FMS could send out SMS message
when your systems fails… or when Google’s value drop below US$ 500. Requirement:
32-bit MS Windows (NT/2000/XP), All POSIX (Linux/BSD/UNIX-like OSes),
Solaris, HP-UX, IBM AIX
Parallel-junit: Small library extensions for JUnit.
Extends the junit.framework.TestSuite class by running tests in
parallel, allowing more efficient test execution. Because TestResult
and TestListener aren’t designed to run tests in parallel, this
implementation coordinates the worker threads and reorder event
callbacks so that the TestResult object receives them in an orderly
manner. In addition, output to System.out and System.err are also
serialized to avoid screen clutter.
Pylot:
Tool for testing performance and scalability of web services. It runs
HTTP load tests, which are useful for capacity planning, benchmarking,
analysis, and system tuning. Pylot generates concurrent load (HTTP
Requests), verifies server responses, and produces reports with
metrics. Tests suites are executed and monitored from a GUI. Requirement:
Python 2.5+. required.Tested on Windows XP, Vista, Cygwin, Ubuntu,
MacOS
Raw Load Tester: This application calls the URL you
select as many times as you choose and tells you how long it took the
server to respond. It writes some additional runtime details to the PHP
log file so you can optionally do more granular analysis afterwards.
Although the server processes most of the statistics, all URL requests
come from the browser. You can run as many browsers and workstations
simultaneously as you want. Requirement: PHP/xxJavaScript
RRDtool:
High performance data logging and graphing system for time series
data. Use it to write your custom monitoring shell scripts or create
whole applications using its Perl, Python, Ruby, TCL or PHP bindings.
Seagull: A multi-protocol traffic generator test
tool. Primary aimed at IMS protocols, Seagull is a powerful traffic
generator for functional, load, endurance, stress and performance tests
for almost any kind of protocol. Currently supports Diameter, XCAP
over HTTP, TCAP (GSM Camel, MAP, Win) protocols. Requirement:
Linux/Unix/Win32-Cygwin
Siege:
A http regression testing and benchmarking utility. It was designed to
let web developers measure the performance of their code under duress,
to see how it will stand up to load on the internet. It lets the user
hit a webserver with a configurable number of concurrent simulated
users. Those users place the webserver “under siege.” SCOUT surveys a
webserver and prepares the urls.txt file for a siege. In order to
perform regression testing, siege loads URLs from a file and runs
through them sequentially or randomly. Scout makes the process of
populating that file easier. You should send out the scout, before you
lay siege. Requirement: GNU/Linux, AIX, BSD, HP-UX and
Solaris.
SIPp: A performance testing tool for the SIP
protocol. Its main features are basic SIPStone scenarios, TCP/UDP
transport, customizable (xml based) scenarios, dynamic adjustement of
call-rate and a comprehensive set of real-time statistics. It can also
generate media (RTP) traffic for audio and video calls. Requirement:
Linux/Unix/Win32-Cygwin
SiteBlaster: A web site load and stress testing
tool. It can be used to rapidly submit requests to a site. Or, it can
pause a random amount of time between submissions; approximating the
behavior of a user. While the testing is being performed, the pages
being tested will be displayed. When testing is complete, a report is
available that can be viewed or printed. Requirement:
Windows Installer v 3.1 and Microsoft .Net Framework 3.5 SP1
SLAMD:
A Java-based application designed for stress testing and performance
analysis of network-based applications. Requirement:
Any system with Java 1.4 or higher
Soap-Stone:
Network benchmark application which can put your network under load
and conduct automatic benchmark and recording activities. Requirement:
OS Independent
stress_driver:
General-purpose stress test tool. Requirement:
Windows NT/2000, Linux
TestMaker: Delivers a rich environment for building
and running intelligent test agents that test Web-enabled applications
for scalability, functionality, and performance. It comes with a
friendly graphical user environment, an object-oriented scripting
language (Jython) to build intelligent test agents, an extensible
library of protocol handlers (HTTP, HTTPS, SOAP, XML-RPC, SMTP, POP3,
IMAP), a new agent wizard featuring an Agent Recorder to write scripts
for you, a library of fully-functional sample test agents, and shell
scripts to run test agents from the command line and from unit test
utilities. Requirement: Java 1.4 or higher virtual
machine on Windows, Linux, Solaris, and Macintosh.
TPTEST:
The purpose with TPTEST is to allow users to measure the speed of
their Internet connection in a simple way. TPTEST measures the
throughput speed to and from various reference servers on the Internet.
The use of TPTEST may help increase the consumer/end user knowledge of
how Internet services work. Requirement: MacOS/Carbon
and Win32
Tsung: A protocol-independent and can currently be
used to stress HTTP, SOAP and Jabber servers (SSL is supported). It
simulates complex user’s behaviour using an XML description file,
reports many measurements in real time (including response times, CPU
and memory usage from servers, customized transactions, etc.). HTML
reports (with graphics) can be generated during the load. For HTTP, it
supports 1.0 and 1.1, has a proxy mode to record sessions, supports GET
and POST methods, Cookies, and Basic WWW-authentication. It has
already been used to simulate thousands of virtual users. Requirement:
Tested on Linux, but should work on MacOSX and Windows.
Valgrind: Suite of tools for debugging and
profiling Linux programs. With the tools that come with Valgrind, you
can automatically detect many memory management and threading bugs,
avoiding hours of frustrating bug-hunting, making your programs more
stable. You can also perform detailed profiling, to speed up and reduce
memory use of your programs. Requirement: Linux
VisualVM: A
free visual tool, originally from Sun, to monitor and troubleshoot Java
applications. Runs on Sun JDK 6, but is able to monitor applications
running on JDK 1.4 and higher. Utilizes various available technologies
like jvmstat, JMX, the Serviceability Agent (SA), and the Attach API to
get data and uses minimal overhead on monitored applications.
Capabilities include: automatically detects and lists locally and
remotely running Java applications; monitor application performance and
memory consumption; profile application performance or analyze memory
allocation; is able to save application configuration and runtime
environment together with all taken thread dumps, heap dumps and
profiler snaphots into a single application snapshot which can be later
processed offline.
Web Application Load Simulator: A web application
load simulator. It allows you to create simulations and have those
simulations run against your webserver. Requirement:
JDK 1.3 or above
WebInject:
It can be used to test individual system components that have HTTP
interfaces (JSP, ASP, CGI, PHP, AJAX, Servlets, HTML Forms, XML/SOAP
Web Services, REST, etc), and can be used as a test harness to create a
suite of [HTTP level] automated functional, acceptance, and regression
tests. A test harness allows you to run many test cases and
collect/report your results. WebInject offers real-time results display
and may also be used for monitoring system response times
Web Polygraph:
Freely available benchmarking tool for caching proxies, origin server
accelerators, L4/7 switches, and other Web intermediaries. Other
features: for high-performance HTTP clients and servers, realistic
traffic generation and content simulation, ready-to-use standard
workloads, powerful domain-specific configuration language, and
portable open-source implementation. C++ source available; binaries
avail for Windows. Requirement: C++ compiler
WebLOAD: A fully functional, commercial-grade
performance testing product based on WebLOAD, Radview’s flagship
product that is already deployed at 1,600 sites. Available for free
download and use, WebLOAD is a commercial-grade open source project
with more than 250 engineering years of product development. Companies
that require commercial support, additional productivity features and
compatibility with third-party protocols have the option of purchasing
WebLOAD Professional directly from RadView. Requirement:
Windows NT/2000/XP
Xceptance
LoadTest: Load testing and regression tool from Xceptance Software
Technologies, Inc for web and Java and other app load testing.
Includes recording capabilities. XLT Cloud Service available. Tests
implemented as JUnit 4 test cases. For web-based tests, the framework
provides a (headless) browser that can emulate Internet Explorer or
Firefox behaviour. Can execute client-side xxJavaScript in the emulated
web browsers and that way it simplifies the creation of test cases for
Web 2.0 applications. Platform independent due to tool being
implemented in Java; test scripting in Java or Ruby. Free for up to five
virtual users.
zapwireless:
A wireless throughput test tool which can be used to fully characterise
the statistical performance of a wireless link.
Please comment below on your thoughts/experience on any of the tools
listed above and/or if you know of any other tools that should be added.