Testing Webservices Using Soapui And Jmeter

Web servers are an integral part of a web application. Choosing the right web server that meets the application�s requirements and the expected load is a key decision in providing a real-time response. This paper studies and compares the performance of different open source web servers like Apache Tomcat and Jetty under different loads. The objective is to carry out load testing using Jmeter, which is an open source tool, written in java. In the process, web service used in load testing is functional tested using an open source tool-soapUI. The paper also explores the capabilities of these two tools. The webservice used for load testing the web servers is a wrapper method around an open source project called Lucene, provided by the Apache foundation. Lucene provides full text indexing and searching capabilities in different libraries. The paper will use the experimental results to calculate the peak TPS of the webservers and determine the capabilities/limitations of the above webservers.

�Discovering the unexpected is more important than confirming the known�-(George Box). Software testing is an essential and integral part of the software development cycle. This can be done either manually or by using automated testing tools. The later method is more preferred in recent times as it saves a lot of time and helps catch the software bugs missed by manual testing. There are different types of software testing techniques that include- functional testing, integration testing, load testing, regression testing etc. Functional testing involves testing the features and operational behavior of a software to ensure they correspond to its specifications. Load testing involves modeling the expected usage of a software program by simulating multiple users accessing the program in parallel. The purpose of the paper is to serve as a case study for evaluating the performance of web servers.

Testing can be aided significantly by several testing tools. This paper performs functional testing of web service using soapUI tool. SoapUI is an open source testing tool which mainly performs functional testing of Web services (SOAP, REST etc). SOAP (Simple Object Access Protocol) is a simple mechanism for exchanging structured and typed information between peers in a decentralized distributed environment using XML [1]. In this paper, wrapper webservice methods are written on top of APIs provided by Lucene. Apache Lucene is a high performance, full-featured text search engine library written entirely in Java with powerful, accurate and efficient search algorithms [3]. It is open source software suitable for most of the applications that require full-text search, especially cross-platform. Applications based on Lucene include: Eclipse, JIRA, Roller, OpenGrok, Nutch, Solr etc [2]. Lucene exposes methods that provide the functionality for indexing text documents and also provides APIs to search indexed documents based on a certain set of predefined fields. These methods exposed by the libraries are used by the webservice to provide indexing and search functionality. This webservice is functional tested using soapUI.

There are different open source web servers available in the market. Jakarta Tomcat, Miniature Java Web server and Jetty are a few examples of open source web servers. Web servers are containers that host web applications/webservices. The lucene webservice is then hosted on two different webservers (Tomcat and Jetty) and these web servers are load tested using JMeter. Apache Jmeter, is another open source testing tool that is used to load test web applications. The webserver�s performance is analyzed under different loads generated using Jmeter. Jmeter is used to simulate heavy load on a server and test the webserver�s capabilitylimitation.

Section 2 of this paper describes the environment setup needed for this project. presents current work and industry acceptance of some of the functional and load testing tools. Section 4 describes Rigi and highlights its limitation. Section 4 describes Understand 2.5 as a reverse engineering tool. Section 5 demonstrates reverse engineering of SimpleDB, an open source code. Section 6 concludes reverse engineering.

2. Environment

Since soapUI and Jmeter are java based tools, it runs on most operating systems including Windows, Mac and Linux. The study will be done on Windows 7. SoapUI and Jmeter require 1.6+ JRE (Java Runtime Environment). It requires at least 1 GB of memory, and about 100 MB of disk space.

Tools: soapUI and Jmeter

Web servers: Apache Tomcat and Jetty

Web service: To avoid the Web service being the bottleneck, a simple Web service that calculates the area of different shapes will be used as the Web service. This Web service can be extended later to perform other functions.

3. Current Practices

This section presents current work and industry acceptance of some of the functional and load testing tools. soapUI and JMeter are two free tools used by the open source community. Section 3.1 talks about different commercial and open source functional testing tools. Section 3.2 concentrates on load testing tools.

3.1. Functional testing tools

Some of the functional testing tools for testing internet applications currently being used by industries are listed below

A) soapUI

soapUI is a tool used for functional testing web services. Many industries are currently using soapUI for testing. It is being used to implement test suits for Alfresco Web services. soapUI is being used in a handful of public sector projects lately. As part of the Czech eGovernment project Datov� Schr�nky soapUI is used for session-based login scenario with Datov� Schr�nky [16]. It is also used in testing ASP.NET Web services. It is a big success in market as it comes with plug-ins for many IDE�s/tools like NetBeans, Maven, Eclipse, IntelliJ IDEA 6+ etc [16].

B) WebLOAD Professional

WebLOAD Professional is RadView�s commercial tool for performance testing of Internet applications [3]. It has built-in support for AJAX technologies, JSON data types and different types of SOAP and XML Web services [3].

C) Quicktest Professional

QTP is widely used commercial test automation tool mainly for functional testing. It allows the users to fill out a Web page and to submit it. It then follows the request from the client workstation through the network. The message is traced to the Web service that processes it [4].

D) TestMaker

TestMaker is a Web service testing application from PushToTest. TestMaker’s tests are embodied in scripts called �test agents. It can also be used to test Web applications [15]. Bundled with TestMaker is a network monitoring tool that can watch HTTP traffic between your browser and a target Web application, and generate test cases from the interaction [15].

3.2. Load testing tools

With applications like twitter and facebook becoming popular day by day, the amount of traffic that web servers handle has increased tremendously. The importance of tuning the performance of web servers has become crucial. Web server performance largely depends of components such as server hardware platform, Web server software, server operating system, network bandwidth, file sizes, caching, etc [5]. The performance of the web servers can be increased tremendously by proper tuning of the components of the server [5]. Most of the web applications run smoothly with minimal traffic. But when thousands of users access a web site in parallel, it tends to run slowly. Hence it is always a good practice to thoroughly load test web site/Web services before it goes live. Industries, when choosing the right web server for the web site being developed, usually take the expected load into account. They will also stress test Web servers by running 20-40 simultaneous requests on one URL [5]. The average time for the responses is then recorded. This will help them make the right decision. Listed are a few of the load testing tools used in the market (particularly open source market).

A) JMeter

JMeter is an open source Apache Jakarta project that can be used as a load testing tool for analyzing and measuring the performance of a variety of services, with a focus on web applications. This will be discussed in detail in section 4. JMeter is used worldwide by many companies for testing. AOL use JMeter for performance testing, QA, and adhoc production requests [14]. PrepMe is another company that uses JMeter. They wrap JMeter with a few scripts to dynamically generate configuration files based on the type of tests to be run [14]. This is integrated with their internal Symfony framework installation. Sapient and orbitz also use JMeter [14]. Advantages of Jmeter are it is open source and it provides the ability to load and performance test different kinds of Web server types like web, SOAP and database via JDBC. It also provides support for displaying the results using different Graphs and will support regression testing in the future.

B) Grinder

The Grinder is an open source load testing tool designed to load test functional behavior and measure performance. Grinder scripts are written in Jython programming language. It adopts a generic approach that allows load testing of anything that has a Java API. For example, HTTP web servers, SOAP and REST Web services [3].

C) NeoLoad

NeoLoad is a commercial load and stress testing tool to measure the performance of the web application and provide pragmatic solutions facilitating design and development of the optimized website. Benefit of NeoLoad is easy creation of tests, which can be performed by non specialists. [3].

Although commercial tools offer richer set of features and are in general easier to use, available open source tools are sufficient to successfully perform given tasks. Their usage requires higher level of technical expertise but they are a lot more flexible and extendable

4. Details

4.1. Functional Testing

The Web servers will host the Web service (CalculateArea) that returns the area of different shapes. Using soapUI, the functionality of the Web service will be tested with different test cases. The following are the methods exposed by the CalculateArea Web service

1) Double AreaOfRectangle (double length, double breadth)

2) Double AreaOfCircle (double radius)

3) Double AreaOfCylinder (double radius, double height)

4) Double AreaOfTriangle (double length, double height)

This service can be extended later to include methods to calculate the area of other shapes.

Figure 1. Functional testing of Web servers using soapUI tool

Testing of Web services has to target three aspects:

the discovery of Web services , the data format exchanged (i.e.,Web Services Description Language, WSDL), and request/response mechanisms (i.e., SOAP). The Web Services Description Language (WSDL) is an XML-based language that provides a model for describing Web services [1][5] . The data format and request/response mechanisms can be tested by invoking requests and observing responses with test data representing valid and invalid data formats[1]. The above three scenarios will be tested using soapUI. soapUI will validate and import Web services from SOAP based WSDL documents. Fig 1 explains the functional testing that will be carried out in details. Each method invocation will be a request to invoke that particular method (ex:AreaOfCircle) on the Web service. The Web server will receive the request and determine which application will handle the request and pass on the request to the particular application. The application (CalculateArea Web service) will receive the request and process the request. The application will throw an exception if there are any issues (missing/invalid parameter value or invalid parameter type) with the request. Otherwise, it will perform the necessary service and return the result in the response which is forwarded to the component that invoked the service (soapUI in this case).

The request and the response will be examined to make sure the response is the expected response for each scenario. If the Web service throws an exception, soapUI throws the same exception so that the user will know what the exception was for. We will use Jakarta Tomcat and Jetty to host the Web service to be tested.

4.2. Load testing of the application and stress testing the web server

Figure 2: Load/stress testing of Web servers using JMeter tool

As mentioned in the previous section, the Web service will be hosted on both Jakarta tomcat and Jetty. Jmeter, an open source load testing tool, will be used to perform the load testing/stress testing on both the web servers. As shown in Figure 2, the load on the Web server will be gradually increased by increasing the number of requests every time. The response time will be noted for each of the response from the web server. This result will help in calculating the time taken by the web server to process these requests. This will be used to calculate the TPS (transactions per second) that each Web server handles under each load. These results should be a good indicator on how the above web server scales for increasing loads. The number of requests per second will be increased by increasing the requests per second from Jmeter and also by using multiple running instances of Jmeter (thereby simulating multiple concurrent users).

The above test will be performed for the following number of transaction on each server and the results will be noted.

Table 1. Load metrics

Requests per second Time taken to handle the requests % Passed transaction % Failed Transaction Transaction Per Second Handled by the Web server

5. Future Directions

When trying to choose Open Source test tools, several factors may be considered, such as how active the development is, the support for the tool, the quality of the documentation, etc. JMeter is free and extensible tool with a substantial community support. One of the JMeter strong points is the distributed testing feature. The master system (controller) carries the user interface or XML test scripts and makes RMI calls to each of the nodes (load generators). Each node receives a mirror of the test plan and performs identical actions on a target Web application [3]. As developers take advantage of JMeter�s pluggable architecture, JMeter’s capabilities will rapidly expand in future. The primary goal of further development is to make JMeter the most useful regression testing tool as possible, without compromising JMeter’s load-testing capabilities. Additional goals are to improve the graphical user interface.

Since there is no support for scripting and advance reporting in soapUI currently, there is scope to add scripting libraries and enhance reporting features in future.

6. Limitations

Open source testing tools are inexpensive (free), and work well for easy to moderately-difficult jobs but they’re somewhat less user-friendly than commercial tools and complex jobs must be built manually. Some limitations of soapUI include- no maintenance as it is a freely available tool. Scripting libraries are not available. No support for advance reporting. These limitations can be overcome by using soapUI pro, a commercial tool used for testing. Also, soapUI test cases cannot be used as examples illustrating the use of the service. Junit test cases can be used to overcome these limitations. soapUI also requires significant coding for building/maintaining test cases and suites. Also, in the place of actual input data for generated test cases, soapUI leaves question marks (or alternatively some pregenerated Latin words or numbers) [7]. In addition, it does not manage the choice and all elements and the occurrence attributes, but introduces comments explaining to the human tester how to handle them.

JMeter has certain drawbacks. There�s no guarantee of support or further development as it is an open source tool. It has limited reporting capabilities. Graphic representation is quite confusing and has no support for scripting. One of the most distinguished disadvantages is insufficient support for test case recording. JMeter uses proxy which can only record HTTP while HTTPS is supported only in so called, spoofing mode. However, there are other tools which can be used in conjunction with JMeter and provide HTTPS test case recording. One such tool is Badboy tool which may be defined as a Web application functional testing tool [3].

7. Discussion and Summary

One of the chief concerns while choosing a testing tool is the cost of the tool. Since software testing usually attracts the least budget in an organization, many people look to the Open Source community for good solutions. This paper is an attempt in showing that there are good open source tools like soapUI and JMeter in the market that accomplishes most of the functionality supported by commercial tools. This paper also studies the performance of different open source web servers for different loads.