Why you should seriously consider switching from Commercial Load Testing solutions to Apache JMeter
Few months ago we wrote a blog on Why Apache JMeter and Ubik Load Pack is your best load testing tool option .
Since then, a lot of things happened in the Load Testing world and the JMeter eco-system which make this quote even more true.
DevOps , CI and CD
With DevOps movement and the increasing popularity of Continuous Integration and Continuous Deployment, Load Testing has become a key player.
You cannot imagine nowadays deploying a new version in Production without integrating Load Testing in the pipeline. This means that you must run more and more frequent load testing on your projects to ensure time to market is met in correct quality conditions and no outages.
This is where Apache JMeter comes into play knowing it is free, almost freely available in the Cloud and easily integrated in your continuous integration pipeline.
For more information on CI/CD, you can read our blog on integrating JMeter in your build process with Maven. You will easily find on the net a lot of blogs, videos on this subject to see all the activities going around it.
JMeter
There has been some hard work on the Apache Project this year and we were one of the major contributors of this work thanks to:
- The donation of our UbikLoadPack JSON Plugin to the project. This feature make JSON a first class citizen in Apache JMeter
- The donation by a Decathlon SA and Ubik-Ingenierie of a new Dynamic Dashboard that allows you generate an HTML report for your Load Test with a wealth of informations:
- Summary report with APDEX, Errors, Results Summary
- 12 dynamic, zoomable graphs showing throughput, response times, errors code ….
The lack of such report in Apache JMeter was one of the major critics made to the product. It is now an old story
- The improvements made on Browser simulation of Downloaded Resources (CSS, JS, Images) which makes JMeter much more realistic in this field and much more scalable. This work has been contributed by Benoit Wiart and Philippe Mouawad, 2 members of our support Team.
The Core team and the community has also done a great job to improve many fields of JMeter,Apache JMeter 3.0 has a lot of awesome new features plus tens of bug fixes that you can discover here.
This version is the beginning of a new Era.
It is also very interesting to note that the project is more dynamic than ever with:
- New commiters invited
- A lot of Pull Requests made on its Github mirror
- An important increase in the project activity that you can see in OpenHub report
Besides, JMeter popularity keeps increasing, since our last post it has become available in the Cloud by more actors:
- Visual Studio ALM now proposes to launch JMeter load test from the Cloud, see <a href= »https://blogs.msdn.microsoft.com/visualstudioalm/2016/02/27/running-apache-jmeter-based-load-tests-in-the-cloud-how-to/ » target= »_blank » >this</a>
- Major APM Dynatrace has now an integration with Apache JMeter. It’s also interesting to know that their load testing offer is now based on JMeter
- Even HP Storm Runner allows you to run a JMeter script
Facts
To convince you it is worth considering switching your load testing to Apache JMeter and UbikLoadPack (for more protocols support and new features sponsoring) let’s compare it with some Major actors.
Cost:
Criteria | LoadRunner (HPE) | JMeter (Apache) | JMeter + UbikLoadPack (Ubik-Ingenierie) | NeoLoad (Neotys) |
Commercial License | Yes | No (Free and Commercial Compatible License (Apache)) | Yes | Yes |
Core Price | Free up to 50 VUs. License per number of Users | Free whatever the number of users | License per number of Users for Plugins only | Free up to 50 VUs. License per number of Users |
Additional Protocol Price | You pay per additional protocol + Core price | Not applicable | You pay ONLY for additional protocol but core (JMeter) is Free. Total Cost Of Ownership ends up being much lower than the competitors. | You pay per additional protocol + Core price, see here |
Cloud Load Testing Price | High | Low | Low | Medium |
Features:
Criteria | HP Loadrunner | Apache JMeter | NeoLoad |
Protocol Coverage | It has the widest protocols support (enterprise software SAP, Oracle ERP…) | HTTP(S), Webservices (Soap, Rest), FTP, LDAP, and Java-related protocols (JMS, JDBC). Support more protocols (Websocket, XMPP…) with Open Source and Commercial Plugins (HLS, GWT, FLEX, JAVA SERIALIZATION) | NeoLoad has good support of many Protocols. It supports enterprise software SAP SAP (Web only), Oracle Siebel etc. |
Record and Playback capability | Yes | Yes | Yes |
Cloud Based Solution | Yes (Amazon , Azure, Google for the Controller) | Yes (Amazon WS, Google Cloud, Azure, IBM BlueMix) | Yes (Custom) |
Continuous Integration (JENKINS, Bamboo…) | Yes | Yes | Yes |
Mobile Load Testing | Yes | Yes (Native + SAAS providers for more features) | Yes |
Real Time Test Monitoring | Yes | Yes (With BackendListener and Graphite based backend (InfluxDB, Graphite)) | Yes |
Script Customization | Hard (C language for all, Javascript added for Web in 12.5 version) and limited | Easy as JMeter is built on a Plugin architecture and made to be customized (JSR223 Elements allow you to use a lot of scripting languages (Groovy, Javascript, Java…) | Medium to Hard (Javascript for extremely complex scenarios as per documentation quote |
Result Reporting | Yes | Yes since JMeter 3.0 (APDEX, Summary and Error Tables, 12 graphs) | Yes |
Load Generation | Depends on Type of License | Unlimited Load Generation. For UbikLoadPack, Unlimited number of Load Agents, Number of Virtual Users depends on Type of License | Depending on Type of License |
IPSpoofing | Yes | Yes | Yes |
Support of authentication schemes and client certificates | Yes | Yes (Basic, Digest, Kerberos, …) | Yes |
Flexible error handling and adjustable pass/fail criteria | Yes | Yes | Yes |
Browser Support | Yes | Yes | Yes |
Windows 7 and above | Yes | Yes | Yes |
Mac OSX | No | Yes | No |
Linux | Yes(Not all Load Generators) | Yes | Yes |
Cluster (Scaling) | Yes | Yes | Yes |
Popularity:
Although not the most important factor, it is an interesting factor to look at, so we used Google Trends restricted to January 2014 to April 2016.
Software category
Programming category
It is interesting to see that JMeter popularity is very high (compared to other tools) and increases in the Programming category which makes it an important actor in the DevOps movement.
ND;LR
As you can see Apache JMeter is no more a tool to « play » or to use only for limited and simple load testing.
It has grown up into a mature, professional tool, used by major organizations for complex load testing use cases.
Furthermore it is available on the Cloud at ultra competitive pricing.
On the protocols side, UbikLoadPack provides important additions to it and this will keep growing.
Besides being provided by Apache, you can find support for it in the market in the following fields:
- Training
- Consulting
- SAAS and Cloud solutions
As a major contributor of JMeter, we also provide training and consulting around Load Testing and if you want to sponsor features you want to add to it, or have some custom protocols you can contact us.
About us:
- Ubik Load Pack is a JMeter based solution for complex protocols.
- Follow us on Twitter and StackOverflow.
- Follow Apache JMeter on Twitter
Cet article Why you should switch from Commercial Load Testing solutions to Apache JMeter est apparu en premier sur Ubik Ingénierie.