Apache JMeter vs Neoload vs HP Load Runner
We see a lot of unfair or very old comparison between Apache JMeter and commercial alternatives.
We felt it was time to write a fair comparison with links to features.
We’ll be happy to take into account any remark regarding the other tools if you contact us by email.
Comparison
To convince you it is worth considering switching your load testing to Apache JMeter and UbikLoadPack (for more protocols support and new features sponsoring) let’s compare it with some Major actors.
Cost:
|
Criteria
|
LoadRunner (HPE)
|
Last JMeter 3.3 (Apache)
|
JMeter + UbikLoadPack (Ubik-Ingenierie)
|
NeoLoad (Neotys)
|
|
Commercial License
|
Yes
|
No (Free and Commercial Compatible License (Apache 2.0))
|
Yes
|
Yes
|
|
Core Price
|
Free up to 50 VUs then License is per number of Users.
|
Free whatever the number of users
|
Free whatever the number of users
|
Free up to 50 VUs.
License depends then on number of Users, site deployment and other non public informations.
|
|
Additional Protocol Price
|
You pay per additional protocol + Core price
|
Not applicable
|
You pay ONLY for additional protocol but core (JMeter) is Free. Total Cost Of Ownership ends up being much lower than the competitors. You can either Rent (Dayly, monthly or annual basis) or Purchase.
|
You pay per additional protocol + Core price, see here
|
|
Cloud Load Testing Price
|
High
|
Low
|
Low
|
Medium
|
Features:
|
Criteria
|
HP Loadrunner
|
Apache JMeter
|
NeoLoad
|
|
Protocol Coverage
|
It has the widest protocols support (enterprise
software SAP, Oracle ERP…) |
HTTP(S), Webservices (Soap, Rest), FTP, LDAP, and Java-related protocols (JMS,
JDBC). Support more protocols (Websocket, HTTP2, XMPP…) with Open Source and Commercial Plugins (HLS, MPEG-DASH, MS SMOOTH/HSS, GWT, FLEX, JAVA SERIALIZATION, AutoCorrelation, Oracle JD Edwards, Vaadin, ….)
|
NeoLoad has good support of many Protocols. It
supports enterprise software SAP SAP (Web only), Oracle Siebel etc. |
|
Record and Playback capability
|
Yes
|
Yes
|
|
|
Support / Training
|
Yes
|
JMeter being an OSS provided by Apache there is no Apache support. But we (UbikLoadPack) provide support and training on the product. We are major contributor to Apache JMeter through features like JSON Extractor, The Web Report, lot of bug-fixes and enhancements and our Team leader being an Apache Committer and Project Member Committer of the project |
Yes
|
|
Cloud Based Solution
|
Yes (Amazon , Azure, Google for the Controller)
|
Yes (Amazon WS, Google Cloud, Azure, IBM BlueMix, OVH with self install)
|
Yes (Custom)
|
|
Continuous Integration (JENKINS, Bamboo…)
|
Yes
|
Yes
|
|
|
Mobile Load Testing
|
Yes
|
Yes (Native + SAAS providers for more features)
|
Yes
|
|
Real Time Test Monitoring
|
Yes
|
Yes (With BackendListener and Graphite based backend (InfluxDB, Graphite))
|
Yes
|
|
Script Customization
|
Hard (C language for all, Javascript added for Web in 12.5 version) and limited
|
Easy as JMeter is built on a Plugin architecture and made to be customized (JSR223 Elements allow you to use a lot of scripting languages (Groovy, Javascript, Java…)). This makes it the most customizable tool compared to others.
|
Medium to Hard (Javascript for extremely complex scenarios as per documentation quote
|
|
Result Reporting
|
Yes
|
Yes since JMeter 3.0 (APDEX, Summary and Error Tables, TOP5 Errors by Sampler, 15 graphs).
Results can also be sent to an Real Time Series Database like Influx DB, see real-time results.
|
Yes
|
|
Load Generation
|
Depends on Type of License
|
Unlimited Load Generation.
For UbikLoadPack, Unlimited number of Load Agents, Number of Virtual Users depends on Type of License
|
Depending on Type of License
|
|
IPSpoofing
|
Yes
|
Yes search Spoofing in this link
|
Yes
|
|
Support of authentication schemes and client certificates
|
Yes
|
Yes (Basic, Digest, Kerberos, …)
|
Yes
|
|
Flexible error handling and adjustable pass/fail criteria
|
Yes
|
Yes
|
|
|
Browser Support
|
Yes
|
Yes
|
Yes
|
|
Windows 7 and above
|
Yes
|
Yes as it’s based on Java
|
Yes
|
|
Mac OSX
|
No
|
Yes as it’s based on Java
|
No
|
|
Linux
|
Yes(Not all Load Generators)
|
Yes as it’s based on Java
|
Yes
|
|
Cluster (Scaling)
|
Yes
|
Yes (Horizontal through client/server mode or SAAS)
|
Yes
|
|
Number of Virtual Users per 8vCPU Machine/4Go RAM
|
1000
|
1000 to 1500
|
1000 to 1500
|
ND;LR
As you can see Apache JMeter is no more a tool to « play » or to use only for limited and simple load testing.
It has grown up into a mature, professional tool, used by major organizations for complex load testing use cases.
Furthermore it is available on the Cloud at ultra competitive pricing.
On the protocols side, UbikLoadPack provides important additions to it and this will keep growing.
Besides being provided by Apache, you can find support for it in the market in the following fields:
- Training
- Consulting
- SAAS and Cloud solutions
As a major contributor of JMeter, we also provide training and consulting around Load Testing and if you want to sponsor features you want to add to it, or have some custom protocols you can contact us.
Cloud Ready
Thanks to our partnership with Redline13 , our plugins are available from AWS cloud and ready to use in few clicks, read more:
About us:
- Ubik Load Pack is a JMeter based solution for complex protocols.
- Follow us on Twitter and StackOverflow.
- Follow Apache JMeter on Twitter
Cet article Apache JMeter vs Neoload vs HP Load Runner est apparu en premier sur Ubik Ingénierie.