Apache JMeter Review

Good for vanilla web testing, less so for more complex requirements.


A mature opensource toolset that has been available for many years. Good for vanilla web testing, less so for more complex requirements. Like most opensource offerings it has relatively poor analytics and limited integration with analytics tools like APM although it is relatively easy to extend with plug-ins and there are quite of these available for free download. A good choice where budget is tight and the tech-stack requirement is straight-forward.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
1 visitor found this review helpful
10 Comments
author avatarpmouawad
Vendor

Jmeter allows testing very complex applications and our experience with it shows that the ability to extend it through Custom Scripting with Groovy, beanshell or JAvascript makes it really powerful.
Regarding reporting, its core os poor but nearly all users use jmeter-plugins as a toolset which provides great graphing and monitoring ability, see this:

- http://www.ubik-ingenierie.com/blog/automatically-generating-nice-graphs-at-end-of-your-load-test-with-apache-jmeter-and-jmeter-plugins/

Finally it is true that OOTB it has less protocol supported than neoload or Load Runnner but there are free anw commercial plugins that answer this need:

- http://ubikloadpack.com for GWT, AMF/Flex, Javaserialization and complex JSON
- Netflix provides a Cassandra Plugin
- There is MQTT and Websocket plugins
...

Disclaimer : Like author of initial review, I am not 100% neutral as being team leader of Ubik Load Pack and PMC member of Apache JMeter.

But have used JMeter for 8 years and much before starting those activities :)

author avatarOperations Expert at a tech services company
Consultant

If you're testing web then there really is no excuse not to consider using using something like JMeter.

JMeter itself can be extended, we do the same for https://flood.io and with the distributed nature of cloud based infrastructure, JMeter scales really well. We see everything from basic API endpoint testing through to simulation of complex user behaviour via the front end.

Disclaimer: I am the co founder of Flood IO and we support both JMeter and Gatling.

author avatarDeveloper at a transportation company with 1,001-5,000 employees
Vendor

Tim, from your point of view, which tool is more adequate for testing web applications, JMeter or Gatling?

author avatarOperations Expert at a tech services company
Consultant

Alin we feel it's entirely subjective. Both tools perform well at reasonably high volume / concurrency. https://flood.io/blog/11-benchmarking-jmeter-and-gatling

JMeter has been around longer and is maintained by a bigger team of contributors. Arguably you might find more examples / documentation for JMeter. Gatling is a newer project but very capable in terms of the load it can generate and once you get the hang of their DSL, quite powerful to use.

I see a mixture of both tools on Flood IO but would say JMeter is used by about 70% of our users.

I don't think you'd go wrong in choosing either.

author avatarDmitri T.
Real User

If your testing is does not require immense load and is limited to HTTP/JDBC/JMS protocols then feel free to choose any of them.

If you need to run tests from more than one machine, you should be considering JMeter as it can be run in http://jmeter.apache.org/usermanual/remote-test.html">distributed clustered mode and has support of specific protocols like https://code.google.com/p/jmeter-ssh-sampler/">SSH2 or http://blazemeter.com/blog/websocket-testing-apache-jmeter">WebSocket and has larger community and knowledgebase.

author avatarConsultant at a consultancy with 501-1,000 employees
Consultant

For additional reporting possibilities you can try https://loadosophia.org/gui/ they got acquired by BlazeMeter not too long ago, but they still offer mostly free reporting at a high level.
Besides the obvious plugins mentioned above. one of the extremely strong points of Jmeter and reporting is that you can simply export the results to a csv file and work your way through that with Excel or something like that.

As far as your statement about vanilla apps but no complex sites, I strongly disagree. I have used Jmeter repeatedly on huge Sharepoint and Microsoft CRM implementations. Very effectively tested and reported on the performance of these highly complext and AJAX driven applications.

author avatarHead of Performance with 51-200 employees
Vendor

Hi Martjin, Sharepoint and Microsoft CRM I would still consider vanilla apps and as you indicate you can use JMeter very successfully however when trying to deal with applications built around long-poll frameworks like SIGNALr and similar it is very problematic. We use JMeter where appropriate and it lends itself well to Dev perf testing however the relatively poor analytics , particularly during test execution and lack of a true scripting capability preclude it from more complex testing. We can achieve a lot more effective testing using tools like Egg Plant Performance, Neoload and Gatling.

KR
Ian

author avatarConsultant at a consultancy with 501-1,000 employees
Consultant

Ian, that simply means you and I have a different definition of vanilla :) I do not consider MS Dynamics CRM a very vanilla application. When talking vanilla I consider the simpler flatter webapplications with a smaller mixture of technologies combined on one page.

For performance testing of realtime transactions I would indeed switch to something else than JMeter.
So far I haven't had too much opportunities to work with Gatling yet, I am looking forward to spending some time with it though.

author avatarKobi Zilberman
User

Hi,
What do you think about using Jmeter on a propraiety protocol over tcp/ip?

author avatarHead of Performance with 51-200 employees
Vendor

Hi Kobi,

JMeter is not well suited for this sort of requirement as you are likely to have to deal with some sort of API. If however the API is web service based then should be ok. Something more proprietary would be better dealt with scripting in C# in my experience so some other tool choice would be better suited.

KR
Ian

Guest