What is most valuable?
The two standout features for us are:
- Team Server Collaboration - as is the case with many MNCs like ours, teams/team members are spread geographically at various locations. The Team Server module is a boon for such teams as multiple users can work on the same project simultaneously and check-in the changes. It also helps manage and allocate virtual users (of shared license) for execution of multiple projects at same time.
- Results & Analysis - NeoLoad provides comprehensive results for any load test. The results can be drilled-down to minute levels (from scenario level all the way to individual request level) of any transaction which makes root cause analysis very effective. NeoLoad can also consolidate results from different sources and the results are graphical (one can build custom graphs for any transaction) and colourful for easier interpretation.
Other advantages are
- GUI driven recording makes it easier to record and prepare test scenarios without need for extensive coding knowledge thereby enabling tool users to ramp up faster
- Supports a ton of protocols plus additional plugins such as Jenkins, AppDynamics, etc.
- Custom scripts and frameworks can be built
- Recording and simulating mobile applications is simple yet realistic
How has it helped my organization?
NeoLoad is a robust and versatile tool. With plugins for CI tools like Jenkins, NeoLoad is configured to automatically trigger small load tests on modules each time there is a build update. This has enabled us to identify performance issues for our clients at a very early stage. It has undoubtedly an impact during larger scale enterprise performance tests when all the modules are integrated and tested as cost to fix performance issues increases with later stages of deployment.
What needs improvement?
We have raised a few support tickets for certain feature sets/improvement areas.
These include permanent fixes for a few stability issues discussed in the following question and implementation of more robust solutions for media streaming protocols, TCP/IP and websocket protocols.
Also, there are a few changes in the latest version, 5.2, which have brought in many improvements but have removed certain features. For example, v5.2 has done away with the 90th percentile reporting for response times, which was present in earlier versions. While this is a small deviation, it still is important for organizations using it.
The tool can hang and take a long time to open projects if the size is large and if many graphs and result templates are stored. Certain protocols although supported by NeoLoad need to have custom solutions built (e.g. web socket connections) while solutions for certain other protocols such as media streaming protocols like HDS, HLS, etc. are still in the nascent stage.
For how long have I used the solution?
We have used NeoLoad since late 2009.
What do I think about the stability of the solution?
There is no specific maximum project size defined (actual NeoLoad project size). This size is dependent on number of user journeys in the project, number of scenarios defined, number of graph templates stored, and number of results stored and so on. The bigger the project, the slower it opens in the tool (as expected) but many times can be excruciatingly slow. The other drawback is that the graphs and results may not get saved correctly. With the latest versions of NeoLoad, this issue has been significantly fixed but it can still occur at an undesirable moment as the optimum size of a project can only be determined by trial-and-error. Our experience has taught us to keep multiple backups of same project under two categories for such exigencies – the first category would contain backup of the project with previous results and graphs while the second one would have only the project (VUs and scenarios) but with previous execution results and graph templates removed.
The other issue we faced was with respect to results generation. If we need to abort a performance test that is in progress, all one has to do is click the stop button. However, if the stop button is clicked twice inadvertently or clicked twice because it still showed “consolidating results” message for eons after the first click, the results ultimately generated would get corrupted. Either the results would not show for all virtual users executed or would not get generated at all. The last time we checked, Neotys did not have a concrete solution to this problem even after analysing the execution logs and troubleshooting it. Their “solution” was to check our stop policy for the scenarios executed (to make sure all virtual users reached their ‘end’ state to generate results) and also to check if any antivirus or firewall was impeding result generation (an antivirus or firewall can prevent NeoLoad from writing results to your folder). From our side, our stop policies were fine and there was no antivirus/firewall to block NeoLoad so the solutions were not that useful. As a workaround, we would click just once to abort an execution run and wait for the results to be generated even if it took a while.
How are customer service and technical support?
The customer service and technical support is excellent if you are a paid customer. The turnaround time and SLA for a priority one - Urgent - issue/ticket is almost immediate. For less priority tickets it is quick (usually a couple of hours) and the representatives even help build a custom solution for your project if need be. However, if you are using the trial version, you would have to depend on NeoLoad forums for answers. On the flip side, Neotys documentation available on their side is detailed and exhaustive which should answer most questions when you are evaluating the product.
Which solution did I use previously and why did I switch?
We have used a host of other solutions such as LoadRunner, JMeter, WebLOAD, SOASTA, etc. While the choice of the solution depends on various factors such as open source vs commercial tool, protocols used, number of virtual users etc. NeoLoad comes up trumps on most factors which were needed for our clients namely:
- Ease of deployment and scalability to test for high user loads (in the range of 100,000s users for large enterprise tests)
- Ability to monitor performance at various levels of architecture
- Graphical representation of results
- Detailed comparison can be seen in the sheet attached
How was the initial setup?
Initial setup is a breeze. The installation is a complete package without having to switch between standalone modules for recording scenarios, execution and analysis which is not the case in say, LoadRunner or WebLOAD which has different modules for each (like VuGen, Controller & Analysis).
Neotys also offers their own cloud infrastructure to leverage upon for tests with larger user loads which can be configured to each project needs. However, if you are setting up your own cloud infrastructure on say, Amazon AWS, you would have to be equipped with enough knowledge to set up the images (AMIs) and configure the necessary IPs & locations for the test.
What about the implementation team?
When we started out in 2009-2010, we did use the vendor team to guide us through and setup the necessary infrastructure. Over the years, we have built a strong in-house team and have setup our own infrastructure and cloud instances.
Depending on the technical expertise of your team, you can choose to implement in-house or avail the support of the vendor team. While the basic installation is straightforward, the vendor team could help you choose the right hardware systems/configuration for localized load generation or help you configure Neotys cloud for IP simulation or handling larger user loads.
What's my experience with pricing, setup cost, and licensing?
Neotys offers various licensing plans to suit every business. The free version comes with 50 users with all protocols included for small load tests. The paid licensing plan depends on number of virtual users needed (to simulate the load), duration, the protocols under test, number of concurrent applications being tested, cloud usage etc. For example, the licence can be availed even for a day, a month, or a year depending on the need. In that sense, the licence cost can range from $1500 to >$20000. A general rule of thumb is the longer the licence period and the greater number of virtual users results in a lower price over the long term.
What other advice do I have?
Leverage the free version of performance tools to its maximum capability to evaluate and cross check against the performance testing needs for your business to make an informed decision on buying the tool. The pros outweigh the cons and so NeoLoad is highly recommended over its competitors.
Which version of this solution are you currently using?
3.0 to 5.2