We just raised a $30M Series A: Read our story

Tricentis NeoLoad OverviewUNIXBusinessApplication

Tricentis NeoLoad is #5 ranked solution in top Load Testing Tools and #6 ranked solution in top Performance Testing Tools. IT Central Station users give Tricentis NeoLoad an average rating of 8 out of 10. Tricentis NeoLoad is most commonly compared to Micro Focus LoadRunner Professional:Tricentis NeoLoad vs Micro Focus LoadRunner Professional. The top industry researching this solution are professionals from a computer software company, accounting for 35% of all views.
What is Tricentis NeoLoad?

The NeoLoad load and performance testing tool for web and mobile apps realistically simulates user activity and monitors infrastructure behavior to eliminate bottlenecks. It covers all performance testing from component and automated tests to system-wide hybrid-cloud load tests.

Tricentis NeoLoad was previously known as NeoLoad, Neotys NeoLoad.

Tricentis NeoLoad Buyer's Guide

Download the Tricentis NeoLoad Buyer's Guide including reviews and more. Updated: November 2021

Tricentis NeoLoad Customers

Dell, H&R Block, Best Buy, Orange, Verizon Wireless, ING, Mazda, Siemens, University of Oxford

Tricentis NeoLoad Video

Archived Tricentis NeoLoad Reviews (more than two years old)

Filter by:
Filter Reviews
Industry
Loading...
Filter Unavailable
Company Size
Loading...
Filter Unavailable
Job Level
Loading...
Filter Unavailable
Rating
Loading...
Filter Unavailable
Considered
Loading...
Filter Unavailable
Order by:
Loading...
  • Date
  • Highest Rating
  • Lowest Rating
  • Review Length
Search:
Showingreviews based on the current filters. Reset all filters
GT
Delivery Manager at cigniti
Vendor
Good scripting and parameterization help to make this a valuable part of our service offering

What is our primary use case?

We use this solution because we want to know the difference between two different tools to suggest for our clients.

How has it helped my organization?

As we are a services company, we have used NeoLoad also as part of our exercise.

What is most valuable?

I like the scripting and parameterization features.

What needs improvement?

While importing the scripts from backup it should not create the new variables because it has created some issues for us.

For how long have I used the solution?

We have been using this solution for one year.

What is our primary use case?

We use this solution because we want to know the difference between two different tools to suggest for our clients.

How has it helped my organization?

As we are a services company, we have used NeoLoad also as part of our exercise.

What is most valuable?

I like the scripting and parameterization features.

What needs improvement?

While importing the scripts from backup it should not create the new variables because it has created some issues for us.

For how long have I used the solution?

We have been using this solution for one year.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
Shivashanker Patil
Senior Performance Engineer at Ascend Learning
Real User
Best tool for testing in production without making many changes to the script or solution

Pros and Cons

  • "It helped in achieving the testing of on-premise applications, as well as cloud-based applications, without much difficulty."
  • "NeoLoad is best tool for testing in production without making many changes to the script or solution."
  • "NeoLoad does not support Citrix-based applications."

What is our primary use case?

Used for performance testing of applications on QA environments as well as production environments. NeoLoad is best tool for testing in production without making many changes to the script or solution.

How has it helped my organization?

It helped in achieving the testing of on-premise applications, as well as cloud-based applications, without much difficulty.

What is most valuable?

  • Integration of all components in one place.
  • GUI-based requests
  • Ease of use compared with other performance tools.

What needs improvement?

NeoLoad does not support Citrix-based applications. Neotys should take a look into this protocol.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

No issues.

What do I think about the scalability of the solution?

No issues. As far as Neotys NeoLoad, it is excellent in scalability.

How are customer service and technical support?

The support is good. 

Which solution did I use previously and why did I switch?

We started with NeoLoad.

How was the initial setup?

The initial setup is easy.

What about the implementation team?

The implementation was done in-house.

What was our ROI?

Our ROI is 100%.

What's my experience with pricing, setup cost, and licensing?

The licensing cost is less compared to other licensing performance testing tools.

Which other solutions did I evaluate?

We tried JMeter, but due to its support (which we found subpar), we had to switch to NeoLoad.

What other advice do I have?

You will love NeoLoad moving from open source and code-based performance tools.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
Learn what your peers think about Tricentis NeoLoad. Get advice and tips from experienced pros sharing their opinions. Updated: November 2021.
554,873 professionals have used our research since 2012.
it_user745425
Senior Quality Assurance Engineer with 10,001+ employees
Real User
A cost-effective source for load, stress and performance testing

What is our primary use case?

It is primarily used for load testing, stress testing and performance testing. It is a very good product.

What is most valuable?

It is a bit easier compared to our LoadRunner, and it is cheaper. It can select to upload and it is based on cloud services. It is really easy to learn and to use. 

What needs improvement?

It needs improvement with post-production. 

For how long have I used the solution?

Three to five years.

How is customer service and technical support?

It has good support. 

What's my experience with pricing, setup cost, and licensing?

It is cheaper than other solutions.

What is our primary use case?

It is primarily used for load testing, stress testing and performance testing. It is a very good product.

What is most valuable?

It is a bit easier compared to our LoadRunner, and it is cheaper. It can select to upload and it is based on cloud services. It is really easy to learn and to use. 

What needs improvement?

It needs improvement with post-production. 

For how long have I used the solution?

Three to five years.

How is customer service and technical support?

It has good support. 

What's my experience with pricing, setup cost, and licensing?

It is cheaper than other solutions.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user380010
Senior Load Test Engineer at a tech vendor with 501-1,000 employees
Vendor
several key features: Jenkins integration, infrastructure monitoring, and results analysis.

Pros and Cons

  • "There are several key features, including Jenkins integration, infrastructure monitoring, and results analysis."
  • "It needs improvements in the UI. It's currently not as friendly as it should be."

What is most valuable?

There are several key features, including Jenkins integration, infrastructure monitoring, and results analysis.

How has it helped my organization?

Implementing this product has allowed us to increase the number of projects that get load testing, and the frequency by which we re-test our performance.

What needs improvement?

It needs improvements in the UI. It's currently not as friendly as it should be.

For how long have I used the solution?

We've used it for seven years.

What was my experience with deployment of the solution?

We haven't had any issues with deployment.

What do I think about the stability of the solution?

We have never experienced any isuses with stability.

What do I think about the scalability of the solution?

The product has scaled very well up until this point.

How are customer service and technical support?

The support from Neotys is very good. Responses are timely and come from experienced, knowledgeable load testing professionals.

Which solution did I use previously and why did I switch?

We previously, and still use JMeter for some projects, in addition to Neoload.

How was the initial setup?

The initial setup is straightforward for basic usage. The setup for higher volume load testing is more involved, but I wouldn't say complex, as should be expected for an enterprise class load testing tool.

What about the implementation team?

We implemented it with our in-house team.

What's my experience with pricing, setup cost, and licensing?

It's priced based on the numbers of VUs, virtual users, that your needs require. The free edition gives you 50 VUs. The pricing is fair for high-volume licensing. Cloud VUs can be rented on-demand for bursts of higher capacity testing needs without permanent purchase.

Which other solutions did I evaluate?

We looked at JMeter and Webload.

What other advice do I have?

I highly recommend understanding load testing concepts before using any automation tool.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
Mukesh Dhingra
Assosiate Project Manager at a tech services company with 10,001+ employees
Consultant
Less manual scripting and ease of setting up monitoring are valuable features.

What is most valuable?

Less manual scripting and ease of setting up monitoring are very valuable features of the tool.

Loadrunner has extensive list of analysis features available whereas NeoLoad had limited when I worked.

How has it helped my organization?

It helped us deliver the solution for various customers within budget and efficiently.

What needs improvement?

Analysis can be further improved. I have been using LoadRunner analysis and it has been a very powerful tool.

For how long have I used the solution?

I have used it for six months.

What was my experience with deployment of the solution?

I have not encountered any deployment issues.

What do I think about the stability of the solution?

I have not encountered any stability issues.

What do I think about the scalability of the solution?

I have not encountered any scalability issues.

How are customer service and technical support?

Technical support's turnaround time for issue response is too good.

Which solution did I use previously and why did I switch?

I have worked on various different performance testing tools, and I found NeoLoad very much within budget without compromising the quality of the product.

How was the initial setup?

Initial setup was straightforward.

What about the implementation team?

A vendor team implemented it.

What other advice do I have?

More protocols can be added, and I would love to see if any profiler is included in it.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user269556
Senior Software Testing Consultant at a tech services company with 501-1,000 employees
Real User
Correlation of dynamic data is the most important advantage of this tool.

Pros and Cons

  • "In my opinion, correlation of dynamic data is the most important advantage of this tool."
  • "It would be good to make some updates on the reporting side."

What is most valuable?

In my opinion, correlation of dynamic data is the most important advantage of this tool. Normally, it is difficult to record performance scripts, but with this tool it is much easier.

How has it helped my organization?

Since this tool is lowering the risk of monthly releases to live environment, it reduces the stress on people. In addition, performance problems that we could not solve for a long time have been solved by the tool with Dynatrace integration.

What needs improvement?

It would be good to make some updates on the reporting side. For example, correlating the performance of server, throughput and hit count in a specific time interval would be a nice functionality.

For how long have I used the solution?

We use the tool full-time on our test environment and we have been releasing it to our live environment for 2 years.

How are customer service and technical support?

Support side of the tool is very strong. We have seen fast feedback and answers to other questions we faced in the past about licensing and other general problems.

Which solution did I use previously and why did I switch?

I have used many performance testing tools. This tool goes one step further with easy integration with code analysis tools, server monitoring, the success of correlation studio and the advantages on the mobile side.

What's my experience with pricing, setup cost, and licensing?

Licence cost is very attractive compared to other vendor tools and also there are many license alternatives. Licence costs are good enough compared to the benefit of the tool to the company.

What other advice do I have?

The ease of recording scripts and the correlation process, the force-on monitoring, success on the mobile side, and easy integration with code analysis tools are the reasons I have rated this product so high.

Disclosure: My company has a business relationship with this vendor other than being a customer: We provide consultancy services for testing and project management. And we have worked with NeoLoad on many projects both inside and outside of Turkey.
ITCS user
Asst Supervisor, QA Automation, Quality Assurance at a insurance company with 1,001-5,000 employees
Vendor
We switched from LoadRunner due to various reasons including cost, ease of use and robust features.

What is most valuable?

Ease of use. Technology support, and robust features.

How has it helped my organization?

We are able to use NeoLoad to quickly expand performance testing to all critical business applications and address some key performance issues.

What needs improvement?

Team Server - Neotys Team Server - This currently stores project data on the local drive. I suggest having Neotys Team Server Data stored on an actual server.

For how long have I used the solution?

Since 2014.

What was my experience with deployment of the solution?

No issues. Simple deployment process.

What do I think about the scalability of the solution?

None.

How are customer service and technical support?

Customer Service: Excellent. Technical Support: Excellent.

Which solution did

What is most valuable?

Ease of use. Technology support, and robust features.

How has it helped my organization?

We are able to use NeoLoad to quickly expand performance testing to all critical business applications and address some key performance issues.

What needs improvement?

Team Server - Neotys Team Server - This currently stores project data on the local drive. I suggest having Neotys Team Server Data stored on an actual server.

For how long have I used the solution?

Since 2014.

What was my experience with deployment of the solution?

No issues. Simple deployment process.

What do I think about the scalability of the solution?

None.

How are customer service and technical support?

Customer Service:

Excellent.

Technical Support:

Excellent.

Which solution did I use previously and why did I switch?

Yes, HP LoadRunner. We switched due various reasons including cost, ease of use and robust features.

How was the initial setup?

Yes, the initial setup process is really straightforward.

What about the implementation team?

In house.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user488832
Performance Engineer at a tech company with 10,001+ employees
Vendor
It allowed us to increase the number of applications we support and test.

What is most valuable?

The ability to quickly record and correlate a script very quickly with a lower learning curve. A complex LoadRunner script could take days to create, but the same script in NeoLoad could be created within hours. We had contractors coming in, and within months they were able to script and start contributing to the team very quickly due to the simplicity of this software.

NeoLoad has the ability to compare a test report to a previous run. We can easily compare to see if this new code base is performing better or on par with the previous code base.


How has it helped my organization?

We were able to create more scripts using NeoLoad within a short time frame. We increased the number of applications we support/test due do the quick scripting time. Now we have more time to do analysis and engineering work which is very important to us.

What needs improvement?

The ability to show transactions per second during the test run. Currently, we have to eyeball the TPS using the graph. NeoLoad 5.2 will have TPS reading after the test is completed but not during the test run.

For how long have I used the solution?

16 months

What was my experience with deployment of the solution?

None.

What do I think about the stability of the solution?

None.

What do I think about the scalability of the solution?

None.

How are customer service and technical support?

9/10

Which solution did I use previously and why did I switch?

We used LoadRunner before we switched over to NeoLoad. LoadRunner is a powerful tool, but the learning curve is very high. It would not be feasible for our team because we have a lot of newcomers and a lot of applications to support. NeoLoad costs a fraction of what Load Runner costs.

How was the initial setup?

Setup is very straightforward. Neotys also provides guidance and support, setting up the team servers and licenses.

What about the implementation team?

We did the installation in-house, with vendor support when needed.

What other advice do I have?

Identify what kind of testing your company needs (mobile, database, web, and so on) and find out if NeoLoad supports it.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
ITCS user
Load Test Engineer at a government with 10,001+ employees
Consultant
All-in-one product with load generators deployed globally.

What is most valuable?

All-in-one product (design, runtime, results), easy scripting and variable correlation, and load generators deployed all around the world to simulate global traffic.

How has it helped my organization?

It is easier to do load tests using NeoLoad.

What needs improvement?

The only improvement that I wanted is support for load tests in agile mode. Neoload 5.2 makes this possible by updating the old versions of scripts with data from the new HTTP requests.

For how long have I used the solution?

I have 4 years of experience.

What was my experience with deployment of the solution?

The only issue that I had was with network virtualization on load generators installed on Windows Server 2008 R2. The link is sometimes cut between the controller and the load generators.

How are customer service and technical support?

10/10. Support is available, they handle issues very quickly, and they are very skillful.

Which solution did I use previously and why did I switch?

I used HPE LoadRunner. I think that NeoLoad is more user-friendly.

How was the initial setup?

NeoLoad is easy to deploy using the install wizard.

What about the implementation team?

Well, first I got trained by a vendor team. After that, I implemented more projects using NeoLoad.

My advice is that there should be a minimum of 4GB of memory on the servers on which you are installing NeoLoad.

What's my experience with pricing, setup cost, and licensing?

Neotys provides a free evaluation licence to use NeoLoad with up to 50 virtual users and all protocols included.

The product licence remains cheaper than HP LoadRunner, for example.

Disclosure: My company has a business relationship with this vendor other than being a customer: We are a partner of Neotys. We do consulting for them.
ITCS user
Test Analyst at a tech services company with 10,001+ employees
Real User
Recording a .NET application worked well.

What is most valuable?

The way it interacts with applications and how the dynamic values are handled.

How has it helped my organization?

LoadRunner was not able to record a .NET application, we had many issues with correlations, requests were not capturing properly, it was time consuming and there were many other issues. When we tried the same application in NeoLoad, it worked like a charm without any hassle.

What needs improvement?

It needs improvements when handling binary values.

For how long have I used the solution?

I am using it for more than 2 years.

What was my experience with deployment of the solution?

None

What do I think about the stability of the solution?

None

What do I think about the scalability of the solution?

None

Which solution did I use previously and why did I switch?

First I tried HP LoadRunner, but it didn't work.

How was the initial setup?

It was straightforward and easy to setup.

What about the implementation team?

Vendor team

What's my experience with pricing, setup cost, and licensing?

The market always pays for the one who comes up with something new and that is easy to customize.

What other advice do I have?

Keep enhancing it and adding new features.

Disclosure: My company has a business relationship with this vendor other than being a customer: My company has a business relationship with this vendor and in the coming days, there could be more demand.
ITCS user
Non Functional Test Manager at a government with 5,001-10,000 employees
Vendor
It enables my team to quickly record and modify test scripts.

Valuable Features:

Wasted time in low-end work (scripting) is greatly reduced with NeoLoad. It enables my team to record and modify test scripts very quickly. This reduces the effort for testing and enables the team to do more high-value work like analysis of results. We use external data analytics tools (Tableau) for result analysis. The export raw data feature in NeoLoad provides the extension with Tableau.

Improvements to My Organization:

Performance testing is done by testing frameworks in which NeoLoad is the driver of load. The ability to extend the tool is crucial for organisations. Our test framework is composed out of NeoLoad as a centralised component surrounded by Tableau for analytics and AppDynamics for deep-dive diagnostics. This load test framework enables the team to more quickly test, find and resolve issues and system bottlenecks.

Room for Improvement:

The weakest feature of most of the load test tools is result analysis. It is important we can customise graphs to visualise results in such a way that stakeholders can understand the outcome of testing. To be able to visualise results better, we extended NeoLoad with Tableau Software.

With Tableau, you can centralize performance related data in a database. Data can come from any tool, from log files or RUM tools and system utilities. Tableau provides you with the ability to connect to any data source, analyse patters and find bottlenecks more easily.  NeoLoad has made huge improvements over the last few years and finally has the ability to view the raw data (and not only aggregated results). This is fundamental for any type of performance engineering.  Aggregated results hide bottlenecks. 

Customer Service:

Customer support is very good.

Cost and Licensing Advice:

The price of the tool is competitive.

Other Advice:

The features introduced in the recently released new version of NeoLoad fit really well into the DevOps way of working, further reducing the time for rescripting. This enables rapid delivery and lean ways of working.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user467850
Senior Test Engineer at a tech vendor with 10,001+ employees
Vendor
The standout features are Team Server Collaboration as well as the test results & analysis.

What is most valuable?

The two standout features for us are:

  • Team Server Collaboration - as is the case with many MNCs like ours, teams/team members are spread geographically at various locations. The Team Server module is a boon for such teams as multiple users can work on the same project simultaneously and check-in the changes. It also helps manage and allocate virtual users (of shared license) for execution of multiple projects at same time.
  • Results & Analysis - NeoLoad provides comprehensive results for any load test. The results can be drilled-down to minute levels (from scenario level all the way to individual request level) of any transaction which makes root cause analysis very effective. NeoLoad can also consolidate results from different sources and the results are graphical (one can build custom graphs for any transaction) and colourful for easier interpretation.

Other advantages are

  1. GUI driven recording makes it easier to record and prepare test scenarios without need for extensive coding knowledge thereby enabling tool users to ramp up faster
  2. Supports a ton of protocols plus additional plugins such as Jenkins, AppDynamics, etc.
  3. Custom scripts and frameworks can be built
  4. Recording and simulating mobile applications is simple yet realistic

How has it helped my organization?

NeoLoad is a robust and versatile tool. With plugins for CI tools like Jenkins, NeoLoad is configured to automatically trigger small load tests on modules each time there is a build update. This has enabled us to identify performance issues for our clients at a very early stage. It has undoubtedly an impact during larger scale enterprise performance tests when all the modules are integrated and tested as cost to fix performance issues increases with later stages of deployment.

What needs improvement?

We have raised a few support tickets for certain feature sets/improvement areas.
These include permanent fixes for a few stability issues discussed in the following question and implementation of more robust solutions for media streaming protocols, TCP/IP and websocket protocols.

Also, there are a few changes in the latest version, 5.2, which have brought in many improvements but have removed certain features. For example, v5.2 has done away with the 90th percentile reporting for response times, which was present in earlier versions. While this is a small deviation, it still is important for organizations using it.

The tool can hang and take a long time to open projects if the size is large and if many graphs and result templates are stored. Certain protocols although supported by NeoLoad need to have custom solutions built (e.g. web socket connections) while solutions for certain other protocols such as media streaming protocols like HDS, HLS, etc. are still in the nascent stage.

For how long have I used the solution?

We have used NeoLoad since late 2009.

What do I think about the stability of the solution?

There is no specific maximum project size defined (actual NeoLoad project size). This size is dependent on number of user journeys in the project, number of scenarios defined, number of graph templates stored, and number of results stored and so on. The bigger the project, the slower it opens in the tool (as expected) but many times can be excruciatingly slow. The other drawback is that the graphs and results may not get saved correctly. With the latest versions of NeoLoad, this issue has been significantly fixed but it can still occur at an undesirable moment as the optimum size of a project can only be determined by trial-and-error. Our experience has taught us to keep multiple backups of same project under two categories for such exigencies – the first category would contain backup of the project with previous results and graphs while the second one would have only the project (VUs and scenarios) but with previous execution results and graph templates removed.

The other issue we faced was with respect to results generation. If we need to abort a performance test that is in progress, all one has to do is click the stop button. However, if the stop button is clicked twice inadvertently or clicked twice because it still showed “consolidating results” message for eons after the first click, the results ultimately generated would get corrupted. Either the results would not show for all virtual users executed or would not get generated at all. The last time we checked, Neotys did not have a concrete solution to this problem even after analysing the execution logs and troubleshooting it. Their “solution” was to check our stop policy for the scenarios executed (to make sure all virtual users reached their ‘end’ state to generate results) and also to check if any antivirus or firewall was impeding result generation (an antivirus or firewall can prevent NeoLoad from writing results to your folder). From our side, our stop policies were fine and there was no antivirus/firewall to block NeoLoad so the solutions were not that useful. As a workaround, we would click just once to abort an execution run and wait for the results to be generated even if it took a while.

How are customer service and technical support?

The customer service and technical support is excellent if you are a paid customer. The turnaround time and SLA for a priority one - Urgent - issue/ticket is almost immediate. For less priority tickets it is quick (usually a couple of hours) and the representatives even help build a custom solution for your project if need be. However, if you are using the trial version, you would have to depend on NeoLoad forums for answers. On the flip side, Neotys documentation available on their side is detailed and exhaustive which should answer most questions when you are evaluating the product.

Which solution did I use previously and why did I switch?

We have used a host of other solutions such as LoadRunner, JMeter, WebLOAD, SOASTA, etc. While the choice of the solution depends on various factors such as open source vs commercial tool, protocols used, number of virtual users etc. NeoLoad comes up trumps on most factors which were needed for our clients namely:

  1. Cost
  2. Ease of deployment and scalability to test for high user loads (in the range of 100,000s users for large enterprise tests)
  3. Ability to monitor performance at various levels of architecture
  4. Graphical representation of results
  5. Detailed comparison can be seen in the sheet attached

How was the initial setup?

Initial setup is a breeze. The installation is a complete package without having to switch between standalone modules for recording scenarios, execution and analysis which is not the case in say, LoadRunner or WebLOAD which has different modules for each (like VuGen, Controller & Analysis).

Neotys also offers their own cloud infrastructure to leverage upon for tests with larger user loads which can be configured to each project needs. However, if you are setting up your own cloud infrastructure on say, Amazon AWS, you would have to be equipped with enough knowledge to set up the images (AMIs) and configure the necessary IPs & locations for the test.

What about the implementation team?

When we started out in 2009-2010, we did use the vendor team to guide us through and setup the necessary infrastructure. Over the years, we have built a strong in-house team and have setup our own infrastructure and cloud instances.

Depending on the technical expertise of your team, you can choose to implement in-house or avail the support of the vendor team. While the basic installation is straightforward, the vendor team could help you choose the right hardware systems/configuration for localized load generation or help you configure Neotys cloud for IP simulation or handling larger user loads.

What's my experience with pricing, setup cost, and licensing?

Neotys offers various licensing plans to suit every business. The free version comes with 50 users with all protocols included for small load tests. The paid licensing plan depends on number of virtual users needed (to simulate the load), duration, the protocols under test, number of concurrent applications being tested, cloud usage etc. For example, the licence can be availed even for a day, a month, or a year depending on the need. In that sense, the licence cost can range from $1500 to >$20000. A general rule of thumb is the longer the licence period and the greater number of virtual users results in a lower price over the long term.

What other advice do I have?

Leverage the free version of performance tools to its maximum capability to evaluate and cross check against the performance testing needs for your business to make an informed decision on buying the tool. The pros outweigh the cons and so NeoLoad is highly recommended over its competitors.

Disclosure: My company has a business relationship with this vendor other than being a customer: We're certified partners with Neotys.
it_user467826
Performance Test Engineer at a tech services company with 10,001+ employees
Real User
It provides easy mobile performance testing solution for our customers.

Valuable Features

  • Codeless scripting
  • Easy correlation
  • Features for mobile such as WAN emulation collaboration with Perfecto

Improvements to My Organization

It provides easy mobile performance testing solution for our customers.

Room for Improvement

The product keeps its pace with the updates, however I would like them to offer online advance training certification and better support material.

Use of Solution

I've used it for around three years.

Customer Service and Technical Support

They provide very quick responses.

Initial Setup

It is very simple compared to other tools on the market that it competes with.

Pricing, Setup Cost and Licensing

Through collaboration with Neotys, we have free licenses for 50 users. I think we would buy more if they could provide hands on training for newly added features to specific sets of people.

Other Advice

It's a great product, especially if you are looking into mobile performance testing.

Disclosure: My company has a business relationship with this vendor other than being a customer: Partners
ITCS user
QA Technical Lead at a financial services firm with 501-1,000 employees
Real User
It has great reports and provides us with good flexibility. The language used to code scripts was very specific to NeoLoad.

What is most valuable?

  • The load testing support for different specialized products.
  • It has great reports and provides us with good flexibility.
  • The product support was from Neotys was good.

How has it helped my organization?

By running NeoLoad just prior to a major system upgrade, we discovered a major performance issue which pointed to a configuration setting on one of the servers. The configuration was amended and the scripts re-run.

What needs improvement?

It wasn't easy to learn how to use NeoLoad and it has a bit of a learning curve. The language used to code scripts was very specific to NeoLoad. It would be better if they incorporated Java and/or other popular coding languages as this would help smooth out the learning curve.

For how long have I used the solution?

I've used it for approximately one year.

What was my experience with deployment of the solution?

We had no issues with the deployment.

What do I think about the stability of the solution?

There have been no issues with the performance.

What do I think about the scalability of the solution?

It's been able to scale for our needs.

How are customer service and technical support?

9/10 - we get a prompt response whether we contact them via telephone or email.

Which solution did I use previously and why did I switch?

We had a product in place and changed because Neotys had good support.

How was the initial setup?

It was a straightforward installation via an executable file.

What about the implementation team?

We used a vendor team. Neither the coding language nor the ease of use of the product was obvious and we needed someone with NeoLoad experience.

What's my experience with pricing, setup cost, and licensing?

The best advice I can give you is to go for a perpetual licensing option so you just pay per user.

Which other solutions did I evaluate?

We were struggling to find a good load testing tool that could support Siebel. NeoLoad was one which ticked the correct boxes.

What other advice do I have?

It’s a very powerful tool which meets all our load and performance testing requirements, along with very sophisticated monitoring tools and reports.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user465003
Software Engineer at Mitacs Inc.
Vendor
Easy for a person without any performance testing experience to use.

What is most valuable?

It’s very easy for a person without any performance testing experience to use. It was a breeze to install and start using.

Monitoring Agents, Dynamic Parameters and Automatically Recording of all the Requests and Responses.

How has it helped my organization?

We did not have any idea as to why our application was breaking during normal loads. NeoLoad helped us to determine the huge issues and factors which were completely hidden if not for NeoLoad.

What needs improvement?

It would be nice if they could make it so that the dynamic parameterization in the requests came back without the values from the previous responses. Also, it needs the ability to capture SQL queries that are being fired.

For how long have I used the solution?

I've been using it for eight months.

What do I think about the stability of the solution?

We had scalability issues for loads over 1000 when using one Load Generator. Now we have increased the Load Generator to two.

How are customer service and technical support?

They are very good and prompt in resolving tickets that we raise with them. They did help us in resolving certain issues over the phone call with a NeoLoad Expert.

Which solution did I use previously and why did I switch?

This was our first performance testing tool.

How was the initial setup?

It was very straightforward as it comes with an installer and you just need some space and a server to install it on.

What about the implementation team?

We implemented it in house and my advice is to follow the installation document.

What other advice do I have?

If you install it, you can have fun.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user464934
Technical Test Specialist, Performance Test Lead at a tech services company with 10,001+ employees
MSP
Executable binaries made life easier during the implementation. They need to update their online documentation.

What is most valuable?

The support for HTTP, JMS, REST API, and XBAP protocols is good. NeoLoad is very user friendly and works well with HTTP and REST AO’s. It's very effective during cloud based testing and the cost is a fraction of LoadRunner. v5.1.4 introduced the JMS testing protocol which is a copy of SOAP UI. I was pleasantly surprised that with a little tweaking and support from Neotys, NeoLoad worked with XBAP applications very efficiently.

How has it helped my organization?

It comes with new features, notably JMS which enables us to be on top of our performance testing tool list.

What needs improvement?

The documentation for the new JMS protocol is in its initial stages and Neotys needs to update their online support documentation with more information about it.

For how long have I used the solution?

I've been using it for over three years.

What was my experience with deployment of the solution?

We've not experienced any deployment issues.

What do I think about the stability of the solution?

There were a couple of times during a load run where NeoLoad crashed out but they were due to issues with our virtual machines rather than NeoLoad.

How are customer service and technical support?

Customer Service:

9/10

Technical Support:

9/10 - the Neotys support team is superb and very efficient.

Which solution did I use previously and why did I switch?

I have used LoadRunner for web, web services and Siebel protocols, but running and licensing costs were very high.

How was the initial setup?

It's very straightforward.

What about the implementation team?

Executable binaries made life easier during the implementation and do not require any advice.

What's my experience with pricing, setup cost, and licensing?

Compared to LoadRunner, NeoLoad licensing of users is very cost effective. The cost is very simple and well defined and you can check it on the Neotys site.

What other advice do I have?

You should attend a three day training course to fully understand the functioanlity of NeoLoad.

Screenshots:


Disclosure: I am a real user, and this review is based on my own experience and opinions.
ITCS user
Performance Architect at a tech services company with 10,001+ employees
Consultant
I like the integration with various monitoring tools.

What is most valuable?

  • The easy of flagging requests while performing manual correlation.
  • Its integration with various monitoring tool.
  • Ease of creation population mimicking various browser emulations and network profiles.

How has it helped my organization?

In the projects where we deployed NeoLoad, it helped to reduce the time taken to script workflows. Also, the ease with which load generators could be added significantly reduced the time taken to design and initiate the load tests.

What needs improvement?

The current product is very good and the roadmap which is laid out for Neoload is sound. However, they need to offer more protocol support.

For how long have I used the solution?

I have been using this solution for about four years.

What do I think about the scalability of the solution?

There were projects which we had to simulate more than 50,000 users and had some scalability issues which were quickly resolved with help from Neotys support.

How are customer service and technical support?

Customer support is excellent, with a very quick turn around time for any question asked.

Which solution did I use previously and why did I switch?

Traditionally we have been using HP LoadRunner, and were looking for a more cost effective solution for performance testing. JMeter was the initial choice, but the lack of support was a cause for concern.

How was the initial setup?

The initial step was relatively easy with a one click installation and setup. The complexity was in learning the features of the tool. The initial learning curve was a bit steep, but once the process and steps were determined, then the way ahead was easy.

What about the implementation team?

The implementation was taken care in-house. Implementation is a fairly straightforward process if the load generators and controllers are in the same subnet. It takes a bit of time to setup the network firewall rules and to open up all the necessary ports if the load generators are in various subnets.

What's my experience with pricing, setup cost, and licensing?

It is fairly cost effective when compared to the cost of LoadRunner.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user367824
Senior Consultant - Software Development and Integration at Sogeti Group
Consultant
We are able to generalize all of the performance testers to the point that any tester can update and execute any script instead of having individual domain experts who may be unavailable or busy.

What is most valuable?

  • Graphical test creation
  • Centralized test and results repository
  • License management

How has it helped my organization?

With the simplicity of the test design interface, little domain knowledge is required for each scripted application, so we are able to generalize all of the performance testers to the point that any tester can update and execute any script instead of having individual domain experts who may be unavailable or busy. This has allowed us to keep more scripts updated easier than ever and keep a constant flow of performance tests in the pipeline, removing some dependence on human interaction.

What needs improvement?

If there was ever a way to bring the same level of ease to advanced scripting with Javascript to make it as easy as a regular test in NeoLoad, I would love to see it.

Some type of global test scheduler where different tests could be arranged on a calendar to automatically execute would help scheduling and provide visibility into the test environment.

Tighter integration with NeoSense and perhaps TFS integration to open defects for performance issues would also be nice.

For how long have I used the solution?

I've used it for 12 months.

What was my experience with deployment of the solution?

We've had no issues with deployment.

What do I think about the stability of the solution?

There have been no stability issues.

What do I think about the scalability of the solution?

Running very large populations on one machine would max out NIC-level capacity far before CPU/memory would provide indication of bottlenecks, but running a distributed test alleviated this issue.

How are customer service and technical support?

Customer Service:

8/10

Technical Support:

8/10

Which solution did I use previously and why did I switch?

We previously used HP LoadRunner, and we switched due to cost, support competence, and the scripting interface.

How was the initial setup?

It's straightforward to install and configure, but complex to convert scripts from a previous product to work in NeoLoad.

What about the implementation team?

We implemented it in-house.

What was our ROI?

We've gotten 20% increase in tests-per-sprint and 35% decrease in time taken to create a script. We also have cost savings of over one million dollars annually.

What's my experience with pricing, setup cost, and licensing?

Purchase a few extra licenses to allow for flexibility, utilizing the team server as much as possible.

Which other solutions did I evaluate?

  • HP LoadRunner
  • JMeter
  • Appvance
  • Rational Performance Tester

What other advice do I have?

Take the time to see what NeoLoad can do when scripted from scratch and remember that it takes a bit of time to become an expert with the advanced features. Some complex actions may require written code, but there is very little that is actually impossible to test with NeoLoad. The limit is usually a function of experience with the software.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
ITCS user
Performance Test Architect and HP ALM Expert at a tech services company with 10,001+ employees
Consultant
I was able to use this product in the past for simulating mobile performance after recording a script over the LAN. I ran into installation issues when the procedure kept hanging.

Valuable Features

The most valuable feature that we've found useful is that NeoLoad provides a variety of bandwidths.

Improvements to My Organization

I was able to use this product in the past for simulating mobile performance after recording a script over the LAN. This feature wasn't available in other testing products that I evaluated.

Room for Improvement

Neotys was extremely promising when they launched and it had features that older versions of HP LoadRunner and HP Performance Center did not (automatic correlation, for example). However with newer releases of HP ALM (especially 11.52 onward), these features were readily available with HP products.

I also ran into installation issues with NeoLoad when the installation never completed.

Use of Solution

I've used it for over two years.

Deployment Issues

I uninstalled the product and tried to install the latest version. However, it sort of kept hanging during the installation procedure.

Stability Issues

We've had no stability issues.

Scalability Issues

There have been no issues scaling it for our needs.

Customer Service and Technical Support

In my experience, technical support was pretty good in those times I had to contact them for issues.

Initial Setup

It was a simple initial setup.

Implementation Team

We implemented it through our in-house team.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user157959
Digital Performance Test Consultant at a tech services company with 10,001+ employees
Real User
When it comes to mobile HTTP load testing, this helps to script quicker.​ ​I would like to see reporting improved.

What is most valuable?

Script designing and "free" network emulation are the most valuable features for us.

How has it helped my organization?

I have seen good productivity improvements of over 30% in script design.

What needs improvement?

Reporting needs to be better. I would like to see information such as Send Time, Receive Time, SSL, Waiting Time, and Blocking Time in my response times.

For how long have I used the solution?

I've been using it for two years.

What was my experience with deployment of the solution?

We've had no issues with the deployment.

What do I think about the stability of the solution?

There have been no stability issues.

What do I think about the scalability of the solution?

We've had no issues scaling it for our needs.

How are customer service and technical support?

When I've presented technical support with an issue, they're prompt in responding to me.

Which solution did I use previously and why did I switch?

I had been using HP LoadRunner, Apache JMeter, SOASTA CloudTest, BlazeMeter and Rational Performance Tester. I chose this because when it comes to mobile HTTP load testing, Neoload helps script quicker.

How was the initial setup?

The initial setup was straightforward.

What about the implementation team?

We implemented it with our in-house team.

What's my experience with pricing, setup cost, and licensing?

Compared to other tools in the market, this is priced moderately. However, I see a lot of short term performance activities, and the license costs are higher in comparison. Most of the time, I just use JMeter in such cases.

What other advice do I have?

Go ahead and give it a try.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user406503
Technology Manager at a pharma/biotech company with 10,001+ employees
Real User
We like the native mobile application testing. I would like to see it support SAP GUI based applications.

What is most valuable?

  • Flexible license model
  • Cloud integration
  • Native mobile application testing

How has it helped my organization?

I had a requirement to test a mobile application for corporate that is used by users across the globe. I needed to do the test setup quickly to generate load across the region, and to do so with a solution that was cheaper than what we had. A NeoLoad license of four days with load generation from the cloud across regions helped to do the setup in three to four days and perform the testing.

What needs improvement?

It needs to support SAP GUI-based applications, which forms 60-70% of our portfolio, as well as Windows (client install) and Citrix-based applications.

For how long have I used the solution?

I've used it for two years.

What was my experience with deployment of the solution?

We had no issues deploying it.

What do I think about the stability of the solution?

There have been no stability issues.

What do I think about the scalability of the solution?

We tested up to 1,000 users across four regions - North America, EMEA Latin America, and Asia-Pacific. It scaled sufficiently for us to be able to do that cross-regional testing.

How are customer service and technical support?

It's excellent as the support is always available and helps us to troubleshoot any issues during execution with live support.

Which solution did I use previously and why did I switch?

We were using HP Performance Center in-house for the previous six years. NeoLoad was tried as there was a need to test mobile applications with load generated from the cloud.

How was the initial setup?

The initial setup is straightforward for both an in-house and a cloud one.

What about the implementation team?

The implementation was done by an in-house team.

What's my experience with pricing, setup cost, and licensing?

The licensing model is very flexible which supports a broad range of budgets, from small projects to enterprise-level setups.

What other advice do I have?

Choose this if your portfolio is predominantly web and mobile. You need to reconsider if your portfolio has many SAP GUI-based applications.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
Shivashanker Patil
Senior Performance Engineer at Ascend Learning
Real User
With features such as easy correlations and a user-friendly GUI, it helped us to improve the performance of our applications.

What is most valuable?

  • Easy correlations
  • User-friendly GUI
  • All components in one place

How has it helped my organization?

It helped us to improve the performance of all our applications.

What needs improvement?

Almost everything about NewLoad looks good, but I would like to have an indication of the variables for error messages appearing in the error tab. This would help us identify the users triggering the errors.

For how long have I used the solution?

I've been using it for a year.

What was my experience with deployment of the solution?

We've had no deployment issues.

What do I think about the stability of the solution?

There have been no issues with the stability.

What do I think about the scalability of the solution?

We have had no issues scaling it for our needs.

How are customer service and technical support?

Customer Service:

In my experience, customer service is good.

Technical Support:

The times that I've contacted technical support have been good experiences.

Which solution did I use previously and why did I switch?

I worked with HP LoadRunner in my previous projects. I started using NeoLoad when I joined a project that was already utilizing it.

How was the initial setup?

I wasn't involved, but my colleagues have said that the initial setup is straightforward.

What about the implementation team?

An in-house team implemented NeoLoad for the current project.

What's my experience with pricing, setup cost, and licensing?

Compared to HP LoadRunner, NeoLoad is cheaper.

What other advice do I have?

It's a good tool to use if you are not willing to spend lots of money on HP LoadRunner licenses.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user371118
Software Engineer at a healthcare company with 1,001-5,000 employees
Vendor
Easy to implement and you can start using it fairly quickly.

What is most valuable?

It gives us the ability to quickly create a script.

How has it helped my organization?

We are able to create load test scripts quickly in a fast paced environment, which in turn helps us identify performance issues.

What needs improvement?

The support needs to be improved, especially when you are looking for a quick turnaround because of the fast-paced environment we work in.

For how long have I used the solution?

We've used it for five years.

What was my experience with deployment of the solution?

We've had no issues with deployment.

What do I think about the stability of the solution?

It's rare, but sometimes the application will shut down.

What do I think about the scalability of the solution?

We've had no issues with scalability. You just have to add additional hardware as you increase the number of users.

How are customer service and technical support?

Customer Service:

8/10.

We have a very good partnership. They are always willing to work with us and help resolve any issues or concerns we have. They keep us updated on new releases and features that will be delivered.

Technical Support:

7/10.

Which solution did I use previously and why did I switch?

We previously used HP LoadRunner, but it was not easy to create scripts for a flex application.

How was the initial setup?

The initial setup was very easy.

What about the implementation team?

We implemented it with our in-house team.

What's my experience with pricing, setup cost, and licensing?

As you add more protocols and monitoring it does get expensive.

Which other solutions did I evaluate?

We looked at HP LoadRunner, SmartBear LoadComplete, and Appvance.

What other advice do I have?

It is very easy to implement and you can start using it fairly quickly.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
it_user371331
Chief Technical Officer at PA Department of Human Services
Vendor
The real time performance monitoring allowed us to find out the troublesome queries and increase resource efficiency.

What is most valuable?

The ability to test mobile applications, support cloud based testing, real-time monitoring during a test, advanced auto-correlation feature, infrastructure monitoring, detailed results reporting etc. It also gives the ability to manipulate the script by inserting JavaScript. It's one of the features that very few other tools have.

How has it helped my organization?

First and foremost, it took out the painful resource hours out that went in to manually correlate each and every action of the script while using LoadRunner. NeoLoad does not require much of scripting efforts, and correlation and parameterization are fairly easy.

What needs improvement?

We would like the product to be able to handle large project suits with large number of test scripts. Currently, we are not able to upload/publish huge projects back and forth from the team server to the NeoLoad controller. There is no support for VT3270 terminal emulation to support mainframe-hosted applications. There is no summary graph that shows the distribution of all of the response times on a single graph. We currently export the data as a coma separated variable (CSV) file and import into Excel to generate this summary graph.

For how long have I used the solution?

4 years.

What do I think about the stability of the solution?

A small number of times, when load testing huge projects with huge number (~10,000) of vUsers, we have seen that it completely froze. We had to recover from the previously saved version of the project.

How are customer service and technical support?

Customer Service:

Very good, the vendor has been very accommodating with the ebb and flow of our business needs. Neotys is a partner.

Technical Support:

Good. (They decommissioned the support for 4.2 though)

Which solution did I use previously and why did I switch?

LoadRunner. Cost effective and with almost same features that are required for our project.

How was the initial setup?

Straightforward and the technical support folks will be available to help.

What about the implementation team?

We utilized a vendor team. They were fairly knowledgeable and knew how to quickly troubleshoot the issues or contact Neotys.

What was our ROI?

A great deal of improvement in application performance happened because we could pin point the bottlenecks. The real time performance monitoring allowed to find out the troublesome queries and tune them. Increased resource efficiency.

What's my experience with pricing, setup cost, and licensing?

Low in cost when compared to tools like HP LoadRunner. Talking about licensing, they offer a cool feature that enables us to split the Vuser bundle across multiple machines. In other words, we can have multiple controllers running multiple load tests at the same time.

Which other solutions did I evaluate?

JMeter. Figured that it would not be suitable tool for our project and requirements.

What other advice do I have?

NeoLoad is the performance testing solution for Web and Mobile applications that can also be used to run the tests from cloud by having load generator’s from various locations. If there is a strong performance team that can analyze and correlate the test results with the raw data, then NeoLoad would be the right solution.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
it_user367815
Enterprise IT at a tech consulting company with 10,001+ employees
Consultant
Prior to NeoLoad, we needed to simulate heavy volumes. We're now able to generate loads from our side and quickly spin the load tests.

Valuable Features:

We pretty much use the majority of the features, but two of the key things are the cloud integration and the ability to spin up the load tests. That is the key driver to using this tool.

Improvements to My Organization:

It's impacted us in a positive way. Prior to NeoLoad, we needed to simulate heavy volumes. We're now able to generate loads from our side and quickly spin the load tests. This process has been greatly improved.

Room for Improvement:

We had some initial issues with the number of tests, memory leaks, and the controller itself.

Use of Solution:

I've used it for three years.

Stability Issues:

In the beginning we had some issues with the number of tests, memory leaks, and the controller itself. We worked with them and they came out quickly. After some time now, it's pretty stable.

Scalability Issues:

It didn't scale well at first but now it does.

Initial Setup:

The setup itself is straightforward. We are in a strict environment so we had some challenges installing it on some machines, but that was down to us internally. We're OK now.

Implementation Team:

The implementation was done in-house.

Other Solutions Considered:

We looked at some others, but NeoLoad really stood out in the end.

Other Advice:

I would strongly recommend this tool highly. It's improved our business a lot.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
it_user61302
IS QA Manager with 1,001-5,000 employees
Vendor
It's provided a faster turnaround on load-testing creation and the ability to use the team servers so multiple people can use this at once, which we couldn't do before.

Valuable Features

It makes it easy to create load tests. We used HP LoadRunner previously, and this is far better.

Improvements to My Organization

It's provided a faster turnaround on load-testing creation and the ability to use the team servers so multiple people can use this at once, which we couldn't do before.

Room for Improvement

It would be nice if this could be cheaper.

Use of Solution

I've used it for one-and-a-half years.

Stability Issues

It's been 100% stable, which we are very happy about.

Scalability Issues

Some of the additional modules can be pricey, but it is scalable.

Customer Service and Technical Support

Compared to support at our previous product, they're more responsive and they're more knowledgeable. Overall, it's three times better than our previous solution.

Initial Setup

The initial setup was very straightforward.

Implementation Team

We implemented it in-house.

Other Solutions Considered

We looked at JMeter, but NeoLoad had what we needed. We were using LoadRunner for six years and then we replaced it with NeoLoad.

Other Advice

Everyone's needs are a little bit different, so make sure you do a good POC.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
it_user367827
Web Engineer at a tech services company with 501-1,000 employees
Consultant
It’s allowed us the ability to look at the performance of an application prior to it going to production utilizing an automated tool.

What is most valuable?

  1. The GUI based scripting. Other tools are more script/code oriented. This is a huge time saver for us since we don’t have a dedicated performance team (yet). I can create scripts quickly and get tests going.
  2. The comparison report feature is good for pre/post-change result comparisons.
  3. The ability to write custom JavaScript actions. This has allowed us to handle JSON posts in web requests, which has been great.
  4. LAN/WAN emulator has been great in being able to simulate remote locations in their network connections.
  5. It ties into Dynatrace out of the box which we’ve used for deep-dive application performance testing as well as defining business transaction SLAs.
  6. The ability to create custom Frameworks for correlation values of custom, in-house apps.

How has it helped my organization?

It’s allowed us the ability to look at the performance of an application prior to it going to production utilizing an automated tool. Up to this point, most, if not all, of it was done manually. Projects are now making time for performance testing into their timelines.

What needs improvement?

Some of the documentation is geared more towards Java. We are an all-Windows shop and it would be great to see some C# examples on how to integrate a custom performance counter from functional testing tools into NeoLoad.

For how long have I used the solution?

It's been almost two years since we began using it.

What was my experience with deployment of the solution?

We didn't encounter any issues with deployment.

What do I think about the stability of the solution?

We haven't had any issues with stability up to this point.

What do I think about the scalability of the solution?

You can deploy as many LGs as you have a license for.

How are customer service and technical support?

Customer Service:

8/10 - Our account rep, Evan, and engineer, Steve, were great to work with when we were looking at the product. Ever since, they’ve been responsive and helpful.

Technical Support:

10/10 - The engineers know their product and know performance engineering.

Which solution did I use previously and why did I switch?

Prior to NeoLoad, we weren’t using any automated tool for performance testing.

How was the initial setup?

The initial setup was straightforward. The setup of an LG or Controller is as easy as a selection during the install. It’s up to the user how complex they want their environment to be.

What about the implementation team?

We implemented it in-house ourselves. Evan and Steve provided guidance while doing so.

What was our ROI?

I wouldn’t know the monetary side, but the ability to handle SignalR requests by dragging a ‘fork’ action onto the script speaks for itself. No extra coding or anything is needed.

To take it a step further, NeoLoad is smart enough, post recording, to identify and account for the presence of something like SignalR, and will create a copy of the script with the ‘fork’ action already present.

What's my experience with pricing, setup cost, and licensing?

At the time of purchase, it was a competition between Neotys and HP. Neotys’ licensing model was also different from HP, although HP was transitioning into a similar model as Neotys while we were going through the process.

Which other solutions did I evaluate?

We evaluated HP LoadRunner. I’ve used it before and it is/was the industry standard tool.

What other advice do I have?

If you’re coming from a code-based solution, spend time with Actions to become familiar with them. Additionally, if you’re using an APM tool like Dynatrace, test out the integration just to see what it gives you.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
it_user356646
Senior Manager Online Performance Engineering at Dell
Vendor
It took just a few minutes to set up Neotys Cloud, as opposed to a few days with other solutions. We also want to monitor applications within various environments, which NeoSense will soon allow.

Valuable Features

It is convenient to be able to keep up the pace in which the application code was delivered to us. Since we manage nearly 500-600 applications and we need to make sure they are tested, we wanted something robust with an option to check against previous baselines with minimal human intervention without sacrificing criteria.

We wanted something with less risk when incorporating or shifting scripts.

The fact that it loads from an external cloud is a plus as well.

It took just a few minutes to set up Neotys Cloud, as opposed to a few days with other solutions.

Room for Improvement

They are coming out soon with NeoSense. It's a hybrid between app management tools and a testing tool. This will be helpful for us because we'll be able to monitor applications within various different environments. This is precisely what we have wanted.

Use of Solution

We've been using it for a year.

Deployment Issues

We had no issues with deployment.

Scalability Issues

We used to have only 50 online users. Later we moved to where we are.

Customer Service and Technical Support

We have used their gold support option. Yes, we have been happy. They usually answer between 36 and 48 hours for non-critical issues, and with critical issues they hold your hand.

Implementation Team

We did it in-house.

Pricing, Setup Cost and Licensing

We didn't negotiate the price. To give perspective, we went from spending three million dollars to 160 thousand dollars, which is a huge saving and good value.

Other Solutions Considered

One of our goals was to find a platform that would have the functionalities that we were used to while incorporating in other ones. ProLoad was a competitor, and one other (can't make out name).

We wanted it to support agile and devops, while being able to upgrade to a number of releases over the course of a day, so it was a fit.

Other Advice

The company really strives on experience, always making sure that I as a customer am happy. For example, my team at Dell had an issue with our new version, which we had just updated to. Another team used a different version which required different protocols, and there were discrepancies between the versions. The company found out that our errors were user errors, and helped us fix the problem.

I helped them in terms of product development since we at Dell told them requirements and they added that to product life cycles. So we are a customer, but told them some requirements we'd like to see.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user347766
Manager IT Performance Testing at a retailer with 10,001+ employees
Vendor
I particularly like its ease of use, built in monitoring, and their extensive library of wireless devices and protocols.

What is most valuable?

It's got a number of valuable features, the best of which I think are the mobile testing capabilities. It allows for load testing to make sure users can experience apps properly and checks for performance issues on the server side.

I particularly like Neoload’s ease of use, built in monitoring and their extensive library of wireless devices and protocols. We are able to simulate various wireless communication standards such as 3G, 4G etc. with ease.

How has it helped my organization?

A lot of our customers are now using mobile technology, so it's provided us the platform and ability to do mobile testing, which in turn enhances their experience in our store.

What needs improvement?

NeoLoad’s lack of support for a complete protocol suite is somewhat limiting. While they do offer some protocols (Kronos, GWT V2.6) that others do not, they are not a one stop shop for protocol testing. Neotys is considered a forward looking company in the software performance testing world and legacy protocols such as RTE, SAP GUI and RDP are not included in their suite of protocols. I would like to see a more comprehensive offering of protocols with future releases.

For how long have I used the solution?

I've used it for a little over a year.

What was my experience with deployment of the solution?

No, not at all. This was probably the easiest part of it. iIt’s working out really well as it is not taking up much resource and it was very easy to install.

What do I think about the stability of the solution?

We didn't encounter any issues with stability.

What do I think about the scalability of the solution?

No, but to be fair, we don’t have a very large license right now, so this hasn’t become an issue for us. If I plan accordingly, this shouldn’t be an issue over time either.

How are customer service and technical support?

The questions that we’ve had for them, they have answered very quickly. The team here is very skilled and we've managed to resolve any issues we've encountered ourselves. But when we have reached out to them they have been very responsive and they've solved our issues quickly.

Which solution did I use previously and why did I switch?

We haven’t completely switched over yet. We are also using HP Performance Center. We are starting to slowly migrate. The mobile capabilities and the cloud testing capabilities, HP are lacking. I think in the next three to five years we will totally migrate to Neoload.

What about the implementation team?

We did it in-house.

What was our ROI?

We certainly have an ROI, but it is hard to put in to numbers. If I don’t do the testing, how much sales do I lose? Millions of dollars. I can definitely tell you it would be significant.

What other advice do I have?

They have been very responsive. They have even made an on-site visit. It’s been very good, they have had recent turnover but they are managing the transition well and they have even had a top team member reach out to us.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
it_user344052
Vice President of Technology at a retailer with 1,001-5,000 employees
Vendor
It provides data that provides executive-level comfort during peak periods, driving direction for future performance-oriented projects and goals. However, more community collaboration would be good.

What is most valuable?

It gives us the ability to quickly script a load test scenario and to develop complex testing scenarios. If you ever want to know more about your application, just recording a session can be invaluable.

How has it helped my organization?

Outside of the obvious benefits of load testing, the data we get back and are able to provide at the executive level provide a level comfort in the given application at peak periods, and drives direction for future performance-oriented projects and goals.

What needs improvement?

They have done well with patching and enhancements in each version. More of a “community” of shared experiences, help, and collaboration would be a good enhancement to their site.

For how long have I used the solution?

We've been using it since 2011.

What was my experience with deployment of the solution?

No issues encountered.

What do I think about the stability of the solution?

At times we have had issues with load generator agent stability, but tuning the heap and documented network hacks greatly reduced this.

What do I think about the scalability of the solution?

No, it's very easy and quick to scale.

How are customer service and technical support?

Customer Service:

7/10

Technical Support:

7/10

Which solution did I use previously and why did I switch?

No, not previously but we have demonstrated or performed proof of concepts of other products.

How was the initial setup?

The infrastructure setup was very straightforward, and getting started with our first test was relatively easy with online videos and resources.

What about the implementation team?

We did it in-house.

What was our ROI?

Difficult to really estimate, however we do know that by load testing we have found countless issues over the years that would have affected our e-commerce revenue had these issues made it to our production environment.

What's my experience with pricing, setup cost, and licensing?

We evaluated other products of similar nature as well other team member’s prior experiences with similar products.

Which other solutions did I evaluate?

We evaluated other products of similar nature as well other team member’s prior experiences with similar products.

What other advice do I have?

Start with some small, simple scenarios and expand complexity from there. Your project will take shape quickly, and something that seems overwhelming and complex will all of a sudden will not be so hard. Use the online resources and videos.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
it_user104961
Penetration and Neoload Tester at a university with 501-1,000 employees
Vendor
LoadRunner vs NeoLoad
The six phases of an IT project: Enthusiasm Disillusionment Panic Search for the guilty Punishment of the innocent (the performance tester) Praise and rewards for the incompetent non-participants This article has been put together as part of an evaluation of the performance test tools NeoLoad and LoadRunner. I consulted a variety of sources, including user groups, discussions with colleagues, telephone chats with HP (the vendors of LoadRunner) and Neotys (the vendors of NeoLoad), and of course the Oracle – Google. The opinions in this article are my own and do not represent the views of any particular company, the software vendors or any organisation (and I’ve assumed that readers have some knowledge of web application performance testing). The versions discussed in this article…

The six phases of an IT project:

  1. Enthusiasm
  2. Disillusionment
  3. Panic
  4. Search for the guilty
  5. Punishment of the innocent (the performance tester)
  6. Praise and rewards for the incompetent non-participants

This article has been put together as part of an evaluation of the performance test tools NeoLoad and LoadRunner. I consulted a variety of sources, including user groups, discussions with colleagues, telephone chats with HP (the vendors of LoadRunner) and Neotys (the vendors of NeoLoad), and of course the Oracle – Google. The opinions in this article are my own and do not represent the views of any particular company, the software vendors or any organisation (and I’ve assumed that readers have some knowledge of web application performance testing).

The versions discussed in this article as of April 2014 are:

LoadRunner 11.52

NeoLoad 4.2.2

I hope this evaluation and comparison will assist you with your own evaluation of LoadRunner and NeoLoad.

Introduction

LoadRunner and NeoLoad are considered the top two best performance testing tools on the market. Comparing the two tools is just like comparing a Mercedes Benz and a BMW: both are high performance, perfectly engineered machines that also have an associated prestige.

Performance testing is a critical component of the software testing process. It determines the actual operational boundaries that will simulate the real world use of an application. Performance testing is load testing, stress testing and scalability testing.

Why are we here?

The philosophy behind performance testing is we don’t want any of our IT systems to crash – it’s terribly embarrassing and can be very damaging, never mind expensive, when a crash happens.

A load test determines how a system behaves under a particular workload. The main objective is to see how various components react to a gradual increased ramp up of workload. The usual outcome of load testing is throughput, response times, CPU load and memory usage.

Stress testing determines the breaking point or the unacceptable performance point in a system. It reveals the maximum service level the system service can sustain.

Scalability testing evaluates the effects of adding additional hardware, virtual or physical, in order to distribute work among system components.

The following evaluation considerations have been taken into account for LoadRunner and NeoLoad:

  1. Record and playback
  2. Recognizing dynamic web components known as ‘correlation’
  3. Data functions – the ability to use data from a text file to populate web forms resulting in realistic test data
  4. Randomising – picking random datasets in rows, check boxes and radio buttons
  5. Putting it all together and orchestrating the load simulation and analysing the system response

LoadRunner is written in C++, and NeoLoad is written in Java. LoadRunner scripts after recording can be manipulated and debugged using C. NeoLoad is a scriptless tool – instead, it involves dragging and dropping functions into a script. Also IF statements and loops can be dragged and dropped. Even exception handlers are controlled the same way within NeoLoad. There are some good and some cumbersome qualities attached to both – the NeoLoad feature is less powerful than the LoadRunner vugen feature, albeit LoadRunner takes more hours to script record, parameterise and debug.

Recording scenarios

If you’ve ever used a proxy debugger like fiddler, you’ll see what’s happening behind the scenes when surfing the internet. There are send requests and responses from your computer’s browser to the website’s web server. Recording load testing scenarios works just like a proxy: when the record button is pressed, it actives a proxy that acts like a ‘man in the middle’ capturing http requests and responses from the system’s web servers.

LoadRunner architecture

LoadRunner, which has a trial version, consists of three components: the virtual user generator, also known as vugen (script recorder and editor), the controller where all the vugen scripts are controlled into a scenario, and the Analysis, the results component. These enable the user to keep scripting in vugen while a scenario is running in the controller component, which usually happens because the load tester is hired at the end of the System Development Life Cycle (SDLC) and is under a huge amount of pressure.

NeoLoad architecture

NeoLoad consists of one component encompassing all-in-one software, which means that if a load scenario is running you cannot create and edit scripts.

Recording web with LoadRunner

Nowadays Chrome and Firefox work perfectly as recording browsers in LoadRunner’s vugen. Once upon a time, just Internet Explorer worked. Other browsers made vugen crash when the recording button was pressed, so LoadRunner consultants were limited to the browser people use to download other browsers.

If you’d like to get the time it takes for a user to login to the system under test, press a simple ‘Start Transaction’ button during recording – you can name the transaction appropriately, log the user in, then end the transaction when the logged in page has downloaded.

The transactions look like this for a simple login script:

Recording web with NeoLoad

NeoLoad records scripts in a ‘tree-view’ like structure. A wizard pops up after recording to automatically correlate the recorded script for you. Start and end transactions are not like those in LoadRunner – they are called ‘containers’ in NeoLoad and there is no end to the container as the ‘end’ of the container is actually the beginning of the new one, like this:

Correlating

Correlating is extracting dynamic values and placing each dynamic value into a variable from GET requests and substituting it into the POST response, usually in the request after the GET. This is the most time-consuming task in load testing – just ask any performance test analyst.

Correlating with LoadRunner

Design Studio is the tool LoadRunner’s vugen uses to try to correlate dynamic values in a newly recorded script.

‘No correlation parameters found’: I glanced at the recorded script below and found many dynamic parameters.

So why didn't the automatic correlation utility work for LoadRunner? This is a good question considering some of the other protocols that LoadRunner uses correlate really well. Like me, many of my colleagues have become used to manual correlation when using LoadRunner for http/web.

In fairness to LoadRunner, it can find one or two dynamic parameters, depending on the size of the recorded script. It looks like this:

It works (sometimes).

Manually correlating with LoadRunner involves placing this statement to grab a parameter from every dynamic GET request, which you’ll have to then place before a POST request if required.

Correlating with NeoLoad

Correlating is definitely where NeoLoad has an edge over LoadRunner. NeoLoad takes hours of painstaking manual correlation work out of the equation, and that saves bucket loads of cash in consultant fees and maintains deadlines. Any IT manager will know how much a LoadRunner contractor costs daily. If that figure is going to be potentially halved (or otherwise cut down by the speed NeoLoad can do the job), then maybe the next word that enters the manager’s head is ‘bonus’ or even ‘gratitude’, for finishing the project on time and within allocated budget parameters. Gratitude goes a long way.

There is some manual correlation in NeoLoad that involves a not-so-very-cumbersome two clicks of a mouse.

NeoLoad is so intuitive that if the same extracted value exists in subsequent requests within the scenario, this pops up (music to my ears):

And if you’d like to replace all, then:

The music keeps on playing – click, click, click ... and done.

Playing back debugging/recorded script with LoadRunner

Debugging is instantly more informative in LoadRunner than in NeoLoad as you can see the script executing and outputting to the log (circled below). It’s easy to watch variables and check for errors in real-time script compilation and execution.

Playing back/debugging recorded script with NeoLoad

One quick and easy way to check and debug a script in NeoLoad is to use the virtual user checking option by right clicking on your script. This runs the script and shows errors in requests that need to be addressed.

In NeoLoad, a more involved debugging analysis (after checking using one method) is to use JavaScript. Do this by dragging the JavaScript function into a script and coding the appropriate write to a text file. All variables can then be sent to a text file and used as a watch list.

NeoLoad will also check the JavaScript syntax for you.

Parameterising

Parameterising is adding a text or csv file to a load testing script. For example, when a script is recorded, there is a hard-coded user id and password in the script. Creating a file of five user ids and passwords and thus adding to the script while replacing the hard coded values is parameterising. You can control this, for example, using the five user names – the script can randomly pick the user id and password or it can pick them one line at a time incrementally.

Parameterising with LoadRunner

I’ve always found parameterising with LoadRunner straightforward, as long as you’re aware that creating a new parameter creates a new .dat file as well. It’s easy to have a user name and password in the same file (like above) instead of in two separate files. You can also edit the text file in Notepad, which makes pasting large amounts of data relatively easy.

Variables in LoadRunner

Where variables need to be declared as well as the variable type, like char or int etc., and values need to be added to them during the script execution, then do the following:

Parameterising with NeoLoad

Adding datasets to NeoLoad isn’t too dissimilar to LoadRunner. Also you can import spreadsheets into the variable manager or just create datasets manually.

Variables in NeoLoad

NeoLoad has an edge over LoadRunner, as none of the above coding is necessary. Variable types are added using the variable manager as in the screenshot below:

NeoLoad has an added SQL variables feature that queries the backend database for test data.

Randomising user input/actions

A very important aspect of a realistic load test is to have as many virtual users doing as many different things as possible to the system under test. The more random actions the better, such as if there are 10 links for two vusers to click, then there should be a one-in-forty-five chance that the two users click the same link (if you don’t know what I mean, check out permutations on Google).

Randomising with LoadRunner

Picking a random row from a result set you’ve just recorded with LoadRunner involves doing something like the code below. First declare the variables, then using the web_reg_save_param_ex() built in LoadRunner function, grab the number of rows, place the number of rows into a random number function between 1 and 10 (if there are 10 rows), and finally get the script to click on any of the rows – this should be different for each iteration in your script, depending on the test you’d like to conduct.

Randomising with NeoLoad

Randomising with NeoLoad is very easy, but to randomise you have to record every request and place them in a container. To give you an example, imagine you are filling out an online form to register for an online shopping site (see the screen below). Just before you submit the form you have the option of subscribing to the store’s newsletter, and also entering into a draw if a radio button is checked. Now that’s four combinations:

  • 1.Submit form
  • 2.Submit form + tick newsletter
  • 3.Submit form + tick prize draw
  • 4.Submit form + tick newsletter + tick prize draw

In NeoLoad you have to record this randomising scenario four times and place each into a container. It’s then possible to control the randomise settings for a realistic and valid load test.

The fun part – performance testing

After all the hard work is done – the correlation, the parameterisation and getting admin access to every single pre-production server in the system – it’s time for the real fun. Every performance tester loves this part because it’s the breaking stuff part: Lego for grown-ups.

Other forms of load testing include what hackers do – distributed denial of service (DDOS) attacks. Well, that’s more like a stress test.

IP spoofing can be implemented with both LoadRunner and NeoLoad – this means that an even more realistic performance test can take place. You don’t want a router in the network to reroute the web requests to a honeypot if there’s a rule in place that redirects heavy traffic away from the system under test.

Load Testing using the LoadRunner controller

All servers in the system under test can of course be added into the load scenario, so that real-time monitoring of server resources can be directly correlated to the number of vusers being ramped up, to a peak hour load test. So for the layman, let’s see what maxes out first if 1,000 users login to your website all at once, or all in the space of five minutes – things will start to break depending on a lot of factors, too many to discuss in this article.

Server statistic graphs can be added to the load scenario, and all the gathered stats as well as transaction time taken can be collated in LoadRunner’s Analysis component. A performance Test Summary Report (TSR) uses a template that can be modified to your liking and then presented to your IT Test Manager. Make sure you let him or her know that you typed the report from scratch – work is all about perception after all.

The client controller component of LoadRunner has all the bells and whistles needed for a realistic load test. There’s only one drawback: there’s a limit on the number of performance counters that can be added to a scenario. If there is a requirement for more performance counters you have to install and configure monitoring software called SiteScope. All servers in the system have to then be added to SiteScope after SiteScope and the SiteScope license server are installed.

Load testing using NeoLoad

The ‘runtime’ component of NeoLoad and the LoadRunner controller are on a par in terms of the capabilities they have for you to add the appropriate graphs.

LoadRunner vs NeoLoad – the cost

Here’s what every IT manager wants to know: Which tool is cheaper? Is it value for money? Now this is a tricky one: How much does LoadRunner cost?

Performance testers have always known that when it comes to price, LoadRunner is not a BMW or Mercedes – it’s a Ferrari. I wasn’t very successful in my endeavours to obtain a quote for LoadRunner. I contacted HP in London, and they referred me one of their local partners here in Australia. My enquiry was pretty simple: I wanted a quote for a 5,000 user Web vusers license, but HP’s partners said via email that they needed more information.

Using Google to ascertain the cost of a LoadRunner license had mixed results. I came across an old article mentioning that a 5,000 vuser perpetual web license with support for three years was about a million dollars. That article was written in 2005.

When I contacted Neotys in France, they were a little more helpful with a licensing quote for NeoLand, and there was a lot of room for negotiation as well, including email support, monitoring for Oracle and a 5,000 vuser license that never expires.

Based on my very ball-park figures, there’s an approximate cost correlation between NeoLoad and LoadRunner. Imagine that LoadRunner costs $100:

LoadRunner: $100

NeoLoad: $10

NeoLoad is about 10% of the cost of LoadRunner, and Neoload includes technical support as well as perpetual licensing.

There are also several product packages associated with both HP and Neotys that can be negotiated as per your allocated budget, so keep that in mind. All you have to do is pick up the phone and talk to them.

So there you go – my take on the top two best performance tools on the planet. Taking into account industry experience, time frames scripting, hours spent completing the job (usually after working hours) and downright value for money, the winner has to go to NeoLoad. It just works.

If your company has spent its money on LoadRunner and highly paid consultants developing test scenarios for your application, then the money has not gone to waste. If your company wants a cheaper product that does the same job as LoadRunner, then evaluate NeoLoad and ask an experienced performance tester’s opinion on the tool. The downside to this option is finding experienced NeoLoad consultants.

LoadRunner is the preferred tool for large organisations, such as banks and financial institutions. Right now, NeoLoad is the preferred performance testing tool for universities and smaller organisations.

Some final questions

My employer has been using LoadRunner for performance testing for many years. How expensive would it be to hire people to convert these scripts to NeoLoad?

Weighted Rankings 1 – 5 (5 is excellent)


Can LoadRunner scripts be migrated to NeoLoad?

Yes, and vice versa. It doesn’t take as much time or resources as you think. Companies don’t need to hire specialists to re-code LoadRunner scripts into NeoLoad scenarios.

Disclosure: I am a real user, and this review is based on my own experience and opinions.