Tricentis NeoLoad Reviews

We are primarily testing APIs but also testing the UI in some cases. In my last job, I also developed a plugin to enable NeoLoad to measure the end-user experience. If you have a modern web application, you usually only exchange JSON files between the client and the server. On the client side, you will render the JSON information to an HTML side or a dynamic DOM structure.

NeoLoad is used for testing the response time, which is not the same as the time it takes for the user to see the page. When the JSON document arrives on the client side, the client has to render the information. If you are considering the user experience from the end user's perspective, you are not interested in how long it takes to get the JSON. You want to know how long it will take to get the JSON and see the information on the page.

NeoLoad can't do this because it does not render the data from the server. The browser engine uses considerable RAM, so we cannot simulate many users simultaneously. We enabled NeoLoad to control a typical test. For example, a QF Test like Robot Framework tests the web UI. It starts and renders the application to measure how long it takes to see some special information on the page. 

These response times are fed into NeoLoad through a data exchange API, enabling us to visualize the actual user experience inside NeoLoad. That's NeoLoad's primary advantage. It has incredible APIs to implement special additional features or plugins you need for special applications.

NeoLoad helped us to transition from load testing to continuous performance measurement. It is integrated into our CI/CD pipeline. We implemented performance testing like you would have unit or functional tests. Every day, we get information if something happens in our application that affects the performance in a negative way.

NeoLoad offers better reporting than most competing tools. It is effortless to analyze and measure the reported data. It's also simple to generate a report that most people can read and management can understand. NeoLoad helps you figure out the main issues inside the application. 

Some users may find NeoLoad too technical, while other users may prefer a scripting language instead of a UI with figures and forms they have to fill in.

I began using NeoLoad when Neotys still owned the product. In total, I have about 13 years of experience working with the solution. 

Setting up NeoLoad is straightforward.

We deployed NeoLoad in-house. I already had experience with the solution from my previous job, so we didn't need Tricentis to send a consultant to help us. However, it's helpful to take advantage of their consultants if you need them. Tricentis has a network of freelancers, so it's easy to find a consultant near you, and you can usually find somebody who speaks your mother tongue.

We had three specific use cases we targeted Tricentis NeoLoad for. The first was for services, the second for terminal emulation mainframe applications, and the third was for the web.  We are looking for a more cost effective, efficient (low code) alternative to our current framework. 

Tricentis provides great documentation and collaboration.

The most useful aspect of Tricentis NeoLoad was the low/no code for the web.

Tricentis NeoLoad could improve the range of codeless scripting such as with terminal emulation apps. 

We did a trial of Tricentis NeoLoad for approximately three months.

Tricentis NeoLoad is a stable solution.

We did not have any problems with the scalability of Tricentis NeoLoad but did not fully push the product.  We have approximately six or seven people involved in the POC. This included senior engineers and test developers.

We do not plan to expand the use of this solution at this time because it does not meet all of our needs.  Would consider a second look in the future.

The technical support from Tricentis NeoLoad is very helpful.

We have used Micro Focus LoadRunner. However, it is expensive and there appears to be a declining number of users.  Tricentis NeoLoad is more affordable and would better enable our pipeline but has fewer features.  We are not in a hurry to change but will continue to review products of interest and will continue monitoring Tricentis product roadmap.

The initial setup of Tricentis NeoLoad was straightforward. We had the installation instructions and the packages, and it took us approximately four hours to complete.

We did the implementation in-house.

Tricentis NeoLoad price is a benefit of using this tool, it is less expensive than some of the other solutions.

I would encourage others to take a look at this solution. It is a good solution and they maintain the technical expertise needed for success.

I rate Tricentis NeoLoad an eight out of ten.

We are using it for performance testing.

It's improving our test lifecycle time. It's taking less time compared to the previous tool.

The Frameworks feature is valuable. NeoLoad Web and the API are also valuable. It provides API support.

The overall stability of the GUI should be improved. The GUI component is not stable enough. We have observed crashes several times. 

There should also be support for more protocols for desktop applications.

I have been using this solution since 2019. It has been four years. 

I'd rate it a four out of ten in terms of stability. The GUI isn't stable enough.

I'd rate it a seven out of ten in terms of scalability. 

Within our team, we have around 25 users. Their roles are testers, test leads, and test managers.

We aren't planning to increase its usage. The way we are using NeoLoad and other tools is pretty streamlined. We are using NeoLoad 90% as per our requirements, and we aren't planning to increase its usage.

I'd rate their support a four out of ten. The problem is that we have to raise a support ticket in the portal. They usually take a long time to reply. The first or second response from the team usually covers the basic tasks, which the team would have already done before reaching out to their support team. So, by the time we get a proper response from them, we might have moved away from that particular project or we might have found some alternative. That part is a bit bad.

Neutral

We were using Micro Focus LoadRunner. We switched because of the organization's requirements.

As compared to other tools, the initial setup is easy, but there is some complexity that comes with the Kubernetes-based setup of one of the components. So, it's a bit easier than others, but there are certain aspects that make it difficult. I'd rate it a six out of ten in terms of the ease of the setup.

Its deployment took a couple of weeks, which also included testing. 

It was deployed in-house. We deployed it ourselves based on the documentation.

For deployment, we only need one employee who has experience in cloud deployment. For maintenance also, one admin is enough. 

I'd rate it a seven out of ten in terms of pricing.

We evaluated open-source tools such as Scala and JMeter. We went for NeoLoad because of the collaboration feature, which isn't there in Scala and JMeter. Another reason was the enterprise support, which doesn't come with open-source tools.

It's definitely worth a try. Compared to LoadRunner and other tools, it gives more flexibility and is easier to maintain. 

Overall, I'd rate Tricentis NeoLoad a seven out of ten.

I am a senior performance engineer providing end-to-end performance testing. This involves gathering all non-functional requirements, creating a test plan, and creating a test strategy document. From there, we would start our benchmark and baseline testing based on the tool and protocol of the specific client in order to gather the scripting and the load testing. NeoLoad helps us identify bottlenecks and memory leaks which will help us increase the hardware capacity for the client and/or look at things on the software end including scalability, reliability, testing, and failure testing.

It offered us an easy to use, limited code option for end-to-end performance testing.

Each of the main components - design, runtime, and results - are unique in their own way.

The design offers four components: user part (which is where we would be creating and enhancing scripts while implementing things like pacing, think time, and error handling concepts), servers (which is where we have the service testing of API testing), population (which is where we can plug scripts, create multiple populations, and perform bandwidth testing), and monitors (which is a server-side where we can add credentials to a database or server via SSH details).

Runtime allows us to create scenarios, populations, low variation policies, load generators, and custom load profiles. Other functions include graphs, templates, check alerts, set alerts, check response time, etc.

The results tab provides us with the NeoLoad report which we can segregate.

We would like NeoLoad to be able to support more protocols.

Testing can also be a little tricky at times.

I have been using this tool for almost nine years.

This tool is easy to use. It requires little code implementation thanks to many predefined areas. I have not experienced any stability issues over the past nine years I have been using it.

I would rate the stability as five out of five. Right now we have about 500 users but in the past, we have had roughly 200k. They range from senior performance engineers to performance architects to performance test engineers.

Customer service is fantastic. I have been working with one customer service rep for the past nine years. 

Initial installation, registration, and setup were all straightforward and very easy, especially in comparison to other tools.

We were able to implement the tool with our in-house team.

We were able to set up per user basis licensing.

Newer additions such as NeoLoad SAP, Citrix, NeoLoad Web, etc. are excellent extensions with a broad range of new functions.

Organizations should be aware that there are limited videos available on the internet regarding NeoLoad so they should ensure that they receive proper training from the reseller/integrator upon purchase. 

The two standout features for us are:

• Team Server Collaboration - as is the case with many MNCs like ours, teams/team members are spread geographically at various locations. The Team Server module is a boon for such teams as multiple users can work on the same project simultaneously and check-in the changes. It also helps manage and allocate virtual users (of shared license) for execution of multiple projects at same time.
• Results & Analysis - NeoLoad provides comprehensive results for any load test. The results can be drilled-down to minute levels (from scenario level all the way to individual request level) of any transaction which makes root cause analysis very effective. NeoLoad can also consolidate results from different sources and the results are graphical (one can build custom graphs for any transaction) and colourful for easier interpretation.

Other advantages are

1. GUI driven recording makes it easier to record and prepare test scenarios without need for extensive coding knowledge thereby enabling tool users to ramp up faster
2. Supports a ton of protocols plus additional plugins such as Jenkins, AppDynamics, etc.
3. Custom scripts and frameworks can be built
4. Recording and simulating mobile applications is simple yet realistic

NeoLoad is a robust and versatile tool. With plugins for CI tools like Jenkins, NeoLoad is configured to automatically trigger small load tests on modules each time there is a build update. This has enabled us to identify performance issues for our clients at a very early stage. It has undoubtedly an impact during larger scale enterprise performance tests when all the modules are integrated and tested as cost to fix performance issues increases with later stages of deployment.

We have raised a few support tickets for certain feature sets/improvement areas.
These include permanent fixes for a few stability issues discussed in the following question and implementation of more robust solutions for media streaming protocols, TCP/IP and websocket protocols.

Also, there are a few changes in the latest version, 5.2, which have brought in many improvements but have removed certain features. For example, v5.2 has done away with the 90th percentile reporting for response times, which was present in earlier versions. While this is a small deviation, it still is important for organizations using it.

The tool can hang and take a long time to open projects if the size is large and if many graphs and result templates are stored. Certain protocols although supported by NeoLoad need to have custom solutions built (e.g. web socket connections) while solutions for certain other protocols such as media streaming protocols like HDS, HLS, etc. are still in the nascent stage.

We have used NeoLoad since late 2009.

There is no specific maximum project size defined (actual NeoLoad project size). This size is dependent on number of user journeys in the project, number of scenarios defined, number of graph templates stored, and number of results stored and so on. The bigger the project, the slower it opens in the tool (as expected) but many times can be excruciatingly slow. The other drawback is that the graphs and results may not get saved correctly. With the latest versions of NeoLoad, this issue has been significantly fixed but it can still occur at an undesirable moment as the optimum size of a project can only be determined by trial-and-error. Our experience has taught us to keep multiple backups of same project under two categories for such exigencies – the first category would contain backup of the project with previous results and graphs while the second one would have only the project (VUs and scenarios) but with previous execution results and graph templates removed.

The other issue we faced was with respect to results generation. If we need to abort a performance test that is in progress, all one has to do is click the stop button. However, if the stop button is clicked twice inadvertently or clicked twice because it still showed “consolidating results” message for eons after the first click, the results ultimately generated would get corrupted. Either the results would not show for all virtual users executed or would not get generated at all. The last time we checked, Neotys did not have a concrete solution to this problem even after analysing the execution logs and troubleshooting it. Their “solution” was to check our stop policy for the scenarios executed (to make sure all virtual users reached their ‘end’ state to generate results) and also to check if any antivirus or firewall was impeding result generation (an antivirus or firewall can prevent NeoLoad from writing results to your folder). From our side, our stop policies were fine and there was no antivirus/firewall to block NeoLoad so the solutions were not that useful. As a workaround, we would click just once to abort an execution run and wait for the results to be generated even if it took a while.

The customer service and technical support is excellent if you are a paid customer. The turnaround time and SLA for a priority one - Urgent - issue/ticket is almost immediate. For less priority tickets it is quick (usually a couple of hours) and the representatives even help build a custom solution for your project if need be. However, if you are using the trial version, you would have to depend on NeoLoad forums for answers. On the flip side, Neotys documentation available on their side is detailed and exhaustive which should answer most questions when you are evaluating the product.

We have used a host of other solutions such as LoadRunner, JMeter, WebLOAD, SOASTA, etc. While the choice of the solution depends on various factors such as open source vs commercial tool, protocols used, number of virtual users etc. NeoLoad comes up trumps on most factors which were needed for our clients namely:

1. Cost
2. Ease of deployment and scalability to test for high user loads (in the range of 100,000s users for large enterprise tests)
3. Ability to monitor performance at various levels of architecture
4. Graphical representation of results
5. Detailed comparison can be seen in the sheet attached

Initial setup is a breeze. The installation is a complete package without having to switch between standalone modules for recording scenarios, execution and analysis which is not the case in say, LoadRunner or WebLOAD which has different modules for each (like VuGen, Controller & Analysis).

Neotys also offers their own cloud infrastructure to leverage upon for tests with larger user loads which can be configured to each project needs. However, if you are setting up your own cloud infrastructure on say, Amazon AWS, you would have to be equipped with enough knowledge to set up the images (AMIs) and configure the necessary IPs & locations for the test.

When we started out in 2009-2010, we did use the vendor team to guide us through and setup the necessary infrastructure. Over the years, we have built a strong in-house team and have setup our own infrastructure and cloud instances.

Depending on the technical expertise of your team, you can choose to implement in-house or avail the support of the vendor team. While the basic installation is straightforward, the vendor team could help you choose the right hardware systems/configuration for localized load generation or help you configure Neotys cloud for IP simulation or handling larger user loads.

Neotys offers various licensing plans to suit every business. The free version comes with 50 users with all protocols included for small load tests. The paid licensing plan depends on number of virtual users needed (to simulate the load), duration, the protocols under test, number of concurrent applications being tested, cloud usage etc. For example, the licence can be availed even for a day, a month, or a year depending on the need. In that sense, the licence cost can range from $1500 to >$20000. A general rule of thumb is the longer the licence period and the greater number of virtual users results in a lower price over the long term.

Leverage the free version of performance tools to its maximum capability to evaluate and cross check against the performance testing needs for your business to make an informed decision on buying the tool. The pros outweigh the cons and so NeoLoad is highly recommended over its competitors.

We used it for putting the load on the system. For example, for a big sale of an online marketplace, we created the scripts with performance testing tools such as Neotys and LoadRunner. We used to search for a product and add it to the cart. We used to capture all this traffic through these tools, and then, we used to do the real-time testing. So, we used to simulate the real-world user scenario. For example, if the company was expecting around 20,000 users on a specific day, we used to simulate the volume of 20,000 users on the application. 

The deployment model depends on a client's requirements and licensing. If we have a sufficient budget, we always go for the SaaS model. If we have a limitation of licensing, then we prefer to go for on-premises deployment. We usually need to get support from the admin team to set up everything. We used to take care of this aspect so that the support team will be able to do things on our behalf, such as setting up things in their environment.

We can quickly do scripting with NeoLoad, which makes it easier to give timelines and meet them. We can easily tell a client that within this much time, we can provide the end-to-end scripts, and we'll be able to do the execution along with all the activities. 

The licensing cost is very less for NeoLoad. It is user-friendly and easy to understand because they have created so many useful functionalities. When I started working with this tool, we just had to do the initial assessment about whether this tool will be able to support our daily work or not. I could easily understand it. I didn't have to search Google or watch YouTube videos. In just 15 to 20 minutes, I was able to understand the tool.

We can easily do scripting with NeoLoad. We just have to understand the basic functionality to create a script. There is not that much effort that we have to put in. If I have to transfer knowledge to a new person or a beginner, he will also be able to pick it up quickly. 

LoadRunner supports multiple protocols, whereas NeoLoad supports only three protocols. With NeoLoad, we can go for the SAP technology, web-based HTTP, and Remote Desktop Protocol (RDP). If I have to simulate .NET application-based traffic, I won't be able to do that. So, protocol support is a limitation for NeoLoad. It should support more protocols.

One issue that we faced was that multiple users weren't able to work on the same application. We used to create multiple scripts based on the application and based on the projects, and then we used to integrate all of them in a single place. With NeoLoad, if you have to do this activity, to import a script, one user has to log off, and the second user has to copy the script and improve it, which is a time-consuming activity. These are the things that can block any further activity.

I have been working with this solution for 18 months.

I would rate it a nine out of 10 in terms of scalability. It is currently being used extensively. My organization is huge, and we have an employee count of more than 300,000. I wouldn't be able to provide the exact count, but for my project, 20 people are using this tool.

They usually clarify any queries or issues within three hours. They usually come back to us within three hours, which is as per our SLA, so that's good. I would rate them an eight out of 10.

Previously, we were using Micro Focus LoadRunner Enterprise. It is still in use. We haven't completely removed it. We switched because of the licensing and the time taken to create a script.

Neotys has taken some references from LoadRunner to create NeoLoad. So, in terms of functionality, NeoLoad and LoadRunner are quite the same.

The licensing cost of NeoLoad is lesser than LoadRunner. LoadRunner supports a higher number of protocols than NeoLoad. LoadRunner has more protocols for interacting with the application than NeoLoad. So, there are multiple things that we can simulate with LoadRunner, such as a desktop application or traffic for a banking domain.

If I have to create a script in LoadRunner, it usually takes around six hours, but if I do the same thing in Neotys, it usually takes around one and a half hours to complete everything. So, we can provide a deadline for deploying a script. Time is money, so NeoLoad is better in terms of time.

It is straightforward. You don't need much understanding from the installation perspective. You just need to download the .exe and install it. You just need the license, and if you are going for the trial version, you just upload the license. If you are going for a business license, you just have to tie up with them and reach an agreement. That's it. 

It is a one-day thing. One day is enough to complete the installation and the setup.

It was an in-house job. In terms of maintenance, we usually have an admin team and a security team to put patches, etc.

Its licensing cost is very less.

If you are dealing only with web HTTP, you can definitely go ahead with this solution because time is money. 

I would rate NeoLoad an eight out of 10.

One of our use cases, it was Oracle E-Business Suite (Oracle EBS) application testing. The other one was a migration project for SAP HANA. In those two use cases, we needed to perform benchmarking and increase the coverage from an end-to-end point of view rather than just focusing on the API endpoints. 

We had to automate some of the Oracle form-specific transactions using NeoLoad, as we wanted to expand the coverage. So, I used NeoLoad as a hybrid solution alongside JMeter. Initially, the majority of the volume was on JMeter, but for the Oracle forms usage, which was low, we utilized a low-volume license from NeoLoad.

NeoLoad is actually really good, mainly because they have a world-class support service.

NeoLoad can improve the correlation templates, which are specific to frameworks. There's room for improvement in that area.

In future releases, I think they definitely need a full UI revamp because I believe it's built on Java technology, which feels a bit outdated. I would like to see that being improved. Additionally, increasing the coverage of the correlation templates capabilities would be very useful.

I've used it across around three projects mainly because we provide the best options for our clients. We combine open-source tools with commercial ones. So, in two cases, we encountered limitations with JMeter, mainly because it doesn't offer off-the-shelf support for Oracle forms and SAP GUI. That's when we decided to introduce NeoLoad.

It is a stable solution. When I worked with this tool, the main pain point was finding solutions. They have some good documentation, but it could be improved because sometimes the feature I was looking for was already there, but it took me a while to find it due to the documentation. 

So, NeoLoad can improve the documentation. Otherwise, the tool itself is stable. For our purposes, using it with around 100 users, it was very stable.

It is a scalable solution. I have integrated NeoLoad three times in your personal projects. I would say the main feature that might lag for enterprise usage is the capability to manage scripts in a team environment. You should be able to maintain scripts across multiple users through a source code repository. I'm not sure whether NeoLoad fully supports managing Java programming-like stuff with ease through a source code repository. That might be one aspect that could make it more qualified for enterprise use. 

Other than that,  NeoLoad can support enterprise needs, but I'm not sure about the team environment support with a source code repository. But as for the tool itself, because we are using it at kind of an enterprise level, not extensively, but it caters to most of the requirements.

The customer service and support were good. The initial support I received from them was world-class. They treated the issues as their own, and I would rate it eleven out of ten. It was that good.

Positive

The initial setup is not that complex unless you try to use it against a complex application like Oracle Forms. Oracle always tries to encourage the use of its own proprietary performance testing tool. They make it harder for the clients to use other tools, pushing them to buy their own. That's when the complexity arises. Other than that, it's straightforward and very intuitive. 

As long as you have a basic understanding, it's not that hard to get started. Personally, I came from a JMeter open-source background, so transitioning to NeoLoad wasn't too hard for me.

Initially, we faced some configuration issues, and the documentation was a little lacking, but the support team was very helpful in resolving the problems and guiding me through the setup. After that, using the tool became a walk in the park. However,  NeoLoad can still improve certain aspects.

NeoLoad has a cloud solution, mainly for executing tasks, but the tool itself is an on-premises one. The main issue with these cloud providers is that they should come up with CNA, which is the security certification. Otherwise, people, especially in the government sector, don't want to use them. They are very conservative about security in New Zealand, so they prefer everything to be on-premises mostly. Cloud solutions, like Platform as a Service, are not easily embraced there; they prefer on-premises solutions. So, everything is on-premises.

NeoLoad now has a much more flexible licensing process. Earlier, it was hectic with people. Nowadays, licensing itself is a killer for most products.

For example, we purchased the Tableau license, and they had this on-premise licensing server setup, which is not working at all. So we had to ask them to give us a cloud license, and the discussion is still going on. They are trying to charge us additional stuff for Tableau. So our clients are now in a situation where they might abandon the product due to licensing issues, making it complex. People don't want to go through this. Really, it's easier to use the tool, but licensing itself takes a lot of unwanted complications.

So, with NeoLoad, we gave feedback and asked them to simplify the licensing model, and they came back with a better proposal.

If it weren't for the limitations in supporting Oracle forms and SAP GUI, I would never choose any other tool over JMeter. As for SAP GUI, we shouldn't fix anything unless it's broken, so I prefer sticking with JMeter. These commercial load test tools are insanely priced, especially the load test tools. 

That's why I appreciate that NeoLoad has come down with better, smaller licenses now. It's a good improvement. But I still think they are overpriced. This applies to all these tools, not just NeoLoad. If you look at them, they sometimes charge for virtual user hours, which I don't understand. When you buy a tool and a certain number of virtual users, you shouldn't be charged for hours of virtual use. Instead, you should be charged based on the number of virtual users, and the clients should be able to use it for the period they purchased the license.

Overall, I would rate the solution an eight out of ten. There are certain areas that need improvements in NeoLoad, like UI, correlation templates, and more.

We use Tricentis NeoLoad to test risk with API calls and make some small automation with the API calls using small Java scripts to take data from the file.

The solution is on-premises and we use a web dashboard which is SaaS but we use on-premise because we mostly work with the clients in its sensitive data that's required, like, only on-premise usage.

The solution is on-premises and we use a web dashboard which is Software as a Service. We prefer on-premise usage because we mostly work with clients dealing with sensitive data that requires on-premise usage.

The scalability, support protocols, and features of Tricentis NeoLoad made it easy and straightforward for us to complete the same tasks for our customers in the server cloud. We had to verify the certificate and attach the file within a protocol, which was a challenge in the server cloud.

From a functional perspective, the range of tools provided with Tricentis NeoLoad is perhaps the widest.

There is room for improvement with the support and community documentation as it can be difficult to find answers to questions quickly. 

I have been using the solution for six months.

The solution is stable. In the past six months, I have only encountered a few issues or bugs when the program crashed. I believe this may have been due to my own configuration settings.

The solution is scalable.

The technical support is not very good.

I previously used Cisco SMARTnet but the generators for reports were difficult to configure on the backend so we switched to Tricentis NeoLoad because it is more user-friendly.

The initial setup is straightforward. We deployed using the manual and did not integrate the solution with anything, such as the CID tool; it was only for small projects.

We used a 60-day trial with ten hours of work per month. The cost is increasing each year.

I give the solution an eight out of ten.

We have three people using the solution in our organization.

As long as the organization's requirements are met, I recommend Tricentis NeoLoad.