Overall, it's helped our ability to address test data challenges. The test data features on their own are very good, but version control test data isn't included yet. I think that's an area for improvement.

We can update the test data on the cloud. That's a good feature. There's also test data management, which is good. [Runscope] doesn't have the test data management yet. Mock services do, and performance testing has it. We can do the same test through JMeter, validating the same criteria, but the feedback from [Runscope] is quite visible. We can see the request and the response, what data comes back, and add the validation criteria. We can manage the test environments and test data, but running the same API request for multiple test data is missing. We cloned the test cases multiple times to run it. They need to work on that.

Version controlling of the test cases and the information, the ability to compare the current version and the previous version within [Runscope] would be really nice. The history shows who made the changes, but it doesn't compare the changes.

In the future, I would like to see integrations with GitLab and external Git reports so we could have some sort of version control outside as well. There is no current mechanism for that. The ability to have direct imports of spoken API specifications instead of converting them to JSON would be nice. There are some features they could work on.

The seamless integration with mobiles could be improved. Right now, they have the UI testing capability, which provides the browsers. They made Safari available, which is amazing. We're able to test on Chrome, Firefox, and Safari. We want the capability to test Chrome on Windows, Chrome on Mac OS, and the capability to test Chrome on Android OS and iOS.

BlazeMeter needs more granular access control. Currently, BlazeMeter controls everything at a workspace level, so a user can view or modify anything inside that workspace depending on their role. It would be nice if there was a more granular control where you could say, "This person can only do A, B, and C," or, "This user only has access to functional testing. This user only has access to mock services." That feature set doesn't currently exist.

An area for improvement could be enhancing BlazeMeter's integration with automation scripts. 

It would be beneficial if BlazeMeter could support automation frameworks more effectively, including the use of Selenium scripts for both manual and automated load testing.

Integration is one of the things lacking in BlazeMeter compared to some newer options. A lot of products are coming out, and BlazeMeter pricing is a factor. 

For example, LoadStorm by Neustar is integrated with built-in APMs. It won't capture all server stats, but it will collect the minimum important aspects – CPU consumption, utilization rate, and how much a single server is being stressed. If BlazeMeter offered similar functionality, it would be fantastic.

The performance could be better. When reviewing finished cases, it sometimes takes a while for BlazeMeter to load. That has improved recently, but it's still a problem with unusually large test cases. The same goes for editing test cases. When editing test cases, it starts to take a long time to open those action groups. 

For a new user of BlazeMeter, it might be difficult to understand it from a programming perspective. BlazeMeter should provide its users with a seamless experience in the area of programming. The tool should be made in such a way that in whatever scenario a need arises for it, new users should be able to use it without difficulty. It will be better if BlazeMeter can handle call scenarios using behavior-driven development, allowing technical and non-technical people to understand the tool.

The technical support team's turnaround time or response time is high, making it one of the product's shortcomings that requires improvement.

Our biggest challenge is the skill set required to operate the solution because we used to have a centralized performance testing team. Now we've opened it up to other teams; some needed to onboard new resources. The solution is simple and user-friendly, but we still need the right staff to use it.

We encountered some minor bugs, and I would like to have the ability to add load generators to workspaces without having to use APIs. We can't do that now, so we're beholden to the APIs.

The only downside of BlazeMeter is that it is a bit expensive.

The tool fails to offer better parameterization to allow it to run the same script across different environments, making it a feature that needs a little improvement. The tool should offer some ease of use across environments.

The solution's scalability is an area of concern where improvements are required.

BlazeMeter is a very handy tool requiring drag and drop to operate., but I don't think I can generate a JMX file unless I run JMeter, which is one of my concerns when it comes to BlazeMeter. In our company, we are mostly unable to capture logs or events with BlazeMeter. We want BlazeMeter to assimilate a mobile app, especially sincere company deals in mobile apps, and we wish to conduct testing using BlazeMeter. The solution has been good so far, but JMeter is one area that has been tricky for me since I cannot generate events.

I cannot speak about a particular weakness in the tool, but it is a tricky product since those who want to use it need to depend on another tool called JMeter. JMeter is required to get the scripts and JMX file before being able to run on BlazeMeter.

In our company, an APK is generated whenever we develop mobile apps, and when I drag and drop it as a script, a JMX file should be generated, which is a feature not included in the solution. The aforementioned area where the solution lacks can be considered for improvement.

BlazeMeter has room for improvement in terms of its integration with GitLab, particularly in the context of CI/CD processes. While it has multiple integrations available, the level of integration with GitLab may need further enhancements. It is known to work well with Git and Jenkins, although the extent of compatibility with GitLab is uncertain.

We sometimes experience downtime, but not so frequently.

Potential areas for improvement could include pricing, configuration, setup, and addressing certain limitations. Enhancements in data import/export and integration with other tools could be beneficial. Additionally, providing support for certain tools like Grafana, which some competitors offer, would be a good extension to consider.

Whenever we use BlazeMeter for the ramp-up and designing the scenarios in our company, we also use JMeter or other load testing tools, which provide some convenience in areas where the granularity can be maintained in seconds. The ramp-up and ramp-down require our company to use the granularity for a few minutes, making it an area where improvements are required to be able to use the granularity in seconds. From a performance perspective, BlazeMeter needs to be improved.

Whenever we discuss the development stage, JMeter has plug-ins and other extensions in the area of WebSockets, and it is the same case in terms of the kind of extensions provided by JMeter that are available in LoadRunner. BlazeMeter has not found the extensions for WebSockets or Java Applet. Decoding the scripts that contain the applications with Java Applet is not possible with BlazeMeter or even with JMeter, and it includes some Oracle and desktop applications, too.

The Timeline Report panel has no customization options. One feature that I missed was not having a time filter, which I had in ELK. For example, there are only filter requests for a time of less than 5 seconds.

There is a tab in Blazemeter named Request Stats Report where improvements are required. If I have to go and find a particular timeline, which might be in a 15 or 20-minute window, there are no places or options where I can enter the perfect or exact timelines I want to find. You have the pointers to drag from one place to another, but that doesn't give me much freedom to get or find a timeline. Basically, on a tab in Blazemeter named Request Stats Report, there should be the start time and end time. In BlazeMeter's Request Stats Report, users should have an option where they can select and manually enter the test start and test end time to get the stats for a particular time period.

Scalability is an area of concern in BlazeMeter, where improvements are required.

I believe that data management and test server virtualization are things that Perforce is working on, or should be working on.

If the solution had better support and the documentation was efficient it would do better in the market.

One problem, while we are executing a test, is that it will take some time to download data. Let's say I'm performance testing with a high-end load configuration. It takes a minimum of three minutes or so to start the test itself. That's the bad part of the performance testing.

I don't think they can reduce that time because that's the functionality they have implemented in our BlazeMeter performance testing. But it's a pain point whenever we are running performance testing in a call or a demo, as well as in our live testing when all the business people are there.

The first time I run a given test, if it takes three minutes to download onto my server that's understandable. But every time I rerun the same test, it is downloaded again, because once the test is completed the files that were downloaded are removed. That means I have to wait for three to four minutes again.

We also had a call last week regarding secret keys. In JMX we have some Backend Listeners, such as Kibana, and there are usernames and passwords for them that we have to manually enter. When we upload the JMX file into BlazeMeter for performance testing, the usernames and passwords are viewable. Anyone who has access to BlazeMeter can download the JMX file and the usernames and passwords are visible to all those people. That's an issue with the performance testing.

Also, all the competitors have MQ protocol support, which is lacking in BlazeMeter's Mock Services. Having MQ protocol support in the Mock Services would be great for us. JDBC, the database communication, is also lacking. If we had those things, we would be completely satisfied with BlazeMeter's Mock Services. 

And for the API monitoring, we are missing a data-driven approach. If, for a single API call, we have 50 to 100 test cases, there should be no need for us to create multiple steps or to duplicate the test steps. Instead, if we had a data-driven approach available, we could directly add the test data into an Excel sheet and call it into the single test steps and achieve what we need to do. We have raised this concern to the Perforce team as well, and they said they are working on it.

The scanning capability needs improvement. 

Integration with APM tools is an area where the product has certain shortcomings and needs improvement. Integration with APM tools like Dynatrace or AppDynamics needs to be improved.

The product currently doesn't allow users to run parallel thread groups, making it an area that should be considered for improvement.

We have already provided some sort of feedback for our BlazeMeter vendor who is directly interacting with us. We would like, for example, for there to be some sort of grouping features available. 

The should be some visibility into load testing. I'd like to capture items via snapshots. 

While they are in the cloud, it would be good to also offer on-premises options. 

Reporting.

Documentation for the solution could be improved because there are some areas, such as licensing costs, where there is a lack of information regarding structure. I'd also like the ability to see a comparison feature after carrying out several tests. I'd like to know the difference in terms of response times and other details. That would be a great feature for them to provide. 

Sometimes we'd like to include additional users during a test run to check application sustainability. We can do it from a scripting end but it would be great if BlazeMeter would provide the option of adding a few more users while running a test.

At the moment, it's heavily based on putting a load on systems through API endpoints/HTTP endpoints. There is a lot of existing tooling out there in organizations that deal with a Selenium-based test and they only have a Selenium web driver option right now, which is not the nicest thing to use. Organizations currently have Selenium tests written and, like us for example, a huge suite in Java. It would be nice to be able to just throw them in there.

It would be very useful if there was a mechanism to schedule performance runs.

For example, a mechanism to automatically schedule performance tests execution, by using the calendar scheduler. This way, it would be possible to schedule multiple runs overnight, without the need to manually start each performance test.

I would like to have better insights when tests fail to run properly or when the load engines occasionally break. There have been times when a test fails to start, sometimes due to a bug in my script and other times due to a problem on BlazeMeter’s side, that have required contacting support. It would be better if the interface proved enough feedback via logs, etc., for me to determine the cause and possibly fix it without needing to wait for BlazeMeter support.

It's a rather expensive tool for running just a few tests — which is our primary use case, if there was a pricing structure that met our use case price wouldn’t be of concern. Worth noting is that we requested they donate our use of the tool — and they did donate the tool for our use — so it did not cost us anything (which we are very thankful for as a nonprofit).

I cannot recall coming across any missing features.

The reporting capabilities could be improved.

It would be ideal if it could incorporate tools such as APM or Dynatrace.

My only complaint is about the technical support, see further details below regarding customer support.

It would improve the product if their Chrome extension allowed you to modify the JMeter settings.

There was one feature that I wanted that they didn't have. The feature was starting two tests simultaneously, but keeping the data separate, and showing separate graphs from two different test instances on the same display.

What happened is that I was trying to run two tests at the same time so I would have to do one in one window, one in the other window, and there was a few seconds difference, which was okay. Then, I would have to take the tests in one browser, and compare them to the tests in another browser.

If they could combine functionality and put them side by side with the same scale for me, that would've been ideal. I was using one standard testing configuration.

Having more options for customization would be helpful.

A possible improvement could be the integration with APM tools. Different plugins for the most common APM tools could help to reconcile traffic charges with application behavior. For example, a root cause analysis of bottlenecks or analysis of non-linear behavior in applications.

In terms of improvement, I would like it to have the ability to customize reports. 

More runs per month for the basic plan would be useful. Also, being able to stream logs from AWS and integrate them with the test reports would be excellent.

The ability to run node.js scripts needs to improve.

Running of multi-tests could be improved to make it easier to create them. Apart from that nothing else springs to mind.

I have used BlazeMeter personally and my experience has been:

Whenever we want to provision more load generators (10 or more), it takes too long to initiate the test. Sometimes, the load generators have not even started and after waiting for 10 minutes or more, we get to know that we need to restart. This creates an awkward situation, when the client is also monitoring live execution and waiting for the test to start.

I have come across multiple instances, where the actual test was run for x hours but the server hours that were deducted from my account, were on the basis of x+1 hours. I have to keep monitoring the usage report frequently and request for credit back of all such server hours. Although, I get them back without any argument with the BlazeMeter team but still, it creates an overhead for me.

It needs to improve the server hour calculation and the provisioning of load generation machines.

There are only certain features that actually work. Their URL testing and JMeter tests work only if testing URLs. If testing anything else, like WebDriver tests using JMeter and BlazeMeter, they do not work.