Because I have not compared it with other toolsets, I don't have a lot of insight. The only area that I can think of at the moment is the building of automated APA test cases. Based on your services, this is an area that requires improvement.

I would like to have more flexibility towards the mainframe virtualization and also in JDBC virtualization.

At the point of next release, we would really like to see the issues being resolved which we are facing right now with dynamic applications. Now, most of the applications in the market, in the industry, and most of the companies are rolling out responsive apps. If you can get rid of that plugin interface, you can get straight into the application itself, that would really work for us.

The cost is an area that needs improvement. There are a couple of other tools which provide support for performance testing with the base version itself, but Broadcom needs a separate component to support virtualization for performance testing. This is a costly component.

From a reporting perspective I think we would like to have a more user-friendly approach. I think that's primarily where we would like to see some improvement. Otherwise we are pretty happy with what we have.

This is what I have spoken to the CA people as well, I would like them to include mainframe testing within this tool.

Right now, we can do service virtualization but there is no mainframe testing feature in this tool such as batch processing, etc.; it is the feature that I need.

I think the ones we've wanted have come through, actually. Because ours was such an old version, and we said at the time, "We'd like to see x, y, and zed". They're already in 9, and what I've seen of version 10, it's even more closely bundled with some of the other offerings as well. From what we saw, the biggest challenge we had at the time was that it didn't link in well with the repository, but it's now there. It's now got the open interfaces to get hold of the other stuff that it works closely with. Once we get the upgrade nailed, we'll probably get more benefit out of it.

I think for future versions, CA need to help customers get to an app store mentality where, when it comes to upgrades, I shouldn't have to go through a whole loop again every time I want to upgrade. If I tick a box, push it out to me. That's what I'd like to see. I'm sure they're probably working on that in some way, somewhere, but that would help me as a customer. For small customers, it's probably a lot easier if you've only got a few servers to upgrade. I've got 35 different servers for SV. I've got 800 people who downloaded it and are using it. It's a big user base, so to actually plan and implement the upgrade is a mammoth task in itself. Anything that would help me do that better next time would be a big positive for me.

So far nothing, it has everything I've needed. Totally everything.

However, it's a pretty heavyweight tool for my personal needs. I like to do stuff on my own and would normally prefer to use my own solution for service virtualization, rather than such a heavyweight, extensive product. In half an afternoon, I could make a few scripts to meet all the needs myself.

There were things early where we couldn’t do a few things in Service Virtualization that have since been updated. The concern I have right now with Release Automation is the concept of what we call immutable objects, so I build it once in development, and then I move to all of my other environments.

My challenge is looking at this from a perspective of “I'm kind of looking at … do I look at containers, do I look at what the RA type of product provides?” I also have to consider being cloud ready. As I deploy into an environment, I want to make sure I have enough of a stable and a repeatable environment type of model where once it comes out of development, I know that everything is staying the same.

There’s different ways we can slice it, and depending on whether you do containers or a combination. That's some of the things I'm trying to understand more with Release Automation right now. I've been pretty happy with the product, I'm sure there are always things that we could ask for, but it's really done a nice job for us.

It's interesting that it still takes teams a little bit to wrap their heads around. They're so used to kind of stubbing. I feel like the product has been able to scale with us, I think more of my challenges are how best to handle the processes. For example, who should be responsible for virtualizing services? I think that generally, the best thing would be for the middleware folks who are building a new service. They should also be responsible for creating the template virtualized service, and you build it right up front so people can start using it. I think there's a part of it where people could then also modify and update data, so there could be other types of responses for their scenarios, and it doesn't have to be done by the middleware folks. We went with a model of a center of excellence where they know the key, and then we start trying to educate and train other teams to build out more. Initially, we were not to that spot with the middleware teams for a variety of reasons, so we had to rely on the individual teams to build the services.

There is what to be careful about - let's just say I have a customer lookup call, I really don't want to have to end up in a situation where two different UI teams who need to use the same customer lookup call did their own virtualizations of the same call. Those are some challenges for us that we've got to figure out. That's more procedural than it is limitations of the tool.

I'd like to see more of the newer technologies included in there, looking mainly from a mobile perspective, possibly, so you can virtualize some of the aspects that we're going to be doing for mobile testing.

Regarding additional features for future releases, that is something we'll come to know when we start using it at full scale. But so far, we're pretty content with what has been offered.

In terms of improvements, I think a little bit more of use cases from existing clients. That could show that it's a workable solution used with other clients as well.

There is need to improve the DevTest Workstation with UI-based testing features.

We still have mainframe systems and while CA provides instructions on how to integrate Linux-based applications, they don’t provide integration steps for mainframe.

We would like to see improvements in mobile integration. Integration is partial, but we would like to have full-fledged functionality. Android mobile solutions are out there, so you can download an app and have full-fledged mobile coverage. You can set the model and Android operating system. But we can’t integrate these with development testing. If we get that feature, we don't need to look at other products like HPE Mobile Center, or something else. Having mobile would be good for us.

We are also looking for more knowledge to share with our developers, and others, on how to use Selenium scripts within CA Service Virtualization. The process is pretty simple. You build everything in Selenium, convert the script to JSON, and dump it to Service Virtualization; alternatively, copy the script and put it in CA Service Virtualization steps. But these guidelines are not clear enough. Clarifying this process would definitely be a benefit for CA.

It is complex technology which takes a high end skillset to set up. You are very dependent on consultants from CA or a small subset of CA partners to implement CA Service Virtualization in the beginning. The training offered by CA is also not up to par. It is not realistic to expect that you can purchase CA Service Virtualization, send a couple of folks to CA training, and get yourself up an running with CA Service Virtualization environments without outside help.

It would be great to have improved logging for trace requests and responses at the virtualized service level. The existing feature needs improving.

I would like to see more integration support with Docker for release automation, a few things on service monitoring that's pretty much provided in the prior version of the product. I would like to see that a little bit more usable in the future.

Currently we are in the early cycle. It meets our current needs. I see where there are things we potentially may need, but nothing in the near future.

There is always room for improvement. Microservices are coming, and I don't know how microservices are going to be virtualized.

They still have a long way to go because service virtualization is a new area and there is a lot happening in this world. They have to keep building on their product and services for us to get it in the future. They have to adapt in more areas.

They already know what we are looking for. We asked them to support the MASL. They worked it out and then they provided us support. Other areas they can improve on are the scalability, failover capabilities, so that you don't have to depend upon a single instance of the server.

We want to see more automation; an automated way of creating service virtualization without using the tool. That's the feature that we would like to see in the next release.

This product provides an interface with current test management tools and that’s what we like about it.

However, we would like to see more interfaces with all open source platforms, especially in the CI/CD framework such as Jenkins, TFS tools.

I wouldn't say that there's a tremendous amount that could use improvement. I think that with applications like this, it's easy to use it for the wrong reasons. Early on, I think we had ideas about leveraging it in places where it wasn't a good fit for us, where it wasn't going to have a large impact. We had to have a bit of guidance from CA as to where we should be focusing. After that, it wasn't as much of a problem. There are some interactions with customers, depending on what they use. Where maybe we were thinking about doing something that was more internal, but didn't make a tremendous amount of sense for us. There was a case where we tried to virtualize too much. There are limits to what can be done in that kind of testing. It's easy to just take the tool and go virtualize something that you really didn't need to virtualize.

It does require a higher level of support. This is not the kind of tool that you can just haul off just go make decisions on. You need some understanding as to what the right move is and I think CA's the only group that's going to be able to help you with that.

The learning curve is an area that needs improvement, as well as stability and performance.

Stability has to be improved. It is not a stable solution. Especially, when you create custom solutions there are issues with memory accumulation. I would prefer that they stop adding features, which is what they do. Instead, focus on stabilizing it and make it work. 

What's the point of adding new features? It just increases the cost.

I'll give you an example; CA and I have continuous applications, and insight is one of the service activation components. It has never worked in any project, just for the namesake and for showing they have this one. 

Support needs improvement, they don't support the migration from an older version to the current version.

The price can be reduced.

This portal, after one week, if it is enterprise-assigned, will stop working. Now we have a policy that every week, we have a scheduled restart of everything, and maintain it. The stability is so poor that this is what is required. 

Broadcom acquired it recently, approximately a year ago. Broadcom was not involved with the software before. They were known for having a history with the hardware, physical devices, and appliances. 

Now, they are focusing on both. Much of their technical expertise, have gone. There is no expertise.

As far as the next version of it is concerned, probably some additional lightweight, portable elements. For instance, we would like to not have to rely on deploying things to a server, be able to carry it and make it more portable (maybe through the mobile devices), etc. Some of the mobility aspects are what we are looking for, and I do not mean running it on a mobile device when I say mobility, but being able to port it between workstations, and being able to take it and work with it at a smaller scale.

The one main thing we want to see is more on the reporting side. That's the one thing that we're missing. We need more detailed reporting. In particular with some of the integration, we want to see more detailed reports so that we can drill down to more scenarios.

They still need to improve the reporting functionality for better customization. Currently the report functionality does not allow us to go beyond 5000 records where the display show all the records also more filters needs to provide to generate/customize test reports.

We had to implement an external service catalog. We put it in ServiceNow. I would like to see an integrated service catalog. 

See below.

I would like to see better GUI testing possibilities.

It only supports web testing by working with Selenium: DevTest is positioned as a test automation tool. People want to use it as that, but then they find out it needs Selenium with all the downsides of that. We also have a couple of client applications they can’t be tested at all with DevTest/Selenium.

Beyond that, you need a lot of ports to be opened to enable working with it. That introduces a lot of problems trying to use it, especially the database connection from the client. A solution would be to add all of the functionality to the webclient.

The full client needs a database connection. That is a problem for us, because that is not allowed anymore.

I think some of the monitoring capabilities could be enhanced. Those aren't necessarily major gaps for us, but I think that would be good. Just some of the real time monitoring and reporting capabilities. Some of the solutions that we've had to create weren't necessarily out of the box. They were successful and again, CA supported them. Some of the reporting capabilities would be good. Additionally, when you're starting to identify your value ads, some of that reporting would help in identifying and calculating your savings.

The product is very robust and is obviously one of the front runners in the service virtualization space. Competitors have more user friendly user interfaces and some have front end systems which are easy to toggle and manipulate.

I have said many times that CA LISA is more of a technical programmer's tool, whereas their competitors are appealing both to the programmers as well as the basic user/business user/tester.

CA Service Virtualization has its pros and cons. It's a strong framework and the feedback is good, however when it comes to acting as a test tool some improvements are required.

UI should be more user friendly: better usability, more testing oriented. This product emerged from the services virtualization domain and evolved to include application testing. They are very good in service virtualization, but in application testing they need to make some improvements. One of these improvements is to be more test oriented, which will allow the user to manage and control the test execution better and to organize the packages in a better, more user-friendly way. The users should be able to organize the test, test suites, and execution in a better way, such that scenarios can be integrated into the testing part. Currently its outside.

The application test should be more feature rich. I would also like to see seamless integration with DevOps. Currently it needs some scripting to integrate with DevOps.

We had a complex initial setup in our environment, but CA helped us with it.

I don't think we've gotten enough experience with this version to talk about what should be in the next version.

It could be easier to setup and to deploy.

The things that we're interested mostly are deployment kinds of capability. We would like to deploy these simulators as part of our continuous deployment process. We would like to be able to auto-deploy them as part of the test mechanism to help test teams without having to set up the infrastructure in advance.

• UI testing could be improved.
• Also, it should have predefined plug-ins to integrate with other automation tools to help in tool-chain enablement.

The user interface and usability of the tool is not the best. They could definitely be improved upon in the next release.

Also, we are hamstrung by different releases and are not using all of the features, but it is useful.

The problem is getting the developers to do more than just hooking into QA backends. They don't do integrated testing so that the first time you bump into an integrative problem is when it hits our QA environment, which are integrated. They discover a defect later on down the cycle and it's more costly to fix. The developers are not finding their own bugs and fixing them.

This tool would allow them to do simulated integration tests without having all this backend infrastructure. They can run on it so we build the services for them, but they're not using it for that capability because it hasn't been presented it to them in that way.

So really the biggest area of improvement is just in terms of who CA should be focusing the tool's use. It would be best to focus on the developers.

The workstation component has a very out-dated UI and is in dire need of a facelift.

There are too many fragmented web components that could easily and logically be merged. For example, Service Catalog, Enterprise Dashboard, Portal, and Identity and Access Manager are all separate web applications. Why not merge them in one web app?

I think the size of DevTest is pretty massive. It's hard to tell what different parts of it can be used to do different things. It gets kind of big, so if they could modulize it more, and lead you to a common work flow, that would be pretty awesome.

They're constantly making improvements, so it's gotten better every time they have released a new version.

My fallback would be scalability.

Oddly, when my capacity folks pull stats memory and CPU both show within normal limits. My users, app administrator, however, see degradation during peak periods.

We're working with our capacity management engineers and our CA SV specialist to analyze the performance stats and recommend a temporary solutions. I'm planning to upgrade to v10 in 2017, so I'm going to insure we have h/w that supports current needs and v10 +24 months.

We would like to see more robust dependency management. This is the only thing that comes to my mind.

The GUI can be hard to navigate. The ITR keeps pulling the screen to the front. The speed can be improved as well.

Integration with Selenium and GUI testing needs improvement. Supporting more new technologies and existing technologies such as Flex, Salesforce, etc. would also be nice.

VSEasy is a nice feature but it needs to have some catches added to it to keep from inadvertently overwriting someone else's virtual service.

To the best of my knowledge (and memory), listed below are the few challenges we were facing at that time. It could be possible that CA would have already addressed a few of these limitations and short falls.

• Simultaneous execution of automation scripts was not possible; or, at least, not readily available. We had to write some wrappers to handle this situation.
• Integration with Jenkins was not available. We had to write small utilities to achieve this.
• Integration with JIRA was not available to automatically log a defect if there were script failures.
• It consumes more memory and makes the system very slow, which was stopping simultaneous execution.

There should be an option to search a transaction that is VSI based through a transaction number. It would really be a great addition.

Even though there is no comparison with its peers and all the necessary things are right in hand, there is room for improvement with the Virtualizing Mainframe components. The concept of agent installation is a tedious process and should have been much simpler.

The most important thing is to have integration with Excel. It has a way to read the Excel sheet but not to write onto it. If I need to generate customized reports in LISA in Excel sheets for management purposes (which can show me number of test cases pass/fail/not started, etc.), I can't do this since you can't ask Service Virtualization to put specific values in particular cells with a particular color, etc. We were able to achieve it only using Java, which meant a lot of scripting, error debugging and rework, and this defeats the purpose of it offering a codeless environment to testers. A simple groovy scripting in SoapUI can achieve all of the above.

The Service Virtualization section needs to be more intelligent in offering a good solution to customers. For example, the virtual service created via the request response pairs needs spoon feeding of responses before the service starts performing. And I am not sure if it serves the purpose of virtual environment till the actual system becomes available. I have yet to see a proper implementation and successful usage of a virtual service through it.

CA should provide a free trial version of the tool (was not available till version 7.0.10) so that clients can try it out before making a purchase.

The biggest area of improvement is in the licensing. The licensing costs are very high.

For our current needs, it's doing what we need right now, although it should automatically generate tests to further make our development to implementation process more efficient.

I feel a few areas can be improved, but I'm not sure if CA has already fixed these in the latest version:

1. Agents for different systems. I am not sure if existing agents can support all types of servers or systems.

2. Along with application details, can we capture network details to simulate network behavior?

3. Can we separate utility to monitor the CAI details rather than the portal, because Portal is accessible to all.

Not all protocols are supported, but I've heard there will be a feature in version 8 with the help of which we can create models for those scenarios as well (surely a few limitations will exist).

I would like to see improvements in the naming conventions. The current naming conventions of CA Service Virtualization are hard to use.

Performance and cost.

With Service Virtualization being a newer concept and market, there is a lack of high quality/experienced resources available across the board for any of the tools (as of mid 2013)...with that said, a good developer can come up to speed fairly quickly with this tool.

One major feature I would like to see is on the user administration part. Right now, anybody can access any of the folders and any of the projects. If they could bring in someone amount of user administration, that would help us a lot.

It's currently very sensitive to app changes/responses & the setup to capture the responses is a bit cumbersome .

I think the biggest improvement can come in usability. It's still very complex to use. Also, we've found a fair number of bugs, but CA always resolves them.

I would like to see a completely web-oriented automated test case design (currently need to use desktop application called DevTest Workstation). I'd also like to see IntelliSense-like functionality for custom scripting, as well as improvements to reports for test cases using datasets to quickly point to/locate results for specific data iteration.

The features that I think we need weren't in 7.5.2 and the 8s. They're all going to be in version 9. These are some reporting and scalability features, as well an expanded virtual service in version 9.

The automated generation of tests could be improved. Right now, we have to generate them all ourselves. We want to be able to run that against the service and have it creates our tests.

I think the one thing that we need that it doesn't do is performance testing for UI applications. So it integrates with Selenium, but we can't use it for performance testing.

We'd like to be able to do performance testing for in-house applications with more than five to 10 users, but we can't do that with the tool.

There's a couple of things with the UI that make it a little bit hard for new people to use. It's not intuitive in some places. You have to know what you're doing. It's not something that somebody could pick up and just start using the same day without somebody explaining it a little bit.

More docs/code samples to develop custom steps: Sometimes we need to develop in web services custom behaviors that simulate technically and functionally the real service.

On the testing side of the functionality, there are a few things that are missing. For instance, a comparative report. But since it's open source, we can add the extension and we do this by ourselves.

It is difficult and requires a lot of manual effort to debug/troubleshoot any virtual service. The overall UI must improve and must be user friendly.

The tool is growing with every release. I don't have any real concerns. Also, the development team usually prioritizes requests once they are raised with them.

In addition to a workstation, the tool has a web portal, with which (without getting into a lot of technicalities) one can generate virtual services quickly. However, even though it launched more than two years back, it still seems to be in beta version. Cross-browser support is not that good for that portal. On top of that, it doesn’t provide most of the features provided by workstation.

Coming back to the DevTest workstation, there are a few areas of improvement there as well. For instance, virtual services are not supposed to be of the same type all the time. With varying nature, DevTest fails to meet expectations sometimes.

The UI is a pain point. If they had a better UI, it would have a much higher rating.

Integration of RAML files so that you can generate simulations directly from the specifications would be another improvement. For example, some guidance within the tool – if we could customize Service Virtualization, to disable functions we do not need. There are a bunch of functionalities which are part of the solution and we have to go through a number of cascading menus to find what we need. In other words, to simplify the front end.

We don’t, for example, use SAP or Oracle databases that we need to attach, and, therefore, it would be nice to disable these un-needed functions, to reduce the complexity of the GUI. Another point is that if something fails during the testing process, there is a lot of logged information that is useful because the error messages are Service Virtualization specific. It would be helpful to get more meaningful error messages. For example, a malformed JSON file, the error that is returned which is some internal error, but it’s not clear, for example, that it might be a syntax error.

Sometimes we are faced with some problems, but CA Service Virtualization is a really felxible tool. When we encounter some problems or issues we just use our own method around it and move forward.

It's not really meant for Services Testing and reporting functionality from the CA SV console - it's a very basic one. I'd like to see improvements in these two areas.

I really want to see more of the "express" kind of model, where you get a little bit for free. I'd love to be able to see you be able to edit and author tests without having to be connected to a licensed server. And then, if you want to go and execute tests, then you go and connect to the server. That's a request I've been making, and I'd like to see that. I think it's in line with the open source model, and I think it would unblock people to be able to do a lot more work from home or from remote places, where they can't really connect to the server.

1. The enterprise catalog. Usually what happens is once you have a set of virtual services, then you go to an enterprise. So, people would like to see what is already virtualized and how can they take advantage of it. Right now, it is actually not part of the product, therefore we created something, a custom solution, for it at this point of time. These are two separate things. Sometimes, we have to put some effort into it. If it could be part of the product itself, a service catalog where a user could actually come and see what is already virtualized, how they could actually utilize it, and the value it adds.
2. The access control to the new portal. There is a new portal available and we are making it as a self-service thing. Whoever wants to create their own virtual service so they can actually go there and create the virtual service. The problem we have right now is how do we restrict access to each of those virtual services or the projects. So, that is not implemented in the product yet.
3. Cloud readiness. I know it is available in AWS, but we would like to see how we can actually make it more cloud-friendly.

The following areas of the product have room for improvement: automated GUI testing, performance testing of various HTML5 applications and its recording capabilities. There are SAP, Siebel, Oracle Apps, Citrix, Remote Desktop and RTE applications that require a multi-faceted approach for sniffing the requests and preparing the scripts.

Additionally, it does not have a robust test management repository. This impacts connecting the various phases of testing life cycle.

I spoke to them at a recent conference and I think they're going to be looking at doing some more JUnit integration to be able to convert JUnit into their solutions. We're still using an older version so, hopefully upgrading to a newer version will enhance some of the features that are already out there that we don't have.

I wish the product had a built-in help function.

We don’t see enough value in virtualizing the services just because it requires a huge amount of maintenance and there is not much support available online. I’d like to see a more simple approach to virtualizing the services.

Improved Mac support would be an improvement. Also, when things fail, I’d like to see meaningful error messages. A more stable client-side application would be a big thing – their server side application is pretty solid.

I would really like more relevant online documentation. There’s not really enough of it. They should also provide some case studies so that I can see what others have done and how their experiences might help me in my situations.

UI Testing and Reporting Features.

They can improve their front end testing features.

The product is supported by custom extension coding. On the virtualization front, RIT (IBM) supports more protocols however it's not as user friendly as CA LISA. It would be great to see a new protocol which supports legacy feature.

They can always work on usability and making simple things simple to do. This is true of every product that deals with complexity.

As of today, none of the Microsoft technologies are supported.

Parsers, look and feel

Load & Performance Testing. It is not as good as other tools like HP Performance Center, SilkPerformer, etc.

HPE ALM integration could be better, especially for exporting/importing/reusing test cases written in an older version of CA Service Virtualization. The rework effort to update test cases is high.

We need to get more and more types of services because this is a multidisciplinary platform. CA can develop more services in it.

The overall installation architecture should be simplified. There are so many components involved in the overall installation and the starting process. They should be made internal, such as:

• Enterprise Dashboard
• Registry
• Portal Broker
• VSE

Instead of all these being manual and optional, they should be compulsory, internal processes.

UI testing needs improvements.

• VSE Console should be more user friendly.
• Versioning of Virtual Services as well as Projects.

The REST test step could include other methods. The currently available ones are: PUT, POST, GET and DELETE. It would be great if the DELETE method could have the Content tab enabled so the request payload can be sent.

IMS services in version 7.5.2 are quite unstable.

Also, the Javascript step doesn't support all Java methods.

Reporting needs to be improved for both manual and performance testing.

More examples of portal-based virtualization.