What is most valuable?
To me, the Object Browser has proven to be the most valuable feature of this product. This tab (and its associated tools) allows the test developer to locate objects within the application to be tested, and retrieve properties and methods that can be used in scripting. This saves a significant amount of time over referring to the application source code for such information. It also makes use of the recording feature periodically to rough out scripts for new tests. The recorded scripts are very fragile, but they can provide a good base for writing stable scripts.
How has it helped my organization?
While our test automation initiative is still in its early stages, TestComplete is allowing us to script and run complex business process tests against our desktop applications. Being able to create smaller modules that can be referenced by other scripts or keyword tests allows for a flexible and easy to maintain test suite.
What needs improvement?
A major part of the product that needs improvement is the licenser. This piece of the application is known to have issues installing that requires manual, command-line modifications to correct. While this doesn’t occur with all installations, it has been a frequent problem for me. The larger issue with the licenser is that it is incompatible with virtual machines. The applications themselves will run on physical or virtual machines, but they must call out to a physical server which controls the licenses for them.
The latest version of TestComplete has introduced support for Git, but my initial impression is that it still needs some work before it can replace my VCS tool.
For how long have I used the solution?
The product has been in use at the company for about two years and I have been actively using it for about 8 months.
What do I think about the stability of the solution?
I haven’t experienced issues that I would attribute directly to the application. There have been some issues with timing and other common problems, but many of those have been corrected through modifications to the tests and the environment.
How are customer service and technical support?
My experience with their technical support has been less than satisfactory, but all of my issues were resolved. Of the technicians I have worked with, two of them were very knowledgeable and one of them did little more than email knowledge base articles that had been ruled out in the original issue report. The latter tech had the additional problem of being about 12 hours out of sync so a full day was lost with every email exchange. I received prompt responses and quick resolutions from both of the knowledgeable technicians I have dealt with although one of them was rather gruff and left me feeling like I was bothering her. Overall, SmartBear needs to work on their customer service and technical support.
Which solution did I use previously and why did I switch?
This application did not replace a previous automation effort.
How was the initial setup?
When deploying new installations of TestComplete, I ran into multiple issues with the licensing engine. The first time I experienced the problem, I contacted support and was directed to a knowledge base article (which I had been unable to find while searching for the error message). I have since bookmarked the article and reference each time I need to do a new install. Once the licenser has installed properly, the configuration and licensing goes smoothly.
What about the implementation team?
The implementation was performed in-house, and was quite easy despite the issues with the licenser. The best advice I can offer would be to plan out your testing environment and requirements before you begin. If you plan to use virtual machines for your test systems, make sure you also have a physical box to run the licenser that is dedicated to the lab. Do not attempt to use someone’s PC as it may not always be available when the test machine are trying to run.
What was our ROI?
We have not performed an ROI analysis.
What's my experience with pricing, setup cost, and licensing?
SmartBear’s product licenses are reasonably priced and generally in line with similar companies and products.
Which other solutions did I evaluate?
I reviewed several products before deciding to continue our automation efforts using TestComplete. Among the products reviewed were Telerik TestStudio, Testing Anywhere, and Ranorex.
What other advice do I have?