How has it helped my organization?
We identified an object that was supposed to have a width of 30 characters, but instead had 100,000. No manual tester would have found it, forcing developers to take a second look at all objects which uncovered similar size issues.
While my experience tells me successful automation projects are at 70% coverage of manual test cases, we have been able to hit well into the 90% range of .Net automation with this tool.
What is most valuable?
So the first impression that hits me about HP UFT 14.0 (formerly QTP) is that it seems to be a whole lot faster! But that could be subjective, as I'm running it on a high end gaming system.
And my second impression was "Oh man, why does it still do THAT?"
Let's review the good stuff:
VBscript language -Easy to learn, surprisingly powerful and extendable.
What I will call the "PDM.DLL feature" provides a list view of any object property and methods at run-time from the code as well as the two other windows.
Built in Excel Datatables for Data-driven design
Revamped beautiful HTML results report with screen and movie capture
Terminal Emulator automation.
Modular design (through functions, ALM components and Flows)
Launch through Jenkins brings CI to the test automation development team.
Can leverage Windows API calls as well as custom AutoItScript for enhanced features.
The wide range of supported current and legacy web technologies, desktop apps, and WebServices testing is by far the most valuable feature.
Even in the case where technology is only partially supported, being able to customize out-of-the-box object methods is another time saver.
For example, we recently started to investigate automation of an AngularJS application. The problem was record/playback (UFT 12.54) did not work on it. However, the Object Spy correctly adds objects to the Object Repository. In addition Descriptive Programming worked from our custom framework. We had a basic login/navigate/verify Proof of Concept test operational with AngularJS Buttons, Links and Images quickly. Minor custom coding was required to override .Set methods of WebEdit objects, and more will be needed to support it's Angular WebTable objects. Totally doable for an experienced level team or user.
What needs improvement?
There are always new features and more support for new and legacy technology architectures with each release. But the bad news is a growing list of long-standing issues with the product rarely gets addressed. While I have a larger list of issues that make day to day work harder than it needs to be, these are the Top Five that I do wish would capture someone's attention in upcoming releases. All hit the tool's ROI pretty hard.
#1) Jump To Source - The Silent Code Killer: In older QTP versions a double-click on any function in the Toolbox window would take the developer to the function's source code, while a drag from the Toolbox would add it to the code window. Since 12.0 a double-click on a function in UFT's Toolbox window now ADDS the function (same as drag) to the Code window - to whatever random location the cursor happens to be at - even if it is off screen, and it will replace sections of code if it is highlighted. We are not sure what the intention was, but our Best Practice is to avoid the Toolbox window entirely to avoid the real danger of losing days of work and needless bug hunts.
Now Jump to Source is not all bad. A right-click on any function called from a Script takes us to the code source, which is great! But it only half works: in a Library, only for functions declared within the same library. Our advance designs have well over twelve libs so a whole lot of extra time is spent searching the entire project for a function's source on a daily basis.
Lastly, while we can add custom methods to object, a Jump To Source from these methods is long overdue. So again our only option is to search the entire project.
#2) Object Spy: It needs to have multiple instances so that you can compare multiple object properties side-by-side. It lacks a Refresh button, so that automation engineers can quickly identify the property changes of visible and invisible objects.
Or HP could skip to option #3...
#3) Add RegEx integer support for .Height or .Width object properties when retrieving object collections. If this were possible, our framework could return collections that contain only visible objects that have a .height property greater that zero. (Side Note: the .Visible property has not returned a False value for us in nearly five years - a recent developer decision, not a product issue) Eliminating the need to separate the non-visible objects from visible ones would decrease execution time dramatically. (Another side note: Our experiments to RegEx integer-based .Height properties found that we could get a collection of just invisible objects. Exactly the opposite of what we needed.)
#4) The shortcut to a treasure trove of sample code in the latest release 14.0 has been inexplicably removed. This impeeds new users from having an easy time learning the tool's advanced capability. In fact the only users daring enough to go find it now will be you who is reading this review.
#5) Forced Return to Script Code. This again is a no-brainer design flaw. Let's say we run a script and throw an error somewhere deep in our function library. Hey it happens. In prior QTP versions when the Stop button would be clicked the tool would leave you right there at the point where the error occurred to fix. Now in recent releases, UFT always takes us back to the main Script, far from that code area that needed immediate attention.
What do I think about the stability of the solution?
The product is surprisingly stable. For the flaws that I mention, if stability was an issue we would not have been using it for the better part of two decades.
The only time that the product gets unstable is when you try to do wicked advanced coding. For example, when you are trying to execute dynamic code strings with the Execute command that might not have been generated correctly. My years of experience tells me that if something is going flaky, then it's the developer's fault (me) and not the tool's fault.
What do I think about the scalability of the solution?
Scalability is quick and easy implemented through a framework. Let's say we write a custom function
VerifyValue (oObj, sExpectedValue)
that can compare the expected value, string or integer, to an object's actual Value property. By adding the function as a .VerifyValue method to all WebEdit class object, the functionality is available to all current and future edit field objects regardless if they are in the repository or programmatic descriptions. And it is done with just a single line of code called RegisterUserFunc()
Now Let's say we now want to extend the verification to include value falls within an expected range? Add the code to the VerifyValue() function and all fields support the new functionality.
Scaling to new pages with new objects is not an issue either. The tool allows advanced users to design frameworks that can identify objects on the fly from "Plain English" descriptions ("OK LINK") without using the Object Repository. This may remind you of a Gherkin/Cucumber approach and the tool is certainly flexible enough to design just such a framework.
How is customer service and technical support?
Orasi's technical support is an A+.
HPE's direct technical support is okay.
Having an issue getting UFT to work with your technology stack? Some versions of QTP used to include an oft overlooked document called the Product Availability Matrix (aka the PAM) telling users exactly which Technology version worked with HP's UFT tool versions. Unfortunately, due to a inexplicable "horse-and-cart" decision, HP has chosen to remove this document from UFT 14.0 install and provide access to it only AFTER users have purchased the product. So I have to buy it to learn how to make it work with my technology stack? Wait, what?
How was the initial setup?
The install/uninstall of the updates are fast and easy. Many of the no-brainer configuration settings are set up correctly out-of-the-box. To be fair, we do have a 20-page document that walks through all the settings to check.
What's my experience with pricing, setup cost, and licensing?
Sure, HP UFT is not free. But consider what you get for that cost: A stable product that is easy to use; the kitchen sink of technology stack support; decades of code (which in many cases actually is free); a version that is a stepping stone to an easier Selenium design; and a support base that is more that just the kindness of strangers.
Want to take UFT to another level for free? Add AutoItScript calls.
Which other solutions did I evaluate?
If Selenium is your thing then make it easy on yourself with LeanFT, aka UFT Pro. This gets you the easy object recognition of UFT, in Java, plus execution of concurrent test cases in multiple browsers which is a trick standard UFT does not do.
I am always downloading and evaluating other products. SmartBear TestComplete is the next closest option.
What other advice do I have?
Take tool training by someone with years of experience.
HP, Orasi and RTTS all offer the level of training that gets you to the advanced state quickly.
And if you might be longing for the IDE toolset that Microsoft offers in Visual Studio, then look at Test Design Studio from Patterson Consulting to enhance your UFT toolbelt.
Which version of this solution are you currently using?
12.54 / 14.0