Authored by Sabhari Murugan and Ujwal Unni from Aspire Systems Testing Community.
Overview
Software Quality is dependent on Test Automation for its ability to provide vast coverage with least possible resource. Having said that, automation has its own overhead and maintenance which requires additional effort. To reduce such additional effort, besides ensuring stability, robustness and increase the ROI - there are certain considerations to be focussed at different stages of the project. Here is an insight to such areas of considerations and how it can be dealt with ‘TestComplete’ – a potential test automation tool.
What is TestComplete?
Started as a tool that is preferred for Test Automation of Desktop applications, today TestComplete is one of the top competitors and a noticeable performer in the commercial market. The recent versions of the tool support Desktop, Web and mobile applications to a great extent. The tool gained its popularity due to its
- Robust and stable execution engine
- Rich in-built libraries
- Support for multiple Scripting Language
- Minimal knowledge required for beginners
- Access to System level operations like system restart, Process management and monitoring
- Cost effective, when compared with similar tools in the market
- Events (Decide what to do at - Beginning of Test, End of Test, Error encountered, etc...)
Doing it right:
Test Automation is not an area with constant or limited needs and expectations. New technologies, frequent changes, emerging software systems and new feature additions create constant challenges for tools and testers. There are certain considerations in each phase of Test Automation solution to build a suite that could provide high ROI, with least effort and maintenance. Here is a brief about some of these challenges / requirements that are more likely in Test Automation solutions that use TestComplete.
1. Frequent Changes
“Change is good” – in many ways, for instance it makes way for new features and enhances user experience. Besides, who likes to use software that has no updates? For any product to be successful, it should have the ability to change and adapt to the cutting edge technology.
Implementation stage: Framework design.
Solution:
1. Name Mapping – an object repository used to store the instance of controls. By doing so any change in the Properties of controls such as size, colour or caption can be isolated leaving the automation scripts unaffected.
2. Modularity Framework- a Test Automation framework in which all the common functions such as entering text in text box, selecting items in a combo box, clicking buttons are developed as reusable functions and used in Test Case Script
Doing so, the Test Case Script will not have any code that directly interacts with the product, the controls are referred through Name Mapping and actions are performed through the re-usable functions. Hence Test Case Script is isolated and no change will be required unless there is a change in the workflow of the product.
2. Testing Multi-Language Products
When testing applications that are released in different language versions it is crucial to test all the language versions to ensure quality.
Implementation stage: Framework design.
Solution:
1. Using an XML or ini file. The file contains a common ID for each caption and their equivalent value in the respective language. For example, below is a representation of the xml file content for the caption of Ok button in three languages.
<caption>
<id=’YesButtonCaption’>
<English=’Yes’>
<German=’Ja’>
<Finnish=’Kylla’>
</caption>
In the Test Case Script or in the reusable functions instead of directly using the caption, use a re-usable function that fetches the appropriate language text from the XML when the id and the language are passed as parameters. For example: getCaption(“YesButtonCaption”, “German”)
3. Different Levels of Execution
Test Suites may vary from less than hundred cases to more than 20,000 cases. It is not always the case that the entire test suite has to be executed; hence cases should be grouped based on features or workflows. For example automation suite of a banking application can be grouped based on features like money transfer, account creation, payments and so on to enable execution at three levels – Case level, Selected Group level and suite level.
Implementation stage: Framework design.
Solution:
1. Driver Script – Instead of matching each test Item to a test case, group them based on features and create a driver script for each feature. A driver script is just a function that calls all the test cases under the respective feature one after the other.
2. Further enhancement can be done by using XLS file to have list of Cases and a column with Yes or No values that determine if the particular case has to be executed or not.
4. Windows event log monitoring
There are scenarios where the application may seem to work fine, even when there are some issues while updating data in the database or storing data to a file. Monitoring the Windows Event log gives a better insight into such situations and helps to identify them earlier.
Implementation stage: At the time of requirement.
Solution:
1. TestComplete provides full read access to Windows event log, leveraging this feature at suite level can capture all the events logged related to the application under test during execution.
2. TestComplete also provides access to real-time resource usage of any process such as CPU usage and Memory usage, logging these details will help identifying memory leakages.
5. Memory issues due to large suites
Scalability is one essential aspect to be considered in Test Automation. TestComplete is designed to handle any number of Test Cases which means almost any Lines of Code (LOC). But having too many script units will have a huge memory foot print which accumulates over execution, which eventually leads to tool crash or other performance issues.
Implementation stage: At the time of requirement.
Solution:
1. If there are way too many Test Scripts in an Automation suite, the suite can be split into multiple smaller suites. All the smaller suites are independent of each other. In this case, when a whole suite execution is required Batch Files can be used to continue the execution from one suite to another.
6. Continuous Long-time Execution
Having around a few thousand Test Cases in a Test Suite means executions can go on continuously for days. This makes the system vulnerable for performance and memory issues.
Implementation stage: At the time of requirement.
Solution:
1. The application under test can be closed and restarted in certain intervals to free memory and have better performance
2. It is not just the application but also the tool that is in running state for too long, hence it makes a difference if the tool is restarted in some required intervals approximately once a day. This can be made possible by two ways- TestComplete’s in-built function “RebootandContinue”,which restarts the system and continues further execution or using batch file to close TestComplete after sometime into execution, let the system be idle for a while, then start TestComplete and continue the execution.
7. Automatic Report generation and mail notifications
When execution continues for days together it is tedious to monitor the progress since the logs are available only in the tool, this means either the execution has to be paused or we will have to wait till it completes to view the report or log
Implementation stage: At the time of requirement.
Solution:
1. The in-built function to Save Logs can be used at regular intervals to automatically export logs to the specified location as and when required.
2. TestComplete also supports automatic mail notifications with the logs as attachments. Once configured results will be mailed automatically as per the configuration
8. Scheduling Tests in Test Execute
SmartBear has an execution tool that is relatively less expensive than TestComplete that can be used only for placing executions. Unfortunately, TestExecute does not enable us to select the Tests that has to be executed. So one way or another TestComplete is required to modify the Tests to be executed.
Implementation stage: At the time of requirement.
Solution:
1. The .PJS and .MDS files in a Test Suite are the two XML files that stores the data about the project and Test items to be executed. Developing a simple tool to read and edit these XML files will completely eradicate the need of TestComplete in placing Executions.
Other Best Practices:
- Always ensure each test script works independently.
- Avoid test data dependencies between test scripts.
- Test data is always externalized.
- Write the test scripts from scratch instead of using record and playback.
- Always ensured test automation scripts sequence are configurable for execution.
-
Frequent catch-up with Dev & Design team to understand the impact on automation scripts due to upcoming releases / changes.
- Tool & Environment upgrades should be taken up as early as possible to avoid surprises at neck of the moment.
A test automation project and all its value additions and work arounds are hosted on a Test Automation framework. Automation Framework is the basic building block of a robust automation suite and framework design is the most important phase of an automation project. This is the phase where all the above considerations should be predicted in order to design a framework that meets all the requirements of the project and accommodate further improvements and new requirements that may arise over the course of the project. Such Anticipation, preparation and worked arounds are the way to ’Doing it right, with TestComplete’.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
My experience with the web support in TestComplete matches yours. As such I have implemented all of our web-based automation using Selenium. I am currently using TestComplete for our desktop applications only.