Recently, I was asked a set of questions regarding my experience with TOSCA. Here are the questions and my answers:
Is there any need for coding or scripting? Based on my research, I expect that the module is created when scanning has been finished. But some users commented that coding or scripting is needed.
- Depends on the application and technology you are trying to automate. TOSCA has a lot of engines (HTML, .NET, Java, Webservices, etc.) with control library's that can be used out of the box. For something that is not covered by these engines (like a custom java widget) you will need to code for steering.
Are control statements like if/then/else and loops supported?
- Loops are not directly supported. If/then/else are supported for templates (data driven test cases).
Test case organization:
Is it possible to group test steps and according to our needs include/exclude particular test groups?
Is it possible to rearrange test steps once they are created?
Is it possible to perform multiple execution for test steps and test groups?
Is there a possibility to create some variables whose values can be adjusted in some of the test steps?
- Yes. In TOSCA variables are called "buffers"
Is possible to add references in a test to some other test?
- It's possible to organize test steps in the library and reuse them in TC's. For other kinds of references you are free to create custom fields (to store path to other TC for example).
Is it possible to modify verdicts, for example, if a test fails that is declared as passed?
Is a test progress view supported? Is it possible to stop/start a test on an error or some other event?
- Yes. You can define different behavior on an error (for example requesting a manual action to continue).
Is it possible to schedule tests? This is useful to have when night runs are executed
- You can trigger TOSCA execution using the TOSCA API from cmd line using windows scheduled jobs.
When some test step fails in some test group, how is that reflected to the corresponding test group? Does the parent test group also fail?
- The parent test groups are organized as "Business execution lists". If one TC fails, Business execution list will also fail.
Is it possible to filter results?
How is logged data is stored?
- Execution results stored in a TOSCA repository (local or shared (DB))
Is it possible to export logs/results?
- Yes. You can print repository content to various formats (excel, PDF, jpg, etc.)
How does TOSCA work with the command line?
- The TOSCA API offers all operations in the cmd as the TOSCA Commander shell. Run TC's, Create TC's, print report, everything...
Is it possible to configure test speed? For example, to insert some delay when GUI is slow.
- Yes, there are multiple settings and options to play with the speed.
Is there any possibility to explore a linux system with TOSCA? Maybe via putty or WinSCP?
- I don't have experience with TOSCA and LInux. Theoretically TOSCA can run external applications. We have created integration of TOSCA and JUnit tests for our needs.
When a test is created on one system, can it be easily transferred to another system? For example, when I create a test, I want to deliver it to another colleague who is working on another machine. Or does he have to create it on his own machine?
- You can use multi-user repository, which has checkin/checkout operations like SVN or CVS. You can also export your TC's or other TOSCA objects to file and import to other local/shared repository.
In the end, what is your opinion/experience about TOSCA?
- Overall impression about TOSCA is quite positive - it's a very rich automation/test management framework. In the right hands it can be used for creation of a really powerful and complex solution. There are many more features that are useful for automation but there can really be a need for some coding effort to integrate with your application - this needs to be considered.