Q. Does Automature support the ability to execute external (non-native) test cases/suites?
Spark is able to execute any external program or batch script in its own process context (within a spawned command prompt window), or in the context of the Spark Process. These programs or scripts are called atoms. In-process atoms are simply functions or methods that are packaged as DLLs or JAR files (which may contain one or more such functions), but can be accessed through a COM or an abstract interface.
Atoms can be written in any programming language (e.g. Java, C#, C, etc...), or can use any interpreter (e.g. bash, Perl, Ruby, etc...).
In case atoms are called in their own process context all console output is captured and fed into the common log.
Note that the external testcase has to be wrapped inside a Chur shell, which provides contextual information to Spark for reporting purposes.
Automature provides a rich set of in-process atoms that are well documented and tested, and minimize the need for creating new atoms for common tasks.
Q. What happens if the external test suite hangs?
Spark supports concept of a global default timeout for each test step, testcase, or the entire test suite. This value can also be modified per atom, or the entire test suite. If an external test suite accidentally hangs and does not return any result, then Spark is able to execute targeted cleanup for the failing atom, test case, or the test suite, after the timeout has expired.
Q. Can Automature execute tasks in parallel?
Spark allows tasks of various granularity (e.g. test cases, a collection of test steps, or atoms) to execute concurrently. This feature can be used to simulate stress, load, or a multi-user environment.
Q. Does Automature support remote test deployment and execution in a distributed environment?
The Nyon product allows users to deploy the Appication under Test, as well as Test Suites to any number of remote machines, and execute them sequentially or concurrently.
Q. What kind of Logging is supported?
Spark produces its internal logs, as well as custom logs from atom execution. Additionally, it is able to automatically compress and archive both its own logs, as well as application logs and system logs from the test session. These logs are kept in the filesystem, but the archives can be accessed through the test session summaries in Zermatt.
Q. How are test cases stored?
Automature stores of test cases in two ways. Test steps, and verification steps for each test case is stored in the Zermatt database in a free-form textual representation. Automated tests are also stored in a more formal manner using the CHUR language in a spreadsheet. The testcases are automatically entered into the Zermatt database during automated execution of the tests.
Q. How is test data stored?
Test data may be stored in different places, depending on its complexity and structure. It may be stored in the test suite spreadsheet inside a worksheet. It may be stored in an XML or JSON file. Or, it may be stored in a database table. Zuoz provides atoms that support SQL, xpath or JsonPath locators to access data from any of these sources.
Q. What kinds of application testing does Automature support?
Spark has been used to test many different kinds of applications, including Browser-based applications, Web Services, mobile, and graphical console applications. It can also spoof a website to test client side code. Since Spark relies on Zuoz to provide domain specific intelligence, it is easy to extend by adding custom atom packages to drive any type of tests.
Q. How portable is it?
Zermatt is based on Perl, Apache, MySQL. So, it can run on Windows or Linux environments. Basel, Spark, Nyon & Zuoz are implemented in Java, and can therefore run on any platform that supports a JVM. These have been certified on Windows, Mac and Linux.
Q. Can Automature interface with other testing tools?
Yes. Spark comes with out-of-the-box support for reporting into JIRA/ZEPHYR, TESTLINK, in addition to Automature's own ZERMATT. ZERMATT comes with documented REST APIs so that any third party tools can easily report results to it. Additionally, we have released a Jenkins plugin for Automature Reporting, that is able to upload test execution results, as well as code coverage numbers for test sessions, and aggregates for the build.
Q. What is Bug Correlation?
A major task in analyzing test cycle results is to understand which failures are new, or if multiple failures emanate from the same root cause. This becomes especially complicated when thousands of tests are run and scores or hundreds of tests fail. Bug correlation allows the user to group failures to known failure footprints using fuzzy match techniques, thus improving efficiency in failure analysis. Zermatt is the only test management system which has this feature. Currently Zermatt aupports this through JIRA.
Q. Can it measure time of execution?
Spark automatically measures execution times for each test case, and records it for every execution. It can also measure and report the execution times for each step of the logic. It is possible to compare execution times for the same test case under different environmental conditions or builds.
Q. Can Automature email test results?
Yes. Zermatt can email test plan result summaries, including any changes to specific testcase results compared to the previous execution on the same environment. Also, Spark stores up-to-the-second results in Zermatt, where it can be browsed through any web browser, so progress can be monitored at will.
Q. Can Automature perform basic file operations?
Automature supports the execution of any system level commands, such as create, delete, rename, etc... as atoms. Whatever commands can be executed at the command prompt, can be used as an atomic step within a test case. Such atoms can be combined to describe test logic to do more complex operations.
Q. Does Automature allow test sessions to be scheduled?
Nyon used with Jenkins provides support for this feature.
Q. What kind of licensing schemes does Automature support?
The free version comes with an unlimited use license, that must be installed and activated. The enterprise version is both time based, as well as locked to the specific (set of) machine(s) for which the license is obtained.