e-Quallity is a leading provider of software testing services in Latin America with expertise in test design, execution, and automation. In this case study on software testing tools, e-Quallity, a software testing company was selected to compare the performance of their current testing processes versus those of MuukTest during an actual customer’s testing engagement. MuukTest automated tests 20x faster and found 20% more defects than e-Quallity’s traditional testing methods, not to mention that the automation engineering cost was 95% lower.
Product Used for Benchmarking
e-Quallity ran the comparison test on a typical customer project. The specific application tested was for a SaaS web application written in PHP that targeted school systems. The application captures and shares students’ grades while enabling messaging between stakeholders, including teachers, administrators, tutors, and students.
How MuukTest Performed – Benchmarking
In summary, MuukTest vastly outperformed today’s methods:
| e-Quallity | MuukTest | MuukTest Value Added | |
|---|---|---|---|
| # Test Cases Designed | 288 | 841 | + 3X more tests |
| Test Design | 80 hours | 6.85 hours | + 12X faster |
| Test Automation | 136 hours | 6.85 hours | + 20X faster |
| Automation Engineering Cost (US)1 | $7,072 USD | $356 USD | 95% lower |
In the course of benchmarking, e-Quallity identified an opportunity for improvement. During the initial execution of MuukTest automated tests, the test engineer needed to review the results to confirm validity. This review slowed the process down. As a result, for initial execution, the performance time was slowed down to 2.4X. However, following initial execution, performance time improved up to 2.6X faster than manual execution. Given this result, we have prioritized eliminating the need to review and confirm automated test validity by creating functionality that will perform the test engineer assessment.
Benefits & Outcomes
As noted in the benchmarking exercise, MuukTest improved test engineers’ performance in every metric.
MuukTest auto-generated 738 test cases with variations after the user created six journey scenarios with MuukTest extension. The remaining 103 scenarios were created with the extension tool. As a result, all 738 test cases can be executed at any time, based on the user’s needs —for example, when they send a new build to production.
In addition, it is worth mentioning that the test engineer in e-Quallity was able to use MuukTest from day one, after a 1-hour onboarding. Industry competitors need more than three months of models training to provide any value.
Note: e-Quallity is an expert software testing company. Their productivity using traditional testing methods is likely much higher than that of the average software company; this means that MuukTest will perform even better than the average software company.
What’s Next for MuukTest
The benchmark also provided information to design our roadmap:
The 103 test cases created with the Extension tool will be examined to determine how to automate them with fewer inputs — improving the automation metrics.
We have shifted our focus in prioritizing our roadmap user stories related to automating test results review. We want to help the user automatically analyze the expected results of a test, which in this benchmark, required a test engineer analysis time.
