test IO, a leader in crowdtesting for mobile and web applications, today released a white paper based on data from tens of thousands of test cycles. The data provides insight into the most common causes of app crashes across iOS and Android devices, how to test safely in production, and the most common “conversion killer” bugs found in e-commerce tests.
The paper thus serves as a guide for QA and product teams testing across a wide range of devices and user scenarios. The white paper is available as a free download at https://hi.test.io/qa-whitepaper-2018.
Why Mobile Apps Crash
test IO’s testers use real devices in real-world conditions to test mobile apps, so their findings — in aggregate — represent a picture of users’ real-world experiences with mobile app crashes. Analysis of thousands of detailed crash reports yields the following as top causes of app crashes:
“internet connection” (18% of mobile app crashes)
“device rotation” (15% of crashes)
“back button(s)” (25% of crashes)
Mobile apps create specific challenges for developers and testers. The white paper discusses those challenges, as well as solutions, in more detail.
How to Safely Test in Production
The white paper documents that a surprising number of tests now occur in production rather than in staging environments. A key enabler: speed. When customers run tests optimized for speed in production, 90% of the critical bugs are found within two hours. Even in tests not optimized for speed, 70% of the important bugs are found in the first two hours; this enables organizations to test real user experiences on real devices without slowing down release pipelines.
For the largest teams, testing with their own employees or enabling toggles on external releases may be enough to control the risk of rapid software releases. The white paper discusses these and other strategies that allow organizations to keep the production train running while managing quality risks.
The Biggest Threats to Conversion Rates
Similar to test IO’s investigation into bug crashes common to mobile apps, a look at the issues found around conversion rates reveals that the commonest “conversion killer” bugs occur in:
Shopping Cart and Pricing (10%)
Shipping and Validation (8%)
Buttons (i.e. back button, 5%)
By using skilled human testers to ensure the validity of complicated interactions, such as beginning the payment process to checking out of the cart, organizations can focus on creating automated tests for the commonest use cases. In our analysis of each grouping of bugs, the paper provides suggestions for how to address conversion killers in development and testing.