Analytics in Testing Drive Priorities for QA

I am continually surprised when I ask my clients — most of whom are quality assurance (QA) managers — if they have access to their company’s analytics data; the answer is almost always no. I’m shocked when they follow up my query asking why they’d need access to it in the first place. The fact that the (seemingly requisite) need for incorporating user analytics in testing planning and execution isn’t part of their regular testing consideration always opens the door for me to discuss the power of first-party data collection — and most importantly — how to use it.

Analytics in testing sidesteps the need to test everything

Today, few businesses make decisions without consulting data. They collect every conceivable type of user information from their apps; it helps drive strategy and execution in almost all aspects of their business. From marketing to content, revenue to customer service, first-party information is the lifeblood that helps businesses to run faster and more efficiently, saving on costs while maintaining a happy user base.

So, if the overall business is using analytics data to drive decisions, then why don’t my friends in QA have access to the same data to inform their approach to testing? Without access to first-party user data, the default becomes a test-everything approach, which is costly in terms of both time and money and has the potential to delay delivery of the latest release.

How to use analytics in testing to your advantage?

To answer this, you have to know what the majority of your users care about. Here are some key areas to consider:

• environments

• demographics

• behaviors

• revenue

Environments

What devices do the majority of your users use to access your application? What models? What operating systems and versions? Environmental data can help you shift the amount of testing effort to get the most bang for your buck. If 60% of your user base is using iOS version 14 on an iPhone 12, that’s a pretty good clue that you should focus a good amount of effort there.

Demographics

Who are your users? Where are they from? What languages do they speak? What networks are they using? Understanding these analytics in testing can help you plan and execute. If your application is available in markets around the world, which markets have the most users? You may want to focus the majority of testing on these markets and their local languages and networks.

Behaviors

How are your users using your application? The data doesn’t lie. Look for data that shows which features are being used the most and prioritize these features for testing. But don’t stop there, dive deeper to see how users are using each feature and you can prioritize those paths over the less traveled ones.

Revenue

How does your app make money? First-party data can help here as well. If your application generates revenue through direct sales, where is the majority of that revenue being generated? Which markets generate the most? What types of payment instruments are being used most often? Is your revenue ad generated? Where are your ads being served the most? All of this data can help you prioritize and focus your testing.

Take a holistic view of your efforts

I’ve outlined just a few areas where first-party data can give you insight to optimize your test plan. But we can take it a step further by taking them all into account. Since businesses care most about revenue we start there, then look to behaviors, demographics and environment. The data may tell you that the vast majority of revenue is being generated by users in a certain market, using a certain feature flow on a particular device model, OS version and network.

As a QA manager, you are continuously held to high standards while being limited by both time and resources. Sure, if you had unlimited resources you could take the test-everything approach and deliver results within the deadlines set by the business. Unfortunately, that is not the reality of the world we live in. So, we have to play the odds when it comes to our test planning. First-party data from your analytics can help you be smart about it. Incorporating data analytics in testing should be a priority for the overall organization, as siloed data repositories become a thing of the past.

Why does a QA team need access to its company’s applications’ analytics data? Because the decisions you make on a daily basis are just as critical to the overall business.

Get more data about your tests with the Applause Quality Score™. Read the whitepaper for more details.