Modern businesses create heaps of data. They also run countless models on a consistent basis to find results. These results help to prove a concept or point they are trying to make. With these results, stakeholders can make crucial data-backed business decisions. But this can all mean nothing if you don’t understand whether or not the models you’re running and resulting data are even valid.
That’s where statistical significance analysis comes in handy. Statistical significance is a measurement that tells you the likelihood that a specific event, versus by chance, influences your result. Calculating statistical significance the right way is crucial to getting valid results. Here’s how to get it done.
Statistical significance can affect your business in several ways. It can mean the difference between choosing one content strategy over another to help lead prospective home buyers down your sales funnel for your mortgage business. It can also mean deciding to switch your call-to-action (CTAs) in your email newsletters to improve retention among current subscribers to your online academy. A statistical significance calculation helps you see what changes you can make to improve your business or processes. Sometimes these changes are as simple as changing the color of a radio button on your landing page. Sometimes, they’re so simple that they feel unnecessary.
While minor tweaks in funnel analytics may seem unimportant, these changes can have a huge impact on your organization’s profitability and, ultimately, the longevity of your business. Think about it: if you were 61 percent confident that switching your CTA from a statement to a question would increase your conversion rate and potentially make you thousands more each month, would you do it? With such a drastic change, it’s easier to see how analyzing the statistical significance of an event can impact your business.
Not every statistic is meaningful. What’s more confusing is that when you get your statistical significance calculation wrong, you risk getting misleading results. Research shows marketers sometimes interfere with results without even recognizing their actions. A 2018 Wharton study highlights how 57 percent of marketers running A/B or split tests practice “p-hacking“—a practice that involves “peeking” or checking on your results before you complete your test. Whenever the results reached a 90 percent significance, the marketers would stop the test.
The problem with this is that you can easily overlook if that result is by chance. Suppose you stop the test the first time. If you ran the test a second or third time and found the significance shows an extreme change, the result wouldn’t be as meaningful. That means you can miss out on knowing when it’s best to tweak your landing page. It can also mean you’re making the wrong change to that landing page.
So, it’s crucial to understand how to measure the statistical significance calculation. There are several ways to get this calculation. But here’s a brief breakdown of some common ways to calculate this important metric:
Statistical significance analysis is crucial to understand because now, more than ever before, companies are making data-backed decisions. So, the decision-makers of the organization must understand the data they rely on and the conclusions they use to make crucial business decisions.
Looking at the evolution of statistical significance over the years makes it easier to understand why it’s crucial to understand if you’re calculating this metric correctly. The concept of statistical significance analysis came about in the 18th century. Then Karl Pearson introduced the Pearson chi-squared test in 1900. Scottish physician John Arbuthnot and the French scholar Pierre-Simon Laplace introduced the world to p-values or probability values. But Ronald A. Fisher refined the standard use of p-values along with his colleagues in 1925. He and his team also popularized the concept of “0.05” as the cut-off point for statistical significance. At first, scientists applied these calculations to academic and scientific experiments. But what may be important for a scientific test may not hold true in the world of business? So, setting the metric for statistical significance goes beyond focusing on the standard of “0.05.”
Calculating statistical significance may seem complicated at first when you do it manually. Moreover, when you apply a method like a chi-square test when doing a split test, you may add more work to your plate. That’s because there are multiple steps you’ll need to take to come to a solid conclusion. Performing this test means you’ll have to identify a hypothesis where there’s no significance and a different hypothesis you want to prove. You’re also going to set your parameters and run the test all before actually performing the chi-square test. But what’s even more important is that you’ll need to analyze your results. The importance of analyzing statistical significance is that it ensures the results are valid and applicable to your organization’s goals.
There are several real-world cases–and misuses–of statistical significance analysis. One example of an ideal way to use statistical significance analysis includes comScore. The company wanted to generate more leads via its product pages using social proof. Initially, it only showcased its customer quote. These quotes appeared alongside other types of content. The content and its layout overshadowed the customer quotes, which may have impacted their conversion rates. To solve this issue, the company leveraged A/B split testing. They used split testing to determine which change to their website’s layout may increase conversion. After performing various tests among 2,500 site visitors, they saw a 69-percent increase in conversion when they used the vertical layout and added a custom logo. This helped them to make a business decision to change the layout of their site across all their product pages.
However, you can also misuse the use of statistical significance analysis when you come up with conclusions that are not actually statistically significant. This can happen when you end your split test too early. Ending your test too early means you may miss an important event that is more of a chance event than fact-based. Running too many tests can also change the significance of your data. For example, if you try to test whether the color blue, red, yellow, green, or black impacts, whether prospects are clicking on your “Try Now” button may be meaningless if you’re also experimenting with the size of the button or shape of its letters.
It’s crucial to understand whether the data results from your A/B split tests are valid. Without this knowledge, you can draw error-based conclusions. In the end, this can hurt your bottom-line. Consider adding a human touch to help you produce and interpret data output. ironFocus offers consulting to help simplify these issues. The goal is to make your data work for your business. ironFocus helps you get it done by offering custom funnel analytics and data services that leverage statistical significance calculations and many others mathematically sound principles. Contact us today for a free consultation.