Significance Testing

What is Significance Testing?

In a nutshell, Significance Testing allows you to verify if differences observed in your sample data are statistically significant.

If the test results say “statistically significant” it means that differences observed in the sample are most likely also appearing in the general population and that your survey results are not purely random.

Significance tests are usually used to compare differences in proportions or means between specific segments.

Currently supported significance tests

Currently, we support the following significance tests:

1. Z-test for proportions (two-sided, unpooled)
2. Independent T-test for means (two-sided, pooled)

Tests are selected and applied automatically based on the type of table you create.

You can select a 90%, 95%, or 99% significance level for each test.

How to conduct Significance Testing with Survalyzer

Step 1: Start with a survey and research question

Let’s assume you have an overall satisfaction question in your survey with a scale of 1-10. You have Male, Female, and Divers segments in your survey.

Research question:

Is there a statistically significant difference in overall satisfaction between gender-based segments in my sample?

Step 2: Create a Professional Analytics report and start the table wizard

Significance testing is available only for Professional Analytics reports. Once you create one, navigate to the Table wizard:

Step 3: Select segments that you wish to compare

In the segmentation step, you need to select segments that you want to test. In our example, we would select variable based segmentation and a “Gender” variable.

Step 4: Select the question that you want to test and select the “significance test” checkbox

At this step, you create your desired table. Significance testing is available when you select either Percent or Mean aggregator. Tests are selected and applied automatically based on the type of table you create:

1. Z-test for proportions if you select “Percent” aggregator
2. Independent T-test for means if you select “Mean” aggregator

Step 5: Finish the table wizard and examine the results in the table.

How to read the results:

Each calculated percentage or mean value will receive alphabetic letters indicating which values from other segments are significantly different.

In our example: Column A (Male) shows a significant difference when compared to Column C (Divers). Column C (Divers) shows a significant difference from both Columns A (Male) and B (Female). Though the Male segment (A) might have a higher mean than the Female segment (B), this difference is not big enough to be statistically significant, with the selected significance level.

Caution! Remember that Professional Report Builder has a default “Top50” setting in the top right, meaning that you only see results for the 50 interviews. In order to see results for the whole data set, switch to “All data” setting or put the table on Report Pages and view the report through the live link version (share report tab or eye icon).

Limits of Significance Testing

Significance testing is a useful tool for data analysis, however, it has to be used with caution. The sample size is an important point to keep in mind. Although any number of responses can be used for a significance test, results from a small sample may not be reliable.

For instance, it is better to have a sizable number of responses from the Male, Female, and Divers segments if you are examining satisfaction levels among these groups. To avoid performing a significance test too soon, we advise holding off until you have at least 50 replies from each group. Less answers on a test can produce results that could lead to misleading insights.