Uncertainty and ‘confidence intervals’
Statistics can’t tell you the whole truth
Statistics come from research that is conducted on groups of people, for example a group of people with back pain. But the group of people in the study are only a small proportion of all of the people in the world with back pain.
We can only measure directly, the effect of the medication on the group involved in the research. We use that measurement to predict that it will have the same effect on everyone in the world with back pain.
Measuring and predicting
But there are lots of reasons why the size of the effect might be different in the research group compared to the whole worldwide population of back pain patients.
We might try to do research with a very similar group to all back pain patients in the world – the same age range, the same gender balance, etc. But even then there would still be some random variation.
So researchers could measure the effect of a treatment for back pain patients in their study. But we also need to know how likely it is that a study like the one they did would tell us the real effect in all the back pain patients in the world. This is a prediction, which means it isn’t certain.
How likely is our prediction to be correct?
This is where a ‘confidence interval’ comes in. Again, because it’s used for making a prediction, we can’t be sure of an exact value. So the confidence interval is a range rather than a single number. It is generated by a set of calculations.
First, a researcher has to choose how certain they want to be that the results of a study like theirs would be able to indicate the real effect on the whole population. Generally 95% is chosen as how sure we want to be that a study like theirs would give an indication of the real effect.
That would mean that there’s a 95% chance that the real effect size in the whole population of patients is somewhere in the range of values calculated from the group in this study.
It is important to understand the uncertainty associated with the kind of statistical information that we provide. In our resources, we generally don’t show confidence intervals. We design resources to rapidly communicate the gist of the information. Displaying confidence intervals would make the resources take longer to use, and be more difficult to understand.
With every resource, we include links to the original data source. The majority have plain language summaries of the results. In many cases this includes a description of the confidence interval. Since this provides a more accurate estimation of the range of possible effects of a treatment, we recommend that you also review these sources.