In research methods you learn about factors of good experimental and correlational research design.
Experimental design 101
As a 101 refresher, experimental designs by nature less easily generalize to the real world. You’re making your own little world with your own structure and then saying, “see! people love dark chocolate,” when we all know the darker the chocolate the grosser it is.
The benefit to experimental design is that you can choose your inputs and then watch as they affect whatever outputs.
At the very least you know that under the strict conditions you imposed (say, putting people in a locked room for three hours with a bar of dark chocolate), they will eat it.
Causation is cool, but not easy to generalize
This, for sure, demonstrates causation, but by what mechanisms and under what conditions we can’t be sure. Causation in a bubble does not generalize to causation in the real world.
IRL you got noise and confounding variables you can’t ever hope to control for in a simple clean experimental design.
We call this external validity, or how well our results generalize to relevant situations and environments we care about.
A simple example
If we leave students in a tiny research room with some dark chocolate alone for 3 hours and they eat the chocolate, what conclusions can we draw?
Can we really assume anything?
Does it only show people eat dark chocolate in a tiny research room?
When they have to sit for x time?
Does it only work for college students or real people too?
Did we really learn anything other than students are willing to eat this yucky treat after being starved for x time?
Correlational research 101
On the other side, we have correlational research. We look at real world trends and see that sales for chocolate increase around Valentine’s Day. We can clearly see the correlation. It’s happening in the real world.
Someone is buying this chocolate.
But because we cannot create or cancel Valentine’s Day whenever we want we can’t be sure that dark chocolate purchases are a result of Valentine’s Day being a thing. It could be something else, like one of those improv everywhere pranks, or someone kindly buying up all the dark chocolate so there is none left for people to inflict as misguided gifts on other people.
Utility of correlations for market research
What we can do is make further hypotheses from the correlations.
If our big question is, “Do people like dark chocolate?” let’s think about it and assume we have all the dark chocolate sales data for the year.
A great lesson I learned from a social psych professor was that when experimental research and correlational research findings agree, you can be pretty confident of the results.
The web as the real world, your website as the tiny research room
When we decide what campaigns to run, we can look around and see what appears to be working for others. This competitor is killing it. Are they running ads? Are they manually building backlinks to their best content? Are they internally linking in an organized and seemingly strategic fashion? Do their calls to action on different pages congruent with the topic matter of those pages?