What is Regression Analysis?
Explanation
Regression analysis is the foundation of almost all statistical techniques, which is precisely why it is often explained either too generally or too specifically. If one were to explain in a single sentence what regression analysis is to someone who is simply curious about it, one could say it is a method for uncovering the relationships between variables.
This useful and remarkable method of analysis was born from the ideas of francis Galton, the man who created eugenics.
While studying genetics, Galton came across data on the heights of fathers and sons, and in general it seemed that when the father was tall the son was also tall, and when the father was short the son was also short. This relationship itself had been known to everyone even before, but Galton took note of the phenomenon in which, as generations passed, it regressed toward the average of the whole.
The son of a tall father is indeed tall, but tends to be shorter than his father, and the son of a short father is indeed short, but tends to be taller than his father. Considered from a commonsense point of view, this is only natural; had it been otherwise, then as generations passed heights would have diverged to infinity or converged to $0$.
On the other hand, it is not always the case that things regress to the average, because there are unavoidable errors such as growth environment or mutations. Nevertheless, the clearly appearing linear relationship must have convinced Galton that ‘height is something one is born with.’
Then, even if not exactly, and even with some degree of error, might it not be possible to roughly guess the son’s height just by looking at the father’s height? If the parents’ height $x$ and the son’s height $y$ have a relationship such as $y = a + b x$, then by substituting the father’s height into $x$, one estimates the son’s height. Of course it will not match up this precisely, but on average it should roughly fit.
This is exactly how regression analysis came about. Of course, by now regression analysis is applied in an enormous variety of fields, and there is no longer any need for talk of generations passing, so the word ‘regression’ has lost its meaning. It is fine to simply move on, noting that this was merely its etymology.