Knowledge · Foundations
Linear regression
The oldest supervised learning algorithm and still the first thing to try on a numeric target. How the line gets fitted, what the coefficients mean, which assumptions it quietly makes, and why its simplicity makes it easy to steal and easy to poison.
Linear regression predicts a number by drawing a straight line through the data. Give it house sizes and sale prices and it finds the line that, on average, misses the known prices by the least. Give it a new size and it reads the price off the line. That is the whole idea, and a surprising amount of production machine learning is still exactly this with more columns.
It is a supervised algorithm: it learns from examples that already carry the answer. And it is a regression algorithm, which means the answer is a continuous value (a price, a temperature, a latency, a risk score on a scale) rather than a category. If the target is “spam or not”, you want classification, and the linear-model equivalent there is logistic regression, which fits the same kind of line but pushes the output through a squashing function so it lands between 0 and 1.
The model
With one predictor the fitted line is
y = m·x + c
x is the input, y the prediction, m the slope (how much y moves for one unit of x) and c the intercept (the prediction when x is zero). Training means finding the m and c that fit the examples best.
With several predictors the line becomes a plane, or a hyperplane, but the equation only grows sideways:
y = b0 + b1·x1 + b2·x2 + … + bn·xn
b0 is the intercept and each bi is the coefficient on predictor xi. The coefficient is the thing people actually want from a linear model: b1 = 340 on a house-size column means each extra square metre adds about 340 to the predicted price, holding the other predictors fixed. No other mainstream algorithm hands you that sentence for free, which is why linear regression keeps its job in finance, medicine, and anywhere a regulator will ask “why this number”.
Fitting the line
For each training example, the residual is the real value minus the model’s prediction. A good line has small residuals. Ordinary least squares, the standard fitting method, picks the line that minimises the sum of the squared residuals: square each miss, add them up, and find the coefficients that make the total as small as possible.
Squaring does two things. It makes every miss positive, so overshoots and undershoots cannot cancel out. And it punishes large misses much more than small ones: a residual of 10 costs 100, a residual of 1 costs 1. Hold on to that second property, because it comes back under security.
There are two ways to actually find the minimum. For modest datasets there is a closed-form solution (the normal equation) that computes the coefficients in one shot with some matrix algebra. For very large datasets, or when the same code path has to serve neural networks too, gradient descent walks downhill on the squared error a step at a time. Both arrive at the same line for ordinary least squares.
Judging the fit
R² is the share of the target’s variance the model accounts for. 0.9 means the line explains 90% of the spread in the target; 0 means it does no better than predicting the mean every time. It climbs whenever you add a predictor, even a useless one, so adjusted R² exists to penalise that.
Root mean squared error is in the units of the target: an RMSE of 12,000 on house prices means a typical miss is around twelve thousand. Mean absolute error is the same idea without the squaring, so it is less dominated by the worst predictions.
Residual plots are the diagnostic that matters. Plot residuals against predicted values. A healthy model shows a shapeless cloud around zero. A curve means the relationship is not linear. A fan that widens as predictions grow means the errors are not constant. A few points far from everything else are outliers or leverage points, and the next section is about why you care.
The assumptions, and what breaks when they fail
Linear regression is honest about what it assumes, which is more than most algorithms manage.
- Linearity. The relationship really is a straight line. If price rises with size and then flattens, a line will underpredict small houses and overpredict mansions. Fix: transform a predictor (log of size), add a squared term, or use a different model.
- Independence. Each observation is its own event. Daily readings from one sensor are not independent; today looks like yesterday. Standard errors on the coefficients will be wrong, and the model will look more confident than it should.
- Constant error variance (homoscedasticity). The spread of residuals is the same across the range. When it is not, predictions at the wide end are far less reliable than the single RMSE figure suggests.
- Normal errors. The residuals follow a bell curve. This matters for confidence intervals and p-values on the coefficients, less so for the point predictions themselves.
Two more that the textbook list usually leaves out:
- No severe multicollinearity. When two predictors move together (square metres and number of rooms), the model cannot tell which one deserves the credit, and the coefficients swing wildly between training runs even though predictions stay stable. Ridge regression, which adds an L2 penalty, is the usual fix.
- Interpolation only. The line is fitted on a range of inputs and says nothing trustworthy outside it. A model trained on 60 to 300 square metre homes will cheerfully price a 2,000 square metre warehouse, and the number will be fiction.
Where it fits today
Linear regression is the baseline. Before a boosted tree or a neural network earns its place, it should beat the linear model on held-out data by enough to justify losing the readable coefficients. Often it does not. The regularised variants, ridge (L2) and lasso (L1), handle wide datasets with many correlated columns and are covered in the supervised learning entry. Logistic regression carries the same machinery into classification and is still the most deployed classifier in fraud, credit, and medical screening.
Security: simple to read, simple to attack
Everything that makes linear regression attractive also makes it the easiest model to attack, and the attacks are worth knowing because they are the clean versions of what happens to bigger models.
Model theft is a linear algebra exercise. A scoring endpoint that returns the raw prediction is leaking the coefficients. With n predictors there are n + 1 unknowns, so n + 1 well-chosen queries recover the model exactly, not approximately. Round the outputs, add noise, rate-limit, or return only a bucket instead of the number if the model itself is the thing you are protecting.
Poisoning works through the squared loss. Because large residuals dominate the objective, a handful of extreme points pull the line hard toward themselves. An attacker who can insert a few rows into the training data, far out along one predictor, can tilt the slope on purpose. The classic demonstration moves a spam or pricing model with a tiny fraction of the dataset. Robust regression (Huber loss, RANSAC) and plain outlier screening before training blunt this.
The fit can leak the data. A model with few training rows and many predictors can fit them nearly exactly, and the coefficients then encode the rows. Combined with the theft above, that turns a prediction API into a slow read of the training set. Regularisation helps; so does not training a 50-column model on 60 rows.
Extrapolation is an input-validation problem. A model that will score anything will score adversarial inputs far outside its training range, and the output will be confidently absurd. Clamp inputs to the training range, or refuse to score outside it, the same way you would validate any other untrusted parameter.
None of these need the attacker to understand machine learning beyond a first lecture. That is the point: when a vendor says their risk model is “just a regression”, read it as a model that is cheap to copy and cheap to steer, and check what sits in front of the endpoint.