Vol. I · 2026
RAGrets

No RAGrets, yet another AI security blog

Knowledge · Foundations

Supervised learning

Training a model on examples that already have the answer attached. The vocabulary you need to read a model card or a vendor's accuracy claim, and the parts that matter when the model is a security control or a target.

Supervised learning is training a model on examples where the right answer is already attached. A spam filter learns from mail that someone has already marked as spam or not. A house-price model learns from sales that already closed. The model’s job is to find the relationship between the inputs it is shown and the answers it was given, well enough that it gets the answer right on inputs it has never seen.

Most of the machine learning you will meet in a security context is supervised: malware classifiers, fraud scoring, phishing detection, content moderation, the reward models used to steer chatbots. The vocabulary below is enough to read a model card or an evaluation table and know what it is claiming and what it is quietly leaving out.

The pieces

Training data is the set of examples with answers. Each row is a bundle of inputs and the known outcome for that row.

Features are the inputs: the measurable properties the model gets to see. For an email, that might be the age of the sender’s domain, the number of links, whether the display name matches the address, and the words in the body. For a house, size, bedrooms, postcode, year built. Picking and computing features used to be most of the work. Deep learning pushed a lot of that inside the model, but the question “what does the model actually see” still matters, because anything it cannot see it cannot use, and anything it can see an attacker can tamper with.

Labels are the answers: spam or not, the sale price, the malware family. Where labels come from is the first thing to ask about any model. Human annotators, user reports, the output of an older system, a heuristic someone wrote in an afternoon. Whoever controls labelling controls what the model learns.

The model is a function with adjustable parameters that turns features into a prediction. It could be a straight line through the data, a tree of if-then rules, or a network with billions of weights. The shape is chosen up front; the parameters are learned.

Training is the loop that sets those parameters. The model makes predictions on the training data, a loss function scores how far off they are, and an optimiser (nearly always some form of gradient descent) nudges every parameter in the direction that lowers the loss. Repeat for many passes over the data until the loss stops improving.

Prediction is using the trained model: feed in features, get out a label, or more often a score that gets turned into a label by a threshold.

Inference is the broader word for running a trained model. In statistics it also covers using the model to understand the data, such as which features carry the most weight or how a coefficient should be read. In ML engineering it almost always just means “the model running in production”, which is why you see inference servers, inference endpoints, and inference cost. Ollama and vLLM are inference servers.

Two kinds of problem

Classification predicts a category. Spam or not spam (binary). Which of twelve malware families (multi-class). Which of several tags apply to an image at once (multi-label).

Regression predicts a number. A price, a time to failure, a risk score on a continuous scale.

The split matters mostly for evaluation. Classification mistakes are counted; regression mistakes are measured by how far off they were.

Algorithms you will see named

Algorithm Good for Can you read it?
Linear and logistic regression Baselines, small tabular data, anything where you need to explain the coefficients Yes
Decision trees Rules a person can follow Yes, until they get deep
Random forests, gradient-boosted trees (XGBoost, LightGBM) Most tabular problems. The default choice for fraud and intrusion scoring Partly, via feature importance
Support vector machines Smaller datasets with clear margins Not really
k-nearest neighbours Quick prototypes, similarity search Yes, by looking at the neighbours
Neural networks Images, audio, text, anything with raw unstructured input No

On tabular data, boosted trees still beat neural networks more often than not. Neural networks win when the input is pixels, waveforms, or tokens.

Evaluating a classifier

Every claim about a classifier’s quality comes from a confusion matrix: how many positives it caught (true positives), how many it invented (false positives), how many negatives it got right (true negatives), and how many it missed (false negatives).

Accuracy is the fraction of predictions that were correct. It is the number vendors quote and the one that tells you least. If one login in a thousand is fraudulent, a model that says “legitimate” every time is 99.9% accurate and catches nothing.

Precision is, of the things the model flagged, how many were actually positive. Low precision means alert fatigue: the analysts stop reading the queue.

Recall is, of the real positives, how many the model flagged. Low recall means misses. For a malware classifier this is the number that matters; for a content filter that blocks user posts, precision usually matters more.

F1 is the harmonic mean of precision and recall, so it is low if either one is low. One number, easier to compare, hides which way the model is wrong.

Most classifiers output a score, and the threshold that turns a score into a decision is a choice, not a property of the model. Raising it buys precision at the cost of recall. A ROC curve plots that trade-off across every threshold, and the area under it (AUC) summarises it. When someone reports a single accuracy figure, ask for precision and recall at the threshold they actually run in production, on data that looks like yours.

Regression is simpler: mean absolute error and root mean squared error, both in the units of the thing being predicted. RMSE punishes large misses more.

Generalisation, and the ways it fails

A model that gets the training set right has learned something. Whether it learned the pattern or just the examples shows up only on data it has not seen.

Overfitting is learning the examples. The model fits the noise, the outliers, and the quirks of the particular training set, scores well on it, and falls over on new data. The usual tell is a large gap between training performance and held-out performance. Overfitting is also a privacy problem: a model that memorised its training data can be made to give it back. Membership inference asks whether a specific record was in the training set; extraction attacks pull records out of language models verbatim. Both get easier the more the model overfit.

Underfitting is the opposite: the model is too simple or was not trained long enough, and does badly on the training data too.

Holding data out is how you tell. Split the data into a training set, a validation set used to tune choices like the threshold or the tree depth, and a test set that is touched once at the very end. Reusing the test set during development turns it into a second validation set and the final number stops meaning anything.

Leakage is when information the model will not have at prediction time sneaks into training. A fraud dataset where the chargeback timestamp is a feature. The same malware sample appearing in both the training and test splits under two hashes. A feature computed over the whole dataset including the rows being predicted. Leaked models look excellent and fail on day one. It is the most common reason a published result does not reproduce.

Cross-validation gets more out of a small dataset. Split it into k folds, train on k minus one, validate on the one left out, rotate, and average. The variance across folds is useful too: a model whose score swings widely between folds is not one to trust.

Regularisation is any technique that penalises complexity so the model has less room to memorise. L1 adds a penalty on the absolute size of the parameters and tends to push many of them to exactly zero, which doubles as feature selection. L2 penalises the squared size and shrinks everything toward zero without eliminating it. Dropout randomly disables parts of a neural network during training. Early stopping halts training when validation loss starts rising. Data augmentation makes the training set larger and messier on purpose.

Where attackers come in

Each piece above is something an adversary can push on.

  • Labels that come from users (report as spam, mark as safe) let users train the model. Flip enough labels and the classifier learns the attacker’s preferences. This is data poisoning, and it is cheap when labelling is crowdsourced.
  • Features the attacker can see and change are a decision boundary they can walk up to. Adversarial examples are inputs nudged just far enough to cross it: a malware sample padded with benign-looking bytes, an image with imperceptible noise that flips the class. The model is applying the boundary it learned. The attacker just found where it runs.
  • A prediction API that returns scores, not just labels, leaks enough to rebuild the model. Query it with chosen inputs, train your own on the answers, and attack the copy offline.
  • Overfitting leaks training data, as above. The more sensitive the training set, the more that matters.
  • A model trained on last year’s phishing kits has not seen this year’s. Distribution shift is a reliability problem in general and an attack technique when someone changes the distribution on purpose. Any supervised model in a security role needs a retraining schedule and a way to notice that its inputs no longer look like its training data.

Supervised models still make good security controls. Just count the training pipeline, the evaluation, and the threshold as part of the control when you review one, because an attacker will.