ML//supervised learning//logistic regression

2026-07-04

Logistic regression is a linear classifier: a weighted sum of features passed through a sigmoid to output a probability, trained by minimizing cross-entropy. It is used wherever a yes or no outcome needs a probability that someone can audit: will this drone battery fail its capacity test, given its cycles, its operating temperature and its internal resistance?

p(y=1∣x)=σ(wTx+b),σ(z)=11+e−z.p(y=1\mid x)=\sigma(w^{\mathsf T}x+b),\qquad \sigma(z)=\frac{1}{1+e^{-z}} .p(y=1∣x)=σ(wTx+b),σ(z)=1+e−z1.

The score z=wTx+bz=w^{\mathsf T}x+bz=wTx+b is the logit, the logarithm of the odds p/(1−p)p/(1-p)p/(1−p), so each weight says how much one unit of its feature multiplies the odds. The decision boundary z=0z=0z=0, where the probability crosses one half, is a hyperplane. Fitting is Bernoulli maximum likelihood, which is the same as minimizing the cross-entropy; the loss punishes hard a confident 0.01 for a case that turned out to be 1. There is no closed-form solution, but the problem is convex, a single valley, so any reasonable optimizer finds the same answer.

It is readable. The coefficients say how each feature pushes the odds, which is easy to sanity-check against domain knowledge: a negative weight on cycle count for battery failure flags a data problem before any metric does.

It is the default scientific control before any deep model. If a small logistic model over hand-built features matches the network, the problem was mostly linear. A network with its hidden layers removed is exactly this model.

It cannot capture feature interactions or curved boundaries on its own; either the engineer writes features that straighten the boundary (products, ratios, physics-based features), or gradient boosting or a neural net pulls ahead.

Complete separation breaks the fit. When some hyperplane splits the training classes perfectly, the likelihood keeps improving as the weights grow, and they run off toward infinity with probabilities of exactly 0 and 1. A small penalty (regularization) is the standard fix, and almost every library applies one by default.

Its probabilities are usually well calibrated on data like the training set, because the loss it minimizes rewards calibration; that is what makes a cost threshold p∗=CFP/(CFP+CFN)p^*=C_{FP}/(C_{FP}+C_{FN})p∗=CFP/(CFP+CFN) meaningful on its output (probability calibration). A rare class needs on the order of ten examples per feature for the weights to settle.

Its regression sibling is linear regression, and the margin-based alternative with the same hyperplane is the support vector machine.