ML//training//dataset

A dataset is the collection of examples a model is trained and evaluated on, the fuel of the model, and its quality matters more than its quantity. A language model's data comes from Common Crawl, Wikipedia, books, code repositories and curated conversations; an industrial model's comes from a historian of process variables, PLC alarms and the maintenance work orders that hold its labels (industrial labels).

A dataset is the collection of examples a model is trained and evaluated on, the fuel of the model, and its quality matters more than its quantity. A language model's data comes from Common Crawl, Wikipedia, books, code repositories and curated conversations; an industrial model's comes from a historian of process variables, PLC alarms and the maintenance work orders that hold its labels (industrial labels).

Garbage in, garbage out still applies, which makes data curation underrated engineering, and the recipe for the data mix is every lab's most guarded secret. In a plant the same work goes into aligning sources that were never designed to talk to each other, with clocks that disagree, and it usually eats most of the project.

Coverage before volume.

A thousand hours of summer operation say nothing about winter, and a thousand failures without a cause written down say little about why machines fail. A dataset is worth the range of conditions it spans, and a model asked about conditions outside that range answers with the usual confidence (out-of-distribution).

Rows are not information. Consecutive samples of a slow signal are near copies: an hour at 1 kHz is 3.6 million rows, but with a second of memory in the signal it carries about as much as 1,800 independent samples (effective sample size). A full historian is not the same as a lot of data, and splitting such rows at random between training and test measures memory (data leakage).

For rare events the events count, not the samples. Estimating a failure rate to within 10 % takes about a hundred failures, however many healthy hours surround them, which is why rare failures are certified with models, accelerated tests and simulation (sample size).

When real data cannot be had, data augmentation reuses the examples under transformations that keep their label, and synthetic data generates new ones from a model or a simulator; both inherit the assumptions of whatever produced them.