ML//TinyML

TinyML is the practice of running ML inference on microcontrollers (MCU) and ultra-low-power devices, with models measured in kilobytes running on Cortex-M cores within sub-milliwatt budgets. Keeping data and inference entirely on the device bypasses cloud latency and connectivity requirements; keyword spotting, anomaly detection and gesture recognition are the typical uses.

TinyML is the practice of running ML inference on microcontrollers (MCU) and ultra-low-power devices, with models measured in kilobytes running on Cortex-M cores within sub-milliwatt budgets. Keeping data and inference entirely on the device bypasses cloud latency and connectivity requirements; keyword spotting, anomaly detection and gesture recognition are the typical uses.

The budget is concrete. A Cortex-M at 100 to 500 MHz with a few hundred KB of RAM runs int8 networks of tens to hundreds of KB in milliseconds and milliwatts, so a vibration sensor on a pump can run its own anomaly detector and send only the alert, instead of streaming raw data over a battery-powered radio. Two hidden layers of 64 neurons are about 5,000 multiply-accumulates, tens of microseconds on a fast core, which fits inside a 1 kHz control loop.

A fixed-size network has a predictable worst case.

It performs the same operations on every input, so its latency is nearly constant and is measured once, which makes it friendlier to a real-time loop than an iterative optimizer such as MPC (worst-case execution time). What nobody gives you for free is knowing when its input has left the training distribution: watch it (out-of-distribution).

The usual frameworks are TensorFlow Lite Micro, CMSIS-NN and Edge Impulse, and quantization is what makes a model fit: going from 32-bit to 8-bit weights cuts the model to a quarter of its size (quantization). The weights end up compiled into the program:

1C++ takes the input2TensorFlow Lite3const_float model_weights

ML can be deployed at three tiers, each with its own kind of event:

On the sensorAn ASIC or MCU does pattern filtering (decision trees, CNNs), stuck with the manufacturer's logic. It suits a sensor that is always listening for an event that occurs briefly (vibration).

On an MCU with an NPUMultiple sensor inputs and neural networks. It is required when the event is continuous (sound); the NPU is passive and wakes the MCU only if needed.

On an MPU with an NPUHeavy neural networks that classify and predict, placed there to avoid killing the CPU (AirPods noise cancellation, Face ID, Alexa).

An Alexa-style pipeline splits the work between the device and a datacenter:

1A DSP detects the keyworda 3 s buffer in MCU RAM2The audio goes over Wi-Fi and the internet3An H100 runs the inference