elementfromgaze: an eye tracker on a laptop webcam, accurate to about one degree
A gaze tracker for Windows built on a laptop's colour webcam, in which two small CNNs read the eye's rotation from straightened eye patches and the head pose turns it into a ray that hits the screen. Posture was the largest source of error, and an 80-second warm-up each session takes a posture never calibrated from 4.3° to about 1.1°, after which clicks and the text caret keep teaching it. Measured on one person, an error of about one degree picks a button 2° tall nine times in ten, while the menus of Windows are a fifth of that height.
document.elementFromPoint(x, y) tells a web page which element sits under the mouse pointer. I wanted the same call for the eyes, elementFromGaze(): which button or which word I am looking at right now. Behind it sat a larger wish, to drive the whole Windows desktop by looking, with the pointer landing on whatever button of whatever app my eyes rest on and a long blink clicking it. I gave myself the laptop as it is, a ThinkPad with a colour webcam and an infrared camera that Windows Hello uses to recognise my face. Its AMD integrated GPU is one PyTorch does not support on Windows (the same machine runs a 35B model in the local LLM piece, where that GPU at least had a job), so every network here runs on the CPU, and there was no tracker to buy.
It works now, within limits I can put numbers on. After an 80-second warm-up the dot lands about one degree from where I look (0.89° in a button-picking test, 1.46° while following a moving ring). That picks a button two degrees tall nine times in ten and fails on the menus of Windows, which are a fifth of that height. It also runs in a browser: the live demo calibrates on your face in 80 seconds and then lights up buttons for you to pick by looking, with the camera image never leaving your computer. The first half of this piece builds up what one degree means to a gaze tracker and why a webcam has a hard time measuring it. The second half is how the tracker got there, and most of it is something that did not work the first time.
A personal note26 · nov 05Read noteI started with EyeTrax, an open-source webcam tracker, and could not get it to work on this laptop, so the core became my own, written on MediaPipe's face landmarks.
One degree of visual angle
A gaze tracker answers one question, where the line of sight meets the screen, and its error could be given in pixels or in centimetres. Both depend on things the tracker does not control: a pixel is smaller on a denser screen, and a centimetre is a smaller angle from further away. What the tracker measures is the direction of the eye, so the field states its error as an angle at the eye, in degrees of visual angle.
I sit about 55 cm from the laptop's screen, and at that distance one degree spans 2⋅550 mm⋅tan0.5∘=9.6 mm2 \cdot 550,\text{mm} \cdot \tan 0.5^\circ = 9.6,\text{mm}2⋅550mm⋅tan0.5∘=9.6mm. The panel puts 1920 pixels across 301 mm, 0.157 mm a pixel, so one degree is 61 pixels. A Windows menu item, 24 to 32 pixels tall, is 0.4 to 0.5°. Those two conversions sit under every number in this piece.
1° spans 9.6 mm = 61 px55 cm, eye to screeneyebutton 2° tall, 4° wide122 × 245 pxfovea0.6 to 1° across37 to 61 pxmicrosaccades:under 0.5°, 31 pxOpenSave asPrintmenu items, 28 px tall0.46° each1° = 61 px on this screena buttonmenu itemsfovea, 0.6°fovea edge, 1°one degree
My laptop's screen seen from 55 cm, the side view and the screen both to scale, the screen at one unit per pixel (1920 px over 301 mm make a degree 61 px). The fovea alone covers half the height of a 2° button, and on a Windows menu it spans two items at once.
The eye sets a floor of its own. Sharp vision comes from the fovea, a patch of the retina that covers 0.6 to 1° of visual angle1. Everything inside that disc is seen in detail at once, so the place I am looking at is already a region about a degree across, and a tracker that reports a point reports its centre. That centre does not hold still either. While I stare at one letter my eye drifts by less than 0.13° at under half a degree a second, and one to three times a second it jumps back in a microsaccade, usually smaller than half a degree2. A perfect tracker would show its dot shivering by a few tenths of a degree on a letter I never stopped looking at. Half a degree to a degree is about as fine as the question gets for a person, whatever the hardware.
1Guestrin and Eizenman, General theory of remote gaze estimation using the pupil center and corneal reflections, IEEE Transactions on Biomedical Engineering 53(6), 2006, which calls it the region of highest acuity. 2Otero-Millan, Macknik and Martinez-Conde, Fixational eye movements and binocular vision, Frontiers in Integrative Neuroscience 8, 2014, which also gives a tremor of a few seconds of arc at around 90 Hz.
The error itself has two parts that are easy to mix up. Accuracy is how close the gaze a tracker reports is to where the person actually looks. Precision is how close repeated measurements of an eye that has not moved are to each other, and makers usually give it as the root mean square of the jumps between successive samples1. They are the two halves of precision and accuracy in any measurement, a tight group off the bull's-eye against a loose one around it. For picking a button, accuracy is the one that decides: a steady dot two degrees off sits steadily on the wrong button.
1Niehorster, Zemblys, Beelders and Holmqvist, Characterizing gaze position signals and synthesizing noise during fixations in eye-tracking data, Behavior Research Methods 52, 2020.
Pupil and glint
Paid trackers solve the problem with light. They light the eye with near-infrared LEDs and film it with a camera that sees that light. In each image the software finds the centre of the pupil and the glint, the LED's reflection on the cornea, and works from the vector between the two, a method Tobii's explainer calls pupil centre corneal reflection.
Pupil centre corneal reflection.
The front of the cornea acts as a convex mirror, with a radius of curvature of about 7.8 mm at its apex, and it makes a small virtual image of each LED: the glint, which optics calls the first Purkinje image (Guestrin and Eizenman, 2006, cited above). The tracker maps the vector from the pupil's centre to the glint onto a point of the screen. The pupil can be made to stand out in two ways: an LED close to the camera's axis makes it appear lit up, the red-eye of a flash photograph, and one placed away from the axis makes it appear black1. The method carries two limits. The eye aims with its visual axis, which ends at the fovea, and in a typical adult the fovea sits about 4 to 5° to the temple side and about 1.5° below the point where the optic axis meets the retina; that offset, the angle kappa, is different for every person, which is why every person calibrates. And with one camera and one light source, the point of gaze can be estimated only if the head is completely stationary; following a free head takes several light sources and a calibration on several points (Guestrin and Eizenman, 2006).
1Gibaldi, Vanegas, Bex and Maiello, Evaluation of the Tobii EyeX eye tracking controller and Matlab toolkit for research, Behavior Research Methods 49, 2017.
The drawing puts that geometry to scale for the parts the paper measures, the cornea's 7.8 mm radius and the 13.1 mm from its apex back to the centre about which the eye turns.
foveavisual axisoptic axiskappa: 4 to 5° sidewaysand about 1.5° downrotation centre13.1 mmcornea, radius 7.8 mmglintIR LEDcamerawhat the camera seesLED off axisdark pupilLED on axisbright pupilthe eyeinfrared lightvisual axis, foveapupil to glint vectordashed: optic axis
One eye seen from above, its cornea and centre of rotation to scale at six units per millimetre and the globe around them schematic; the LED and the camera sit about 55 cm away and are drawn close. The glint is the point where the cornea mirrors the LED into the camera, and the visual axis leaves the optic axis by kappa, an angle that differs from person to person.
That is how the infrared trackers on sale reach the size of the fovea. Tobii's Pro Spectrum, a research tracker that samples at up to 1200 Hz, reports a median accuracy of 0.30° with the head on a chin rest and 0.43° with the head free1. SR Research gives 0.25 to 0.50° as typical for the EyeLink 1000 Plus in its remote mode2. Gazepoint sells the GP3, 0.5 to 1° at 60 Hz, for $9853. The consumer end is quieter. The Tobii Eye Tracker 5, a gaming device at 279 euros, publishes neither its accuracy nor its sampling rate4. Head-worn trackers trade some of the accuracy for freedom: Pupil Labs lists 1.8° uncalibrated and 1.3° with an offset correction for its Neon glasses5.
1Tobii, Tobii Pro Spectrum specifications: 0.30° median over 20 participants with the head supported, in optimal conditions, and 0.43° over more than 400 with the head free. 2SR Research, EyeLink 1000 Plus technical specifications. 3Gazepoint, GP3 eye tracker product page. 4Tobii, Tobii Eye Tracker 5 product page, which gives the working distance and the screen sizes and no accuracy. 5Pupil Labs, Neon technical specifications.
Webcam methods live an order of magnitude higher. WebGazer, which runs in the browser and learns from the user's clicks instead of asking for a calibration, measured 4.17° in the lab with a chin rest1. GazeCapture, with more than 1,450 people on phones and tablets, gave its errors in centimetres, 1.71 cm on phones and 2.53 cm on tablets without calibration2. Centimetres suit a screen held in the hand, where the distance, and with it the angle, changes all the time. A network that reads the whole face, trained on MPIIFaceGaze, reaches 4.8° on people it never saw3. Trained on one dataset and tested on another, networks lose more, 7.08° from ETH-XGaze to MPIIGaze in the best result I found4.
1Papoutsaki et al., WebGazer: scalable webcam eye tracking using user interactions, IJCAI 2016: 169 px on a 24-inch screen at 59 cm, against a Tobii EyeX. 2Krafka et al., Eye tracking for everyone, CVPR 2016. 3Zhang, Sugano, Fritz and Bulling, It's written all over your face: full-face appearance-based gaze estimation, CVPR Workshops 2017. 4Cheng, Bao and Lu, PureGaze: purifying gaze feature for generalizable gaze estimation, AAAI 2022, with no samples of the target dataset.
foveainfraredTobii Pro Spectrum, chin rest0.30Tobii Pro Spectrum, head free0.43EyeLink 1000 Plus, remote0.25 to 0.5Tobii Pro Fusion0.3Tobii Pro Spark0.45Gazepoint GP30.5 to 1Pupil Labs Neon, wearable1.3 to 1.8Tobii Eye Tracker 5not publishedwebcamthis project, warmed up0.89 to 1.46this project, as a new user1.80WebGazer, chin rest4.17MPIIFaceGaze CNN, unseen people4.8trained on ETH-XGaze, on MPIIGaze7.080.25°0.5°1°2°4°8°accuracy, log scaleinfrared, maker figurewebcam, published methodthis project, measured on mefovea, 0.6 to 1°bar: a range · tick: a median
Published accuracy of eye trackers in degrees, each under its own protocol (chin rests, optimal conditions, people seen or unseen), so the rows are not one benchmark. The axis is logarithmic, each tick double the last. Infrared reaches about the size of the fovea, and published webcam methods for people they never saw land four to seven times higher.
The infrared hardware is what buys the difference: its own light, and a reference on the eye, the glint, that a room's light cannot move. A webcam has neither, and the published webcam figures are for strangers with no calibration or a short one. I only needed the tracker to work for one person, with as much calibration as I could stand, and on paper that put a degree within reach.
Pixels per eye
What a webcam offers instead is pixels, and the question is how many of them fall on the part that moves. The iris is 11.7 mm across, give or take half a millimetre, in nearly everyone1. How many pixels it covers follows the pinhole camera model, its size times the focal length in pixels divided by the distance, but instead of looking up my webcam's field of view I measured the answer. Across the sessions I recorded at 1920 × 1080 while sitting as I work, MediaPipe's iris landmarks put the iris at a median of 25 pixels wide (22 to 33 from one session to the next, and 53 in a session where I sat close on purpose). One pixel at my eye is therefore about 11.7/25≈0.46 mm11.7 / 25 \approx 0.46,\text{mm}11.7/25≈0.46mm.
1Google Research, MediaPipe Iris: real-time iris tracking and depth estimation, 2020, which uses that near-constant size to estimate the distance to the camera.
Now turn the eye by one degree. It rotates about a centre 13.1 mm behind the front of the cornea, so the front of the cornea moves 13.1 mm⋅tan1∘=0.23 mm13.1,\text{mm} \cdot \tan 1^\circ = 0.23,\text{mm}13.1mm⋅tan1∘=0.23mm, and the pupil, which sits a little behind it, moves less. One degree of gaze is about half a pixel in the camera frame, and at the camera's default of 640 × 480, where I started and where the iris spans 10 pixels, it is a fifth of one. What can still see it is the grey level of every pixel along the edge of the iris and the lids: when an edge moves by a fraction of a pixel, each pixel it crosses turns a little lighter or darker, and the visible edge of an iris 25 pixels across runs through a few dozen of them.
1920 × 1080 framethe eye: 57 of 1920 pxone eye, 1.9 units a pixeliris 25 px = 11.7 mmcorners 57 px apartpatch, 60 × 36 pxcorners pinned36 px apart1° of eye rotation: the cornea moves 0.23 mm1920 × 1080 frame0.50 px640 × 480 frame0.20 px60 × 36 patch0.32 pxone pixel, drawn 120 units wideone pixelone degree of eye rotationwhat the network reads
The webcam’s pixel budget for one of my eyes as I usually sit: a median of 25 px across an iris of 11.7 mm in my 1080p sessions, by MediaPipe’s iris landmarks, and 10 px at 640 × 480. A one-degree turn of the eye moves the front of the cornea about half a pixel at 1080p, so the network has to read shifts smaller than a pixel from the grey levels along the iris edge.
MediaPipe's Face Landmarker does the first part of the work. For each frame it returns 478 points in 3D, 468 on the face and 10 on the irises. Alongside them come 52 blendshape scores (how closed each eye is, among others) and a 4 × 4 matrix with the head's position and rotation1. My first model used those numbers directly, the iris's place inside the eye and how open the lids were, fed to a small regressor. Then I compared it with a network that reads the pixels around each eye, leaving one recorded session out at a time: at 640 × 480 the landmarks left 3.5 to 7.8° of error and the pixels 1.7 to 4.1°, and at 1080p 2.9° against 1.9°.
1Google, Face Landmarker, MediaPipe documentation.
The network never sees the whole face. Each eye is cut out and warped so that its two corners land on fixed points 36 pixels apart in a patch of 60 × 36 pixels, which takes out the head's roll and its distance from the camera, and the left eye is mirrored so one network serves both. In the 1080p frame my eye corners sit about 57 pixels apart, so the patch shrinks the eye by a third; larger patches gave no clear gain. The gaze literature straightens eyes another way, by turning a virtual camera to face each eye in 3D, and a vision researcher would have started there. I tried it, with a network pretrained on MPIIGaze, and it lost to the plain warp on the corners.
Two small convolutional networks read the patches, each three convolution blocks and a layer of 64 units, 113 thousand weights, and their answers are averaged. Each returns two angles, the eye's yaw and pitch inside the head. On the laptop's CPU they take about 5 ms of the 13 ms the whole frame needs.
The head pose turns those angles into a ray. MediaPipe's matrix rotates the eye's direction from the head's frame into the camera's and places the eye in space; on a laptop the camera sits in the plane of the screen, so the point where the ray crosses that plane is the point I look at, and a fitted map turns its centimetres into pixels. This part is geometry with a handful of numbers fitted per user, and as a systems engineer it was the first piece I trusted. A regressor that goes straight from features to the screen sees that the iris moved and cannot tell whether the eye turned or the head did, so it either ignores the pose, and then the dot moved 15 to 68 pixels for every degree the laptop turned, or it learns the pose and missed by 11° on a new session.
webcam1920 × 108030 fps, colourMediaPipe478 landmarks52 blendshapeseye patches2 × 60 × 36 pxcorners pinned2 small CNNs113k eacheye yaw, pitchgaze rayturned by poseonto the screencorrectiona Kalman shiftfrom labelssnapposterior 0.8release at 0.4pointeron the buttonblink: clickhead pose, 4 × 4warm-up, 80 s: fine-tunes the top layersposture bank: reuses a warm-upclicks and caret: labels for the correctiontrain: refits on every saved sessionlabelsfine-tuneclicks,keyshardware, outside librarydata and labelstrained weightslive correctionwhat I getdashed: learning, between frames
One frame of the live path on the laptop’s CPU: about 13 ms of compute in the 33 ms between frames at 30 fps, of which the two networks take about 5. Learning sits outside the frame and only changes the networks and the correction between frames.
A warm-up for every posture
The first real test was reading. I opened a page of text in the browser with a layer that highlights the line the gaze is on, and the highlight sat 3.5° off vertically, five lines away from the one I was reading. Reading is an unforgiving first test: the eye jumps about two degrees at a time, seven to nine letters, and rests a quarter of a second on each stop1. The log said where the error came from. I was sitting 37% closer to the screen than during the warm-up that session had started with, and every frame was being read by a network tuned to a man who sat further back.
1Rayner, Eye movements and attention in reading, scene perception, and visual search, Quarterly Journal of Experimental Psychology 62(8), 2009: mean fixations of 225 to 250 ms and saccades of 7 to 9 letter spaces.
That turned out to be the rule, and the largest source of error in the whole project is posture. The eye seen from another angle, under another light, is a different input to the network while the rule from eye rotation to gaze stays the same, which in the vocabulary of machine learning is covariate shift. A network trained on my earlier sessions was 4.3° off in a posture it had never seen. After an 80-second warm-up in that posture it was at 1.1 to 1.2°.
1Follow a ring across the screenpursuit2Fixate the four corners3Turn the head while fixatinga guide shows how far4Look at nine buttons as they lightmeasures what is left
The warm-up fine-tunes the top of the network on those 80 seconds, and I removed each of its parts in turn to check that each one helps. The head turn teaches the network that turning the head is not looking elsewhere: I keep my eyes on a ring and steer a red circle, my nose, onto a blue dot. The nine buttons at the end measure what the fine-tune left and seed the correction of the next section. Every warm-up is kept with the posture it was fitted to, and when I sit like that again for about ten seconds the dot switches to it; a posture it has never seen sets off a short recalibration. I first made that recalibration ten seconds long, and on held-out sessions it gave 1.77° against 1.33° for the full 45-second path, so it now follows the warm-up's path.
The warm-up's first screen, 78 seconds announced (the app speaks Spanish).The head turn: eyes on the white ring, nose (the red circle) steered onto the blue dot.
A personal note26 · nov 07Read noteThis morning the room was dim, the webcam quietly dropped to 16.7 frames a second, and the dot started to jump. The camera now asks for 30 and the app warns when it gets fewer than 24.
Clicks and the caret as labels
I almost always look at what I click, and at the place where I type. That makes every click and every typed character a free label of where the gaze truly is, collected while I work, which is online learning with the labels supplied by my own hands. A low-level hook records that a click or a key press happened and when, never which key, and the caret's position comes from the accessibility interface that Chrome and VS Code expose, among others.
The labels feed a Kalman filter on one shift for the whole screen, the way I would follow a slow bias on any sensor. It took two tries to get there. A single offset nudged by every click fixed my usual posture, from 2.6° to 1.2° with three clicks, and made the near and far ones worse. Then a burst of clicks in one corner bent the whole screen. Now the network fine-tunes live only when the labels cover at least four of nine zones of the screen, and clicks and the caret move only the shift. Replayed on the button test's trials, five clicks took the error from 1.67 to 0.97°, and the nine buttons at the end of the warm-up, which feed the same filter, took it from 1.74 to 1.14° on replayed trials. Retraining the networks on every saved session of everyday use did the most for the worst case, a cold start in a close posture, which went from 5.75° to 2.43°.
Learn only from labels the model did not make.
When the pointer snaps to a button on a page, the snap is the tracker's own guess. Fed back as a label, it teaches the network to agree with itself, and an error that once pulled the dot onto the wrong button is reinforced every time it happens again. The network now fine-tunes only on clicks and the caret, and a click made by a blink does not count either, for the same reason.
The laptop's infrared camera
I was sure infrared was the way, because it is how every paid tracker works and the laptop already has an infrared camera, the one Windows Hello uses. OpenCV cannot open it, but the Windows Runtime frame interface can: 640 × 360 pixels at 15 frames a second, every other frame lit by its emitter, so about seven useful frames a second, and in the lit ones each eye shows a glint. I fed its eye patches to the network beside the colour ones and measured. After the warm-up the colour camera alone was at 1.62°, and with infrared added at 1.81°.
Fusing the two cameras made the tracker worse, so infrared stays out of the live path. My guess is that seven small frames a second had little to add to thirty large ones once the warm-up had done its job, and I have not tested it further. Without a warm-up the order flips: infrared alone is the most robust to posture and light, 3.15° against 4.26° for colour, so calibration sessions still record it for a mode that would skip the warm-up.
A long blink as the click
My first click gesture was a squint held for a moment. MediaPipe has a blendshape for it, and when I squinted its score went down, from 0.31 to 0.11, into the range of my ordinary frames. The click became a long blink instead, both eyes closed past a threshold set from my own blinks. A 25-second recording of normal and long blinks found that my normal ones last up to 0.47 s and my long ones from 1.19 s, so the threshold sits at 0.8 s, with room on both sides. The button is frozen as the lids start to close, because closing lids drag the vertical estimate down and the pointer would slide off it at the moment of clicking.
Button size against gaze error
With a click, the desktop mode could do what I wanted from the start. Windows UI Automation, the interface screen readers use, lists the clickable elements of the window in front and of the taskbar with their rectangles, and a background thread refreshes them (one window takes 30 to 240 ms). Each frame then asks which element I mean with Bayes' rule. The estimate ggg is the true gaze plus a normal error; looking inside element kkk spreads the true gaze evenly over its rectangle, and looking at anything else spreads it over the whole screen:
p(g∣k)=1∣Rk∣ [Φ(x1−gxσx)−Φ(x0−gxσx)]× [Φ(y1−gyσy)−Φ(y0−gyσy)]\begin{aligned} p(g \mid k) = \frac{1}{|\htmlData{sym=0}{R_k}|}\, &\Big[\htmlData{sym=1}{\Phi}\big(\tfrac{x_1-g_x}{\htmlData{sym=2}{\sigma_x}}\big)-\htmlData{sym=1}{\Phi}\big(\tfrac{x_0-g_x}{\htmlData{sym=2}{\sigma_x}}\big)\Big] \\ \times\, &\Big[\htmlData{sym=1}{\Phi}\big(\tfrac{y_1-g_y}{\htmlData{sym=3}{\sigma_y}}\big)-\htmlData{sym=1}{\Phi}\big(\tfrac{y_0-g_y}{\htmlData{sym=3}{\sigma_y}}\big)\Big] \end{aligned}p(g∣k)=∣Rk∣1×[Φ(σxx1−gx)−Φ(σxx0−gx)][Φ(σyy1−gy)−Φ(σyy0−gy)]
RkR_kRkthe rectangle of element kkk, from x0x_0x0 to x1x_1x1 and y0y_0y0 to y1y_1y1 Φ\PhiΦthe cumulative distribution of the standard normal σx\sigma_xσxthe error across the screen σy\sigma_yσythe error up and down
The pointer snaps to an element when its posterior passes 0.8 and lets go below 0.4. When two neighbours sit too close for the error to tell apart, they split the posterior between them and nothing snaps, which I prefer to a confident wrong pick.
On large buttons and the taskbar it works. On the menus of Windows it does not, and the button test says by how much. It lights one button among many, in blocks of shrinking size, and decides from the median dot over a one-second look which button I meant. One run after the warm-up, 56 looks, gave 23% at 0.75°, 39% at 1°, 64% at 1.5°, 88% at 2° and 95% at 3°. The curve follows from a single number. If the error follows a two-dimensional normal distribution whose median distance is that run's 0.89°, the error along each axis is σ=0.89∘/2ln2=0.76∘\sigma = 0.89^\circ/\sqrt{2\ln 2} = 0.76^\circσ=0.89∘/2ln2=0.76∘, and the share of looks that land on a button hhh tall and 2h2h2h wide is
P(h)=erf (h22 σ)erf (h2 σ),P(h) = \operatorname{erf}\!\left(\frac{h}{2\sqrt{2}\,\sigma}\right)\operatorname{erf}\!\left(\frac{h}{\sqrt{2}\,\sigma}\right),P(h)=erf(22σh)erf(2σh),
which lands within seven points of every measured size.
Windows menu items0%25%50%75%100%0°0 px0.5°31 px1°61 px1.5°92 px2°122 px2.5°153 px3°184 px3.5°214 px4°245 px2°: 88%3°: 95%one menu item28 px: 24%share of looks on the right buttonheight h of a button h tall, 2h widemeasured, 56 looksmodel: normal error, median 0.89°Windows menus, 24 to 32 pxmodel, a menu item
One targets run of mine after the warm-up: 56 looks, each decided by the median dot over a one-second dwell and snapped to the nearest button of a grid of buttons h tall and 2h wide. The line assumes only a two-dimensional normal error with the run’s median, 0.89°, and lands within seven points of every measured size.
A Windows menu item, 28 pixels or 0.46° tall, has neighbours only above and below, so only the first factor counts, and it gives one item a 24% chance. For nine picks in ten the median error would have to fall to about 0.16°, and even the best infrared figure above would pick a menu item only six times in ten. So the fix I have planned enlarges the target instead: a magnifier that blows up the region I look at, where I pick at ease.
The button test: one lit button among many, in blocks of shrinking size.The moving-ring test after a warm-up: 1.46° median while following a ring, with the live dot shown half the time and hidden the rest.
A base trained on strangers
Everything so far starts from networks trained on about thirty sessions of mine. Anyone else would start from nothing, so the last piece was a base trained on other people. MPIIFaceGaze holds photographs of 15 people taken by their laptop webcams, each labelled with the true 3D direction of gaze. I ran ten of them through exactly my pipeline, MediaPipe's landmarks and the same warped patches, and trained the same network to match their angles, turned by their head pose, with no screen involved, so people with different laptops can be mixed. This is transfer learning on a small scale. A newcomer's warm-up fits the geometry of their screen and fine-tunes the top layers for 16 rounds while replaying the strangers' patches, which keeps the fine-tune from wiping out what the base knew, an old guard against catastrophic forgetting. Replay took the error from 2.19 to 1.86° in the same test.
I have only tested it with myself playing the newcomer, a base that never saw me plus my 80-second warm-up, on five held-out sessions: 1.61, 1.80, 1.48, 1.93 and 3.80°, a median of 1.80° against 1.34° for my own network. On five of the dataset's people it never saw, calibrated with 150 images each, the error is 4.7 to 4.8°, their photographs being harder than a webcam at arm's length. A second real person has not sat in front of it yet.
The ledger of changes
Every change that stayed in the project moved a measured number on my own sessions, and the figure gathers the ones that moved it most, with the one that moved it the wrong way.
warm-up, a posture never calibrated4.31.1 to 1.2retraining on everyday use, cold5.752.43eye pixels instead of landmarks2.91.9nine-button check, trials replayed1.741.14five clicks, trials replayed1.670.97recalibrating for 45 s instead of 101.771.33smoother, on a moving target1.781.36five crops averaged per frame1.451.35strangers replayed, as a new user2.191.86infrared fused with colour1.621.810°1°2°3°4°5°6°beforeafter, keptafter, rejectedmedian error, degrees
Median gaze error in degrees before and after each change I measured, on my own sessions or on replayed trials, as recorded in the project’s commit messages. Posture was the largest lever: the two changes aimed at it moved the error by about three degrees each, and none of the rest by more than one.
The ideas that did not survive are worth listing so nobody retries them blind, me included:
Infrared fused with colour1.81° against 1.62° for colour alone, after the warm-up.
3D normalisation pretrained on MPIIGazeLost to the plain warp on the eye corners.
Larger eye patchesNo clear gain over 60 × 36.
Calibration-aware trainingA tie with the plain training.
A change-point filter for the correctionIdentical to the current filter on real and synthetic jumps.
Meta-learning (Reptile) for faster warm-upsA tie.
Synthetic postures by rotating the cameraNo information, since the patch changed by 0.35 of 255 grey levels.
A loss built on fixation physics, on unlabelled videoThe outputs collapsed.
One ray per eye, fused by confidenceBetter cold, slightly worse after the warm-up.
Head pose as an extra input to the networkA tie.
Clicking by squintingThe squint score fell when I squinted.
Published weights and next steps
The code is under the MIT licence. I publish my two eye networks, 0.9 MB under CC BY 4.0, and they are only the networks: none of the images or sessions they were trained on leave the laptop. The strangers' base is 21 MB under CC BY-NC-SA 4.0, the licence of the dataset it comes from. Both files are Python pickles, which can run code when they load, so the download checks each one's SHA-256 against the value in the code and refuses anything else.
What comes next, in the order I expect to do it:
The magnifier for small targets, because the menus are where I wanted to use this in the first place
Retraining by itself, on a schedule, instead of when I remember to run it
A second and a third person, and a base trained on many more faces, with and without glasses
The same tracker in a browser
The strangers' base also runs in a page that anyone with a laptop can open. MediaPipe has a web build of the same face landmarker, and the two eye networks are small enough to write out by hand in JavaScript: three convolutions and three dense layers each, 0.9 MB of weights, about 7 ms per network and frame in a background worker. On 24 MPIIFaceGaze photographs the page and the Python pipeline agree to 0.02° on the eye angles, so the port adds no error of its own.
What the page cannot do is the warm-up's fine-tune, sixteen rounds of backpropagation through the top layers with the strangers replayed. It fits less: a single pixel size, the centre of the screen, the eye's offset and the distance scale, each held inside physical limits, plus a straight-line correction of the network's angles, thirteen numbers per network solved by least squares in about a tenth of a second. A first version let the screen mapping move freely and it cheated, fitting the warm-up with a screen whose pixels came out three times their real size, and failing on the next session. With the limits in place the page's calibration matches the desktop one: on my five held-out sessions, with the base that never saw me, 1.65, 2.08, 1.55, 2.36 and 2.96°, a mean of 2.12° against 2.12° for the full fine-tune. Before the demos it measures the visitor's own error on nine buttons the fit never saw.
The page asks for the camera, runs MediaPipe and both networks in the tab, and sends nothing anywhere. It has been tested end to end only with a recorded video standing in for a camera; the first real faces in front of it will say whether a degree on mine is a degree on theirs.