Forecasting visual auras from event streams: a simulation-first study

A simulation-first rehearsal for forecasting a visual epileptic aura from the small visual events that may precede it, and a stricter measurement of what a model would actually be allowed to claim.

When people hear epilepsy they picture the big obvious version, the full seizure with the whole body involved, and that picture covers a small corner of the space.

A lot of epilepsy is far more local. A small region of the brain becomes unstable for a while, and depending on where that happens the output looks completely different: spread into motor areas and you get the version people know; stay near the visual system and you get something much quieter from the outside and very vivid to the person having it. This project lives in that quieter regime.

The relevant event is a bright point in the field of view, planted where the gaze wants to go, like a small, invasive white sun refusing to move. For the person it happens to, whatever they were doing (reading, writing, working) is over for a while.

I am going to avoid pretending this is a neurology lecture. I am not a neurologist, and for this project the exact boundary between migraine aura , epileptic aura , cortical spreading depression, inhibition, excitation, GABA, glutamate, etc. is beside the point.

For the data, the useful version is much simpler: before the real aura there are often smaller visual events. Tiny flashes, brief interruptions, little points of light. The proper words are something like phosphenes or elementary pre-aura visual events (EPVs from here on).

Most EPVs go nowhere. A flash appears, disappears, and the system returns to baseline. But sometimes they stop looking like independent little glitches. They begin to cluster, the spacing between them shortens, the density goes up.

That is the premise. Whatever signal exists lives in the local statistics of the sequence (event density, clustering, shortening inter-arrival times) rather than in any one flash.

In statistical terms this is close to a Hawkes-process problem: a sequence of events where each event can temporarily raise the probability of the next one. The bet is that EPVs are weakly self-exciting. One event makes the next slightly more likely, the sequence gets denser, and the system moves closer to a transition.

The question is whether that density contains a learnable pre-ictal signal, or whether it only feels predictive because the human brain is very good at drawing a line backwards after something scary has happened.

The important word is whether. Seizure prediction is famously unsolved, and whether a clean pre-ictal state exists at all is still the disputed part, so the job here is to measure how much predictability the stream carries and to report that number even when it is small.

One caveat governs everything below: the data are synthetic event streams, generated from a deliberately small phenomenological model. That keeps the result from masquerading as physiology, and it leaves the engineering question standing, which is hard enough on its own: if a world contains sparse self-exciting visual events, rare transitions, session-level fragility, and nuisance context, can a small forecasting pipeline recover the part of the signal that is actually recoverable, without fooling itself? That question is the experiment.

Simulation first

The project starts with a world for the model to be wrong in, and with the argument for why that world is allowed to be fake.

Why the first pass is a rehearsal

The target is rare, subjective, noisy and only partly observable, so a first pass cannot be a discovery paper. It is a modeling rehearsal.

A rehearsal sounds like a demotion, and it is the useful version of the project, because a rare-event predictor fails in boring ways long before it fails in interesting ones. It leaks across time. It learns a session-specific artifact and calls it physiology. It leans on a plausible feature because the feature sounds like a trigger. It reports ROC-AUC as if that were a warning system. It adds a neural network because the word sequence appears in the problem statement. So the first thing built is the little world in which the predictor is allowed to embarrass itself.

A microscope shows you the world; a simulator shows you your assumptions. Mix the two up and every conclusion is fake. Keep the boundary clean and a simulator earns its keep for the one reason a microscope cannot: you know the data-generating process, so you can ask whether your analysis is disciplined enough to recover only the claims it deserves.

The literature keeps demonstrating why (the papers are at the end; here I only need their headline). Serious seizure-forecasting work is full of models that beat a random predictor in one setting and quietly deflate in another. Temporal models can find signal in EEG, and that signal is patient-specific, validation-sensitive, and far easier to overstate than to use. LSTMs are fine. The validation protocol carries almost as much weight as the architecture, and it is the part that keeps getting skipped.

What synthetic data can and cannot do. It can test a pipeline, compare modeling choices, expose leakage, estimate sample-size pressure, and define which claims are off limits. It cannot discover a clinical fact. A model that works on data shaped by my own assumptions has learned nothing about the brain; it has cleared a sanity floor, meaning that in a world where the signal exists the pipeline can see some of it. A low bar, and the one most personal ML projects quietly step over.

Building the simulator

The simulator has three moving parts.

The first is the baseline rate. EPVs can happen even when nothing is building. A flash appears, disappears, and the system returns to baseline. In point-process language, this is the boring part.

The second is self-excitation. After an EPV the probability of another EPV is briefly higher, not forever, not deterministically, and not enough to make every flash meaningful. Just enough to create bursts. This is the Hawkes intuition from the intro: each event leaves a small temporary dent in the future, and the dents can stack.

The third is session-level fragility. Some sessions are simply more loaded than others. Poor sleep, longer wake time, fatigue, or an already-irritable visual system do not create the next event directly. They change the slope of the hill. A burst on a fresh day and the same burst on a loaded day should not be treated as the same object.

That gives the simulator a simple causal shape: EPVs raise short-term intensity; accumulated intensity raises fragility; fragility raises the probability that a dense local cluster tips into a full aura. Then random timing noise decides whether the possibility actually lands in this window or not.

That last clause carries the design. The generator is noisy because the question being simulated is noisy. Without noise the task is too clean; with too much of it no learner can do anything. The interesting regime is the annoying middle: enough structure that a simple model can beat chance, enough randomness that no model gets to feel prophetic. The simulator lets the model watch the weather forming, never the lightning bolt.

The risk map: each simulated session as a trajectory through the density-load space; a few climb toward the danger corner. The phase portrait: how the fast clock (recent density) and the slow clock (accumulated load) push each other around.

The risk map and phase portrait summarize the latent state. The raw material underneath them is the event stream itself, one session per row:

The event raster: EPV streams stacked session by session, so bursts and quiet stretches line up across days.

What the model actually estimates. A local hazard, never the next flash: given the recent density of flashes and the slower state of the session, how dangerous is this window compared with a quiet one? Hazard can be real while exact timing stays stubbornly uncertain, and a good part of this article turns on that difference.

Why more synthetic data isn't more evidence

The tempting objection writes itself: if the data are synthetic, why simulate scarcity? Why not generate a million windows and let the LSTM win or lose cleanly? Because more synthetic rows shrink the Monte Carlo error and leave the epistemic error exactly where it was. A million windows from the same simulator tell you the simulator more precisely; they do not make the simulator more true.

So the synthetic cohort is deliberately kept in the small-data regime, and the question becomes the practical one: under the evidence budget a real logging project would actually have (few positives, correlated windows, uneven sessions), which claims survive? The scarcity is part of the stress test rather than a limitation inherited from reality. Large runs are used only to check the stability of the comparison, never as extra evidence about biology.

The features

Turning events into features

You cannot hand a model a flash happened at 14:32. You hand it numbers sampled on a clock. So the first modeling decision comes before any model: which number should stand in for the system is getting excited right now.

The main feature is a recency-weighted event rate: an exponentially-weighted moving average over the EPV stream. Each event bumps it up; between events it decays. Recent flashes count for a lot, old ones fade. A single isolated flash barely moves it; a cluster pushes the curve up and keeps it there.

In the language of the simulator, that average is an observable proxy for the Hawkes intensity, the running estimate of how self-excited the local stream has become. The premise that events matter when they cluster becomes one number the model can read.

That number is paired with a slower clock: accumulated load. The fast clock asks whether a burst is happening now. The slow clock asks whether the session has been simmering long enough for the same burst to matter more. A short dense burst on a fresh system and the same burst after an hour of subthreshold activity are different objects. One is a spark, the other a spark in dry grass.

The two clocks reading one EPV stream (top), with six more sessions for scale (bottom). Each tick is an EPV; the fast clock (recency-weighted density, cyan) spikes and decays, while the slow clock of accumulated load (amber) creeps up underneath. Aura sessions climb into a racha before the red bar; calm ones never do. The figure draws the hypothesis rather than evidence for it: that auras follow a racha is built into the simulator, so what you see is the assumption the model is asked to recover.

The session scatter: every window placed by its two clocks. The rare positive windows are the bright cluster drifting up and to the right, exactly the corner the model has to find.

Around those clocks sit a few context variables: sleep, wake duration, screen exposure, session metadata. They are allowed in because a real logger would collect them, and kept from dominating because the simulated mechanism is event-centric on purpose. The claim under test is narrow, local EPV density may contain a weak pre-aura signal, and a small hypothesis is easier to kill.

Adding a negative control

The most dangerous feature in the project is also the most psychologically satisfying: screen-context switching.

Laptop to phone. Phone to monitor. IDE to browser. White page to dark terminal. It feels like the villain: modern, visual, jittery, exactly the sort of thing a person who works on screens expects to blame. So it goes into the simulator, hard, and it goes in as a negative control.

A negative control is a feature plausible enough to tempt you, built so that it carries no predictive information. Context switching looks realistic as a behavioral stream and is never wired into the aura hazard. If the model gives it weight anyway, the pipeline has a problem: it is grabbing correlation-shaped noise because the dataset is small and the feature sounds good.

The pipeline rejects it. It receives no stable weight, does not survive feature selection, and does not improve session-blocked validation (though here I am getting ahead of the project, because both of those procedures come in the next two sections). I am giving the decoy's verdict before the machinery that delivers it, because a negative control is easier to hold as a goal (a feature that should fail) than as a result buried three sections later.

You can also just look at it. Stacked one row per simulated day, with every aura drawn as a red bar, the behavioral streams and the auras share a screen but not a rhythm: the blue typing trace and the amber window-switch marks wander everywhere, and the red bars fall where they fall. Nothing in the behavior lines up with the moments that matter, which is exactly what a decoy is supposed to look like.

None of this says anything about whether screen switching is a real trigger; I wrote the answer into the world myself. The conclusion is smaller and more useful: given a plausible decoy, the feature-selection procedure does not automatically crown it. A real version of this project will be full of features that feel meaningful (brightness, caffeine, sleep debt, stress, notifications, posture, ambient light), and the question is whether the pipeline can say no.

A variable can be useful precisely because it should not work: it is a trap you set for your own model. If the model starts believing in the decoy, that is telling you something ugly about leakage, feature selection, or sample size, and refusing it is one of the few ways a small pipeline can earn trust.

Pruning the feature set

At the start the candidate table has more columns than the final model deserves: the clocks, the context, simple counts, recent maxima, inter-arrival summaries, session age, and the decoy. Eleven candidates is fine for exploration.

Eleven is too many for the evaluation you actually want to believe, and this is the part people get backwards with synthetic data. Because the generator can produce unlimited rows, it is tempting to keep the feature set wide and drown the uncertainty in volume. But the benchmark is tied to a future logging regime where positives are rare, and the effective sample size sits much closer to the number of independent sessions and positive transitions than to the number of rows. Consecutive windows from the same session do not give you eleven independent chances to learn eleven effects. They give you one correlated stretch of evidence sliced into many rows.

So the pruning happens before looking at the outcome. Redundant features go, by correlation and by meaning: if two columns are two ways of saying recent density, one survives. If a feature is a post-hoc summary unavailable at prediction time, it is out. If a feature exists only because it was easy to log, because the spreadsheet has it does not count as a reason. What is left is five knobs: the fast clock, the short-term density slope, accumulated session load, sleep debt, and time awake. The exact count matters less than the discipline. The model is forced to stay small enough that a human can still argue with it.

The eleven candidates correlated against each other. The fast-clock family in the top-left block is almost the same column written six ways. Burst and seconds-since-event are anti-correlated at -0.98. That redundancy is why eleven collapses to five: keep one reading of recent density and drop its synonyms. The decoy, window-switching, correlates with nothing, exactly as it was built to. In rare-event work, feature selection is debt control. Every extra variable is a loan against evidence you probably do not have, and in a simulation study it is worse, because extra features are extra assumptions with column names.

Validation and models

Splitting the data

The easiest way to make this project look impressive is to split the data by row. It is also the easiest way to make the number meaningless.

Rows inside the same session are siblings. A window ending at 14:45 and a window ending at 15:00 share the same simulated day, the same slow load, the same context, and often most of the same events. If one lands in training and its near-twin lands in testing, the model is recognizing a cousin, and the metric comes back beautiful and means nothing.

So the split is by session. A whole session goes into training or into testing, never both. Five folds, stratified so the rare positives do not all pile into one fold. The bootstrap resamples whole sessions too, because rows are not independent evidence units. Resampling rows would produce narrow intervals and false confidence; resampling sessions gives uglier uncertainty and a number worth reading.

Unglamorous, and decisive: once the split is honest, the scores get less exciting and more useful.

Why the split is by session. In a time series, data leakage does not need to look like cheating. It can look like a clean random split. But if near-duplicate windows from the same session appear on both sides, the model has already seen the shape of the test case. Session-level splitting is the minimum condition for the metric to mean anything, well short of conservative.

From simple baselines to the LSTM

Before the LSTM, there is a threshold. If recent EPV density crosses a fixed line, raise the flag. No memory cell, no hidden state, no learned representation, just the claim in its most primitive form: dense EPVs are more dangerous than sparse ones. It is too stupid to be the final answer, which is exactly why it belongs first. It gives every later model something humiliatingly simple to beat.

The second model is logistic regression over the five retained features. Still boring, still readable: you can look at the coefficients and ask whether the model uses the world the way the simulator actually works. The fast clock should matter. The slow clock should matter. The negative control should not. If the signs come out strange, the representation or the validation is wrong, and no LSTM fixes that.

In this scarce synthetic benchmark, the logistic model lands around ROC 0.72.

Take one pre-aura window and one calm window at random, and ask the model which is riskier. It picks correctly about 72 times out of 100. A coin gets 50. A perfect oracle gets 100. So there is real signal inside the toy world, clearly more than chance, and nowhere near something you would trust as a warning system.

The logistic model's decision surface over the two clocks. Risk rises up and to the right (more recent density, more accumulated load), and the red pre-aura windows do lean that way. But look at the overlap: most of the field is mixed blue and red. That is ROC 0.72 up close: a real tilt, and no clean line.

The rarity makes it harsher than the number sounds. Even when the model concentrates risk, most alarms are still false, because most windows are non-events. The top-risk windows are enriched for positives, never purified. It is a metal detector that triples your odds in a field where almost every beep is still a bottle cap. A model can be directionally right and operationally useless at the same time.

So you reach for the LSTM. It is the tempting model because the data are sequential, and the temptation is reasonable: a stream of events really does have temporal structure, and a recurrent model is built to consume it. If you want one assembled gate by gate from a single neuron, that is LSTMs from scratch. Here it is just the obvious next thing to reach for. You would bet on it: a sequence model reading the raw stream should beat a four-feature logistic regression. That is the entire promise of deep learning.

It does not pay rent.

The LSTM learns the simulated stream without trouble; generalization under the evidence budget is where it comes apart. With few positive sessions, the extra capacity mostly buys more ways to memorize local accidents. Under session-blocked cross-validation, the heavier model shows no reliable edge over the small logistic model, and its apparent advantage over the threshold rule is unstable. The handcrafted clocks are already close to sufficient statistics for the simulated mechanism, while the LSTM has to learn those clocks from scarce, correlated windows. It has more freedom than evidence, so it spends the freedom badly.

Training loss keeps dropping while validation loss bottoms out early and drifts up. The network is memorizing scarce, correlated windows instead of the mechanism. *More freedom than evidence*, made visible: the gap between the curves is the overfitting the bake-off then pays for, and early stopping is taken at the validation minimum, not the training one.

And to rule out that the network was simply mis-sized or under-trained, I built both LSTMs the way you would if you wanted them to win. Each reads the raw EPV stream as a sequence instead of the handcrafted clocks, trains with a class-weighted cross-entropy so the rare positive windows are not drowned by the majority, and uses early stopping on validation PR-AUC (not training loss) with the stopping fold held out under the same session-level split, so model selection never peeks at a test session. A small network at 16 hidden units, and one at roughly double the capacity, 32. Neither earns its keep. Scored by PR-AUC (the metric that actually respects rare positives) and averaged over the session-blocked folds, the picture repeats across every forecast horizon: the four-feature logistic is the tallest bar, the one-feature rule sits just behind, and both LSTM sizes trail, worst in the hardest 5–15 minute window.

The bake-off. In all three horizons the four-feature logistic (cyan) is the highest bar; the two LSTM sizes (orange, red) never clear it and collapse hardest in the 5–15 minute window. Doubling the LSTM's capacity did not buy more skill, only more ways to overfit a scarce, correlated dataset. And mind the axis: even the winner sits near PR-AUC 0.1, which is real signal and still nowhere near a usable alarm.

The bootstrap interval on the logistic model's edge over the dumb threshold crosses zero, and the LSTM is no cleaner: in this finite simulated study I cannot rule out that the cleverer models add nothing meaningful over a density rule.

The verdict lands on this evidence regime and on nothing else (LSTMs, brains and aura forecasting all walk away unjudged). It is the cleanest result in the project because it flatters nobody and because it travels: at this sample size, capacity buys overfitting rather than skill, which is a fact about the volume of evidence and would hold in any domain.

Revealing the hidden state

There is one more useful abuse of the simulator, the experiment no real study gets to run.

Because I built the world, it has a hidden variable a real logger would not have: the session-level latent fragility, the thing that decides whether a day is generally dangerous before any local window is considered. So I can cheat and hand that latent state directly to the model. Give it the variable it normally has to infer, and ask how much the score improves.

The score barely moves.

That says nothing about real brains and a good deal about this generator, which is still worth knowing, because it tells you where the simulated difficulty lives: even once the session is known to be loaded, the exact window keeps its stochastic timing. A loaded die tells you the distribution, never the next roll. Under this data-generating process, better access to session fragility does not solve event timing.

Fragility versus timing. Predicting that a system is fragile and predicting the moment it breaks are different jobs. Most risk models do the first well; this project asks for the second. In this simulator most of the remaining error lives in timing, which proves nothing about reality and does map the wall for the next real experiment: if richer sensors suddenly improve timing, the simulator was missing a driver; if they do not, the bottleneck may really be short-horizon stochasticity.

Taking stock

What the EEG literature contributes

I have saved this literature for last, but in the order things actually happened it came first. These are the papers that talked me out of a discovery paper and into a rehearsal. The serious seizure-forecasting literature comes at the problem from the other side (real neural recordings), and the comparison calibrates expectations rather than borrowing authority.

Payne et al. used long-term intracranial EEG from the NeuroVista dataset, passed one-minute segments through CNN/LSTM models, and forecast seizure-onset windows across several horizons. Performance was above a random predictor, but it varied patient to patient: better for some, worse for others. Chambers et al. later tested LSTMs directly on unprocessed intracranial EEG, again reporting above-chance prediction under a carefully defined framework rather than a solved problem.

That matters because this project works with a poorer signal, subjective visual events instead of electrodes. If deep models over real neural measurements still need careful patient-specific evaluation, a personal EPV stream has no right to expect a clean predictive dial. What transfers is the structure: both stories resist the fantasy of a green-to-red dial, and both produce the same flavor of answer, some signal, lots of uncertainty, heavy dependence on validation choices, and no permission to skip the hard prospective test.

It is also why I like the single-subject framing as a target. If subjective visual precursors are useful at all, they are likely to be personal. The relevant distribution is one person's private stream, their EPVs, their sleep, their wake duration, their threshold, their logging discipline, and only distantly people with epilepsy in the abstract. A population model may be scientifically richer. A personal hazard model may be practically closer to what someone in that situation actually wants, at the cost of one brutal requirement: enough personal data, collected prospectively by whoever has the stream, before any conclusion is earned.

What the synthetic pass actually buys

There is a fair objection to all of this: synthetic data cannot tell you whether a real aura is predictable. Was that not obvious from the first line?

Yes. If the whole conclusion were go get real data, this would be a long way to say nothing, and the residue is more specific than that.

First, the pipeline can reject a plausible decoy. Future real logs will be full of emotionally satisfying features; without negative controls you cannot know whether the model is learning physiology or your narrative.

Second, the simple clocks are strong baselines. Any future deep model has to beat them under session-level validation, and looking better under a row split does not count. That saves a lot of wasted neural-network theater.

Third, the relevant metric is a bundle: the score, the bootstrap interval, the base-rate behavior, and the edge over a dumb rule. A model that beats chance and fails the operational threshold is a correct measurement, and a paper is allowed to say so.

Fourth, the sample-size question becomes concrete. The next pass needs more independent sessions, more positive transitions, cleaner event timestamps, and labels recorded before the analysis has a chance to rewrite them.

Go get this kind of real data, split this way, with these controls, and expect this much wobble is an instruction you can act on, and it is what the synthetic pass buys.

What this was worth

So, can you forecast a visual aura from the little flashes that seem to precede it?

In the simulator: a little. Better than chance, not enough to trust, and not clearly improved by an LSTM under honest validation. In the world: unanswered. That boundary is the point.

What the project produced is a pipeline harder to fool than the first version of the idea: it builds the right clocks, keeps the sample scarce on purpose, splits by session, uses a negative control, compares against a dumb threshold before trusting a neural network, bootstraps uncertainty at the right level, and, above all, knows which conclusions it is not allowed to make. That is the instrument. The predictor can wait.

One part does not fit in a methods section. I wanted this to work. There is a particular pull to a problem like this (the sense that the signal is right there, that one more feature or one more layer will finally surface it), and that pull is exactly the thing a rare-event study has to be built to resist. Most of the discipline in this writeup, the session splits and the negative control and the dumb baseline I kept in the report even when it embarrassed the network, is really scaffolding to protect the result from the person who wanted a different one. The number I trust most is the one I was least hoping for.

A real version would be straightforward and annoying to run, which is usually a good sign. One button per event, pressed by whoever actually has the stream. One session boundary. Sleep and wake duration recorded before the outcome is known. Everything the synthetic pass just argued for (session-level validation, the negative control, the fixed feature budget, the dumb baseline kept in the report even when it embarrasses the LSTM) moved from principle to protocol. Then ask whether the real stream lands above the synthetic sanity floor.

I am a systems person and not a clinician, and a neurologist would probably frame the whole thing differently. But the engineering lesson travels past brains.

You do not get to build the dial until you have earned the right to trust the needle.

This is the writeup of trying to earn that right in a small simulated world first, and of finding that even there, under honest scarcity, the needle barely moves. The measurement did its job. Most of seizure prediction is exactly this honest; most of it just does not say so out loud.

References

Payne, D. E., Chambers, J. D., Burkitt, A. N., Cook, M. J., Kuhlman, L., Freestone, D. R., & Grayden, D. B. (2023). Epileptic seizure forecasting with long short-term memory (LSTM) neural networks . arXiv:2309.09471.

Chambers, J. D., Cook, M. J., Burkitt, A. N., & Grayden, D. B. (2024). Using Long Short-Term Memory (LSTM) recurrent neural networks to classify unprocessed EEG for seizure prediction . Frontiers in Neuroscience, 18, 1472747.

Bernabeu, A., Zhuang, J., & Mateu, J. (2025). Spatio-Temporal Hawkes Point Processes: A Review . Journal of Agricultural, Biological and Environmental Statistics, 30, 89–119.

Reinhart, A. (2018). A Review of Self-Exciting Spatio-Temporal Point Processes and Their Applications . Statistical Science, 33(3), 299–318.