Flood-filling network · Lobeworks/17

A flood-filling network is a recurrent three-dimensional convolutional network that segments a volume one object at a time: starting from a seed inside a cell, it repeatedly predicts which voxels around it belong to that same cell, feeds its own prediction back in as an input, and moves its field of view along the obje

Flood-filling network. A flood-filling network is a recurrent three-dimensional convolutional network that segments a volume one object at a time: starting from a seed inside a cell, it repeatedly predicts which voxels around it belong to that same cell, feeds its own prediction back in as an input, and moves its field of view along the object until the object ends.

Segmentation gives every voxel of an electron microscopy volume the identity of the cell it belongs to, and it is the step that turns grey images into a connectome. The older pipelines first marked membranes and then grouped the regions between them, so one stretch of membrane that failed to stain let two cells run together. A flood-filling network starts like the paint bucket of an image editor, from one point, and differs in the rule for growing: a convolutional network looks at a small cube of the image and at its current guess of which voxels in that cube belong to the object it is tracing, and returns a better guess. When the guess reaches a face of the cube with high confidence, the cube moves that way.

The second input is what makes it work. Knowing which object it is following, the network treats an ambiguous gap in a membrane as a gap in a wall it is walking beside, and stays on its side.

Its errors are measured as distance. On a volume of zebra finch brain it traced an average of 1.1 mm of neurite before the first mistake and made four merges in 97 mm of test path.

It is expensive. Each object is filled separately with overlapping fields of view that pass over the same voxels many times, which trades computation for fewer errors that a person would otherwise have to find.

It built two fly connectomes. Google segmented the hemibrain and the male central nervous system with it; FlyWire used a different pipeline that predicts affinities between neighbouring voxels and joins fragments by their mean affinity.

A merge costs far more than a split.

A split leaves a cell in pieces that are easy to join; a merge credits one cell's synapses to another and wires the graph wrong, so a method that avoids merges at the price of computation saves the most expensive resource, human proofreading.

Questions: How does a flood-filling network get past a gap in a membrane without leaking? It is never asked whether a voxel is membrane in general; it is asked whether the voxel belongs to the object it is already tracing. Its input is the image and its own current guess of the object, so at a gap where the stain failed it can see that the shape it has been following continues on one side and stays there. A method that first marks membranes and then floods the regions between them has no such memory, and pours through the hole into the neighbouring cell. How does a proofreader find a merge in three dimensions? By looking for anatomy that cannot exist. A segmented neuron with two cell bodies is almost always two neurons joined, a branch that stops in the middle of tissue is almost always a split, and a flat flange or a piece that jumps across the brain is suspect. The proofreader then goes to the electron micrographs at that spot, finds where two strands were confused, and cuts or joins. Why is a merge worse than a split in a connectome? A split leaves one neuron in two pieces, which a proofreader sees at once as a branch ending in the middle of tissue and repairs by joining them. A merge fuses two neurons, so every synapse of one is credited to the other and the graph is wired wrong in a way that can look plausible. Segmentation methods are judged largely by how few merges they make, and flood-filling networks trade a great deal of computation for that. Why is neuropil the hardest tissue to segment? Because its processes are thinner than 100 nm, packed together and run in every direction. A membrane that failed to stain for a few nanometres can let a segmentation pour from one neurite into the next, and a process running along the plane of the cut is smeared across one section. Flood-filling networks and proofreaders spend most of their effort there.