A latent diffusion model that invents kanji, trained on a laptop CPU

Stable Diffusion's recipe (an autoencoder, a frozen text encoder, a denoiser with cross-attention, classifier-free guidance) scaled down to 64-pixel glyphs so that an eight-core laptop CPU trains it in about six hours. Each choice of size, loss and schedule is explained with the alternative it replaced, and the model is evaluated against its own training set: how often a dictionary word brings back its kanji, how varied the samples of one prompt are, and whether any of 4,908 generated characters is a copy.

The program takes an English word and draws a kanji for it. For rain it draws 雨, for tree 木. For internet, which no kanji means, it draws a character under 門, the gate, with something of its own inside, and for algorithm a character on the thread radical 糸. None of those last characters exist. That is the point of the project: a model that has learned how kanji are built, from strokes up to which component carries which meaning, and that can be asked for a character the writing system never made.

Characters drawn by the model for words with no kanji, three samples per prompt at guidance 3

The idea is not mine. In January 2023 David Ha shared a Stable Diffusion model trained on images of kanji that came up with characters for skyscraper, Pikachu, YouTube and Elon Musk, and he had done something similar with a recurrent network on KanjiVG strokes back in 2015. That model was a full Stable Diffusion, pretrained on billions of photographs and then fine-tuned on kanji. This one rebuilds the same recipe from scratch at a size a laptop CPU can train, which makes every part of it small enough to open and look at, and it sets up the experiment the rest of the write-up runs: what a model knows about kanji when kanji is all it has seen, and what changes when it starts from a model that has seen the world.

A personal note24 · sep 16Read noteFirst try: a fine-tune of Stable Diffusion in a notebook, against a Google Drive cache. It never trained. The loss compared the prediction with the noisy image instead of the noise, so no amount of compute would have fixed it.

Why the 2024 version never trained

The repository from September 2024 had the shape of a diffusion project and none of the working parts. It ran the denoiser directly on pixels with no autoencoder, its loss compared the network's output against the noisy input (a model trained that way learns to copy its input, which is useless at sampling time), its configuration referred to a key the code never defined, and its data path pointed at a Colab cache that no longer exists. Fixing it piece by piece would have kept its first decision, a full-size Stable Diffusion fine-tuned on 512-pixel images, and that decision is what made it unaffordable on the machine at hand: an AMD Ryzen AI 7 PRO 450 with eight cores, 64 GB of memory and an integrated Radeon that PyTorch cannot use on Windows1.

1There are no ROCm builds of PyTorch for Windows that support the gfx1152 graphics core of this chip, so everything ran on the CPU, at roughly 200 GFLOPS in fp32. Mixed precision and channels-last layouts gave no speedup.

The rewrite kept the architecture and dropped the scale. Every number below comes from that constraint: a budget of about a second of CPU per training step, which allows a night of training to see each character a hundred times or more.

Stage Stable Diffusion 1.x This model

Image 512×512, colour 64×64, one channel of ink

Autoencoder latent 64×64×4, downsampling 8 16×16×4, downsampling 4

Text encoder, frozen CLIP ViT-L/14, 77 tokens CLIP ViT-B/32, 32 tokens

Denoiser U-Net, 860M parameters U-Net, 17.4M parameters

Training target the noise ϵ\epsilonϵ the velocity vvv

Sampler PNDM or DDIM, 50 steps DPM-Solver++, 30 steps

Data: 6,413 characters drawn from their strokes

KanjiVG (Ulrich Apel, CC BY-SA 3.0) describes each character as SVG paths, one per stroke in writing order. KANJIDIC2 (the Electronic Dictionary Research and Development Group, CC BY-SA 4.0) gives each character its English meanings. The characters present in both, leaving out KanjiVG's stylistic variants, are 6,413.

The glyphs are drawn from the stroke paths instead of from a font, for two reasons. A font brings its own design (serifs, brush endings, contrast between thick and thin) and every character in it shares that design, so a model trained on one font learns the font as much as the writing system; strokes of one uniform width with round ends leave only the structure. And the stroke data is openly licensed, which a commercial font usually is not. The renderer samples each Bézier segment of a stroke at sixteen fixed points, draws the polyline at four times the target size and downsamples, which takes under five minutes for the whole set on a process pool1.

1The first renderer walked each curve by arc length with a pure-Python SVG library and took over half an hour; at 64 pixels the two are indistinguishable.

Sixteen of the 6,413 training characters as rendered from their strokes, with the first KANJIDIC2 meaning under each

The resolution is 64×64 because it is the smallest size at which the densest characters stay legible. The strokes are about three pixels wide at that size, so a character with twenty strokes still has a pixel of white between most of them, while 32×32 merges strokes in anything past a dozen. Ink covers 13% of a glyph on average, which matters for the loss of the autoencoder below.

Each character gets two captions: all its meanings joined (heart, mind, spirit, heart radical (no. 61)) and the first one alone (heart). The long one carries everything the dictionary says; the short one makes a single-word prompt at sampling time look like something seen in training. A training step picks one of the two at random. The choice has a side effect that shows up in the results: KANJIDIC2 lists 日 as day, sun, Japan, counter for days, so the short caption the model learned for 日 is day, and the prompt sun is not one it ever saw alone.

The model

The model is three networks with separate jobs, trained in two stages. Each stage optimises its own loss, and each loss reaches only the networks that are trained in that stage.

Training and sampling. Stage 1 trains the encoder and decoder to reproduce a glyph. Stage 2 freezes the encoder, adds noise to the latent and trains only the U-Net to predict the velocity, reading the frozen CLIP embedding of the caption. Sampling runs the U-Net thirty times from pure noise and decodes once.

Stage 1, the autoencoder

An autoencoder is an encoder E\mathcal{E}E and a decoder D\mathcal{D}D trained as one: the encoder squeezes the 4,096 pixels of a glyph into a latent zzz of 1,024 numbers, the decoder redraws the glyph from zzz, and the only signal either of them receives is how far the redrawing x~=D(E(x))\tilde x = \mathcal{D}(\mathcal{E}(x))x~=D(E(x)) is from the original xxx. Nothing tells them what a stroke or a radical is. Whatever survives the squeeze is whatever turned out to be needed to redraw kanji.

The reason to have it at all is cost. The denoiser of stage 2 runs on every training step and, at sampling time, thirty times per character; an autoencoder trained once lets that expensive network work on a quarter of the numbers. Measured on this CPU, the same U-Net takes 21 seconds per step of batch 64 on 64×64 pixels and 1.8 seconds on the 16×16×4 latent, eleven times less, which is the difference between five hours of training and more than two days. That is the argument of latent diffusion, which splits the job into perceptual compression, done once by the autoencoder, and semantic compression, the part the diffusion model has to learn1.

1Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022, where the two terms and the trade-off between downsampling factors are introduced.

(a) The autoencoder: two stride-2 convolutions take 64×64 to 16×16, residual blocks at each resolution, a 3×3 convolution to the 8 channels of mean and log-variance. (b) The U-Net: three resolutions with 64, 128 and 192 channels, transformer blocks with cross-attention at 8×8 and 4×4, skip connections concatenated into the decoder.

Each part of the autoencoder was chosen against an alternative.

DOWNSAMPLING 4Stable Diffusion downsamples by 8. Here that would leave an 8×8 grid, and a cell of the grid would cover 8×8 pixels of a glyph whose strokes are three pixels wide and often three apart: one cell would have to describe two or three strokes at once, and fine detail is exactly what kanji are made of. The latent diffusion paper found factors 4 to 16 to be the useful range for photographs; for line drawings at this size, 4 is the one that keeps a cell to about one stroke.

FOUR CHANNELSThe same as Stable Diffusion 1, which gives 1,024 numbers in all. A latent much richer than that costs the denoiser more without the decoder needing it: reconstructions are already within half a percent per pixel.

ALMOST NO KLA VAE adds a KL term that pulls the latent towards a standard Gaussian. Weighted heavily, it blurs reconstructions, because the encoder is pushed to forget detail; left out, the latent can drift to any scale, and the diffusion model then adds unit-variance noise to a signal of unknown size. The weight is 10−510^{-5}10−5, the same order as Stable Diffusion's 10−610^{-6}10−6: enough to keep the latent well scaled, too little to blur. The latent is then divided by its measured standard deviation (0.765) so that it reaches the denoiser at unit variance.

BCE AND L1Ink covers 13% of a glyph. A squared-error loss on a mostly white image is minimised early by drawing faint grey, because a missing stroke costs little; binary cross-entropy on the ink treats every pixel as a yes-or-no decision and punishes a confident miss, and an L1 term on top keeps edges sharp.

NEAREST UPSAMPLINGThe decoder upsamples by repeating pixels and then convolving, instead of using transposed convolutions, which are known to print a checkerboard pattern on their output^[Odena, Dumoulin and Olah, Deconvolution and Checkerboard Artifacts, Distill, 2016.]. On thin strokes over a white ground such a pattern would be the most visible artefact in the picture.

Trained for 3,000 steps (43 minutes), the autoencoder misses by 0.0055 per pixel on the 2% of characters held out of training.

Six glyphs through the autoencoder, with the error magnified four times. The error sits on the edges of strokes.

What the latent does and does not know shows in a mixture of two characters. Averaging the pixels of 木 and 米 gives a double exposure; averaging their latents and decoding gives a drawing that holds 木 for the first third of the way and 米 for the last, with only a short stretch of broken strokes in between, because the decoder knows how to draw strokes and does not know how to draw half a stroke.

Mixing 木 and 米 with weight λ, in pixel space (above) and in the latent space, decoded (below).

The middle of the lower row is not a real character, and nothing in the autoencoder could make it one. It knows how kanji are drawn; it does not know which ones exist. That knowledge is the job of stage 2.

Stage 2, the denoiser

A diffusion model learns to undo noise. Training takes the latent z0z_0z0​ of a real character, picks a noise level ttt between 1 and 1,000 and mixes in Gaussian noise,

zt=αˉt z0+1−αˉt ϵ,ϵ∼N(0,I),z_t = \sqrt{\bar\alpha_t}\,z_0 + \sqrt{1-\bar\alpha_t}\,\epsilon, \qquad \epsilon \sim \mathcal{N}(0, I),zt​=αˉt​​z0​+1−αˉt​​ϵ,ϵ∼N(0,I),

and asks the network what was done. A network that can answer at every level can be run backwards from pure noise, one level at a time, until a latent is left that looks as if it came from the training set1.

1Ho, Jain and Abbeel, Denoising Diffusion Probabilistic Models, 2020, whose Figure 2 this one redraws.

The forward process q adds noise; the U-Net learns the reverse step pθ. Images are the latents decoded only to be seen.

The choices in this stage are about what the network predicts, how the noise levels are spread, and how big the network can be. The first changes the target of the loss from the noise to the velocity,

v=αˉt ϵ−1−αˉt z0.v = \sqrt{\bar\alpha_t}\,\epsilon - \sqrt{1-\bar\alpha_t}\,z_0 .v=αˉt​​ϵ−1−αˉt​​z0​.

VELOCITYThe network predicts vvv instead of the noise ϵ\epsilonϵ. At the noisiest levels ztz_tzt​ is almost all noise, so predicting ϵ\epsilonϵ there means predicting almost the whole input, a trivial target that teaches little, and turning that prediction back into a clean glyph divides by a number close to zero. The velocity is a mix that weights the clean signal more as the noise grows, so the target stays informative and well scaled at both ends^[Salimans and Ho, Progressive Distillation for Fast Sampling of Diffusion Models, ICLR 2022.].

COSINE SCHEDULEThe noise levels follow a cosine curve instead of the linear schedule of the original DDPM, whose authors' successors found that the linear schedule was sub-optimal for images of resolution 64 × 64 and 32 × 32: its last steps are so noisy that a fifth of the reverse process can be skipped without loss^[Nichol and Dhariwal, Improved Denoising Diffusion Probabilistic Models, 2021, section 3.2.]. A 16×16 latent is smaller still.

SIZEThree resolutions of 64, 128 and 192 channels, two residual blocks each, 17.4M parameters, which is what fits the budget of a second or so per step. Attention costs grow with the square of the number of positions, so it is used only at 8×8 and 4×4 (64 and 16 positions) and left out at 16×16 (256 positions), where convolutions do the work.

CROSS-ATTENTIONThe text enters through cross-attention. At each position the U-Net forms a query from what it has drawn so far, compares it with the 32 vectors of the caption and mixes in the ones that match. That lets different parts of the character read different words of the caption, which a single caption vector added everywhere would not.

CAPTION DROPOUTOne caption in ten is replaced by the empty string, so the same network also learns to denoise with no prompt. That is what makes guidance possible at sampling time; the guidance paper found that dropping 10 to 20% worked better than 50%^[Ho and Salimans, Classifier-Free Diffusion Guidance, 2021.].

EMAThe weights used for sampling are an exponential moving average of the trained weights with decay 0.999, about the last thousand steps. It smooths out the step-to-step noise of the optimiser; DDPM used 0.9999 over far longer runs.

The text encoder is CLIP ViT-B/32, a transformer trained with contrastive learning to place a caption near its image1, frozen and run once over all the captions before training, which takes it out of the training loop entirely. It is the small sibling of the ViT-L/14 in Stable Diffusion 1, with 512-dimensional vectors instead of 768, and it is frozen because 6,413 captions are far too few to teach a text encoder anything without wrecking what it already knows. Captions are cut at 32 tokens: the median caption is 7 tokens long and only 6 of 6,413 are longer than 32.

1Radford et al., Learning Transferable Visual Models From Natural Language Supervision, 2021.

The optimiser is AdamW at 4×10−44\times10^{-4}4×10−4 with batch 64, warmed up over 300 steps and decayed on a cosine. The first 10,000 steps took 3 hours and 17 minutes; the run was then resumed for 6,000 more with a second, lower cosine, which took about two hours.

Running mean of the training loss. The run was resumed at step 10,000 with a second cosine schedule to 16,000.

Sampling

Sampling starts from Gaussian noise zTz_TzT​ and runs the U-Net thirty times with DPM-Solver++, a solver that treats denoising as an ordinary differential equation and takes large high-order steps along it, so thirty evaluations replace the thousand steps of the training schedule1. At each step the network is evaluated twice, with the prompt and with the empty caption, and the two predictions are combined with classifier-free guidance,

1Lu et al., DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models, 2022.

v^=v^θ(zt,t,∅)+w (v^θ(zt,t,y)−v^θ(zt,t,∅)),\hat v = \hat v_\theta(z_t, t, \varnothing) + w\,\big(\hat v_\theta(z_t, t, y) - \hat v_\theta(z_t, t, \varnothing)\big),v^=v^θ​(zt​,t,∅)+w(v^θ​(zt​,t,y)−v^θ​(zt​,t,∅)),

which at w=1w=1w=1 follows the prompt as learned and above 1 pushes past it in the direction the prompt adds. The final latent is decoded once by D\mathcal{D}D. A batch of 64 characters takes about 20 seconds on the CPU.

Each step's prediction also implies an estimate of the finished latent, z^0=αˉt zt−1−αˉt v^\hat z_0 = \sqrt{\bar\alpha_t},z_t - \sqrt{1-\bar\alpha_t},\hat vz^0​=αˉt​​zt​−1−αˉt​​v^. Decoding those estimates shows the order in which a character is decided.

The decoded estimate of the finished glyph at steps 1 to 30 for four prompts.

The layout and a main component appear at the first step, and they are not final: the sample for internet starts as 門 and turns into a different character around the seventh step, and the one for algorithm settles only by the eleventh. After that only the ends of strokes move.

Evaluation

A model of kanji can fail in two opposite ways: by drawing nothing recognisable, or by drawing only characters it has already seen. Both are measured against the training set itself, with one similarity and three numbers derived from it.

SIMILARITYThe ink of two glyphs, thresholded and thickened by one pixel, compared by intersection over union (the Jaccard index). The thickening lets a stroke drawn one pixel off still count as the same stroke.

BASELINEFor 1,000 real kanji, the similarity to the nearest other real kanji: 0.57 at the median, 0.78 at the 99th percentile. A generated glyph above 0.78 is closer to some training character than 99% of real characters are to their nearest neighbour, which is the definition of a near-copy used here.

FIDELITYFor twelve words with a dictionary character (tree, rain, woman), the share of samples whose nearest training glyph is that character.

VARIETYThe mean similarity between the four samples of one prompt; lower means more varied.

The similarity has a known weakness. A correct 女 drawn a few pixels smaller than the training glyph overlaps it less than two different characters of the same frame do, so fidelity undercounts: it is a lower bound, and the samples are also read by eye. A standard image metric such as FID was not used because its feature extractor was trained on photographs and has no notion of a kanji.

Results

At 500 steps the samples already have the density and frame of a kanji, with strokes that break into dots. By 2,000 the strokes are continuous and the components have settled: the sample for fish carries 魚, the one for rain sits under 雨, the one for computer has picked the metal radical 金. The later steps clean lines and, for the simple characters, close in on the dictionary one.

The same prompts and seed at 500, 1,000, 2,000, 4,000, 10,000 and 16,000 steps.

The last six thousand steps did more for the dictionary words than anything before them: at 10,000 steps the dictionary character was the nearest glyph for 3 of 48 samples, at 16,000 for 18, while the invented words stayed as far from every training character as before.

Words that have a kanji

Tree, rain and woman give 木, 雨 and 女 in all four samples, mountain gives 山 in three, and bird and gold give 鳥 and 金 or a close variant. The misses follow the data. The prompt for fish gives 魚 with a second component, because most training characters that mention fish are particular fish (鮭 salmon, 鯖 mackerel, 鮪 tuna) written as 魚 beside a phonetic part, and the model learned the family better than its root. The prompts for sun and moon give compounds under 日 and 月, because neither word was ever the short caption of those characters. Water, fire and heart give loose dots and short strokes, the shapes 水, 火 and 心 take as components (氵, 灬, 忄), which is how most characters that mention them contain them.

Samples for words that have a kanji, each over the training glyph most similar to it, with that character and the similarity.

Words that have none

The samples for internet draw 門 in three of four, with something new inside. Computer and smartphone put the tree radical 木, which marks wooden things and also stands in 機 and 械, the two characters for machine, on the left of all eight samples. Pizza builds on 欠, the open mouth that 飲 (to drink) and 歌 (song) share, and one of its samples comes close to 飲 itself. Algorithm takes the thread radical 糸 in three of four, a radical that besides textiles marks order and sequence (系 lineage, 紀 chronicle, 級 rank).

Samples for words with no kanji, each over the most similar training glyph.

No copies

To test for memorisation at a scale where a rare copy would show, the model drew four characters for each of 1,227 words and names from science, technology, culture and games, 4,908 glyphs in all, and each was compared with all 6,413 training characters, the procedure of studies of copying in image diffusion models1.

1Somepalli et al., Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models, CVPR 2023.

Similarity to the nearest training glyph, for real kanji against their nearest neighbour and for 4,908 generated glyphs.

None of the 4,908 is above the near-copy line, and the closest reaches 0.73. The generated distribution sits slightly to the left of the real one (median 0.54 against 0.57): a generated glyph is, if anything, a little less like its nearest training character than real characters are like each other.

Guidance

The same evaluation was run at seven guidance scales.

Left, fidelity against variety for guidance w from 0 to 8. Right, mean similarity of invented words to their nearest training glyph.

At w=0w=0w=0 the prompt is ignored and fidelity is zero; at w=1w=1w=1 it reaches 19%; from w=1.5w=1.5w=1.5 on it stays between 38 and 42% while variety keeps falling. So any guidance past 1.5 buys no fidelity and costs variety, which is the trade-off the guidance paper describes, measured here on kanji. The samples in this write-up use w=3w=3w=3, chosen before the sweep; w=1.5w=1.5w=1.5 would have given the same fidelity with more varied samples. Guidance never pushes the invented words towards copying: their similarity to the training set stays flat at every scale.

6,413characters with both KanjiVG strokes and KANJIDIC2 meaningsKanjiVG r20260714, KANJIDIC2 11×training step time of the same U-Net on pixels against the latent, 21 s against 1.8 sbatch 64, measured on the CPU 0.0055mean absolute error per pixel of the autoencoder on held-out glyphs2% of characters held out 18 of 48samples of dictionary words whose nearest training glyph is the dictionary character, against 3 at 10,000 stepstwelve words, four samples each, w = 3 0 of 4,908generated glyphs above the 99th percentile of real-to-real similarity1,227 prompts, four samples each w = 1.5smallest guidance with full fidelity, and more variety than w = 3sweep over w from 0 to 8

Second arm: the same data on a pretrained Stable Diffusion

The model above knows kanji and nothing else. Its only visual vocabulary is the set of components of 6,413 characters, so when a prompt has no kanji it recombines components, and the results look like plausible, slightly ordinary kanji. The model David Ha shared drew pictures into its characters instead, and the obvious explanation is that it started from a Stable Diffusion that had seen billions of photographs.

The second arm tests that explanation: Stable Diffusion 1.5 fine-tuned on the same characters, with the same two captions and the same caption dropout, rendered at 512 pixels and brought back to 64 for the same evaluation. It ran on a rented A100 80GB, at $1.79 an hour; how that rental works is a piece of its own.

The first attempt learned the style and threw away the meaning

The first run was a standard fine-tune: the whole U-Net trained with the base model's objective (predict the noise), a learning rate of 10−510^{-5}10−5 decayed on a cosine, batch 32, 6,000 steps, an hour and three quarters. Within a thousand steps it drew kanji, in a stroke heavier and more varied than the small model's, with dense, ambitious compositions and a grey vignette where the background should be white. From there the sheets stopped changing, and the meanings never arrived: asked for tree, mountain or rain, it drew the same kind of overloaded character it drew for everything else.

The same eight prompts from the small model (above) and from the first Stable Diffusion fine-tune (below), one sample each.

The evaluation put a number on it: the dictionary character was the nearest training glyph in 0 of 48 samples at every guidance scale tried, against 18 of 48 for the small model. To see why, a diagnostic took 256 training characters, noised them, and asked each model for its estimate of the clean character twice, once with the character's own caption and once with the caption of a random other one. If a model reads the meaning, the right caption should give a clearly smaller error.

Error reduction from the right caption t = 200 t = 500 t = 800

Small model, from scratch 3.1% 17.5% 47.0%

Stable Diffusion 1.5, original 0.0% 0.0% 0.5%

Stable Diffusion 1.5, fine-tuned 0.0% 0.0% 0.0%

The fine-tuned model reconstructs kanji far better than the original (its error at t = 800 fell from 0.216 to 0.087), and it does so without using the caption at all. The fine-tune made it an unconditional kanji generator: it learned what a kanji looks like, and that the caption, for it, predicts nothing.

The reason is the training target. Stable Diffusion 1.5 learns to predict the noise, and at the noisy steps, which are the ones where a character's components are decided, the noisy latent is almost all noise: predicting the noise is close to copying the input, the loss is tiny there (0.003 at t = 800, against 0.02 at t = 200), and it barely depends on what the caption says. The gradient that would teach the model to use the meaning is therefore almost nil, and with a learning rate meant for teaching a style, the model took the cheap route. The small model predicts the velocity, which at high noise amounts to predicting the clean character, and there the caption is worth almost half the accuracy; that is why it had the radicals in place by step 2,000. The grey backgrounds have a related cause: Stable Diffusion's noise schedule never fully destroys an image's mean brightness, so sampling starts from mid-grey and the model never learned to reach pure white1.

1Lin et al., Common Diffusion Noise Schedules and Sample Steps Are Flawed, WACV 2024, which proposes zero terminal SNR with v-prediction as the fix.

A diagnostic of whether a conditional diffusion model reads its condition has to compare errors on the clean-image estimate. Measured on the predicted noise, the same test gave 0.0% for the original Stable Diffusion too, a model that certainly reads its prompt: at high noise the noise is almost the input, and predicting it says nothing about the caption.

The second attempt trains it like the small model

TODO(final): attempt 2 results. Changes: v-prediction, zero terminal SNR with trailing DDIM and guidance rescale 0.7, learning rate 1e-4 on the cross-attention and 2e-5 elsewhere, constant; caption gain and fidelity measured every 1,000 steps; started from attempt 1's weights.

The comparison changes three things at once (pretraining, size and text encoder), so a third run would keep the small model and swap in the larger CLIP, to separate the text encoder from the pretraining.

Where the recipe stands in 2026

The structure used here, an autoencoder with a generator working in its latent space, is still how the leading open image and video models are built. Stable Diffusion 3 and Flux work in the latent of an autoencoder, Wan and HunyuanVideo in that of a video autoencoder, and OpenAI described Sora as a network that compresses video into a latent and a diffusion model trained on patches of it1. What has changed since Stable Diffusion 1 sits inside each box.

1OpenAI, Video generation models as world simulators, February 2024; Wan team, Wan: Open and Advanced Large-Scale Video Generative Models, 2025; Kong et al., HunyuanVideo, 2024.

AUTOENCODERMore channels in the latent. Stable Diffusion 3 moved from 4 to 16 and measured the gain: reconstruction FID fell from 2.41 to 1.06 at the same downsampling^[Esser et al., Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, 2024, section 5.2.1.]. A richer latent costs the generator more and only pays once the generator is large.

GENERATORTransformers instead of U-Nets. DiT cut the latent into patches and handed them to a plain transformer, whose quality improved steadily as its compute grew^[Peebles and Xie, Scalable Diffusion Models with Transformers, ICCV 2023.]; Stable Diffusion 3 and Flux let text and image tokens attend to each other in the same blocks instead of through cross-attention. At 17 million parameters and a 16×16 grid, the U-Net's built-in locality is still the better deal.

OBJECTIVEStraight paths instead of noisy ones. Flow matching and rectified flow train the network to move along a straight line between noise and data, which allows fewer sampling steps^[Lipman et al., Flow Matching for Generative Modeling, and Liu et al., Flow Straight and Fast, both ICLR 2023.]. The velocity this model predicts is a relative: both targets are a weighted mix of the noise and the clean latent, and what differs is the path between them.

SAMPLINGOne to four steps instead of thirty. Consistency models and adversarial distillation teach a second network to jump to the end of the path, at some cost in variety^[Song et al., Consistency Models, ICML 2023; Sauer et al., Adversarial Diffusion Distillation, 2023.].

Two currents run against the recipe. On ImageNet, pixel-space models have caught up again without an autoencoder1, though neither has been shown at the scale of the text-to-image products. And the image generator inside GPT-4o drops diffusion altogether: in OpenAI's words, unlike DALL·E, which operates as a diffusion model, 4o image generation is an autoregressive model natively embedded within ChatGPT2, with the image written as tokens by the same model that writes the text.

1Hoogeboom et al., Simpler Diffusion (SiD2), CVPR 2025; Li and He, Back to Basics: Let Denoising Generative Models Denoise, 2025. 2OpenAI, Addendum to GPT-4o System Card: Native image generation, March 2025.

The word encoder in all of this means one thing at the level of the interface, a network that turns an input into vectors another network can use, and something different in each case at the level of what the vectors keep, because a network keeps what its training asked for. The glyph encoder here was trained to make reconstruction possible and keeps the drawing, with no idea of meaning. CLIP's text encoder was trained to match captions to photographs and keeps what can be seen, which is why pizza and coffee lead the U-Net to similar places. BERT was trained to fill in missing words and keeps what predicts the surrounding text. GPT-style language models have no separate encoder at all. The autoencoder also lives on inside language models: interpretability work trains sparse autoencoders on an LLM's activations, with a wide waist instead of a narrow one and a penalty that keeps few units active, so that each unit ends up standing for one feature1.

1Anthropic, Towards Monosemanticity, 2023.

Running it

The repository is kanji-diffusion, with the trained weights in its v0.2.0 release. Four commands go from nothing to samples; the first downloads both datasets, so neither is redistributed.

BASH Copy

python -m kanji_diffusion.prepare # render and caption 6,413 kanji, about 5 min python -m kanji_diffusion.autoencoder --steps 3000 # about 45 min on a CPU python -m kanji_diffusion.train --steps 16000 # about 5 h on a CPU, resumable python -m kanji_diffusion.sample runs/small "pizza" "internet" -n 4 --guidance 1.5

docs/paper_figures.py redraws every figure of this write-up from a trained run, and kanji_diffusion.evaluate prints the numbers.