When Aligning an Image Model Makes It Worse at Spelling

Ask an image generator for "a bakery storefront with a hand-painted sign that reads OPEN," and a lot of models will nail the warm lighting, the pastries in the window, the weathered brick. Then you look at the sign. It says "OPEEN." Or the letters trail off into something that isn't quite any alphabet.

That's not a random failure. A paper out this week, "Manifold Drift in Flow Preference Optimization," argues it's the predictable result of a specific, fixable mistake in how these models get fine-tuned to match human preferences. The fix they propose, called ThermoDPO, improves OCR accuracy on generated images by 47.5% on one benchmark model. For a site that lives and dies by how well machines read text, that's worth unpacking.

KINCAID'S HAMBURGERS: City Says Sign Needs To Go Or Popular Burger Joint Faces Daily Fines — by Robert W. Howington via flickr, BY-SA 2.0.

What flow matching actually does

Skip the math for a second and think about the training process like this: a flow-matching model learns to gradually turn random noise into a realistic image (or audio clip, or video frame) by following a smooth path, step by step, from pure static to a finished picture. "Flow matching" is the current leading recipe for training these path-following generators, and it's the engine behind models like Stable Diffusion 3.

Crucially, when a flow model is trained well, the endpoints of all those paths land squarely inside what researchers call the data manifold: the space of images that actually look like real photographs, real documents, real signage, real anything. Feed it random noise and it reliably produces something that resembles the training data, because that's what it was mathematically built to do. The paper's first real contribution is proving this cleanly: optimal flow matching recovers the true data distribution. No drift, no surprises.

Where preference tuning breaks the deal

Here's the part that trips people up. Once you have a working image generator, the next step in most modern pipelines is alignment: showing the model pairs of outputs, telling it "humans liked this one better," and nudging its weights toward the preferred choice. This is the same idea behind DPO (Direct Preference Optimization), the technique that took over language model fine-tuning a couple of years ago. It works well for text. Porting it to flow matching is where things get messy.

The problem is that a flow model's transport paths, the routes from noise to image, have no built-in leash keeping them on the data manifold once you start tugging on them with a reward signal. The paper formalizes this as "manifold drift": a preference update moves the model's outputs away from realistic image space whenever that update has what they call a nonzero normal component, meaning the push has a piece pointing perpendicular to the manifold rather than along it.

Picture the manifold as the surface of a pond and the model's outputs as boats on that surface. A good update rows the boat to a better spot on the pond. A drifting update yanks the boat up and out of the water entirely, and now it's technically closer to wherever the reward function wanted it to go, but it's not a boat on a pond anymore. It's just wrong in a way the reward model failed to notice.

That last clause is the real story. This is reward hacking: the model finds a way to score well on the metric that's supposed to represent "humans preferred this" without actually producing something humans would prefer. Garbled sign text is a visible, easy-to-catch version of this failure. Less visible versions, subtle texture artifacts, anatomically odd hands, colors that are almost but not quite right, are harder to catch and probably happening in production models right now.

ThermoDPO: keeping the boat on the water

The fix the authors propose starts from a simple instinct: if the danger is drifting away from real, known-good samples, anchor the optimization to those samples directly. ThermoDPO is a temperature-controlled objective that pins pairwise preference updates to the preferred (winning) sample in each comparison pair, rather than letting the update float freely based only on the reward gap between the two options.

The "temperature-controlled" part matters because it gives the method a dial. At one setting, the objective behaves like rejection sampling fine-tuning, or RFT: you generate a bunch of candidates, keep the ones that score well, and train on those. RFT is safe and stays close to the data manifold almost by construction, but it throws away a lot of the signal in a preference pair (it never compares low-quality output against high-quality output directly, it just filters). At another temperature setting, the objective converges toward FlowDPO, the more aggressive preference-optimization approach that uses the full comparison but is more prone to drift. ThermoDPO sits on a continuum between the two and lets you choose how much you want to trade safety for signal.

The authors also show, and this is the more technical but genuinely useful part, that their objective controls a pointwise reconstruction-based surrogate for manifold distance. In plain terms: they found a proxy measurement that tracks how far a generated sample has wandered from the data manifold, and their training objective keeps that number in check as a side effect of normal training, not as a separate penalty bolted on afterward.

One catch: at low temperatures (where you're closest to the safe, RFT-like regime), the training signal gets weak, because you're mostly telling the model "this sample is fine" rather than giving it a useful gradient toward improvement. Their answer is ThermoDPO-weighted, a variant that reweights the objective to keep the signal strong even when the temperature is turned down toward the cautious end. It's the version that ends up winning most of their benchmarks.

The numbers, and why they're bigger than they look

On the paper's main toy benchmark (a controlled setup designed to isolate this exact failure mode from the noise of a full production model), the scoring metric is something they call StrictScore, which appears to penalize any deviation from the true data manifold rather than just rewarding preference-alignment. Plain FlowDPO scores 0.629. Adding rejection sampling on top of FlowDPO (FlowDPO+RFT) gets to 0.857. ThermoDPO-weighted reaches 0.899. That's a big gap between "aligned to preferences" and "aligned to preferences without wrecking the underlying model," and it's the paper's cleanest demonstration that drift is a real, measurable cost, not a theoretical worry.

The result that'll matter more to this audience comes from testing on SD3.5-M, a real production-scale Stable Diffusion 3.5 model, at a classifier-free guidance (CFG) setting of 4.5, which is a standard knob that controls how strictly the model follows its text prompt. ThermoDPO improved OCR accuracy on generated images by 47.5%, and improved the average across four evaluation metrics by 16.0%.

Read that OCR number carefully, because it's doing something interesting. The researchers aren't running an OCR benchmark for its own sake. They're using OCR accuracy as a stand-in for "did the model actually generate legible, well-formed text instead of hallucinated squiggles," which is exactly the kind of failure that's easy for a human rater to miss at a glance (the sign looks fine at a thumbnail) but trivial for an OCR engine to catch, because an OCR engine either reads real words or it doesn't.

Why it matters for people building or evaluating OCR pipelines

If your work touches document AI, synthetic data generation, or any pipeline where images with embedded text get created or transformed by a generative model, this paper is relevant even though it never mentions document processing directly. A few concrete takeaways:

• Text legibility is a sensitive, cheap check for a much broader problem. If you're evaluating or fine-tuning any image generator and want an early warning sign for reward hacking or manifold drift, run generated outputs through an OCR engine and check accuracy. It's a canary that catches subtler failures before they show up in harder-to-quantify visual quality scores.
• Synthetic training data for OCR systems is only as good as the generator producing it. A lot of document AI teams generate synthetic receipts, forms, and signage to augment training sets. If that generator was preference-tuned without a drift-aware method like ThermoDPO, the synthetic text in those images may be subtly less realistic than it looks, which can quietly poison downstream OCR training.
• A 47.5% OCR improvement on a real production model (not just a toy benchmark) suggests this isn't a niche academic fix. If ThermoDPO or something like it gets adopted by major image-model providers, expect noticeably better text rendering in the next generation of generated documents, signage, and UI mockups.
• Reward hacking isn't just an LLM problem. Anyone building evaluation pipelines around generative models, image, text, or otherwise, should treat "did the reward metric go up" and "did quality actually improve" as two separate questions that need two separate checks.

What's still open

The headline numbers come from a toy benchmark built to isolate this exact effect, plus one production model (SD3.5-M) at one guidance setting. That's a solid proof of concept, not a sweep across the model families and settings people actually deploy. It's worth asking how ThermoDPO performs at different CFG values, on larger models, or on video generation, where flow matching is also increasingly the standard approach and where drift could compound across frames.

The temperature parameter is also a knob you have to tune, and the paper's own results show that getting it wrong (too low) starves the training signal, which is exactly why the weighted variant exists as a patch. That's a reasonable engineering tradeoff, but it does mean ThermoDPO isn't a parameter-free drop-in replacement for existing preference-tuning pipelines. Teams adopting it will need to do their own sweep to find the right regime for their model and data.

Still, the core theoretical contribution, that a preference update leaves the pretrained manifold precisely when it has a nonzero normal component, gives the field a much sharper way to think about why alignment sometimes backfires in continuous-time generative models. That's a useful mental model even for teams not planning to adopt ThermoDPO specifically: when you fine-tune a flow-based generator on preferences, ask whether your update is pushing along the manifold or off of it. The paper gives you a name for the failure and, for the first time, something close to a way to measure it.