One look, one guess is the wrong shape for detection
The standard approach gives the model a single attempt at every object at the same time. It cannot take a closer look at a small object before deciding where its edges are. It cannot notice that a box it already drew is loose and fix it. And it cannot skip an object it is unsure about without also giving up the ones it got right.
A human annotator does all of these things without thinking about it. Each one is a decision, and each decision can be rewarded or penalized on its own. That is what we wanted the model to learn.
Detection also suits reinforcement learning for two specific reasons:
The reward can be checked by a program.
Given the model's boxes and the true boxes, the score is a fixed calculation. No learned reward model sits in the middle, and no person needs to review the output.
What we care about belongs to the whole attempt.
Good detection means finding every object, labelling it correctly, and adding nothing extra. That is a property of the final answer, not of any single word the model writes. Reinforcement learning is built for that kind of goal. Standard supervised fine-tuning is not.
What we built: a gym that owns the answers
The system runs as two separate parts that share only a narrow connection. The gym (the environment) holds the images, the correct answers and the scoring rules. It contains no model and no training code. The learner is an outside program that starts episodes, sends actions and receives back what the model can see, plus a score.
- Images
- Correct answers
- Scoring rules
- Starts episodes
- Sends actions
- Receives a view and a score
This split is about information. Everything the model sees is assembled from an explicit allow-list, so the correct answers cannot reach it by accident. They are read only inside the gym, and only to compute the score. We also withhold the list of object classes, because naming the classes in an image gives away much of the answer. The model proposes its own labels in plain words, and the gym checks them against the ground truth.
How an episode runs
The model gets a view of the image, a small thumbnail of the whole image for orientation, and the boxes it has drawn so far. On each turn it takes exactly one action. An episode lasts at most eight turns.
No separate detector does the hard part. The model writes every coordinate itself, so learning where things are is the job, not something handed off. The model also describes each box relative to what it can currently see, and the gym converts that into whole-image coordinates.
A score that is hard to game
A reinforcement learning model will exploit any loophole in its reward. Most of our design effort went into closing loopholes before the model found them.
Missing an object costs as much as a false alarm.
If the score only counted the boxes the model drew, it could report one easy object, stop, and score perfectly on an image containing ten. Ours counts every miss against it.
Spraying boxes doesn't pay.
The whole score is scaled down by how clean the set of boxes is, so guessing widely in the hope of a lucky hit loses on average.
Nearly right earns partial credit.
Most systems use a pass/fail overlap threshold. A box just under the line earns nothing and, worse, gets no signal about which way to improve. Models then settle just short of the line. We give credit that rises smoothly as a box gets closer, using a measure that still works when a box misses the object entirely.
Hints along the way cannot change the goal.
The final score arrives only at the end, so we add a small signal on each turn for progress, such as a better box or a view framing an unfound object. We use a form of hint that is mathematically proven not to change which behavior scores best (Ng, Harada and Russell, 1999). It speeds learning up without redefining success.
How it learns: six attempts, compared
Each training step asks one question: did this attempt do better than the other attempts on the same image?
The model makes six full attempts on one image. Each attempt is scored, and the update then pushes the model toward whatever the above-average attempts did.
Siblings instead of a critic.
We use group-relative policy optimisation (GRPO), the method introduced with DeepSeekMath (Shao et al., 2024). Classic reinforcement learning trains a second network, a critic, to predict how good each situation is. GRPO drops it: each attempt is judged against the average of its five siblings. That means one fewer network to train and fit in memory.
Train a thin slice of the model.
Rather than retraining the whole model, we use LoRA: small trainable adapters added alongside the frozen pretrained weights, covering under 1% of the model.
Valid output by construction.
As the model writes each token, a parser removes every option that would break the action format. Every action the model produces can be parsed and executed, so no turns are wasted on malformed output.
Small, controlled steps.
Each update is capped in how far it can move the model, because the attempts were collected by a slightly older version of the model and can only be trusted while the two stay close.
What we learned building it
Make leaks impossible, not just unlikely.
Keeping the answers inside the gym and building every observation from an allow-list means no future code change can accidentally show the model the ground truth.
Silence has to cost something.
Before misses counted against the score, the cheapest strategy was to report one confident box and stop.
Pass/fail thresholds stall learning.
A box at 49% overlap under a 50% threshold gets no credit and no direction. Smooth partial credit gives the model a reason to keep improving.
Progress hints must agree with the final score.
If the per-turn signal rewards a different notion of a "good box" than the final score does, training time goes into teaching habits the model later has to unlearn.
Don't ask a small model to learn formatting.
A 2-billion-parameter model asked for strict JSON will often comment on the image instead, or leave a bracket open. We constrain generation so that every output is a valid action by construction. The model's effort goes into which action to take and where the box goes.
Coordinate mapping is where silent bugs live.
If the conversion from "what I see" to "where it is in the full image" is wrong, every zoomed-in box lands in the wrong place, and the score can still look believable.
Only compare attempts on the same image.
Our learner judges each attempt against its siblings, other attempts on the same image. Mix in different images and the comparison measures which image was easy, not how well the model acted.
Clever fixes fail quietly.
Our first version of the formatting-token filter assumed compact JSON. The model writes JSON with spaces, so the filter matched nothing and the training run degraded. It now falls back to scoring every token when the filter comes back empty.