Deepak Vijaykeerthy

As the policy improves, a fixed task distribution stops producing useful training signal, so the environment has to change with it. We study this with search agents, where what the environment teaches matters as much as how hard it is. An agent that has found three affected customers can't tell from those records whether a fourth exists, so stopping is a bet on evidence it hasn't seen. Agents trained with outcome rewards tend to stop too early. Our hypothesis is that they learn the base rates of their training data. If the missing record is rare in training, stopping early is usually correct and saves search cost, so the reward favors it. We're studying how to build and adapt training environments so that stopping depends on the evidence the agent has gathered.

Evaluating an enterprise agent before deployment, or training it with RL, requires an environment that reproduces the coupled services and data stores it will operate on. Their state evolves between the agent's actions, and the agent's own actions condition what follows, so each episode has to be simulated online. Our premise is that fidelity is required only at the interface the agent acts through, since environments that respond identically to every available action are indistinguishable to it. Starting from a seed, we learn this interface from traces and realize each component behind it as a real system, a stub or an LLM simulation, depending on how much of it the agent can observe and how much other state depends on it. The central difficulty is keeping this hybrid consistent over long episodes, especially under RL, which will exploit any inconsistency that increases reward.

An agent's harness consists of its tools, what they return, its memory and how its context is managed. Harness changes can shift performance as much as further training, yet harnesses are usually designed by hand and kept fixed during training. Each component encodes an assumption about what the model needs, and the same component can help one model and hurt another. Building a strategy into a tool also removes the need for the model to learn it. Our hypothesis is that scaffolding works best as a temporary aid, added between rounds of RL where a model falls short and removed once it has learned the capability. What remains at the end would show which computation and state are better kept outside the weights.

Post-training papers are usually organized by objective family. The choices that determine training behavior lie below that level, in which samples get gradient weight and at what granularity, how stale replayed data may be, and how far a proxy reward can be trusted. Methods from the same family can disagree on any of them. rl-experiments isolates these choices in PyTorch sandboxes small enough to read the gradient directly, so failures that take days to surface at scale show up on a single GPU.

• Group-relative RL gets a direct signal only from prompts the policy solves some of the time. Prompts it never solves give no signal yet, and prompts it always solves give none either. Pretraining, SFT, curricula, larger groups and distillation all bring tasks into that useful range, each by a different route, which is why they aren't interchangeable fixes for sparse reward.
• In our sandbox, GRPO's entropy collapse traces to the standard-deviation normalization. When a group nearly agrees, dividing by its spread inflates a negligible reward gap into a full-sized step. The clip only bounds the size of that step, so tightening it has little effect, and the variants that do slow the collapse, such as Dr. GRPO, change the normalization. Whether the same mechanism dominates in large-scale runs is the next thing to test.
• Assigning credit at a finer grain redistributes the signal a reward already contains and can't create signal where the reward is silent. Creating signal requires reward shaping.

minilab runs the whole stack, from pretraining through RLVR, on a single laptop GPU, with every RL method sharing one rollout loop, so each stage of post-training can be watched at a scale small enough to reason about. The research built on it treats sparse-reward RL as an information problem before an optimization problem, since an update can only reinforce a trajectory that the policy visits and the verifier can tell apart from the failures around it.