TogetherDesign
From real sessions to verifiable tasks
Static benchmarks hand an agent a complete spec up front and grade only its final code. Real coding help is interactive — users clarify goals, add constraints, and correct mistakes across many turns. SWE-Together reconstructs that loop from real user–agent sessions and scores agents as collaborators, not one-shot solvers.
Real sessions
Every task comes from an actual user–agent coding session, not a synthetic prompt — the first-turn instruction is the user's verbatim initial request.
Verifiable state
From 11,260 recorded sessions we keep 109 with a recoverable repository state, a clear user goal, and an observable outcome such as submitted code changes.
Multi-turn intent
Intent is revealed incrementally — clarifications, added requirements, and corrections — so each task preserves the real interaction loop instead of one fixed instruction.
Anchored user simulator
A state-conditional LLM simulator keeps the original user's intents and intervention order, releasing feedback only when its trigger conditions arise in the agent's trajectory.
Scored by frozen rubrics
The final repository state is judged against task-specific, implementation-agnostic rubrics derived from repository inspection and original-session evidence.
Measures interaction, not just success
Beyond final correctness we report User Correction — how much corrective steering the user needed — and Intent Coverage, whether intents are conveyed consistently.
Method
How a session becomes an evaluation
Each benchmark instance packages the original task, the repo state, the user simulator, and the scoring rubric as separate artifacts. That separation lets us test whether an agent can collaborate through corrections without letting the simulator or judge leak the answer.
Rebuild the coding problem from the original session.
A task contains the user's first message, a Docker environment pinned to the pre-fix commit, a structured summary of the original interaction, and frozen evaluation goals. The agent sees the same kind of workspace a developer would see, not the paper's analysis.
The user simulator responds to what the agent actually does.
After each agent turn, the trajectory is summarized into task progress, missing requirements, and observable failures. The simulator compares that state with the original user's intents and decides whether to stay silent, clarify, correct, or approve.
This keeps the interaction conditional: an agent that solves the issue early should not receive unnecessary hints, while an agent that misses a real follow-up requirement should still face it.
Agent, simulator, and judge each get different evidence.
The coding agent operates in the task workspace. The simulator receives progress summaries and original-session intent anchors. The final judge receives the completed patch and frozen rubric. Keeping those views separate reduces leakage and makes the benchmark easier to audit.
We report collaboration signals alongside correctness.
Final correctness is scored by an implementation-agnostic rubric. Interaction quality is tracked separately through User Correction and Intent Coverage, so the leaderboard can distinguish agents that solve tasks smoothly from agents that need repeated steering.
Results
What the benchmark reveals
The clearest story is not another budget curve: stronger cohorts need less corrective steering, and efficiency separates models that finish with similar solve rates.
Composition
What's in the suite
FAQ
Frequently asked questions
What makes TogetherBench different from single-turn benchmarks like SWE-bench?
Single-turn benchmarks test whether an agent can solve a task from one prompt. In practice, developers steer agents through multiple turns — redirecting, clarifying, and reviewing. TogetherBench captures this loop: a user simulator replays the original human interaction pattern, and the headline metric is the gain from turn 0 (first attempt) to the final turn after corrections.
How does the user simulator work?
Each turn, the coding agent's work is distilled into a structured summary — a compact digest of what changed, what's still broken, and what tests pass. A Gemini-based user simulator reads this summary alongside the ground-truth session notes, then decides what a human would say next: a correction, a follow-up question, a new requirement, or "looks good, stop." The simulator never sees the agent's raw code — only the summary.
Why use an agentic judge instead of test suites?
Narrow test suites reject correct solutions that take a different implementation path — OpenAI found 35.5% of SWE-bench test failures were false negatives. Our agentic judge (Opus 4.6 in an E2B sandbox) reads the full diff, runs the tests, and scores against weighted completeness goals. The judge score, not the raw test reward, is the headline ranking metric.
Where do the tasks come from?
Every task is reconstructed from a real developer session on a public GitHub repository (20+ stars). The instruction is the verbatim first user message (PII-redacted). The Docker environment clones the repo at the exact pre-fix commit with all dependencies installed. We currently have 109 tasks spanning TypeScript, Go, Python, Rust, and more — covering bugfixes, features, and refactors.