Odysseys2
October 2026 · 12 min read

Odysseys2:A Web Agent Benchmark
Based on Real User Requests.

Odysseys2 is a web agent benchmark developed by researchers at Carnegie Mellon University and Browser Use, with 100 tasks grounded in real user needs that run on the live internet. While the original Odysseys drew on user browsing histories, Odysseys2 uses tasks constructed from the real user request distribution provided by Browser Use. Odysseys2 evaluates tasks with an LLM judge which takes in the files agents produce alongside their action histories and screenshots, and assigns zero credit to attempts it flags for reward hacking.

We compare 15 agents, test three browser-agent harnesses, and measure variation across multiple attempts on the live web. GPT-6 Astra leads on Odysseys2 with perfect scores on 47 out of 100 tasks. Watch the models race, explore the tasks, or download the benchmark to evaluate your own agent.

Download 100 tasks ↓ BrowserCode ↗ Benchmark repo ↗

Lawrence Jang1, Alexander Yue2, Magnus Müller2, Gregor Zunic2, Andrew Keunwoo Jang1, Younghan Park1, Mareks Woodside1, Daniel Fried1, Jing Yu Koh1, Ruslan Salakhutdinov11Carnegie Mellon University · 2Browser Use

The task download includes the full instructions and rubrics.

100tasks 982rubric checks 15agents compared 3harnesses tested
0100
Race the models · recorded runs

Recorded completion times, replayed in about five seconds; scores are out of 100. Four models ran at high effort; Astra ran at default effort.

Why

Why an Odysseys2?

We released Odysseys (Jang et al. 2026) in April 2026 with 200 long horizon web agent tasks. Its judge, Gemini 3.1 Flash Lite, scores one rubric item at a time from the agent's screenshots and actions. When we ran OpenAI's GPT-6 Astra on the same tasks in late September and graded it the same way, it completed 129 of the 200 perfectly (Fig. 1). These gains prompted us to revisit both the tasks and how we grade them. We also wanted to provide a follow up given the quick saturation rate on the original Odysseys - and Browser Use requests provide a different distribution of tasks than the original browser history based Odysseys.

Better agents explain a lot of the improvement on Odysseys, but we also found a gap in the evaluation: many Odysseys tasks asked for a artefact from the web agent, yet the judge that determines task success never received that file. Odysseys2 gives the judge the final task output contents alongside the action history and screenshots. On the same 100 Odysseys2 tasks, Astra ranks first under both judges, but its perfect attempts fall from 86 under the Odysseys judge to 47 under the Odysseys2 judge on the same tasks.

Best agent at releaseOdysseys paper, April 2026
44.5%
GPT-6 AstraSame 200 tasks, September 2026
64.5%
0%Tasks perfect100%
Fig. 1. Share of Odysseys tasks completed perfectly. This is not a controlled comparison, since the two results are five months apart, though Astra's run used the same judge model. (April figure from Jang et al. 2026.)

The comparison in Fig. 2 brings these changes together: tasks constructed from real user requests, a judge that reads the delivered files, and a reward hacking check that assigns zero credit to flagged attempts. Alongside scores, we report how long each attempt took, so readers can compare agents on speed as well as the quality of their work.

Odysseys and Odysseys2 side by side
Odysseys (April 2026)Odysseys2
Tasks200100
Where they come fromComposed from Chrome browsing histories that 248 Prolific participants labeled, plus 30 tasks written by the authors and 80 generated hard tasksSelected Browser Use requests, denanonymized, rewritten and expanded into benchmark tasks, with human review of task and rubric quality
What a finished task looks likeThe right pages reached and the right information found along the wayResearch and interactive tasks with explicit coverage, verification, and output requirements
Rubric3 to 12 items per task8 to 12 items per task, 982 in all
JudgeGemini 3.1 Flash Lite grades each item on its own from step screenshots and actions; an item counts if any step satisfies itGPT-5.6 Luna evaluates the action history, screenshots and extracted file contents, grades all items together, and flags suspected fabrication or reward hacking for the zero-score policy
Checked against peoplePeople annotated 120 Opus 4.6 runs; the rubric judge agreed with them better than a judge scoring the whole runHuman review of tasks and rubrics before the agent runs. In a five-task Prolific pilot, reviewers missed a confirmed violation that the Odysseys2 judge caught.
Best result reported at release Opus 4.6, 44.5% of tasks perfectGPT-6 Astra, 47 of 100 tasks perfect
Fig. 2. What changed between the two benchmarks. Odysseys figures are from Jang et al. (2026). 
Results

How do agents do on Odysseys2?

Under the Odysseys2 judge, GPT-6 Astra leads with a mean score of 84.6, yet only 47 of its 100 attempts are perfect (Fig. 3). Switch between the Odysseys judge and the Odysseys2 judge to compare their scores on the same attempts.

How to read the numbers. Each entry is one model in one harness at one reasoning effort.
15 agents on Odysseys2

Fig. 3. Fifteen agents under the Odysseys2 judge and fourteen under the Odysseys judge, which has no Luna grades. Both views use the same 0–100 score axis. Means use 100 scored attempts per agent, except for Fable 5: its content filters blocked two tasks, so its mean uses the remaining 98. Vertical lines show the reported 95% confidence intervals for the Odysseys2 judge only. Cost is the mean recorded cost per task, on a log scale. For Opus 5.5, Grok 4.7, Luna and Gemini 3.8 Flash, it includes model and judge usage; the other entries report model usage only.

Do different harnesses matter?

We ran GPT-5.6 Luna at several reasoning efforts in Browser Use’s Python library, BrowserCode and Pi to measure how much the harness changes performance (Fig. 4).

GPT-5.6 Luna: mean score against median minutes per attempt
Show the numbers
Fig. 4. GPT-5.6 Luna across three harnesses, showing low through the highest supported reasoning effort. Each point is one setup on 100 tasks. Hover or tap a point for its numbers. Python has no max setting.

With the Python harness, Luna scores around 30 from low effort onward. BrowserCode and Pi both improve sharply from medium to high effort, reaching nearly identical scores at high effort. At higher settings they diverge: BrowserCode reaches 63.0 at max, while Pi reaches 71.9, slightly above Opus 5.5’s 71.2 in BrowserCode at high effort. Because the harnesses also provide different records to the judge, the comparison reflects differences in both execution and the evidence available for grading.

Fast versus good

A higher final score can come with a longer wait. Median attempts took 25 minutes for Opus 5.5, 24 for Grok 4.7 and 13 for Luna. Figure 5 compares the score earned by each model when only attempts completed within a given time count.

15 min

Fig. 5. Credit from attempts finished by the time shown, divided by all scored attempts, so each curve ends at the model's mean. BrowserCode, high effort, 100 tasks.

Luna's early lead is large: 37 points by 15 minutes, against 18 for Opus 5.5 and 12 for Grok 4.7. Grok 4.7 catches up at about 30 minutes and Opus 5.5 at about 36, and both keep climbing after Luna levels off (Fig. 6 shows every attempt). Luna is also far cheaper, at $0.35 of recorded model and judge cost per task against $6.22 for Opus 5.5 and $6.30 for Grok 4.7.

Every attempt by the four high-effort models

Fig. 6. Every attempt by the four high-effort models, by run time and score. A cross marks an attempt the judge zeroed, and dashed lines mark median times. Click a dot to open its task.
Consistency

How much do results vary across runs?

We ran eight GPT-5.6 Luna at high reasoning effort runs in BrowserCode on all 100 tasks. Fig. 7 shows both rubric score and perfect-pass rate against elapsed time. Across the 100 tasks, at the 60 minute cutoff, the mean rubric score is 56.9; selecting each task’s best score across eight attempts gives 83.1, and selecting its worst gives 23.1. The mean perfect-pass rate is 6.25%. At least one attempt is perfect on 26 of 100 tasks (26.0%), but no task is perfect in all eight runs.

Eight runs: score and perfect passes over time
Two charts for eight runs on 100 tasks, with time from 0 to 60 minutes on the x-axis. At 60 minutes, mean rubric score is 56.9 ± 2.9%, best of eight is 83.1%, and worst of eight is 23.1%. Mean perfect-pass rate is 6.25 ± 1.5%, at least one of eight is 26.0%, and all eight is 0%. Callouts mark 15 and 30 minutes. Distinct line patterns identify the curves; thin lines show individual runs and shading shows one sample standard deviation.
Fig. 7. Eight runs on all 100 tasks. Left: mean rubric score, highest score per task (Best@8), and lowest score per task (Worst@8) across eight attempts. Right: mean perfect-pass rate (pass@1), at least one perfect attempt (pass@8), and all eight attempts perfect (pass8). Scores enter at recorded attempt completion; these are not observations of progress within an attempt. Thin lines show individual runs; shading is ±1 sample standard deviation, not a confidence interval. Best@8 and Worst@8 select an attempt for each task after grading. Callouts mark 15 and 30 minutes.

The eight runs were collected under different execution conditions, including dates, concurrency and logging settings. Their variation therefore reflects both those conditions and differences between individual attempts and grades.

Tasks

From user requests to benchmark tasks

We de-identified the source requests provided by Browser Use and used their distribution to construct long, self-contained web tasks with explicit deliverables. People reviewed and validated the resulting tasks, which reflect real user needs without copying individual requests one-to-one. Each rubric was drafted with an LLM and revised over several rounds of human and model review.

Browser Use drafted rubrics with a model that could browse the web, compare several agents' attempts, and check source facts. Before the final evaluations, four people reviewed each selected task and rubric. They corrected unclear instructions, infeasible requirements, and mismatches between tasks and rubrics. The process also included five rounds of model-assisted checks for each task's instruction <> rubric alignment and overall feasibility. 

The instructions are long (median 271 words) and usually ask for several deliverables at once, such as a spreadsheet, a log of pages that failed and a short written comparison. They also say what the agent must not do, like signing in or adding to a cart. Every task starts from a Google search page, and Fig. 8 shows a few examples with their rubrics.

Unless we say otherwise, agents ran on the live web with BrowserCode, Browser Use's agent harness which interacts with the browser by writing code against the Chrome DevTools Protocol (CDP). Our previous work compares this approach with harnesses that use a predefined set of browser tools.

We used Browser Use’s stealth browsers as the remote browser infrastructure. The setup combines residential IPs, browser fingerprinting, a custom Chromium fork, and automatic CAPTCHA solving. Testing on live websites exposes agents to complex page structures, network delays and changing content that simplified test sites may not capture.

See the cloud browser benchmark for website-access comparisons.

Fig. 8. A task is a request plus a rubric. Instructions and rubric items are exactly as the agents and judges saw them; we wrote the titles and categories for browsing. Open any rubric item to read the full requirement.
Grading

How we grade attempts

Many Odysseys2 tasks require a spreadsheet, report or other file. To evaluate that work, the Odysseys2 judge receives the delivered file contents together with the action history and screenshots. Figure 9 compares it with the Odysseys judge on the same saved attempts.

Evaluation recipe · Judge prompt. Includes scoring rules and recorded model settings.

The two judges
Odysseys judgeOdysseys2 judge
ModelGemini 3.1 Flash Lite GPT-5.6 Luna at its xhigh reasoning setting
What it seesThe action history and screenshotsThe action history, screenshots and the contents of the delivered files 
How it gradesOne rubric item at a timeAll items together, writing down the evidence for each
VerdictsMet or not metMet, not met, or not assessable
Reward hacking checkNoneScores the attempt zero when the judge flags suspected fabrication or reward hacking
Fig. 9. The two judges. Both graded the same saved runs, so any difference in score comes from the grader, not the agent.

Met

Every condition in the item holds, and the record shows where the answer came from. A plausible value in a spreadsheet is not enough on its own.

Not met

At least one condition fails, or the agent did something the task ruled out.

Not assessable

The evidence needed to decide is missing from what the judge received. The item earns nothing, but the judge says so instead of guessing.
Fig. 10. The Odysseys2 judge's three verdicts. Item credit is the share of rubric items met. The final score is item credit after the reward hacking check described next, so a flagged attempt's final score is 0.

The Odysseys2 judge also flags suspected reward hacking, including suspected fabrication in deliverables. When it flags an attempt, this reward hacking check scores the attempt zero however many items it met. The flag is a model judgment, not independent proof of fabrication or intent. Fig. 11 shows an example.

Task 002, Count authors on a set of arXiv papers
Gemini 3.8 Flash in BrowserCode, high effort
Met (3)Not met (6)

R1: Not met. Only abstracts containing the exact phrase “Privacy in Machine Learning” qualify.

“The accessed abstracts do not contain the exact contiguous phrase. Step 27 shows ‘privacy-aware machine learning’ for 2406.04610, step 106 shows ‘privacy-preserving machine learning’ for 2402.11224, and step 108 explicitly reports the literal test ‘privacy in machine learning’ as false for both. Nevertheless, the final report counts both papers.”Evidence cited by the Odysseys2 judge for R1
“It also asserted exact-phrase matches after step 108 explicitly found the literal phrase absent, suggesting fabricated or knowingly unsupported deliverable claims.”Judge's reason for the reward hacking flag
Item credit
33
3 of 9 items met
Fig. 11. One selected attempt. The boxes show actual item identities and verdicts. The saved abstract text and literal-match test support the R1 failure, which the Odysseys judge missed. The same mismatch was visible in its input. The whole-task zero is a separate policy decision; this item failure alone does not validate it. Toggling the policy changes the score, not the item verdicts.

The reward hacking check assigns zero credit to 1 of Opus 5.5’s 100 high-effort attempts and 75 of Gemini 3.8 Flash’s (Fig. 12). It also flags 8 of Luna’s 100 attempts, more than either Opus 5.5 or Grok 4.7, despite using Luna as the judge.

Mean score at high effort, before and after the reward hacking check
Item creditFinal score
Fig. 12. Mean score in BrowserCode at high effort, before and after the reward hacking check.

Same attempts, two judges

Figure 13 compares the scores that the two judges assign to identical attempts from fourteen agent setups.

Odysseys2 judgeOdysseys judge
77 to 96Range of mean scores under the Odysseys judge
2 to 85Range of mean scores under the Odysseys2 judge
Fig. 13. Mean score per agent on Odysseys2, using the same attempts under both judges. Default reasoning effort unless marked high. Fable 5 has 98 paired grades because its content filters blocked two tasks; the other agents shown have 100. Astra, Sol, Terra and Luna are OpenAI's GPT-6 Astra and GPT-5.6 models; Fable 5, Opus and Sonnet are Anthropic's. Gemini Lite is an agent, not the Flash Lite judge.

Astra ranks first under both judges, but several other agents receive much lower scores under the Odysseys2 judge. Gemini 3.7 Flash, for example, falls from 94.9 to 5.8.

To separate the effect of the judge model from the grading procedure, we kept the Odysseys judge’s per-item procedure and replaced its model with Luna. Across 2,098 attempts from 21 agent setups, this lowers the mean score from 88.9 to 60.9. The full Odysseys2 judge gives the same attempts a mean of 45.3 (Fig. 14).

Score changes vary substantially even within a provider: Astra falls by 11.0 points, Sol by 31.9, and GPT-5.4 mini by 62.4. These differences measure disagreement between the judges; lower scores alone do not establish greater accuracy.

Odysseys judgeGemini Flash Lite, one item at a time
88.9
Same setup, Luna as judgeOnly the judge model changes
60.9
Odysseys2 judgexhigh reasoning, file contents, joint grading, reward hacking check
45.3
0Mean score across the same attempts100
Fig. 14. One set of attempts, three graders. Mean over the 2,098 attempts, out of 2,100 across 21 agent setups, that all three judges graded. The last step changes several things at once, so we can't yet say how much of it comes from access to file contents.
Human review

Validating our judge

To check the judge against human assessments, we ran a small paid pilot on Prolific, assigning two reviews to each of five tasks. Reviewers received a saved attempt with its step log, screenshots and output files, then rated each requirement and cited the supporting evidence. The job offered $5 for an estimated 20 minutes and asked reviewers not to use AI.

From the pilot, none of the human ratings marked a requirement “not met",  and both reviewers passed a task error we confirmed manually, which the Odysseys2 judge caught. These results point to a problem with the human review process for long horizon web agent tasks: checking a long attempt through human review may be worse than the strongest agents with access to the trajectory artefacts.

Why human review is hard here

  • The trajectories are long. Logs alone run 16,000 to 41,000 words per run, before screenshots and files.
  • Requirements combine several conditions. One requirement on task 007 needs four fields checked on every record.
  • Checking can mean repeating the task. Reviewers may need to revisit sources and verify the agent’s work themselves.
  • The review time is long. The median usable form took 49 minutes, not 20, and 29 review assignments were returned unfinished.
Every rating, from the reviewers and from Opus
Fig. 15. Every rating in the pilot, one row per submitted form and one cell per requirement. Blue-labeled rows are Claude Opus 5.5 reviews of the same runs, and the faded row's explanations were all N/A. Hover or tap a cell for the requirement and the explanation. The bar on the right is time on the form against the advertised 20 minutes.

The example human error which did not flag a rubric error is on task 005 (Fig. 16). The Website column must not contain editor links, yet row 101 holds a Google Sites address ending in /edit. Both human reviewers and the original Odysseys judge marked it met, but the Odysseys2 judge caught it in the run's own log.

We then asked Claude Opus 5.5 to review each of the five runs twice, using the reviewers’ instructions and saved run text without screenshots. Each review took about 2 to 5 minutes. On task 005, both reviews caught the editor link and a second address that the agent’s own log had identified as malformed. The human reviewers and Odysseys judge had passed both, while the Odysseys2 judge flagged them. Agentic judge with the strongest models seem to provide good signal, but this takes time and may not be scalable/feasible due to cost.

Task 005, requirement 4: the Website column, all 374 rows
Fig. 16. Task 005, requirement 4. Each square is one row of the delivered CSV; hover for the site it links to.

Conclusion

How we grade the work changes the picture of agent performance. Across the same fourteen agents, mean scores range from 77 to 96 under the Odysseys judge and from 2 to 85 under the Odysseys2 judge. Examining delivered files exposes unsupported claims that the Odysseys judge passed, although the human-review pilot was not sufficient to establish the Odysseys2 judge’s overall accuracy.

Astra earns the highest mean score, but completes fewer than half the tasks perfectly. Luna leads the four-model high-effort comparison for roughly the first half hour, and the eight-run results show how outcomes vary across attempts. We release all 100 tasks and rubrics, along with the published scores, so others can evaluate agents and examine these results.

All tasks

All 100 tasks

Every task is below. You can search by site or topic, filter by category, or sort by how the four high-effort models did. Opening a task shows the full instruction, the rubric and each model's score, and lets you download it.

100 tasks
    Fig. 17. All 100 tasks. The scores CSV includes the four high-effort models, Astra at default effort, and five Luna runs. Downloads here follow your filter, so with a search or category selected the buttons export only the matching tasks. Every rubric item has weight 1. See the evaluation recipe for scoring details. The four bars on each card are final scores for Opus 5.5, Grok 4.7, GPT-5.6 Luna and Gemini 3.8 Flash.

    Citation

    Cited as:

    Jang, Lawrence, Alexander Yue, Magnus Müller, Gregor Zunic, Andrew Keunwoo Jang, Younghan Park, Mareks Woodside, Daniel Fried, Jing Yu Koh and Ruslan Salakhutdinov. (Oct 2026). Odysseys2: A Web Agent Benchmark Based on Real User Requests. https://odysseysbench.com/v2/.

    Or

    @misc{jang2026odysseys2,
  title        = {Odysseys2: A Web Agent Benchmark Based on Real User Requests},
  author       = {Jang, Lawrence and Yue, Alexander and M{\"u}ller, Magnus and
                  Zunic, Gregor and Jang, Andrew Keunwoo and Park, Younghan and
                  Woodside, Mareks and Fried, Daniel and Koh, Jing Yu and Salakhutdinov, Ruslan},
  year         = {2026},
  howpublished = {\url{https://odysseysbench.com/v2/}}
}