SentinelBench
A benchmark for long-running monitoring agents. 100 tasks across 10 live web apps, where the right move is often to wait, keep watching, and respond at the right moment.
Agents are getting good at long stretches of work, and most benchmarks test them by asking for continuous action, which means clicking, typing, searching and doing it again. A lot of real work is not like that. No amount of reloading makes concert tickets go on sale sooner. Someone has to keep an eye on the page and act when the moment comes.
SentinelBench measures that. We built working replicas of ten familiar kinds of website, among them an email client, a team chat, a stock trading app and a code host. Each one is a real application with its own server and database, so emails get marked as read, orders go through, and comments post. A task runs inside one of these apps while it changes on its own schedule, and the agent has to notice the one change that matters. We score whether it noticed, how fast it reacted, and how much it spent while it waited. Tasks run for ten minutes by default, and the same tasks can be stretched to forty minutes, to hours, or to days.
Ten apps that change while you watch
These are real screens from the benchmark, captured as its scripted events fire. Pick an app and watch one task play out.
How SentinelBench works
The agent gets one sentence and a browser. Behind the page, a server replays a scripted sequence of events on a clock, such as new emails, price moves, posts and issues. One of them makes the task's condition true. The agent succeeds if it reports after that moment and not before, having done anything the task asked for along the way. How it gets there is up to the agent.
Tasks come in a few kinds. Passive tasks only need the agent to notice something. Active tasks need it to act along the way, such as opening each conversation to check for an attachment. In no-op tasks the condition never happens, so the only correct outcome is to still be watching when time runs out.
Results
We gave each model one of two ways to wait. An ordinary sleep tool pauses for a number of seconds the agent chooses. A purpose-built wait_for tool takes a condition in plain language and waits until the page changes, and it is described further down. We ran three models with each. GPT-5.4 at low reasoning effort stands in for a current agentic model, GPT-4o for an older one, and Qwen 3.5:9B for a small model hosted locally. At the default ten minutes, success ranges from 46% to 75%, and the tool an agent is given changes its cost much more than it changes its success. Every number in this section is for ten minute tasks, the shortest setting, unless a chart says otherwise.
Leaderboard
Success over 100 tasks of 10 minutes, with median cost and median reaction time
Reaction time is measured on successful tasks only, so the two tools are compared on different subsets.
For every model, wait_for gives similar or better success for less money. The two weaker models gain almost nothing in success from the better tool. Most of what they miss are active tasks, where acting is the hard part.
Success by task type
Share of tasks of each kind completed, on 10 minute tasks
Active tasks are the hard part for every model. No-op tasks look easy, but GPT-5.4 with sleep fails five more of them than with wait_for, because it sometimes stops early and reports that nothing happened.
Ten minutes is the short version
At ten minutes the best agent we tested already finishes three quarters of the tasks. That is the easy setting. Every task can be stretched, and the longer the wait, the more an agent's way of waiting matters. With sleep, the agent in practice wakes every few seconds to look again and pays for every look. With wait_for, it pays only when the page changes.
Stretching the same 100 tasks from ten minutes to forty is as far as we have measured. Over that one step the sleep agent drops from 68 to 56 tasks completed and its cost per task quadruples, while the wait_for agent barely moves. Hours and days are the settings the benchmark is built for, and we expect the gap to keep widening there.
GPT-5.4 as tasks get longer
Median per task
A tool built for waiting
Alongside the benchmark we describe wait_for, a deliberately simple tool. The agent calls it once with a condition in plain language and a timeout. The tool snapshots the page text every second, diffs it against what it has already seen, and only when something new appears does it ask the model whether the condition is now true. Quiet stretches cost nothing.
Same page, two ways to wait
A mock-up to show the idea, with no model running behind it. Post something and watch which agent wakes up.
The sleep agent wakes on a timer and reads the whole page each time, whether or not anything is new. The wait_for agent does nothing until the page text changes, then sends the model only the lines that are new.
wait_for(condition, timeout):
seen = snapshot(page)
until timeout:
every second:
now = snapshot(page)
changes = new_blocks(diff(seen, now))
if changes and model_says_yes(changes, condition):
return
reload the page now and then, for pages that do not update themselves
The tool saves money and does not save time. In our runs GPT-5.4 reacted a few seconds slower with wait_for than with sleep, while spending about a fifth as much.
A small world, entirely made up
Every face, post, song, video and company in the benchmark is synthetic. The same 100 personas and 201 organisations appear across all ten apps, so the person who emails you also posts in the feed and shows up in the chat. Nothing here is a real photo or a real name.
Hover or tap a tile to see what it is.
Paper
SentinelBench: A Benchmark for Long-Running Monitoring Agents
1University of Florida 2Microsoft Research, AI Frontiers
arXiv 2606.05342 Blog post Code
Citation
@article{maldaner2026sentinelbench,
title = {SentinelBench: A Benchmark for Long-Running Monitoring Agents},
author = {Kunzler Maldaner, Matheus and Fourney, Adam and Swearngin, Amanda and
Mozannar, Hussein and Bansal, Gagan and Murad, Maya and Hosn, Rafah and
Amershi, Saleema},
journal = {arXiv preprint arXiv:2606.05342},
year = {2026}
}