RealtimeBench introduces 180 tasks to test agents on a live clock

News update: this post was researched and drafted automatically from the sources listed at the end, and published without a human edit. If you spot a mistake, write to me and I will fix it.

I read a paper from arXiv, submitted on 14 October 2026, that puts LLM agents on a live clock. The authors introduce RealtimeBench, 180 tasks across six game types. The important change is that the games keep running while the agent is thinking and calling tools. Most benchmarks pause the environment. RealtimeBench does not.

This matters because an otherwise correct action can arrive too late. A chatbot that takes six seconds to answer a factual question may be fine. A monitoring agent that takes six seconds to respond to a changing state may be useless. The paper reports paired evaluations. GPT-6.1-sol outperforms the faster Jev when the games are paused, but falls behind when they keep running.

In one line: RealtimeBench is a benchmark of 180 live-clock tasks across six game types that shows a slower, stronger model can lose to a faster one when the environment does not wait, and TORSI lifts the fast student Jev from 13.51 to 24.19 on a 0 to 100 progress scale.

What RealtimeBench measures

RealtimeBench evaluates agents while games continue running during inference, according to the paper. Progress is scored on a 0 to 100 scale. The authors compare two models, GPT-6.1-sol and Jev. When games are paused, GPT-6.1-sol does better. When games run, Jev takes the lead. That reversal shows the benchmark is testing something current evals miss.

Why paused benchmarks are not enough

Most agent benchmarks assume the world waits. That assumption is fine for static documents or offline tasks, but not for anything that changes while the agent works. A related paper, When Reality Outpaces Cognition, asks how quickly reality can change before a system cannot reliably perceive, reason and act. It proposes effective Speed of Coherent Cognition and evaluates reinforcement-learning agents as the speed of reality increases by up to 50 times without retraining. The authors report that an architecture called Co4 maintains higher effective speed than Transformer and gated-MLP baselines. A third paper, StepSafe, studies timely safety intervention at the step level. Its authors argue that whether an action is safe depends on prior actions, tool outputs and disclosures. They report a state-of-the-art open-source guardrail on five agent-safety benchmarks with a Youden's J of 53.0 for a 4B model. The focus is safety, but the underlying issue is similar: step-level timing matters.

TORSI: teaching a fast student from a slow teacher

The RealtimeBench paper also proposes a training method called Two-Timescale Offline Recursive Self-Improvement, shortened to TORSI. A slow teacher studies failures and tests corrections in paused games between attempts. It then writes reusable instructions for a fast student. The student acts without teacher calls during gameplay. Validation scores decide which updates are adopted. Two rounds raise Jev's average test score from 13.51 to 24.19 on the 0 to 100 progress scale. The paper reports improvements in all six game types. The instructions also improve average scores for three other models without retuning. That suggests the fast student does not need to become smarter. It needs better standing instructions.

What this does not yet do

RealtimeBench is a benchmark and a proof of concept. The tasks are games, not production systems. The paper does not claim the method works for arbitrary agent workflows. It also does not give a direct recipe for client deployments. The score improvement from 13.51 to 24.19 is meaningful but still far from full progress. For a team shipping software, the main value today is a clearer question: does your agent act before the world changes?

What I would try

For client delivery, I would add a live-clock check to any agent that writes or changes something in a shared system. For RAG chatbots, latency is usually measured as time to first token, but time to correct action is different. I would not adopt TORSI yet. The method is fresh and the benchmark is game based. Probe-and-Resume offers another timing idea: evaluate an agent only when it claims to be done, and give the critic access to a fork of the state so it can probe hidden details. I would watch whether the instruction approach from TORSI transfers to real tool calls and partial state. Nothing to build this week, but a test worth adding to the evaluation suite.