JarvisBench

A focused benchmark for two questions: does Jarvis help an Agent finish the work, and does it keep the user meaningfully in the loop?

Source benchmark WildClawBench 60 real-world OpenClaw tasks

Selected subset 34 text-only tasks Long-horizon, tool-using workflows

Track 01Agent-Collaboration Track 02User-Interaction

JarvisBench selects the 34 text-only tasks from WildClawBench, spanning browser, shell, file, search, communication, scheduling, and safety workflows in a live OpenClaw environment. During execution, GPT serves as an online user simulator with two personas: an expert who provides guidance in the Agent-Collaboration track and a nonexpert who seeks frequent interaction in the User-Interaction track.

TRACK 01

Agent-Collaboration

Does Jarvis help the same worker Agent complete more tasks?

The original OpenClaw loop is compared with the same worker augmented by Jarvis. Jarvis surfaces sparse failure checkpoints, carries expert guidance back into the ongoing trajectory, and never solves the task itself.

Worker model	Without Jarvis	With Jarvis
GPT-5.4-mini	44.24%	49.11%
GPT-5.5	50.54%	58.45%
Gemini-3.1-Pro	43.14%	54.78%
Claude Opus 4.7	64.01%	75.79%

Without Jarvis: The original OpenClaw worker loop.
With Jarvis: The same worker with Jarvis mediation.
Score: WildClaw task-completion percentage over 34 tasks.

TRACK 02

User-Interaction

Can Jarvis help a nonexpert in real time while the Agent keeps working?

The worker is fixed while the Jarvis brain changes. Users ask short progress or task-topic questions without providing task-solving guidance.

Jarvis brain	Latency (s)	Progress	Topic
Qwen3-30B-A3B-Instruct	0.69	3.46	2.93
Qwen3-32B-Thinking	2.89	3.53	3.24
GPT-OSS-120B	1.12	4.07	3.59
GPT-5.4 mini	1.09	4.02	3.43
GPT-5.4	1.39	4.02	3.91

Latency: End of user speech to start of Jarvis speech.
Progress: Average 1-5 score grounded in the current Agent trace.
Topic: Average 1-5 score consistent with the task context.