ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search

Abstract

Person search is normally posed as one-shot retrieval from a complete query, either a cropped image or a full text description, matched against a static gallery. Witness testimony rarely arrives in that form. It comes as fragments about who was seen, roughly where, and roughly when, often hedged and sometimes inconsistent.

We formulate interactive multi-camera person search, in which an agent recovers the target by questioning a witness over several turns while reasoning over the spatio-temporal structure of the camera network. To study the task we build ARGOS, a benchmark of 2,691 tasks across 14 scenarios in an industrial factory and a university campus, derived from 16 synchronized cameras and 1,273 annotated identities in MTMMC.

Each scenario is grounded in a Spatio-Temporal Topology Graph (STTG), a directed weighted graph whose nodes are cameras and whose edges carry measured transition-time statistics. Ground truth is computed from the STTG and the tracking annotations, and a language model is used only to phrase the witness's replies, so benchmark correctness never depends on LLM behavior.

We also release a tool-augmented agent baseline with four modules and eight tools. Across four LLM backbones the best agent reaches a Turn-Weighted Success of 0.383 on the spatial track and 0.590 on the temporal track, against an oracle score of 1.000. Ablations attribute most of that performance to the spatio-temporal tools: removing the temporal tool costs 49.6 points of Top-1 accuracy, and single-pass inference without interaction reaches 11.3%.

Motivation

The query is never complete

Person search research assumes the query arrives whole, the match happens once, and appearance carries all the information. Each of those assumptions breaks as soon as a real witness starts talking.

Prior formulations

ARGOS

✗

Complete query

A visual crop or a full text description, given upfront and assumed sufficient.

✗

One-shot matching

A single similarity search over a static gallery, with no second chance to ask.

✗

Appearance only

Even multi-modal ReID reasons purely about how the person looked.

→→→

✓

Partial clues

Fragments collected over dialogue turns, hedged and sometimes contradictory.

✓

Interactive reasoning

The agent plans, asks, and filters under uncertainty within a turn budget.

✓

Space and time

Camera topology and measured transit times rule candidates out.

New task

Interactive multi-camera person search

An episode opens with a vague witness statement. At each turn the agent picks one action: ask an attribute question, query the gallery, invoke a spatial or temporal tool against the topology graph, or commit to a prediction. It has to decide what to ask, when to spend a turn on a tool call, and how to read a hedged answer, all inside a 20-turn budget.

Walkthrough

Four episodes, replayed turn by turn

Each case below is a single benchmark episode run twice under identical conditions: the same witness, the same initial candidate set, the same turn budget. The only thing that changes between the two columns is the capability named in the tab. Step through the turns to watch the candidate set move.

Initial witness statement

Candidate trajectory

candidates left

Grounding

Spatio-Temporal Topology Graph

A directed weighted graph whose nodes are cameras and whose edges carry measured transition-time statistics. We build it from multi-camera tracking ground truth through transition extraction, priority-based edge labeling with human verification, and directed-edge aggregation, giving 110 edges for the factory and 149 for the university. The graph serves two roles: tasks are derived from it, and the agent queries it as a tool.

Benchmark

2,691 tasks across three progressive tracks

All tasks draw from a common gallery of persons observed by 16 synchronized cameras in two environments, an industrial factory and a university campus, sourced from MTMMC. Track 1 isolates semantic parsing of the witness statement in a single turn; Tracks 2 and 3 add spatial and temporal reasoning on top of it.

2,691

Tasks

Scenarios

1,273

Annotated persons

Synchronized cameras

	Track 1Who	Track 2Where	Track 3When
Task count	989	550	1,152
Interaction	Single-turn	Multi-turn	Multi-turn
Core capability	Semantic parsing	Spatial reasoning	Temporal reasoning
Avg. oracle turns	n/a	2.02	1.89
Primary metric	Top-1 Acc	TWS	TWS

Table 1. Benchmark statistics. Track 1 is independently evaluable, while Tracks 2 and 3 presuppose its semantic parsing. Turn-Weighted Success (TWS) scores correctness weighted by turn efficiency relative to the oracle path, so a perfect agent scores 1.000.

Baseline

ARGOS Agent: four modules, eight tools

A tool-augmented LLM agent running an observe, think, act loop. The Analyst estimates how far each candidate action would narrow the candidate set, the Planner selects the next action, the Interviewer executes it, and the Interpreter converts the witness's reply back into a filter. The environment is deterministic, so failures trace to parsing, reasoning, or strategy rather than to noise.

ARGOS agent system architecture — **Figure 3. System architecture.** Left: the four LLM-driven modules and the eight tools they issue. Right: the evaluation environment, comprising a witness simulator, a deterministic state manager (gallery, filter, and session services), and an evaluator. The agent selects actions; the environment executes them and returns the resulting state.

Experiments

Far from solved

Four LLM backbones, a 20-turn budget, temperature 0. The best agent reaches TWS 0.383 on Track 2 and 0.590 on Track 3, both far below the oracle's 1.000, and no single backbone leads on both tracks.

	Track 2 (Where, spatial)				Track 3 (When, temporal)
Backbone	TWS	Top-1	SR@5	AvgT	TWS	Top-1	SR@5	AvgT
Oracle	1.000	100.0	100.0	2.05	1.000	100.0	100.0	1.88
GPT-5.2	0.338	73.1	33.6	7.04	0.590	88.2	65.8	3.91
GPT-4o	0.323	74.5	31.6	7.49	0.567	80.6	60.8	3.91
GPT-5-mini	0.319	74.9	32.2	7.48	0.556	88.0	60.5	4.40
Claude Sonnet 4	0.383	76.0	38.4	6.71	0.548	83.6	59.5	4.25

Table 2. Main results on Track 2 (spatial) and Track 3 (temporal). SR@5 is success within a 5-turn budget; AvgT is the average number of turns on solved tasks. Best non-oracle TWS per track shown in teal.

		Track 2 (Where)		Track 3 (When)
Agent	Removed	TWS	Top-1	TWS	Top-1
Oracle	upper bound	1.000	100.0	1.000	100.0
ARGOS Agent	none	0.323	74.5	0.567	80.6
w/o Strategy	reasoning	0.136	47.6	0.373	76.9
w/o Spatial	spatial tool	0.063	40.7	n/a	n/a
w/o Temporal	temporal tool	n/a	n/a	0.054	31.0
Single-Pass	interaction	n/a	64.7	n/a	11.3
w/o All Tools	all tools	n/a	n/a	n/a	11.4

Table 3. Ablation with GPT-4o as the backbone. Removing the temporal tool costs 49.6 points of Top-1 on Track 3, and removing the spatial tool costs 33.8 points on Track 2. With no tools at all, Track 3 collapses to 11.4%, which is the level of random guessing.

Grounding tools carry most of the signal

One feasibility query against the STTG narrows 19 candidates to 3. Physical reachability is a hard constraint, whereas attribute answers are frequently hedged or unknown.

The gain comes from the loop

Single-pass inference reaches 11.3% on Track 3 against 80.6% for the agent, and 11.4% with no tools at all. Interaction grounded in the STTG is what produces the difference.

Ground truth is computed, not generated

Answers are derived algorithmically from the STTG and the tracking annotations. A language model only phrases the witness's replies, so correctness never depends on LLM behavior.

The benchmark is far from saturated

Agents that do find the target still spend three to four times the oracle turn count, which is exactly what TWS penalizes. No backbone leads on both tracks.