QMVDet

讓模型自己判斷哪台相機可信

Query-based multiview detection with a camera-aware attention scheduler: reliability is measured as agreement between 2D and 3D, guiding learning only when it matters — a new benchmark on both datasets.

DOI ↗ interactive — scroll to step through the method

▸

Background

A quick primer on multiview detection and why some camera views can't be trusted.

Multiview detection places several overlapping cameras around a scene and fuses their views to locate objects — usually people — on a shared ground plane (a bird's-eye map). Multiple views help because an object hidden from one camera is often visible to another, which is how the method fights occlusion.

But cameras aren't equally trustworthy at every moment — glare, distance, crowding, or a bad angle can make a view unreliable, and naively averaging all cameras lets one bad view corrupt the result. QMVDet is a query-based detector with a camera-aware attention scheduler: it estimates each camera's reliability as the agreement between 2D (per-camera) and 3D (ground-plane) evidence, and leans on a view only when it is trustworthy.

▸ key terms

Multiview detection — fusing several cameras to detect objects. Ground plane / BEV — the shared bird's-eye map the views are projected onto. Occlusion — an object blocked from one camera's view. Attention — letting the model weight the informative inputs more heavily.

METHOD · 2D guides 3D via QBL

93.1% · MultiviewX 95.1% MODA robust even when cameras drop out

Multiview detection fuses many cameras onto one ground plane — but weighting them equally ignores occlusion.

STEP 00 · the problem

Not every camera is equally reliable

Multiview detection projects feature maps from many cameras onto one ground plane to beat occlusion. The strong baseline, MVDetr, weights every camera equally.

But occlusion depends on object position and camera angle — equal weighting leaves accuracy on the table.

STEP 01 · 2D detection

A 2D network to anchor the truth

A single-view detection-by-tracking network (FairMOT on DLA-34) produces reliable 2D foot points, using tracklets to fill in missed detections under occlusion.

STEP 02 · project & encode

Lift each view to the ground plane

Perspective transformation maps each camera's feature map onto the bird's-eye ground plane; a deformable transformer encodes per-camera BEV features.

STEP 03 · 2D–3D consistency

Reliability = agreement between 2D and 3D

Project the predicted 3D foot point back to 2D and measure its discrepancy against the 2D detection. Small discrepancy means a trustworthy camera.

Averaged per camera, this becomes a camera-aware attention weight Ac.

STEP 04 · QBL scheduler

Guide learning — only when it matters

Cameras are aggregated by attention-weighted averaging, F = (1/C)·Σ Ac·fc. Because the consistency computation is expensive, a query-based-learning scheduler only steps in when the camera weight order shifts — gated by an entropy change above 0.1.

The 2D network acts as an oracle that schedules when to teach the 3D detector — a scheduler, just like in my generation work.

STEP 05 · the result

A new benchmark on both datasets

QMVDet sets state-of-the-art MODA on Wildtrack (93.1%) and MultiviewX (95.1%), beating MVDet, MVDetr and 3DROM.

It even stays robust when some cameras drop out.