BLACKBOX

BLACKBOX

A robot-fleet black box that replays a drive in Rerun with tracks, closeness, time to contact, and Cosmos event times.

BLACKBOX is a fleet black box on the VAST video stack. After a close call, the camera, the objects, the distance, and the model's read are on one clock.

The App button opens this page. The Rerun viewer plays Toronto AV drive 01, 13:25–14:00. That recording holds each clip as video, YOLO tracks and one-second trails, in-path proximity for people, bicycles, and vehicles, a time-to-contact lane, and Cosmos3-Reason event lines on the risk-2 clips.

On the 13:55 clip the ego car is slowing on a narrow street. The recording has 65 tracks. A bicycle in the centre path reaches proximity 0.84. The shortest time to contact is 1.8 seconds, on a car. Cosmos times a pedestrian in the path at 13:55, 13:56, 13:58, and 13:59. The stored caption says cyclist.

We did not build the video stack. VAST stores the clips and the index. Cosmos captions each five-second segment. YOLO writes the boxes. Rerun supplies the viewer. We added the episode view, the exporter, the tracks, proximity, time to contact, and the Cosmos timing pass.

Risk and event names on 13:55 are still inferred. Re-ingest has replaced 2 of 6 clips on each of the two parent chunks, and 13:55 still has the old caption.

Team size: 2
Kevin Chen (submitter, already signed in)
Chloe Gao — email: the address she registered with

The form already counts the signed-in person. Enter only Chloe on the extra row.

VAST Data
NVIDIA
SpaceXAI (Cursor)
CoreWeave (Weights & Biases)
Rerun (add your own)
YOLO (add your own)

https://team-27-app.thecosmoslabs.com/app/

https://team-27-app.thecosmoslabs.com/app/replay/

Same recording the App button used to open directly: toronto-av01-cyclist, 13:25–14:00.

https://team-27-app.thecosmoslabs.com/app/blackbox-demo.mp4

2 minutes 15 seconds. Problem, solution, starting point, our work, result, next steps.

https://github.com/labcc/robo-obs

This repo is private. In “Private repository or video? Add judge access”, say how reviewers get in. A private link by itself does not grant access.

The workshop App path rewrites onto one pod. / is this page. /replay/ opens Rerun 0.38 on the recording built at startup.

export_rrd.py reads the BLACKBOX episode, then pulls each clip and its YOLO boxes from the in-cluster video service. It writes one .rrd. Boxes of the same class are linked into tracks when their overlap is at least 0.3, with a one-second trail. In-path proximity is the lowest box edge inside the centre 30 percent of the frame, so a higher number is closer. Time to contact is box-height growth over half a second. There is no depth sensor. For risk-2 clips the exporter asks Cosmos3-Reason for event times and logs them on that timeline.

The Rerun web viewer is proxied at /viewer/ so its relative asset URLs stay under /app. Recordings are served from /recordings/.

1. Repo: https://github.com/labcc/robo-obs on branch cursor/rerun-web-viewer-69c3.
2. From the team environment, run ~/robo-obs/rerun-app/deploy.sh. It expects the blackbox-creds secret created by ~/robo-obs/deploy.sh.
3. The default window is toronto-av01-cyclist, 13:25 to 14:00. Add windows with RERUN_EXPORTS='name=<episode_id>:<from_s>:<to_s>' ./deploy.sh.
4. Open https://team-27-app.thecosmoslabs.com/app/ and choose Open the Rerun viewer.
5. The demo video is https://team-27-app.thecosmoslabs.com/app/blackbox-demo.mp4. A pod restart rebuilds the recording and drops that file until it is copied back.
6. ~/robo-obs/deploy.sh puts the earlier BLACKBOX app back on /app.

The captions and the boxes were already in the stack, on different clocks. Putting them on one timeline is the product.

Pods cannot resolve the external video hostname, so the exporter talks to the in-cluster video service.

Re-ingest can copy the clips and then sit at 2 of 6. A missing job record does not mean the clips were labelled. 13:55 still has the old caption, so its risk and event names are inferred.

Cosmos's own timing can disagree with the stored caption. On this clip it says pedestrian where the caption says cyclist, and another run can differ.

This machine cannot render the Rerun canvas, so the demo video is the real 13:55 clip with the same tracks, proximity, and time to contact drawn on it.

https://team-27-app.thecosmoslabs.com/app/shots/shot-viewer.jpg
https://team-27-app.thecosmoslabs.com/app/replay/

One URL per line in the form’s additional-links field.