How FRC 3847 / 8515 turned an NVIDIA Jetson Orin Nano Super into a GPU-accelerated AprilTag vision coprocessor: what we built, why, and how to redo it. The detailed technical reference (exact versions, commit hashes, every measurement) is in docs/TECHNICAL.md.

🎓 Vision Training Course: Check out our interactive, 17-chapter visual guide on how the Jetson vision system works at spectrum3847.github.io/SpectrumJetson.

Thank you, Austin Schuh. The CUDA AprilTag detector at the heart of this build is Austin's work, first written for FRC 971 Spartan Robotics. He's the reason any of this works. He now develops it in RealtimeRoboticsGroup/aos (frc/orin). We built from his copy in frc971/bos, inside FRC-Team-4143's CUDA PhotonVision. Full credits are at the end.

Built by Spectrum 3847 with Claude Opus 5.5 (Anthropic) in Claude Code, which did the research, code, patches, tests and docs alongside the team.

Last updated September 24, 2026.

Measured on the bench:

• 2 AprilTag cameras at 120 fps each, full resolution (1280x800), about 15 ms latency, using about 25% of the CPU. The cameras are Thrifty Bot Thriftiest Cams: mono, global shutter, USB 2.0, $50 each. The GPU finds the tags in about 2 ms per frame and is only about 12% busy (peaks under 25%).
• Set up for 4 AprilTag cameras on the USB-A ports, plus a 5th camera on a USB-C hub (tested: all 5 streaming). All of them share one USB 2.0 budget, which 5 cameras fill.
• Game-piece detection at 30 fps alongside the AprilTag cameras, with no measurable slowdown to them (76 fps if uncapped). It found FUEL surprisingly well even on our mono camera.
• Rewind: robot code can record every camera at 30 fps, for 3% of one core, and you can download the recordings from the web UI.
• Accurate timing: frames are timestamped at mid-exposure and synced to the robot's clock.
• Match-ready:
  • detecting tags about 20 s after power-on
  • an unplugged camera is back in about 1 s
  • built for power cuts: the system partition is only written by updates, settings and recordings live on their own partitions, and a damaged one never stops a match (see Storage; the power-cut test on this layout is still to be repeated)
  • restarts itself after errors
• Works with stock PhotonLib on a SystemCore (2027 alpha-2).

For a new Jetson, or a spare SSD. You need an Orin Nano (Super) devkit, an NVMe SSD of at least 80 GB, 128 GB or more for Rewind recordings (not a DRAM-less budget drive: see Storage), a USB-C cable, Wi-Fi for the Jetson, and an Ubuntu 22.04 laptop. The 80 GB is the 64 GiB system partition, the 2 GiB settings partition and at least 8 GiB of scratch.

1. Once per laptop: first edit config.env for your team (team number, Wi-Fi connection name, fan mode): the next step bakes the Jetson's user and hostname into the image. Then scripts/host/01-prepare-bsp.sh downloads and unpacks NVIDIA's flashing tools and creates the Jetson's account (about 15 minutes). After 01-prepare-bsp.sh --force (a fresh unpack), run step 2 again.
2. Prepare the image: scripts/host/prestage-rootfs.sh puts passwordless sudo, the laptop's saved Wi-Fi connection and its SSH key into the image, so nobody has to touch the Jetson after the flash.
3. Flash (about 7 minutes): put the Jetson in Force Recovery Mode (jumper FC REC to GND on the button header, pins 9-10, then power on; or sudo reboot forced-recovery on a running Jetson), then scripts/host/02-flash-nvme.sh.
4. Set up (unattended): scripts/host/setup-jetson.sh --prebuilt (about 35 minutes: it installs our newest release's prebuilt libraries and PhotonVision build), or without --prebuilt to build everything from source (about an hour). Optionally add --settings photon.sqlite to restore a saved setup and --model model.onnx for game pieces. It waits for the Jetson to boot, copies everything over and runs scripts/jetson/install.sh there. If a step fails, fix it and run the same command again: it carries on from that step.
5. Check: open PhotonVision at http://192.168.55.1:5800 and its Match Ready page, or run scripts/jetson/health-check.sh on the Jetson.
6. Before competition, once it's tuned: scripts/jetson/ro-root.sh on makes the system partition read-only (and saves the current settings as the fallback copy). To update anything later: ro-root.sh off, update, ro-root.sh on.

Prebuilt releases. A release's spectrum-jetson-prebuilt-<version>.tar.gz holds what the slow steps compile: the three allwpilib libraries the detector needs at runtime (the full allwpilib build is 17 minutes), the CUDA detector, the TensorRT and JPEG-hardware libraries, the field-calibration tool, and our PhotonVision jar. Its MANIFEST names the repo commit it was built from, the L4T and CUDA versions, every patch, and each file's SHA-256; install.sh --prebuilt checks all of it, and refuses a bundle built for another L4T or CUDA. The camera driver is still compiled on the Jetson (41 s), for its exact kernel. To make a bundle after a from-source install: scripts/jetson/make-prebuilt-bundle.sh on that Jetson.

The rest of this README explains what each step does and why.

We turned an NVIDIA Jetson Orin Nano Super into a vision coprocessor that finds AprilTags on its GPU, for team 8515's robot at the October 2026 off-season event.

The robot controller is a SystemCore running 2027 alpha-6 robot code. The Jetson runs PhotonVision, the same software many FRC teams use on an Orange Pi. Ours is a special version that sends the AprilTag math to the GPU using a detector written by FRC team 971. The robot code talks to it through PhotonLib over NetworkTables, like any other PhotonVision camera.

Everything we did is scripted in this repo, so another Jetson can be set up the same way. These notes explain what we did and why, including the mistakes, so you can understand the system and not just copy commands.

A camera frame goes over USB into PhotonVision on the Jetson. PhotonVision finds tags with the CUDA detector on the GPU, then sends results to the robot over NetworkTables.

flowchart LR
  CAM["2-4x Thriftiest Cam<br/>USB 2.0, MJPEG 1280x800"] --> PV["PhotonVision 2026 fork<br/>(4143 CUDA version + our patches)"]
  PV --> DET["971 CUDA AprilTag<br/>detector on the GPU"]
  DET --> PV
  PV -->|NetworkTables| SC["SystemCore<br/>2027 alpha-6 robot code"]
  SC -->|time sync| PV

Why a 2026 PhotonVision works with 2027 robot code: the CUDA version of PhotonVision only exists for 2026. We checked the source to confirm the two versions speak the same language:

• the messages have the same format (PhotonLib compares a hash of the message layout, and it matches exactly),
• the NetworkTables protocol is the same,
• time sync is the same.

The one rule: don't upgrade the robot's PhotonLib past alpha-6. Newer versions changed the message format, and the robot code would crash when it reads from this Jetson.

Flashing writes NVIDIA's operating system onto the Jetson's NVMe SSD and updates the bootloader stored on the board. It's done from an Ubuntu 22.04 laptop over a USB-C cable. The whole flash took about 7 minutes.

Why JetPack 6.2.3 and not 7: JetPack 7 now supports this board, but it moves to Ubuntu 24.04 and CUDA 13. The CUDA detector and PhotonVision fork were built for JetPack 6. 6.2.3 was the newest 6.x release, with bug fixes over the 6.2 in our original plan.

1. Download the two NVIDIA files, the "BSP" (flashing tools) and the "sample root filesystem" (Ubuntu itself), about 2.6 GB.
2. Prepare them on the laptop: scripts/host/01-prepare-bsp.sh. It unpacks everything and installs NVIDIA's flashing prerequisites. It also creates the Jetson's user account (spectrum3847, hostname photonvision-3847), so the first boot doesn't need a monitor and keyboard.
3. Put the Jetson in recovery mode:
   • Unplug power.
   • Jumper the FC REC and GND pins (pins 9 and 10) on the small 12-pin button header under the module. It's not the big 40-pin header.
   • Plug power back in, then remove the jumper.
   • Check with lsusb: the Jetson should show up as 0955:7523 NVIDIA Corp. APX.
4. Flash: scripts/host/02-flash-nvme.sh. The script temporarily stops Ubuntu's network manager from grabbing the Jetson's USB network connection, and temporarily opens the firewall for it. Both are common causes of failed flashes, and both are put back afterward.

Reading the flash log: "Waiting for target to boot-up" repeating for about 30 s is normal, and so are warnings like "backup GPT table is corrupt" and missing mmcblk0boot0. The flash is only finished at "Flash is successful". "Successfully flashed the external device" comes a couple of minutes earlier, and the bootloader is still being written at that point, so don't unplug the board then.

After flashing, the Jetson boots from the SSD and shows up on the laptop as a USB network device. The Jetson is 192.168.55.1, and the laptop gets 192.168.55.100.

1. Set up SSH keys so scripts can log in without a password: ssh-copy-id -i ~/.ssh/jetson_ed25519.pub [email protected].
2. Verify and enable MAXN SUPER: scripts/jetson/01-verify.sh checks three things. The root filesystem must be on the NVMe (/dev/nvme0n1p1), the software version must be R36.5.2, and the MAXN SUPER power mode must exist. It then switches to that mode (nvpmodel -m 2). Afterward, all 6 CPU cores run at 1728 MHz, and the setting survives reboots.
3. Get it online. Over USB it has no internet, and its clock is wrong until it syncs, which makes package downloads fail. Wi-Fi is easiest: sudo nmcli --ask dev wifi connect <SSID>.
4. Install CUDA with scripts/jetson/02-jetpack.sh, which runs apt install nvidia-jetpack. Flashing only installs the base OS, so CUDA 12.6, cuDNN and TensorRT come from this step.

Tip: never put a password in a script or a chat. On our Jetson the team account has passwordless sudo (/etc/sudoers.d/90-spectrum3847-nopasswd, added by the team so setup scripts can run over SSH); on a fresh Jetson, run sudo steps in a terminal where you type the password yourself.

The vision stack has four parts. Two are built on the Jetson, one on the laptop, and one is installed from PhotonVision's installer.

Where the detector code comes from. FRC 971 (Spartan Robotics) wrote the CUDA AprilTag detector. Austin Schuh, its author, now maintains it in the RealtimeRoboticsGroup/aos repo and works with team 1868. We started with FRC-Team-4143's copy (GpuDetectorJNI), which dates from about August 2024. We switched to frc971/bos, which has Austin's current code with a CMake build that works on our exact CUDA version. In a side-by-side test, the new detector found tags exactly as well as the old one, and 30–40% faster (1.7 ms per frame instead of 2.4–3.0 ms).

Why build on the laptop sometimes? The fork's jar is Java plus a web UI, with no native code, so it builds the same anywhere. The detector and allwpilib are native ARM and CUDA code, so they have to be built on the Jetson itself (or cross-compiled, which is more work).

Safe deploys. 06-install-fork-jar.sh refuses to install a jar that isn't a valid zip, and it keeps the previous working jar as photonvision.jar.prev. We added that after a truncated jar took PhotonVision down (see the bugs section).

Robot readiness. jetson/09-robot-tuning.sh prepares the Jetson for the robot: no automatic updates, headless boot, snapd off (it was adding 45 s to every boot), clocks locked at max on boot, USB autosuspend off for cameras, power-cut safety (data on the SSD within 3 s, the system log kept across power cuts), the fan on NVIDIA's quiet profile (FAN=full for full speed, FAN=off for a fanless heatsink: see Fanless), a 30 s hardware watchdog (also while rebooting, where it was 10 minutes), reboot on kernel panic, PhotonVision restarted on any exit, OpenCV's worker threads sleeping instead of spinning, 1 s USB retries (a stuck camera held up the others on its hub for ~65 s each), and dropping a network connection whose other end has vanished after ~13 s instead of ~15 minutes (a dashboard laptop that left without closing kept its camera streams encoding for nobody). The system log keeps up to 2 GB. After it, the Jetson boots in 16.5 s instead of 57 s, and both cameras are detecting about 20 s after power-on. jetson/health-check.sh prints a PASS / WARN / FAIL readiness report you can run over SSH before a match.

With the stock heatsink's fan sealed under an aluminium plate and thermal tape (so it can't move air), FAN=off 09-robot-tuning.sh keeps the fan stopped and takes it away from the kernel, which would otherwise switch it on at 74 °C. If the chip reaches 95 °C, spectrum-fan-guard sets a flag and PhotonVision caps every camera at 60 fps until it's under 88 °C (photonvision-57; Match Ready and Settings > Robot state show it). The chip itself throttles at 99 °C and shuts down at 104 °C.

Measured on the bench, 2026-10-01 (5 cameras, 4 on AprilTags at 122 fps, plate only, no airflow, lights on):

That's the pessimistic case: at an event the robot is off except in queue, on the field and for short pit checks, so every match starts much cooler. tests/thermal/event-sim.sh --arm, then a power cut and a cold start, measures that case. Earlier plate tests that seemed to level off at ~73 °C were held there by the kernel switching the fan on at its 74 °C trip.

scripts/jetson/fake-cameras.sh start [recording] makes virtual cameras (v4l2loopback) that play a Rewind recording frame for frame at its original rate, looping; with no recording given it generates one with AprilTags (detector/far_search_test --write-session). PhotonVision opens them like USB cameras, so the real path runs: cscore, our MJPEG decode, the GPU detector. While they run, PhotonVision uses throwaway settings in RAM (a copy of the real database's network and team settings, no cameras), so your real cameras' settings are never touched; fake-cameras.sh stop puts everything back, and so does a start that fails.

Uses: the browser test suite on a Jetson with no cameras plugged in (tests/ui), replaying a match recording through a new build, and bench load or thermal tests. Fake cameras have no exposure or gain controls and no USB bandwidth, so those features can't be tested this way. Needs sudo apt install v4l2loopback-dkms (install.sh --only fake-cameras).

The first SSD died from power cuts (2026-09-30). The Jetson stopped booting: the kernel couldn't read the root filesystem's journal ("critical medium error", "JBD2: recovery failed"). The drive's own health report told the story: an Inland TN320 (a budget drive with no DRAM cache and no power-loss protection), 15 hours of use, 22 power cycles, 21 of them unsafe (power cut, no shutdown), and 285 media errors: areas of flash it could no longer read. Its wear was 0%.

We had tested power cuts and called them safe: ext4's journal keeps a filesystem consistent through a cut. But that only holds if the drive keeps what it already stored, and this one didn't. No filesystem can protect against a drive that corrupts its own flash, and robots lose power without warning all the time. Our test checked that the Jetson booted again, not the drive's error count, so it missed the damage building up. Lesson: when a test says "it survived", check what the hardware itself reports too.

So the SSD is now split three ways (02-flash-nvme.sh and scripts/jetson/10-data-partition.sh):

Quiet mode (photonvision-56, Settings > Robot state): power cuts only damage an SSD while it's being written, and the robot is switched off a few minutes after every match. So when a match ends (field connected, teleop, then disabled), or after the robot has been disabled for 60 s, the Jetson stops writing: Rewind stops, PhotonVision's log file closes (the log still shows in the UI), the system log moves to RAM, and the scratch partition is remounted read-only. The cameras keep running and tags are still found. Any enable ends it: the partition is writable again about 80 ms later, and vision never waits on it (nothing on the match path writes to the SSD). A recording someone asks for always wins: field calibration, Record now, or robot code's /photonvision/rewind/record keeps quiet mode off, or ends it, for as long as it records. Robot code can also decide with /photonvision/jetson/quiet (true/false); the Jetson publishes quietNow and quietReason. tests/quiet-mode/run.sh plays a fake match and checks all of this: quiet 0.08 s after the match ended, a Record now ended it in 0.3 s, enable to writable in 0.08 s. A PhotonVision restart while quiet hands quiet mode over to the new process (photonvision-64, tests/quiet-mode/restart.sh).

Tested (tests/storage-fallback/run.sh scratch|settings, 2026-10-01): with either partition's superblock destroyed, the Jetson boots and PhotonVision serves. With scratch gone, its logs, the system log and Rewind go to RAM, and nothing lands on the system partition. With settings gone, it runs on the committed copy, with identical database contents. e2fsck -b 32768 then repairs the partition with its files intact. tests/ro-root/run.sh: with the read-only system on, PhotonVision and both partitions work, a file written to the system partition is gone after a reboot, and off makes it writable again (about 40 MB lands in the RAM layer per boot). Still to test: real power cuts on this layout, on a healthy SSD.

Choosing an SSD: use a known brand with a DRAM cache, or better an industrial drive with power-loss protection. Avoid DRAM-less budget drives. health-check.sh reports the drive's media errors and unsafe shutdowns; any media error means replace it. The installer saves the drive's health report at install time (~/install-logs/ssd-smart-at-install.txt).

None of this code was written for our exact setup, so we found and fixed several real bugs. Each fix is a small patch file in patches/, applied automatically by the build scripts. They're good examples of how real systems fail.

Lesson: check results by measuring, not by assuming. The truncated-jar bug happened partly because we trusted a log line from the old process. Now every check looks at the running process's own ID.

We went from 33 fps to the cameras' full 122 fps, on two cameras at once. The GPU was never the bottleneck: the detector takes under 2 ms per frame, fast enough for 500+ fps. The real limits were the camera exposure, how PhotonVision reads frames, and how it decodes them.

Exposure units are a trap. USB cameras count exposure in the UVC standard's 100 µs units, so 295 means 29.5 ms, not 0.3 ms, and stock PhotonVision doesn't say so. We only found this by reading the camera's control directly with v4l2-ctl. Our build now shows milliseconds on the slider, e.g. "Exposure (5.0 ms)" (photonvision-10).

Shorter exposure, lower cutoff. Shorter exposure means less motion blur, but a darker image and a lower decision margin (how confidently a tag decoded): about 44 at 3 ms, about 118 at 8.3 ms. PhotonVision drops tags below its cutoff, 35 by default. We run exposure 50 (5 ms) with the cutoff at 15. The detector still finds real tags, and tag36h11 at Hamming distance 0 almost never gives false positives.

Lights flicker. Mains lighting flickers 120 times per second (every 8.3 ms). At exposures that aren't a multiple of 8.3 ms, each frame can catch a different part of the flicker and pulse in brightness. In our shop at 5 ms we measured no flicker (0.6% brightness change frame to frame) with tests/flicker-check/run.sh. Run it again under the event's lights; if frames pulse by several percent, go back to 83 (8.3 ms).

Measure before optimizing. We added a once-per-second stats line to the detector (calls per second, milliseconds per frame, tags per frame, decision margin). It showed right away that the GPU was idle and the camera pipeline was the problem, which saved us from optimizing the wrong thing.

Four cameras share one GPU. With 4 cameras the GPU is only about a third busy, yet the slowest frames take 6–12 ms instead of ~3. That's queueing: a camera's GPU work sometimes waits behind another camera's, like a short line at a busy store. We tried two settings to shorten the wait: 32 hardware work queues for CUDA instead of 8, and letting the CPU sleep while the GPU works instead of spinning. The first test made the queues look like a big win, but repeating it showed the same settings vary by 2 ms from one restart to the next. Over 11 runs the queues were worth maybe 0.5 ms, and sleeping made no difference. Both are on anyway because neither costs anything (08-select-detector.sh sets the queues; details in TECHNICAL.md). Lesson: measure more than once before believing a speed-up. Also close the dashboard before measuring: 4 open camera streams added about half a CPU core and 1–2 ms to the slowest frames.

Four Thriftiest Cams at 122 fps (2026-09-29). The first time all four ran, PhotonVision used 2.2 CPU cores and the GPU was 40% busy. Three fixes later:

Each frame now takes less time with four cameras than one camera did before. Five cameras also run at 122 fps each: 2.1 ms detection, 44% GPU, 1.9 cores, 11.1 W (facing bright ceiling lights, the hard case). The limit is now USB bandwidth: five capped Thrifties use 6,400 of the ~6,720 bytes the Jetson's USB 2.0 ports share, so a 6th won't fit. What's left of the CPU is mostly inside NVIDIA's own libraries: how CUDA waits for the GPU, and the JPEG hardware's driver, which reopens the graphics device for every frame.

Where the latency goes. A result reaches the robot about 14.5 ms after the middle of the exposure. We timed each step with the camera driver's own log: half the 5 ms exposure (2.5 ms), then 8.1 ms for the camera to send the frame, then 0.15 ms until our decoder starts, 2.8 ms of JPEG decode, and 1.2 ms of detection. The camera streams each frame out over about one frame period while its sensor reads out, so neither USB nor PhotonVision is what's slow. The bos-04 change was checked by replaying ~5,000 recorded frames: every one of 3,567 tag detections came out byte-for-byte identical.

More cameras. Each camera at its full 122 fps now costs about 0.6 of a CPU core (it was 1.4 before the decode fix), and less with the JPEG hardware, so 4 cameras should fit. The limit was USB bandwidth. With the stock driver each camera reserves ~196 Mbps whatever mode it runs, and the Jetson's whole USB 2.0 side holds two of those (see below).

We fixed that with a patched camera driver (scripts/jetson/11-uvcvideo-payload-cap.sh). It caps the Thriftiest Cam's reservation at 82 Mbps (UVC alternate setting 7), still about 1.4x the largest frame we've measured at 120 fps. Now 4 cameras fit on the USB-A ports, plus a 5th capped at 1600 bytes on a USB-C hub. Tested with 2 cameras: 122 fps each, every frame complete. We first estimated this would cost ~2 ms of latency (a 50 KB frame crossing USB in ~4.9 ms instead of ~2 ms). Measured with 4 cameras, it costs nothing: the camera itself spreads each frame over ~8 ms while its sensor reads out, and the capped link keeps up with that. It does not make timestamps less accurate: the driver stamps a frame when its first USB packet arrives. Java's memory isn't a concern: the heap peaked at 28 MB with zero garbage collections in 20 s.

There is only one USB 2.0 bus. The USB-C port doesn't add bandwidth (measured 2026-09-24). The Jetson has one USB controller, with one USB 2.0 bus (lsusb Bus 1) and one USB 3 bus (Bus 2). Every USB 2.0 port is a branch of Bus 1 and shares its budget:

• the four USB-A ports (one hub inside the Jetson);
• the USB-C port;
• the M.2 Key E slot (the Wi-Fi card's Bluetooth).

We tested by setting cameras' USB alternate settings directly, with PhotonVision stopped. The bandwidth figures are bytes reserved per 125 µs microframe.

So about 6700 bytes fit, whichever ports the cameras are on (about 430 Mbps). With 4 cameras at 1280 and a 5th at 1600 it's full: a 6th USB 2.0 camera needs lower caps (check each camera still holds its frame rate), or one of these:

• A USB 3 camera: it streams on Bus 2, the USB 3 bus, which has its own budget. The USB-A ports are USB 3.2, and so is USB-C.
• A USB controller card in the spare M.2 slot: a second controller gives a second USB 2.0 bus.
  • The slot: the Orin Nano dev kit has an empty M.2 Key M 2230 slot, PCIe 3.0 x2 (NVIDIA's hardware layout, mark 11). The 2280 Key M slot holds our SSD.
  • Drivers: Key M slots carry only PCIe, not USB, so this needs a PCIe-to-USB controller card. The Jetson's kernel has the drivers (xhci-pci, plus the Renesas firmware loader).
  • Untested: most such cards are 2280 or need a riser, and the new ports must be mounted on the robot.
  • Not the Key E slot: its USB is Bus 1 again, and it holds the Wi-Fi card.

The Camera Matching page shows all of this (photonvision-32). A USB bandwidth card at the top draws the shared budget as a bar, one colour per camera, with what each actually sends filled in. Per camera it shows:

• FPS and Using: what the camera really sends, in MB/s and as a share of its allocation. At 90% or more it says "squeezed" (see below).
• Largest frame: the biggest recent frame, how many times it fits in one frame time at the camera's allocation ("fits 1.4x"), and how long it takes to cross USB.
• Allocation: what it reserves. Pick another from the list to change it:
  • it's saved for the next boot;
  • the camera is reset, like a replug, to apply it: about 2 s without frames;
  • choices that would go over the budget are greyed out, so to give one camera more, lower another first.

Each camera's card also gets a line like "USB bandwidth: 3.4 of 10.2 MB/s (port 1-2.4)".

What if a camera needs more than its allocation? It can't take more: the USB host lets it send only that many bytes per 125 µs. Its allocation is also guaranteed, so one camera never takes another's.

We tested what the camera itself does by giving two cameras less than they were sending (2026-09-24). Both kept their frame rate and compressed harder instead:

So a squeezed camera:

• sends lower-quality JPEGs, which may cost AprilTag range and precision (not yet measured with tags);
• adds latency, because each frame takes longer to cross USB.

We haven't pushed a camera so far that it drops frames.

Choosing an allocation:

• AprilTag cameras: keep them out of "squeezed" (under 90%), with "fits" at 1.3x or more.
• The cost of going smaller: latency. A 60 KB frame takes 5.9 ms to cross USB at 1280 bytes, and 8.0 ms at 944.
• Slower cameras have room to spare: a 30 fps game-piece camera or a 60 fps camera usually fits 2–6x.

The numbers come from our camera driver, which counts each camera's bytes and largest frames (kernel/uvcvideo-payload-cap.patch). Allocations set on the page are per USB port (e.g. 1-2.4:944) and override the per-model defaults from 11-uvcvideo-payload-cap.sh, which keeps them when reinstalled.

scripts/jetson/usb-bandwidth.py shows the same from the command line: what each camera reserves, which ports failed and why, and the cap command to fix it. The health check runs it too.

Timestamps mark mid-exposure (photonvision-13): the Jetson subtracts half the exposure from every frame's timestamp, so the robot shouldn't. The camera's own delay (readout and JPEG, before its first packet) is still to be measured on the robot with the spin-in-front-of-a-tag test, and set as SPECTRUM_CAMERA_DELAY_US. Frames are timestamped by the camera's own clock (see Steadier timestamps below).

What other teams' vision systems do (Austin's AOS as run by 1868, 4646 and 254, 971's bos and cos, 6328's Northstar, EagleEye, Code Orange's MLTag, 4533's Whacknet), what an ideal system would have, the full profiling story, and future work: docs/VISION-RESEARCH.md.

Both cameras use the same PhotonVision settings. Each one needs its own calibration at the resolution it will run at.

Camera settings (Dashboard, per camera). The camera's Input tab has a Tuning guide (photonvision-26): what to set, in order, with these values and why, and each control's tooltip gives its recommended value.

• The main point: short exposure on a robot. Auto exposure, or tuning by eye with the robot standing still, picks long exposures. Those blur as soon as the robot moves: spinning at 3 rad/s smears a tag about 11 px at 5 ms, and 44 px at 20 ms.

• Calibration is the exception: a long exposure is fine there because the board is still, but don't copy it into the vision pipelines.

• Type: AprilTagCuda

• Resolution: 1280x800 at 120 FPS, MJPEG. Don't use YUYV, which only manages 5 fps at this resolution.

• Auto Exposure off, Exposure 50 (5 ms), Brightness 100

• Camera Gain: the slider appears since photonvision-21 (upstream's Thriftiest Cam support), but our cameras' firmware reports no gain control, so it probably does nothing. Test it before relying on it (docs/UPSTREAM-PORT.md).

• AprilTagCuda tab: decision margin cutoff 15

• Low Latency Mode off

• Processing Mode 3D and multi-tag on (Output tab), once the camera is calibrated. The robot's pose code needs both.

• Stream Resolution: small, to save CPU (it only affects the video you watch in the browser)

• AprilTag field layout: 2026 Rebuilt AndyMark. The robot code must use the same layout.

New cameras start with these settings automatically. A camera PhotonVision has never seen gets AprilTagCuda, 1280x800 MJPEG, exposure 50, decision margin 15, brightness 100, white balance 2800 K, Low Latency off and multi-tag on (TeamCameraDefaults in photonvision-06). 3D can't work without a calibration, so it starts off and turns itself on (with multi-tag) as soon as a calibration is saved or imported for the resolution the camera uses. It never turns 3D off. Existing cameras keep their saved settings.

Cameras are named after their USB port. Every Thriftiest Cam reports the same name and serial number, so PhotonVision tells them apart only by the port they're plugged into, and each name (and its calibration) stays with its port. The robot code uses the same names, e.g. new PhotonCamera("TopLeft").

All the Jetson's USB 2.0 ports share one bandwidth budget: the four USB-A ports, the USB-C port, and anything on a hub (see Performance). Four cameras fit only with our capped camera driver installed (scripts/jetson/11-uvcvideo-payload-cap.sh --install; the health check shows which driver is loaded). With the stock driver, only two fit.

A calibration belongs to one physical camera and lens, so if you move a camera to another port, recalibrate it there.

Focusing a lens (photonvision-34): the Camera page's Focus card scores how sharp the image is, like Limelight's focus tool.

1. Turn on Measure, and point the camera at something detailed that stays still: a tag or the calibration board, about as far away as the tags you care about most.
2. Press Reset. Turn the lens slowly past the sharpest point, then back to where the centre reads 100%.
3. Check the 3x3 grid: each part of the image is shown against its own sharpest. A corner well below 100% when the centre is at 100% means a tilted lens or a soft corner.
4. Lock the lens (glue or silicone), then recalibrate.

The score depends on the scene and the light, so only compare numbers without moving the camera. It's measured only while the card is on, about 5 times a second, off the vision thread.

Copying settings between cameras (photonvision-25): no more photographing one camera's settings and typing them into another.

• Where: in the pipeline menu (☰ next to the pipeline name), Copy settings from… copies from any camera and pipeline into the one you're looking at.
• What: tick the groups: Camera (exposure, brightness, gain, white balance, stream resolution), AprilTag (decision margin and detector settings), 3D and multi-tag, Resolution, or Object detection.
• All cameras at once: it can also copy into the same pipeline number on every other camera.
• Never copied: orientation, names and each camera's exposure limits. Resolution is only copied between cameras with the same video modes.

Profiles (practice field, event field): give every camera the same pipelines at the same numbers, e.g. 0 = Event and 1 = Practice field.

• Making one (photonvision-46): in the new-pipeline dialog (☰ → +), Start from a copy of any camera's pipeline, and tick Create on every camera. Each camera gets it as its next number, and the message says which number each got; cameras that already have the same number of pipelines get the same number.
  • A copy from this camera is exact, like Duplicate.
  • A copy from another camera keeps each camera's own orientation, exposure limits and mask, and its resolution unless the two cameras offer the same video modes.
  • Only the camera on the dashboard switches to the new pipeline.
• Switching from the dashboard: ☰ → Switch every camera to pipeline N (the ⇄ icon) moves every camera to the number you're on. A camera without that number stays where it is, as do cameras in driver mode or calibrating; the message lists them, and warns when a camera's pipeline at that number has a different name.
• Switching automatically at events (photonvision-51): in Settings > Robot state, turn on Event pipeline when the field connects and pick the number. When the field (FMS) connects, every camera switches to it, once. After that anything sticks: a switch on the dashboard, robot code choosing a pipeline, or turning the setting off. The chooser lists what each camera has at each number, the report names any camera whose pipeline there is something else, and Switch now tries it without a field. health-check.sh prints the event pipeline and warns if a camera lacks it or has a different pipeline there.
• Numbers everywhere (photonvision-47): the Pipeline dropdown and the Camera Matching page show each pipeline as "N: name", so a mismatched profile is easy to spot.
• Robot code switches them all together too (the toggle is in issue #10), and the robot log records which one was active.
• Copy settings fills in the other cameras after tuning one.

More camera controls (photonvision-28): the Input tab now has Contrast, Gamma, Sharpness and Backlight Compensation, which stock PhotonVision doesn't show.

• Only the ones the camera has, with its own ranges; each tooltip gives the camera's default.
• Saved per pipeline. A pipeline that never set one uses the camera's default, re-applied on every switch.
• Leave them at the defaults unless you're testing. Sharpness is the one worth trying lower, since the camera's sharpening can put halos on tag edges.
• Copy settings includes them in the Camera group.

Leaving bad tags out of multi-tag (photonvision-22): if a tag is mounted wrong at an event, list it, and every camera's multi-tag solve ignores it.

• You list the bad tags, not the good ones.
• Where: type them on the Settings page's AprilTag Field Layout card (e.g. 7, 12, saved on the Jetson). Robot code can add more on /photonvision/excludedTags.
• Left-out tags are still reported as targets, with single-tag poses, and are dimmed in the tag table.

Measuring a camera's mount from the tags (photonvision-17):

• Where: with the robot level on the floor and 2+ tags in view, the Targets tab's Camera mount estimate shows the camera's height, pitch and roll on the robot, averaged over the last 100 samples with a ± spread.
• Why it works: while the robot is level, the camera's height, pitch and roll on the field are its mount's, wherever the robot is.
• Using it: the numbers use robotToCamera's axes and signs (negative pitch means tilted up), so they go straight into robot code.
• Yaw and X/Y also need the robot's pose. Robot code gets everything from /photonvision/<camera>/mount on NetworkTables, and at events it can compare the estimate with its configured mount to catch a bumped camera.
• Accuracy: it's only as good as the field's tag layout and a flat floor.

Calibration board settings (ChArUco, 5x5 markers, 30 mm squares, 22 mm markers):

The board is labeled "9x12", but PhotonVision needs width 12, height 9. With 9x12, every calibration failed with "Negative corner in reprojection error calc". We found the right setting with tests/charuco-board-check/check_board.py: it tried all four combinations on a live frame, and only 12x9 found corners (78 of 88).

Calibration tips:

• The board settings start at our board: ChArUco, 5x5 markers, 30 mm squares, 22 mm markers, width 12, height 9. Sizes are in millimetres (photonvision-23).
• After clicking Start, set Auto Exposure off and Exposure about 150. Calibration uses its own settings, and its default was nearly black. A long exposure is fine here because the board is held still; don't copy it into the vision pipelines.
• Take at least 100 snapshots. PhotonVision now requires 100 (upstream #2437); mrcal's docs show the returns diminish around there.
  • Click Auto Snapshots (1 per second) and keep moving the board slowly. It keeps a snapshot only when it finds the board.
  • Take Snapshot still takes one by hand, for a specific shot.
  • Cover every corner of the image, tilt the board up to about 45°, and vary the distance.
• Aim for a mean reprojection error under about 0.5 px (under 1 px is fine for FRC).
• If a calibration fails and gets stuck, restart PhotonVision (Settings → Restart Software) to clear the bad snapshots.

Bench calibration results (checked with tests/calibration-check/check_calibration.py, which reads the saved calibration and reports error, outliers and coverage):

We couldn't get the error below 0.5 px handheld, and that's okay. With so few outliers, the data is clean, and the remaining ~0.8–0.9 px is noise from MJPEG compression. The focal length came out at about 737 px in all three of camera 1's calibrations. Outliers are the better warning sign. Our second try had 42% outliers because many snapshots had the board mostly outside the frame. For edge coverage, keep most of the board in the image and just touch the edge.

Our replacement for Limelight Rewind. While robot code sets /photonvision/rewind/record to true in NetworkTables, PhotonVision saves every camera's video to the Jetson's SSD at 30 fps. For real matches that means while enabled with the FMS attached; for a specific test, just around the test. Afterwards, scripts/host/rewind-pull.sh copies a recording to the laptop and turns it into videos. Each frame is stamped with the robot's clock, so the video lines up with the AdvantageKit log.

• It saves the camera's own JPEG frames. Nothing is decoded or re-encoded, so it costs 3% of one CPU core, and detection fps and latency don't change (measured).
• About 1 GB per match with 4 cameras. The oldest recordings are deleted past 100 GB.
• For bench tests there's a Record now switch in Settings → Rewind.
• To download a recording, click the download button next to it in Settings → Rewind. You get a zip with one video (.avi) per camera; VLC or Ubuntu's Videos app plays it.

Everything else, including the robot-code example and how to line video up with a log, is in docs/REWIND.md.

A page in PhotonVision's web UI (Field Calibration in the sidebar, photonvision-30). With the robot on and disabled, pushed by hand:

1. Tune camera settings (optional, about a minute per camera).
   • Each camera has its own card: its live stream, the tags it sees right now with their margins, a slider for every setting, and its own Tune. Turn the robot so that camera faces tags, near and far, and press Tune.
   • The Jetson steps it through exposure, gain, brightness, contrast, gamma and sharpness under the event's lights. It recommends the shortest exposure that still finds every tag (less motion blur), and changes the others only if they clearly help.
   • Set any setting by hand with its slider: it saves to the pipeline in use, like the Camera page. Tick a setting's lock to keep it while the rest are tuned.
   • Apply recommended saves that camera's results. Each camera keeps its results until it's tuned again, so the robot can be turned between cameras.
2. Record the spots. Push the robot to 10–20 spots and hold it still at each until the page says the spot counts (about 2 s, and it beeps). A field map shows which tags need more views, where each camera is, and a suggested next spot. It can also be shown in 3D, using FIRST's own field CAD. Each element is placed by its tags, so it matches the AndyMark or welded layout in use, and after a calibration it shows where things really are. The mouse works like Onshape's: right-drag turns, middle-drag or Ctrl + right-drag moves, the wheel zooms.
3. Solve. The Jetson replays the recording through the same 971 GPU detector it uses in matches, then solves every tag's real pose and every camera's mount. It takes about a minute.
4. Results:
   • Tags: each tag marked as drawn or moved, with its offset.
   • Cameras: each mount compared with robot code's, plus a Java snippet.
   • Use this layout in PhotonVision: loads the corrected layout; Undo puts the old one back.

Robot code has to publish each camera's robotToCamera (a Transform3d struct) at /photonvision/<camera>/robotToCamera, from CAD or the last calibration. Without it, height, pitch and roll are still found, but x, y and yaw are only relative to one camera. The solved mounts come back at /photonvision/<camera>/fieldcal/robotToCamera.

How it works, the tests and the command-line version are in tools/fieldcal. The plan and the shop-test checklist are in docs/FIELD-CALIBRATION-PLAN.md.

PhotonVision's Object Detection pipeline runs YOLO models on the Jetson's GPU through TensorRT. That's our backend: photonvision-14 plus libspectrumtrt.so, built from detector/TensorRtYoloJNI.cu. Robot code gets normal PhotonLib targets, with class and confidence.

• Install a model from an ONNX export (built into a TensorRT engine on the Jetson; takes several minutes, so do it in the pit):

  ~/SpectrumJetson/scripts/jetson/12-install-yolo-model.sh ~/models/fuel.onnx "Fuel" Fuel

  Then give the camera a pipeline of type Object Detection and pick the model.

• The 2026 FUEL model we started with is Team 2826 Wave Robotics' YOLO11n, the same one PhotonVision ships for other hardware. The models we have, their licenses and exports: docs/GAME-PIECE-MODELS.md.

• It works on our mono cameras.

  • The Object Detection pipeline doesn't ask for gray frames, so a mono camera's JPEG is decoded to a colour image whose three channels are identical. The model runs on that.
  • With the hardware decoder on (--jpeg nvjpg), that colour decode runs on the Jetson's JPEG hardware too (photonvision-18): about 1.6 ms of CPU a frame instead of ~5.7 ms.
  • The model was trained on colour photos, where FUEL is yellow, yet it found real FUEL balls on the bench surprisingly well.
  • We judged that by eye on the stream. Detection rate at distance and false positives haven't been measured yet (Rewind recordings are good for that).
  • Inference costs the same on gray or colour, so the speed numbers below hold for a colour camera too.

• Measured on the bench (mono camera, 1280x800 in; measured with nothing else using the GPU):

  	AprilTag camera beside it	AprilTag detect (avg / worst)	GPU	PhotonVision CPU
  FUEL capped at 30 fps (default)	122 fps	1.62 / 3.75 ms (same as without FUEL)	11–30%	107%
  FUEL uncapped (76 fps)	122 fps	1.96 / 28 ms	29–44%	176%

  • Object Detection pipelines are capped at 30 fps by default (SPECTRUM_OD_FPS_LIMIT; a robot-set FPS limit takes precedence). At 30 fps they cost the AprilTag cameras nothing measurable.
  • Raising the cap: the hardware colour decode saves CPU, but uncapped FUEL also pushed the AprilTag camera's worst detect time to 28 ms by sharing the GPU, and hardware decode doesn't change that. Measure the AprilTag worst case before running game pieces faster than 30 fps.
  • Don't build TensorRT engines while measuring. A trtexec build uses the GPU hard for ~8 minutes and made our first measurements look like FUEL doubled the AprilTag detect time. It didn't.

• Which camera: a colour camera should do even better (FUEL is yellow), and a mono Thriftiest Cam works too. With 4 Thriftiest Cams on USB-A, a 5th USB 2.0 camera fits only capped (the USB-C port shares the same budget; see Performance), or use a USB 3 camera.

A Thriftiest Cam can get stuck after a USB hub reset, and only cutting its power fixes it (found 2026-09-25 with tests/usb-hub-reset/run.sh).

• What happens: when the Jetson's USB-A hub resets while the cameras stay powered, both Thriftiest Cams stop answering. We reset it in software; a static shock can do the same. The kernel logs "device descriptor read/64, error -110", then "unable to enumerate USB device".
• The Global Shutter cameras come back by themselves, in about 40 s: each stuck camera holds up the hub for ~20 s of retries first.
• What doesn't fix it:
  • switching the port's power off in software (the hub says it can, but the carrier board's USB 5 V is always on);
  • resetting the USB controller;
  • rebooting the Jetson: a reboot doesn't cut USB power either.
• What does: cutting the camera's power. Replug it, or power-cycle the robot.
• How you'll know: the camera's health/problem topic and the log say "not answering on USB port 1-2.1: replug it, or power-cycle the robot" (photonvision-33). The health check and usb-bandwidth.py name the port.
• Our decision: keep the Jetson's own USB ports. We won't add a hub that can really switch its ports' power. Hot glue and strain relief make loose plugs, a likely cause, less likely. If a camera is out, power-cycle the robot between matches.

A software reboot hung twice (2026-09-25).

• What happened: after sudo reboot the Jetson reset (the fan dropped from full speed) but never finished booting. A power cycle fixed it.
  • The second time, both Thriftiest Cams were stuck (see above). We don't know about the first.
  • The log from the last 2 minutes before that reboot is missing.
• Suspected cause, not confirmed: the boot firmware waiting on the stuck cameras, since a reboot doesn't cut USB power.
• What's covered now:
  • A shutdown that hangs resets itself after 30 s: the hardware watchdog while rebooting, which was 10 minutes.
  • Nothing reboots automatically: the USB watchdog only resets the controller.
• Not covered: a hang in the boot firmware, before Linux starts the watchdog.
• What to do: if cameras are stuck, power-cycle instead of rebooting.
• To find the cause: with a DisplayPort monitor plugged in, run tests/usb-hub-reset/run.sh, then sudo reboot, and see where the boot stops.

How the tags look (photonvision-54): whenever a camera sees a tag big enough (24 px or more), PhotonVision measures the tag's white margin and black border, a few times a second. The Input tab shows it above the camera controls ("white 205, black 38 (contrast 167), 0% clipped: good"), with what to change when it isn't good: whites clipping (shorten exposure, or lower brightness or gamma), low contrast (lengthen exposure, watching motion blur), or blacks lifted (brightness too high). Match Ready's camera tiles and health-check.sh show it too. It explains what the Field Calibration tuner's scores can't. The Camera Gain slider is gone for cameras that have no gain control: the Thriftiest Cam is listed upstream as having gain, but ours has none, so the slider did nothing.

Settings snapshots (photonvision-53, Settings page): save every setting under a name ("Before Houston", "tuned exposure"): each camera's pipelines, settings and calibrations, plus Robot state and Rewind. They're kept on the Jetson (about 2 MB each). Restore brings every camera back to that snapshot and restarts PhotonVision; it saves the current settings first as "Before restoring …", so a restore can be undone. Download one to keep it off the Jetson. PhotonVision also saves one automatically the first time the field connects each day. Logs and calibration images are never touched.

Match Ready page (photonvision-52, in the sidebar): one screen before a match. A big READY / NOT READY from the same checks as health-check.sh (run by PhotonVision every 30 s while the page is open, or with Check again), anything that needs attention listed first, and a live tile per camera: pipeline number and name, fps, latency, targets in view, and whether it's calibrated at its resolution. Every check is underneath, by section.

Checking on it: run scripts/jetson/health-check.sh for a readiness report. journalctl -u photonvision -f shows PhotonVision's live log on the Jetson. The 971 stats lines show frames per second, detection time and decision margin for each camera. The 971 jpeg lines (every 10 s) show which JPEG decoder is running and how its checks went.

Far-tag search (detector/far_search.cc, switch in photonvision-55): the GPU detector normally looks for tags on a half-size image, so it misses tags under about 18–20 px (far away). When no camera has had a good view for 0.25 s (a good view = at least 2 near tags, 40 px or more with a decision margin of 30), the cameras start also searching their full-size images, one camera at a time and at most 30 a second across all of them. Far tags they find are then followed with small full-size crops. As soon as any camera has a good view again, it all stops, so while the pose is good it costs nothing.

• Range: in the bench test it finds tags down to 10 px instead of 18 px, 1.8× the distance (about 6 m → 10 m for a 6.5 in tag face-on).
• Cost while searching (5 cameras, no tags in view, searching the whole time): GPU 16.5% → 19.4%, 9.7 → 10.1 W, average latency +0.2 ms. All cameras stay at 122 fps. Each full-size search is ~2.7 ms.
• Switch it off in Settings > Robot state (Far-tag search when the pose is weak), which also shows whether it's searching and what it has found. health-check.sh says whether it's on.
• Replay a recording with it: tests/far-search/replay.sh SESSION_DIR (on the Jetson) replays every camera of a Rewind session together, far search on and off, and reports the tags only the far search found. Replaying the bench recordings found a detector bug that could freeze a camera during a full-size search of a busy scene (fixed in bos-08).
• Tests: tests/far-search/run.sh on the Jetson renders real tags into frames and checks the range, that its corners match the normal search's (within 0.02 px), the policy, tracking a moving far tag (92 of 92 frames) and the budget. tests/ui checks the switch.

Idle while disabled (photonvision-49): while the Jetson is connected to a robot that's disabled, each camera processes 30 frames a second instead of 122. It's back to full rate within 5 ms of enable. With 5 cameras: 7.8 W disabled against 9.7 W enabled, less heat in the queue and between matches, and 30 results a second is still plenty to set the robot's starting pose. Off the robot (no NetworkTables connection) nothing changes, so tuning on the bench sees every frame. When the robot disables, PhotonVision also runs a garbage collection (27 ms), so the memory is cleaned up before the next match rather than during it. To test at full speed with the robot attached (photonvision-50): press Full speed on the dashboard's "Robot disabled: cameras idle at 30 fps" notice, or switch off Idle while the robot is disabled in Settings > Robot state. The same card sets the idle rate and shows what PhotonVision sees of the robot (connected, enabled, auto, FMS). It's saved across restarts, so health-check.sh warns while it's off.

Steadier timestamps: frames are timestamped by each camera's own clock (the USB video driver's hwtimestamps option) instead of when their first USB packet happens to be processed. The timestamps the robot receives now vary by 0.01 ms instead of 0.95 ms (up to 4 ms before), which matters when the robot is turning: 1 ms is 0.36° at 360°/s. 11-uvcvideo-payload-cap.sh --install sets it, and health-check.sh checks it.

1.2 ms lower latency from the camera driver: our USB video driver now looks at incoming packets every 2 ms instead of every 4 ms (urb_packets=16), so each finished frame reaches PhotonVision sooner: 12.98 → 11.76 ms from capture to result with 5 cameras, for about 0.1 core of CPU. Set by 11-uvcvideo-payload-cap.sh --install, checked by health-check.sh.

Hidden dashboard tabs stop their streams (photonvision-48): a dashboard tab that's behind another tab, or in a minimised window, disconnects its camera streams, and the Jetson stops encoding them. They reconnect when you look at the tab again. This also fixes streams that broke (http://undefined:…) when a dashboard reconnected to PhotonVision after a restart.

After installing a new build: run tests/ui/run.sh on the laptop. It drives the dashboard in Chrome and checks that every slider, switch and menu changes the page, PhotonVision and a second dashboard, and puts each one back. It also tests the mask editor, Start from, Create on every camera, Switch all and hidden-tab streams. It takes about a minute and a half and works on a temporary copy of a pipeline, so your settings aren't touched. See tests/ui.

Detector regression check (tests/regression/run.sh, on the Jetson): replays a fixed corpus of recordings through the detector as built now, far-tag search on as live, and compares every detection with the last accepted output (tests/regression/golden/). It runs before every prebuilt bundle (make-prebuilt-bundle.sh refuses on a change; SKIP_REGRESSION=1 to override).

• What it catches: any detection lost, gained or moved by more than 0.001 px, in any change to the detector: bos, our bos-* patches, the JNI build, far search, or new detector settings. Replays are bit-for-bit repeatable.
• The corpus (tests/regression/corpus.json, checksummed, so a damaged recording reports as that and not as a detector change):
  • fieldcal-synth: a rendered 4-camera drive with a real lens model (2,628 frames, 3,314 detections);
  • synthetic-near-far: near tags and a 14 px far tag (361 detections);
  • ceiling-4cam: four real cameras looking at the shop lights, no tags (false positives: 0).
  • Real tag footage and match recordings go in as we record them: run.sh --add NAME /opt/photonvision/rewind/sessions/SESSION --about "...".
• Where it lives: on the Jetson at /opt/photonvision/rewind/regression (scratch partition, beside Rewind's sessions; Rewind's quota never touches it), with a copy on the laptop at ~/spectrum-regression-corpus. --add holds quiet mode off while it writes.
• A change you meant (a detector improvement): read the report, then run.sh --bless, copy corpus.json and golden/ back to the laptop and commit them.
• Time: about 25 s for the whole corpus.
• Not covered: PhotonVision's Java side (poses, multi-tag, tag quality); the fork's tests and tests/tag-quality cover that.
• Checked by: a run with min white-black difference 5 instead of 20 reported 24 detections lost, 16 new and 3 moved in fieldcal-synth, and 0 false positives on the ceiling.

Robot code sees the same health on NetworkTables (photonvision-16), once a second:

• /photonvision/jetson/: GPU load, temperatures, fan speed, power, the JPEG decoder's state, and a throttle reason (OVER-CURRENT when the supply sags, HIGH TEMP, or clocks capped, photonvision-20). It's also on the Settings page as CPU Throttling.
• /photonvision/<camera>/health/: fps, pipeline time, latency, and failed JPEG decodes.
• The topics and suggested alerts are in docs/TECHNICAL.md and issue #10.
• tests/jetson-telemetry/run.sh prints them on the bench.
• The Settings page's Device Metrics also has a GPU Usage chart (photonvision-19).

How much to trust each tag (photonvision-61): every frame with tags also goes out on /photonvision/<camera>/tagQuality, a double array, just before the result it belongs to. PhotonLib's own message can't carry these: the robot checks its format.

• Layout: [sequenceID, tagCount, then per tag: fiducialId, decisionMargin, edgePx, undistortPx, reprojBestPx, reprojAltPx], in the result's target order. Match it to a PhotonLib result by result.metadata.sequenceID.
• edgePx: how close the tag's nearest corner is to the image edge. Distortion is strongest and the calibration weakest there, and the tag may be cut off. Trust falls off as this shrinks (Northstar and AOS use 25–50 px).
• undistortPx: how far the lens model moved the corner that moved most. Large means the pose relies heavily on the calibration being right out there.
• reprojBestPx, reprojAltPx: RMS pixel error of the tag's corners under its best and alternate single-tag poses. Single-tag ambiguity: keep a tag when best is clearly smaller than alt (AOS: under 0.4×), then pick the candidate closest to the gyro. NaN alt means there's no second candidate. For a tag used in multi-tag, both are under the multi-tag pose instead: a tag that disagrees with the others by several pixels is a bad tag or a bad layout.
• NaN whenever it can't be known: no lens calibration, or 3D off.
• Cost: 48 bytes a tag plus 16 a frame on the network (measured: 111 bytes against a 616-byte result, 18% more), and 3.9 µs a tag of Jetson CPU (tests/tag-quality Bench).
• Checked by: TagQualityTest in the fork. On PhotonVision's real test image the best error is 0.54 px. On two rendered tags through multi-tag it's 0.24 and 0.13 px, and moving one tag 5 cm in the field layout raises it to 4.8 px. tests/tag-quality/run.sh checks the live topic on the Jetson.

NetworkTables bandwidth. Results go to the robot over its wired Ethernet, which has plenty of room: about 0.5 KB a frame with one tag, 1.5 KB with four, so 2.5–7.4 Mbit/s at 5 cameras × 120 fps (measured overhead 107 bytes a frame with no tags; each tag ~294 bytes). What has to stay small is what crosses the radio to the driver station:

• Elastic is safe as it is: it only subscribes to topics with a widget, at its update period, never every value.
• AdvantageScope, on the field: set Live Mode to Low Bandwidth (only what's on screen). "Logging" pulls every value of everything, which is fine on the pit tether.
• AdvantageKit sends whatever robot code logs to every dashboard, so log one compact record per observation, not raw PhotonLib results: timestamp, camera, the multi-tag pose (or single-tag best and alternate), tag IDs and distances, and the tagQuality values. That's enough to replay a changed trust model against a match, at about a tenth of the size. Raw results are 0.3–0.9 MB/s at 5 cameras × 120 fps (2.4–7 Mbit/s over the radio with AdvantageScope in Logging mode, and 45–135 MB of log per match), and 81 of each tag's 294 bytes are always empty for tags (skew, the game-piece fields, minAreaRectCorners), plus 64 more duplicated for tags in multi-tag (the alternate pose and ambiguity), and 72 of the multi-tag result's own ~140 (its single solution is copied into the alternate fields). With 4 tags in multi-tag that's ~650 of ~1.5 KB a frame. Deeper replays (re-detecting tags) use Rewind's frames.
• Don't turn on "Publish protobuf" in the networking settings: it sends every result a second time.
• Robot loop time: NetworkTables' sending and receiving run on its own C++ thread, not in the robot loop. What does run in the loop is PhotonLib decoding results and addVisionMeasurement; if that shows up in loop time, read the cameras on a separate thread and hand the loop finished measurements.

Robot code: use SpectrumVision (robot-vision/), our robot-side library on top of PhotonLib, for WPILib 2026 (roboRIO) and 2027 alpha-6 (SystemCore). It reads every camera's frames, solves them (multi-tag, or one tag's position plus the gyro), gates out bad ones with named, tunable gates, gives each a standard deviation, and sends them to WPILib's or CTRE's estimator in capture-time order. It uses this Jetson's per-tag quality, health topics and excluded tags, and works with stock PhotonVision too. robot-vision/install.sh <robot project> 2026|2027 copies it in.

What robot code must do (from the 2026-10-01 audit, docs/AUDIT-2026-10.md; SpectrumVision handles the ones marked *):

• Stay on PhotonLib v2027.0.0-alpha-2 and WPILib 2027 alpha-6 for October. Later PhotonLib changes the message format (PhotonLib then refuses our results). WPILib alpha-7 changes time-sync units without an error. Log PhotonPipelineResult.photonStruct.getInterfaceUUID() at startup and alert if it isn't 4b2ff16a… (* SpectrumVision alerts on any mismatch with the coprocessor's format). (docs/MAINTENANCE-2027.md)
• Use the Jetson's camera names (TopLeft, TopRight, …), exactly as the dashboard shows them.
• Call setEnabled(true) on every camera at robot init (*). A camera robot code disabled stays disabled if robot code restarts (PhotonVision keeps it in memory), and returns empty results.
• Drop the tags in /photonvision/excludedTagsActive (*). They're left out of multi-tag, but still sent as single-tag targets.
• The event profile wins at field connect. With "switch to the event pipeline when the FMS connects" on, a pipeline robot code chose before the field connected is replaced. Set it again after DriverStation.isFMSAttached() if robot code picks pipelines.
• After a NetworkTables reconnect, results in the first ~1 s can carry an unsynced timestamp: skip results whose metadata.timeSinceLastPong is large (*: the timeSynced gate).
• The robot's control word: both formats work. A 2026 roboRIO publishes /FMSInfo/FMSControlData, and WPILib 2027 publishes the /FMSInfo/ControlWord struct (photonvision-62). Before this, PhotonVision read a 2027 robot as always disabled, so cameras idled at ~30 fps through matches.

Camera settings go in the robot log too (photonvision-24), so you can tell what a camera was set to in any match.

• Per camera: /photonvision/<camera>/settingsJson holds the full current pipeline settings, the video mode, which lens calibration is in use, and the camera's controls as actually set, including ones the UI doesn't show.
• Jetson-wide: /photonvision/jetson/settingsJson holds the PhotonVision build, the detector and decoder settings, the tags left out of multi-tag, and a fingerprint of the field layout.
• Cost: both are rebuilt every 5 s but only sent when something changes, and AdvantageKit records them at the start of every log.
• Rewind: each recording's session.json gets the same snapshot.

Every change in the UI is saved right away (within about a second), with no undo and no "unsaved changes" warning (upstream issue #2614). So at an event:

• Before experimenting, duplicate the pipeline (☰ next to the pipeline name) and try things on the copy. Switch back if it doesn't work out.
• If a good setup gets lost, the settings from any earlier match are in that match's robot log (settingsJson above) and in any Rewind recording's session.json. Nothing restores them in one click: re-enter them from there, or use Copy settings from… if another camera still has good ones.
• Only one person tunes a camera at a time. Two dashboards now stay in sync (photonvision-41), but two people dragging the same slider will still fight.

The detailed technical reference, with exact versions, commits and measurements, is docs/TECHNICAL.md. This README is the overview. What this setup lacks compared with Limelight 4, and what the robot code has to do about it (MegaTag 1/2, gyro heading), is in docs/LIMELIGHT-COMPARISON.md. Other teams' vision systems and our performance work are in docs/VISION-RESEARCH.md. What we took from upstream PhotonVision, and what to test, is in docs/UPSTREAM-PORT.md.

Still to do before the October event:

• Calibrate both cameras at 1280x800 (done on the bench; redo on the robot)
• Deploy the jar with the Device Control and 8-coefficient fixes
• Robot tuning: no auto-updates, headless boot, clocks locked, USB autosuspend off, power-cut safety (data on the SSD within 3 s, system log kept across power cuts)
• Fan on NVIDIA's quiet profile by default (775 rpm at 43 °C on the bench; FAN=full 09-robot-tuning.sh for full speed), 30 s hardware watchdog, reboot on kernel panic, PhotonVision always restarted
• Camera unplug test: the camera detects again ~1 s after it's plugged back in; the other camera is unaffected (tests/camera-replug/run.sh)
• With 4 cameras (2026-09-29): a camera pulled for 10 s detects again 1.3 s after it's back in, and one yanked and pushed back (out 1 s) loses ~1.5–2 s in all. The other 3 never dropped below 119.7 fps. Two cold power cycles: all 4 detecting 21–22 s after power-on, clean both times
• Power-cut test: pulled the plug mid-recording. No filesystem errors, the log survived, PhotonVision came back healthy, 1.4 s of video lost (tests/power-cut/run.sh). But after 21 cuts the SSD itself failed (see Storage); the test must also check the drive's media-error count
• Power-cut test on the split storage, checking the SSD's media errors after every cut
• Reboot test: tuning survives a reboot; boot 57 s → 16.5 s, first detection ~20 s after power-on
• Name the cameras after their ports (TopLeft, TopRight; BottomLeft/BottomRight when added)
• Rename the Global Shutter cameras BottomLeft (port 2.2) and BottomRight (port 2.4), if they stay
• Wi-Fi / Bluetooth switches in PhotonVision (Bluetooth off; Wi-Fi off before events)
• Static IP 10.85.15.15 on Ethernet (set in PhotonVision: Settings > Networking)
• Rewind: record every camera to the SSD when robot code asks (bench-tested, no fps cost)
• Jetson sets its date from the robot's clock when it has no internet (photonvision-08; robot code publishes /photonvision/clock/unixMs, issue #10)
• Test on the robot network with the SystemCore (NetworkTables, time sync, PhotonLib reading results, Rewind's robot-clock timestamps, the Jetson's date from the robot). Include pulling the Ethernet cable for 5 s mid-test: NetworkTables and time sync should reconnect by themselves
• Hot glue the USB connectors and add strain relief. Label the cables and ports: every Thriftiest Cam has the same serial number, so a camera in the wrong port silently swaps calibrations
• Lock each lens after focusing (glue or silicone, like 6328), then recalibrate. Vibration can turn a lens and spoil its calibration
• Turn off Wi-Fi for competition (Bluetooth is already off)
• Write the vision subsystem in 2026-FM-SystemCore using the AndyMark field layout, with photonlib kept at alpha-2
• Check temperatures with the Jetson mounted on the robot (44 °C on the bench with the fan at full speed, 43 °C on the quiet profile). We're working on running it fanless; the tests are in the passive-cooling work
• Decode speedup: both cameras at 122 fps, 13 ms latency, 1.3 of 6 CPU cores
• Upstream PhotonVision v2026.3.4 fixes, setEnabled() support, OpenCV leak fixes (docs/UPSTREAM-PORT.md)
• Frame timestamps moved to mid-exposure (photonvision-13); the camera's own delay is still to be measured with the robot spin test
• Game-piece detection: TensorRT backend, FUEL model working (76 fps uncapped)
• Game-piece pipelines capped at 30 fps by default (no measurable effect on AprilTag cameras)
• JPEG hardware decode for gray and colour cameras, with pixel checks and a memory-leak test (--jpeg nvjpg, tests/jpeg-hw/run.sh)
• Dashboard stream only while someone is watching, capped at 30 fps (photonvision-15)
• Game-piece colour camera on the robot
• USB bandwidth: capped camera driver so 4 cameras fit on USB-A (alt 7, tested with 2: 122 fps, no bad frames)
• 5 cameras on USB 2.0 (4 on USB-A, 1 on a USB-C hub): the whole USB 2.0 side shares one budget, measured
• USB bandwidth card on Camera Matching (photonvision-32): allocated vs used per camera, and a per-camera allocation setting
• Check photonvision-31 live: the health/problem topic, and no USB resets while a camera lacks bandwidth
• Measure whether a squeezed AprilTag camera (lower JPEG quality) loses range or precision
• 4 cameras on the USB-A ports (2 Thriftiest, 2 Global Shutter) and a colour camera on a USB-C hub, all streaming
• Re-measure with tests/perf-snapshot.sh with every camera running
• USB hub reset test (tests/usb-hub-reset/run.sh): the Global Shutter cameras come back by themselves; the Thriftiest Cams stay stuck until their power is cut (Known issues)
• USB controller watchdog (14-usb-watchdog.sh), tested with a faked dead controller: it resets the controller and reports which cameras came back
• Log flood fixed (photonvision-33): a camera failure logged 20,000 lines a minute. The system log also keeps up to 2 GB now, and the hardware watchdog covers reboots (30 s)
• Find why a software reboot can hang: repeat it with a DisplayPort monitor plugged in (Known issues)
• Focus score on the Camera page (photonvision-34), like Limelight's
• Target yaw is positive to the right (clockwise), the opposite of WPILib's angles. PhotonVision's own code and tests say so, but PhotonLib's PhotonTrackedTarget comments and the "getting target data" docs page say left. photonvision-36 corrects those in our fork and labels the Targets tab's columns; aiming code needs turn = -kP * target.getYaw() (training chapter 19)
• Retune exposure and decision margin on the event field, and run tests/flicker-check/run.sh under its lights. Compare decision margin 15 against AOS's 50 on Rewind footage (false positives)
• Later, if we need it: AprilTags and game pieces in one pipeline on the same camera (plan in docs/VISION-RESEARCH.md)
• Field calibration mode: push the robot by hand to 10–20 spots, then solve for the event's real tag positions, every camera's mount and the best camera settings (docs/FIELD-CALIBRATION-PLAN.md). Test in the shop first.
  • Solver: tools/fieldcal. On synthetic recordings: tags to a few mm, camera height to 4 mm, pitch and roll to 0.03°
  • Runs on the Jetson, replaying the recording through the 971 GPU detector at ~400 fps (scripts/jetson/13-build-fieldcal-detect.sh --install)
  • A Field Calibration page in PhotonVision (photonvision-30): tune camera settings, guided recording, solve, results, apply/undo the layout. Tested on the bench through the API (a synthetic recording solved, apply and undo round-trip exact); the page itself not yet looked at
  • Robot code publishes each camera's robotToCamera at /photonvision/<camera>/robotToCamera
  • A 3D view of the field (Onshape mouse controls) with simplified 2026 elements placed by their tags
  • The real field CAD in the 3D view (tools/fieldmodel, 2.1 MB): FIRST's STEP converted, each element placed by its tags, the perimeter by the field's size, so it fits the AndyMark and welded layouts
  • Shop test on our half field, including the settings tuning with real tags
  • Recalibrate the lenses with board views right into the corners: our calibrations can't model the outer 1–4% of the image
• The camera controls PhotonVision's UI didn't show (contrast, gamma, sharpness, backlight compensation) are on the Input tab (photonvision-28), at the camera's defaults unless set
• Measure whether contrast, gamma or sharpness help (decision margin, far-tag range, corner jitter): the field-calibration settings sweep
• A stuck camera recovers by itself (photonvision-29): reconnect, then USB reset. Tested on the bench with its test hook
• Bad tags left out of multi-tag, calibration with 100 auto snapshots and our board as the default, camera settings in the robot log, copy settings between cameras (patches 22–25)
• Test the detector's max_line_fit_mse at 2.5 (upstream #2138: rejects tags cut off at the image edge) against the default 10 with tags in view: range and edge behaviour. Then set it with 08-select-detector.sh bos --mwbd 20 --jpeg nvjpg --mse 2.5
• Test whether the Camera Gain slider does anything on our cameras (photonvision-21)
• Re-test the detector's thread count with several tags per camera, the field case (tests/detector-threads/run.sh, switches live; with one tag it made no difference)
• Robot code, in progress: Rewind starting itself on FMS matches, named by match (issue #10 8a)
• Robot code: drop a bad tag at run time from the dashboard (issue #10 11b and 12). The Jetson side is done (photonvision-22)
• Robot code: the rest of issue #10's robot-side items (trusting tags less near the image edge, single-tag checks, sanity gates, standard deviations, alerts on the Jetson's health topics)
• Full backup image of the SSD with PhotonVision's settings (scripts/host/04-backup-ssd.sh: 8.7 GB, 7 min). Keep it on the team drive, never GitHub (it holds the Wi-Fi password and SSH keys)
• GitHub release v2026.09.24: the PhotonVision jar, TensorRT backend, camera driver and settings
• Clone a spare SSD from the backup (scripts/host/05-restore-ssd.sh)

This setup stands on other people's work. We link to or patch their code rather than copy it into this repo, except where noted. Each project keeps its own license.

Code we build on:

Models:

• FUEL YOLO11n by Team 2826 Wave Robotics. It's the model PhotonVision ships for 2026; Wave gave PhotonVision permission to include it.
• FUEL YOLO26n by Project516 (AGPL-3.0), downloaded for comparison.

We don't redistribute either; see docs/GAME-PIECE-MODELS.md.

Field model:

• Source: assets/field-models/2026-rebuilt.glb, the 3D field on the Field Calibration page, is converted from FIRST's official 2026 field CAD (FE-2026-rev-rebuilt-playing-field.step, from the Playing Field page) by tools/fieldmodel. It's simplified, and the FUEL and fasteners are left out.
• Terms: © FIRST. FIRST publishes the CAD for teams but states no license for it. AdvantageScope redistributes field models converted from the same CAD in the same way. If FIRST asks, we'll remove it and have the build convert it locally instead.

Ideas and research we learned from (no code copied):

• EagleEye by Scythe-Engineering (grayscale-only decode). It's PolyForm Noncommercial, so we took ideas only.
• Team 3476 Code Orange's ML-assisted AprilTags ("MLTag").
• Team 4533's Whacknet (coprocessor constrained solve).
• 971's bos and cos (TensorRT YOLO, hardware JPEG decode).
• Mechanical Advantage 6328's Northstar.
• Many Chief Delphi threads, linked in docs/VISION-RESEARCH.md.

This repo is licensed under the GPL-3.0, following PhotonVision. The kernel patch in kernel/ is GPL-2.0, like the Linux driver it changes. The scripts, tests, docs and our own code (detector/, and the new files in our patches) were written by Spectrum 3847 with Claude Opus 5.5 (Anthropic), working in Claude Code.