Moondream Photon cuts VLM latency to 33ms on NVIDIA B200
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
Photon hides CPU-induced GPU idle time during autoregressive decode by launching the next forward pass before the current token's CPU bookkeeping completes, keeping the sampled token in GPU memory rather than blocking on a CPU round-trip — yielding up to 35% higher decode throughput and ~33ms VLM inference on a B200. If your serving stack still synchronizes per-token before planning the next step, you're leaving roughly a third of your decode throughput on the table; the win comes from pipelining with double-buffered slots and deferred detokenization, not bigger hardware.
Photon reports ~33ms near-realtime VLM inference on NVIDIA B200 and up to 35% higher decode throughput by hiding CPU–GPU synchronization bubbles during autoregressive decoding. The practical takeaway is that decode performance can be materially improved without changing model weights: keep sampled tokens on-GPU, pipeline the next forward pass before CPU bookkeeping finishes, and handle buffer/order/cleanup hazards explicitly in your inference engine.