You Can’t Rerun Sunlight: Designing ML Data Architectures for Physical AI

Large language models are transforming how we build software, but physical AI systems expose a hard limit: you cannot recompute reality. When robots, sensors, and production systems interact with the real world, failures are causal and time-based, not semantic. You cannot go back to record sunlight you missed, an observation you never captured, or robot telemetry lost to fragile connectivity. Backfills can repair a dataset while changing our understanding of what the system actually knew at the time. In this talk, I will show how these constraints change how we design ML data platforms for robotics, agriculture, manufacturing, and other physical-world domains. We will look at how to capture irreversible data reliably at the site where it happens, preserve the difference between observed and reconstructed data, handle late and out-of-order events without rewriting history, and build a shared historical data backbone for analytics, training, and debugging. I will close with where LLMs and agents fit into this workflow, and why they can reason about evidence but cannot recover evidence that was never captured.