We're helping AI to see the 3D world in motion as humans do. 🌐 Enter D4RT: a unified model that turns video into 4D representations faster than previous methods - enabling it to understand space and time. This is how it works 🧵
To perceive a 2D scene captured on video, an AI must track every pixel of every object as it moves. 🔍️️ Capturing this level of geometry and motion requires computationally intensive processes leading to slow and fragmented reconstructions. But D4RT takes a different
D4RT encodes input videos into a compressed version, then processes and queries the data using a lightweight decoder in parallel. This makes it extremely fast and scalable - whether to track just a few points, or to reconstruct an entire scene. 🖼️
Many 4D tasks can now be solved with one model, enabling us to: 👉 Predict a pixel’s 3D trajectory by looking for its location across different time steps. ⏱️ Freeze time and the camera viewpoint to generate a scene's complete 3D structure.
D4RT can even create and align 3D snapshots of a single moment from different viewpoints - easily recovering the camera's trajectory. 🎥
4D reconstruction often fails on dynamic objects, resulting in ghosting artifacts or processing lag. ⏳ D4RT can continuously understand what's moving while running 18x–300x faster than previous methods - processing a 1-minute video in roughly 5 seconds on a single TPU chip.
We believe this research could have unlimited applications in the real-world. From providing spatial awareness in robotics 🤖, leveling up efficiency in AR 🕹️, and expanding the capabilities in world models 🌐 D4RT’s potential is a necessary step on the path towards AGI. Find
