Waabi's Implementation of Neptune for Enhanced Experimentation Workflow and Resource Monitoring

Waabi is a company founded by AI pioneer and visionary Raquel Urtasun, with the goal of building the next generation of self-driving truck technology. They aim to develop a solution for self-driving trucks that can be used on a large scale. To achieve this, they use deep learning, probabilistic inference, and complex optimization to create software that is end-to-end trainable, interpretable, and capable of very complex reasoning. Their Machine Learning teams are organized around different technical pillars, each responsible for delivering technology for a different functional area. These teams are usually self-sufficient and able to deliver features and product capabilities from start to finish.

Waabi, a company focused on developing the next generation of self-driving truck technology, faced a significant challenge in managing their large-scale experimentation workflow. Their Machine Learning teams, organized around different technical pillars, constantly launched experiments for different tasks, seeking model improvements by iteratively fine-tuning them and regularly comparing results against established benchmarks. The data involved in these experiments was diverse, including maps, LiDAR, camera, radar, inertial, and other sensor data. Keeping track of the data collected from these experiments and exporting it in an organized and shareable way became a challenge. The company also identified a lack of tooling for planning and building consistent benchmark datasets. They needed a solution that would allow them to share benchmark results in a constant place and format and retain data for later comparison after the end of a project.

Waabi evaluated open-source solutions and well-known vendor products, looking for feature-rich experiment tracking, collaboration capabilities, and high-quality documentation. They chose Neptune, a tool that provided a single place to organize all the results, allowing everyone in the organization to access and share experiments within and across different teams. During the evaluation phase, Waabi worked closely with Neptune’s Sales and Product teams to find the best way to leverage the tool for their use case and collaborate on the roadmap. Neptune also provided resource monitoring capabilities, which were crucial for Waabi's data-heavy models. The tool allowed them to track utilization and improve it, monitor memory increases over time, and remotely stop cloud training jobs from the Neptune UI without navigating cloud infrastructure-oriented dashboards.

• The implementation of Neptune has not only improved visibility across the organization but also made the teams at Waabi more productive. The tool has made it easier to keep track of the experiments they are running, reducing the amount of overhead spent on organization. Collaboration and knowledge transfer have become much easier, with everyone in the organization able to view and share experiments. The tool has also been beneficial for resource monitoring, particularly for data-heavy models. The team can now track utilization and improve it, monitor memory increases over time, and remotely stop cloud training jobs from the Neptune UI. This has been particularly helpful for debugging and optimizing training runtime, which directly affects cost and productivity.

• Almost all projects in the company are now using Neptune for experiment tracking.

• The tool has significantly reduced the amount of overhead spent on organization.

• The tool has improved the visibility across the organization in large-scale experimentation workflow.