Datadog > Case Studies > Toyota deploys at scale faster and more securely by monitoring AWS with Datadog

Toyota deploys at scale faster and more securely by monitoring AWS with Datadog

Company Size

1,000+

Region

America

Country

United States

Product

Datadog
Chofer
Amazon EC2
Amazon RDS
Amazon EKS

Tech Stack

AWS
Backstage

Implementation Scale

Enterprise-wide Deployment

Impact Metrics

Cost Savings
Digital Expertise
Productivity Improvements

Technology Category

Analytics & Modeling - Real Time Analytics
Infrastructure as a Service (IaaS) - Cloud Computing
Platform as a Service (PaaS) - Application Development Platforms

Applicable Industries

Automotive

Applicable Functions

Discrete Manufacturing
Product Research & Development

Services

Cloud Planning, Design & Implementation Services
Data Science Services

About The Customer

Toyota (NYSE:TM) has been a part of the cultural fabric in the US for more than 65 years, and is committed to advancing sustainable, next-generation mobility through its Toyota and Lexus brands, plus nearly 1,500 dealerships. Toyota Motor North America (TMNA) is the operating subsidiary of the Toyota Motor Corporation in the United States, Canada, Mexico, and Puerto Rico. TMNA works to create high-quality vehicles and find innovative ways to advance society with cutting-edge automotive technology. TMNA began using Amazon Web Services (AWS) in 2015. As it did so, it also wanted to simplify and standardize application development in the cloud and improve time to market. In response, Kishore Jonnalagedda, director of engineering, led the TMNA cloud platform team in building an internal, self-service development platform called Chofer using Backstage running on AWS.

The Challenge

Toyota Motor North America (TMNA) began using Amazon Web Services (AWS) in 2015 to simplify and standardize application development in the cloud and improve time to market. However, the team lacked a consistent monitoring tool, which created reliability concerns. Some developers used open source tools, others used log management tools, and some didn't use anything. As a result, team members often spent multiple hours trying to get to the bottom of an outage because they didn’t know what to look for or where. With 1,600 total applications (300 in the cloud) and more than 100 teams, that was a challenging task. On top of gaining unified visibility, the cloud platform team also sought to improve mean time to detection (MTTD) and ensure they could meet SLAs for 99.9 percent uptime while simultaneously reducing costs and helping engineers become more efficient.

The Solution

Jonnalagedda and his team began looking for an observability solution that could provide full visibility into the health and performance of each layer of TMNA’s environment at a glance, in a single pane of glass. Ultimately, TMNA achieved that by ingesting data from its AWS services into Datadog. It can now maintain visibility into its cloud-hosted apps running on Amazon EC2, Amazon RDS, Amazon EKS, and others, all in one place. Toyota also needed to monitor different parts of its tech stack. Datadog gave Toyota the visibility it needed with its 600+ integrations with key technologies, including support and out-of-the-box dashboards for over 100 AWS services. Datadog’s dashboards helped TMNA develop applications with more transparency, bringing metrics and logs into one place—regardless of their source—and helping the team quickly gain context and troubleshoot problems faster. These visualizations were also an easy way for the organization to look at site reliability engineering practices, visualize service level objectives (SLOs), and manage AWS over-capacity.

Operational Impact

TMNA has saved $10 million over two years using Chofer. Part of that savings can be attributed to using Datadog to monitor its underlying infrastructure, supporting services, applications, and security data in a single observability platform.
With these time savings, teams now ship projects in weeks instead of quarterly.
In addition, since new hires can easily make sense of TMNA’s distributed architecture with Datadog’s centralized platform, onboarding developers and contractors now takes as little as three to four days instead of the eight to twelve weeks previously required.
Finally, Datadog helps Jonnalagedda’s team reduce MTTD. “MTTD is reduced from about six hours to 15 minutes in a large-scale system,” says Jonnalagedda.
In another example, TMNA also used Datadog’s services to help reduce the mean time to resolution (MTTR) from seven days to two hours in one of its manufacturing plants, avoiding hundreds of thousands of dollars of cost from downtime.

Quantitative Benefit

MTTD reduced by 96% from about 6 hours to 15 minutes on average
New developers and contractors onboard 20X faster in 3–4 days instead of 8–12 weeks
Teams now ship projects in weeks instead of quarterly