ClickHouse > Case Studies > Harnessing ClickHouse and Materialized Views for High-Performance Analytics: A Case Study of Inigo

Harnessing ClickHouse and Materialized Views for High-Performance Analytics: A Case Study of Inigo

ClickHouse Logo
Technology Category
  • Application Infrastructure & Middleware - Database Management & Storage
  • Infrastructure as a Service (IaaS) - Cloud Databases
Applicable Industries
  • Equipment & Machinery
  • National Security & Defense
Applicable Functions
  • Maintenance
  • Product Research & Development
Use Cases
  • Real-Time Location System (RTLS)
  • Time Sensitive Networking
Services
  • Cloud Planning, Design & Implementation Services
  • System Integration
About The Customer
Inigo is a leading company in the GraphQL API management industry, specializing in comprehensive security and governance at scale. Their platform empowers API and platform teams to fully control, manage, and secure GraphQL deployments by providing immediate visibility, security, access control, compliance, and continuous delivery. In addition to its security features, Inigo offers key developer experience enhancements and workflow efficiencies, ultimately enabling organizations to expedite GraphQL API development, lower costs, and improve end-user experiences. They process and analyze billions of GraphQL API requests, creating aggregated views on high cardinality data, generating alerts, and creating dashboards based on extensive data.
The Challenge
Inigo, a pioneering company in the GraphQL API management industry, was in search of a database solution that could handle a high volume of raw data for analytics. They explored various alternatives, including SQLite, Snowflake, and PostgreSQL, but none of these options met their needs. Snowflake was too slow and costly for their needs, especially when handling real-time customer data within a product. PostgreSQL, while an excellent transactional database, was unsuitable for large analytic workloads. The company was able to get it to work with around 100K rows, but past that, the indexes were growing out of control and the cost of running a PostgreSQL cluster didn’t make sense. There was significant performance degradation once they hit the 100K - 1M rows mark. Inigo needed a solution that could handle billions of GraphQL API requests, create aggregated views on high cardinality data, generate alerts, and create dashboards based on extensive data.
The Solution
Inigo chose ClickHouse, a columnar database specifically designed to handle high volumes of data while providing fast query execution. They paired ClickHouse with Materialized Views, which allow datasets to be pre-aggregated, significantly improving the performance of analytical queries. ClickHouse's ability to automatically manage and update materialized views as new data is ingested simplified the maintenance process. The open-source nature of ClickHouse and the ability to run it as a simple binary or docker on their laptops were major attractions. Inigo's current ClickHouse setup is a single database where nearly every Inigo service has its own table, with some tables containing billions of rows of data. They consolidated the data into several materialized views, each grouping the data into distinct time window buckets. By creating materialized views with different intervals, they could enhance query performance and enable more efficient data analysis across various timeframes.
Operational Impact
  • The use of ClickHouse and Materialized Views has brought about several operational benefits for Inigo. They utilize Materialized View queries in nearly all aspects of their app, with the exception of one table with Raw data and any queries that involve filtering by a unique field. Switching to use Raw data when necessary is straightforward. They employ the same SQL files in their queries, with minor modifications made for Materialized Views. The convenience of Test-Driven Development (TDD) is enhanced by the speed of their independent integration tests. They've encountered no issues when using Materialized Views, both with standalone instances and within a clustered environment. Confidence in query speed enables them to identify issues outside the database, such as in tracing extensions. Looking ahead, they are excited about the ClickHouse Cloud offering and anticipate moving some of their self-hosted workloads to the cloud.
Quantitative Benefit
  • Significant difference in data size. For instance, Raw data covers up to a week, while Materialized Views can accommodate up to a month's data.
  • Less load on the database and faster operation of the entire App due to the smaller size of MV tables
  • ClickHouse’s strong typing simplifies migration logic which historically needed to allow for weaker typing guarantees in NoSQL solutions.

Case Study missing?

Start adding your own!

Register with your work email and create a new case study profile for your business.

Add New Record

Related Case Studies.

Contact us

Let's talk!
* Required
* Required
* Required
* Invalid email address
By submitting this form, you agree that IoT ONE may contact you with insights and marketing messaging.
No thanks, I don't want to receive any marketing emails from IoT ONE.
Submit

Thank you for your message!
We will contact you soon.