ClickHouse > Case Studies > Building a Unified Data Platform with ClickHouse: A Case Study on Synq

Building a Unified Data Platform with ClickHouse: A Case Study on Synq

Technology Category

Analytics & Modeling - Machine Learning
Platform as a Service (PaaS) - Application Development Platforms

Applicable Industries

Buildings
Cement

Applicable Functions

Product Research & Development
Quality Assurance

Use Cases

Building Automation & Control
Time Sensitive Networking

Services

System Integration
Testing & Certification

About The Customer

Synq is a data observability platform that analyzes log-level data from complex data ecosystems. It is a large-scale log processing engine that ingests and processes data from dozens of systems. The platform is designed to provide deep integration into ClickHouse clusters with capabilities to detect delayed data loads and uncover hidden data anomalies. It also offers automatically created data lineage and tooling for managing data quality. Synq serves teams at companies such as Typeform, Instabee, and LendInvest, helping them monitor their cloud data stacks.

The Challenge

Synq, a data observability platform, faced the challenge of managing the complexity, variety, and increasing volumes of data that powered their software system. The company needed to merge operational and analytical needs into a unified data platform. They were dealing with a continuous stream of data from dozens of systems, with frequent bursts of volume when customers ran large batch processing jobs or when new customers were onboarded. The company had set ambitious performance goals for backfilling data and wanted to provide immediate value to customers as they onboarded their product. They also wanted an infrastructure that could serve their first set of defined use cases and provide functionality to support new use cases quickly. Lastly, they aimed to build a single platform that could store their raw log data and act as a serving layer for most data use cases needed by their applications and APIs.

The Solution

Synq found the solution to their challenges in ClickHouse, a high-performance column-oriented database management system. After a few days of testing, they found that ClickHouse could ingest tens of thousands of rows per second, create query-specific data models, and maintain consistent read query performance under heavy ingest load. To focus their entire development team on the R&D of their platform, they partnered with ClickHouse Cloud. They built a solid ingestion system using the officially maintained Go client clickhouse-go. They also leveraged the ReplacingMergeTree table engine to handle duplicate events. To optimize performance, they created specialized tables that transformed their raw logs data to a format optimized for their queries. They also used the popular data transformation framework dbt to create auditing tables that extract summary statistics about their log data. Finally, they used their ClickHouse cluster as a backbone for many other use cases, including in-app analytics.

Operational Impact

The use of ClickHouse has allowed Synq to fully merge their operations and analytics storage, enabling them to think about their system in terms of use cases, knowing that they have a performant data platform and other necessary building blocks to support them. The ability to control underlying storage engines, ingest mechanics, or query settings has given them extreme control over their storage, which has so far handled any use case they had in mind with performance that can support interactive user-facing experiences. Materialized views and integration with dbt have given them a lot of flexibility to quickly develop new data use cases without any extra ETL code or large migrations. This has made development extremely efficient and has allowed them to provide immediate value to their customers.

Quantitative Benefit