Accelerating GraphQL Hive Performance: Migration from Elasticsearch to ClickHouse
Technology Category
- Application Infrastructure & Middleware - Database Management & Storage
- Infrastructure as a Service (IaaS) - Cloud Databases
Applicable Industries
- Consumer Goods
- Oil & Gas
Use Cases
- Time Sensitive Networking
- Traffic Monitoring
Services
- Cloud Planning, Design & Implementation Services
- System Integration
About The Customer
The customer in this case study is The Guild, a company that specializes in developing open-source tools for GraphQL APIs. One of their products, GraphQL Hive, is a Schema Registry, Monitoring, and Analytics solution for GraphQL APIs. It helps users track the history of changes, prevent API breakage, and analyze API traffic. The Guild's team is familiar with SQL, and they were looking for a database solution that was easy to learn and maintain, had excellent performance, was suitable for analytics and data aggregation, had built-in TTL, a type system, and no issues with high cardinality.
The Challenge
GraphQL Hive, an open-source tool for monitoring and analyzing GraphQL APIs, was facing significant scaling issues. The tool, which tracks the history of changes, prevents API breakage, and analyzes API traffic, was initially using Elasticsearch for data storage. However, as the volume of data increased, the average response time began to slow down significantly. Additionally, the indexing process was problematic, with larger users affecting the query performance of smaller users. Despite attempts to improve performance by creating an index per user, the overall speed of Elasticsearch was still below expectations. The team at The Guild, the company behind GraphQL Hive, also found the JSON-based query language of Elasticsearch challenging, as they were more familiar with SQL.
The Solution
In search of an alternative to Elasticsearch, the team at The Guild evaluated several databases, including InfluxDB, TimescaleDB, Druid, and ClickHouse. Their requirements included ease of learning and maintenance, excellent performance, suitability for analytics and data aggregation, built-in TTL, a type system, and no issues with high cardinality. After testing, they found that ClickHouse offered the best performance, with a query time of approximately 100ms, compared to 10 seconds with Elasticsearch. The migration from Elasticsearch to ClickHouse was done gradually, with data written to both destinations for a full month to ensure zero downtime. The team also began moving into ClickHouse Cloud to prepare for future scaling needs, taking advantage of its out-of-the-box support for sharding and replication.
Operational Impact
Quantitative Benefit
Case Study missing?
Start adding your own!
Register with your work email and create a new case study profile for your business.
Related Case Studies.
Case Study
Taking Oil and Gas Exploration to the Next Level
DownUnder GeoSolutions (DUG) wanted to increase computing performance by 5 to 10 times to improve seismic processing. The solution must build on current architecture software investments without sacrificing existing software and scale computing without scaling IT infrastructure costs.
.png)
Case Study
Improving Vending Machine Profitability with the Internet of Things (IoT)
The vending industry is undergoing a sea change, taking advantage of new technologies to go beyond just delivering snacks to creating a new retail location. Intelligent vending machines can be found in many public locations as well as company facilities, selling different types of goods and services, including even computer accessories, gold bars, tickets, and office supplies. With increasing sophistication, they may also provide time- and location-based data pertaining to sales, inventory, and customer preferences. But at the end of the day, vending machine operators know greater profitability is driven by higher sales and lower operating costs.
Case Study
Remote Wellhead Monitoring
Each wellhead was equipped with various sensors and meters that needed to be monitored and controlled from a central HMI, often miles away from the assets in the field. Redundant solar and wind generators were installed at each wellhead to support the electrical needs of the pumpstations, temperature meters, cameras, and cellular modules. In addition to asset management and remote control capabilities, data logging for remote surveillance and alarm notifications was a key demand from the customer. Terra Ferma’s solution needed to be power efficient, reliable, and capable of supporting high-bandwidth data-feeds. They needed a multi-link cellular connection to a central server that sustained reliable and redundant monitoring and control of flow meters, temperature sensors, power supply, and event-logging; including video and image files. This open-standard network needed to interface with the existing SCADA and proprietary network management software.
Case Study
Refinery Saves Over $700,000 with Smart Wireless
One of the largest petroleum refineries in the world is equipped to refine various types of crude oil and manufacture various grades of fuel from motor gasoline to Aviation Turbine Fuel. Due to wear and tear, eight hydrogen valves in each refinery were leaking, and each cost $1800 per ton of hydrogen vented. The plant also had leakage on nearly 30 flare control hydrocarbon valves. The refinery wanted a continuous, online monitoring system that could catch leaks early, minimize hydrogen and hydrocarbon production losses, and improve safety for maintenance.
