ClickHouse > Case Studies > HIFI's Transition from BigQuery to ClickHouse for Enhanced Music Royalty Data Management

HIFI's Transition from BigQuery to ClickHouse for Enhanced Music Royalty Data Management

ClickHouse Logo
Technology Category
  • Application Infrastructure & Middleware - Database Management & Storage
  • Application Infrastructure & Middleware - Middleware, SDKs & Libraries
Applicable Industries
  • Buildings
  • Cement
Applicable Functions
  • Procurement
Use Cases
  • Inventory Management
  • Retail Store Automation
Services
  • Cloud Planning, Design & Implementation Services
  • System Integration
About The Customer
HIFI is a company that is building innovative technologies to provide critical financial and business insights for its members, primarily music creators. The company offers a product called HIFI Enterprise, which brings intelligent revenue processing to sophisticated businesses handling music royalty data at scale. This includes companies and funds acquiring music rights and business managers who work with the most notable artists in the world. HIFI ingests a lot of royalty data, with a single HIFI Enterprise account having over half a gigabyte of associated royalty data representing over 25 million rows of streaming and other transaction data. The company is backed by top VCs and chart-topping creators across the globe.
The Challenge
HIFI, a company providing financial and business insights to music creators, was facing challenges with its data management system. The company ingests a significant amount of royalty data, with a single HIFI Enterprise account having over half a gigabyte of associated royalty data representing over 25 million rows of streaming and other transaction data. This data needs to load into the user interface as soon as a customer logs in, and there can be multiple customers logging in simultaneously. Previously, it could take up to 30 seconds to load the data, and sometimes it would not load at all due to timeouts. HIFI was using Google Cloud's BigQuery (BQ) to store royalty data, but the pricing structure of BQ was a major challenge. It discouraged data usage and contradicted HIFI's data-driven values. Google's solution to purchase BQ slots ahead of time was not feasible for HIFI as a startup, as usage patterns could change dramatically week to week.
The Solution
HIFI decided to switch to ClickHouse, a fast open-source column-oriented database management system that allows for generating analytical data reports in real-time using SQL queries. ClickHouse's performance exceeded all other column-oriented database management systems, processing billions of rows and tens of gigabytes of data per server per second. With ClickHouse, even the largest datasets loaded in under a few seconds, improving the user experience significantly. ClickHouse also proved beneficial for calculating internal metrics such as total royalties in the system and indicators like total royalties paid out in the last year. However, the speed of ClickHouse came with some surprises, like operations like joins not working as expected. To overcome this, HIFI used a JOIN engine table to update records more efficiently. HIFI also connected ClickHouse to PostgreSQL databases for non-royalty data such as customer account data and metadata.
Operational Impact
  • The transition to ClickHouse has significantly improved HIFI's operational efficiency. The speed of ClickHouse has not only enhanced the user experience but also boosted the company's confidence in its ability to scale as it adds more customers and rolls out new products. The company can now calculate internal metrics such as total royalties in the system and indicators like total royalties paid out in the last year more quickly and accurately. Moreover, the company has overcome the challenges of joins in ClickHouse by using a JOIN engine table, which has made record updating more efficient. HIFI also successfully connected ClickHouse to PostgreSQL databases for non-royalty data, further enhancing its data management capabilities.
Quantitative Benefit
  • Data load time reduced from 30 seconds to a few seconds
  • Ability to process billions of rows and tens of gigabytes of data per server per second
  • Efficient updating of records using JOIN engine table

Case Study missing?

Start adding your own!

Register with your work email and create a new case study profile for your business.

Add New Record

Related Case Studies.

Contact us

Let's talk!
* Required
* Required
* Required
* Invalid email address
By submitting this form, you agree that IoT ONE may contact you with insights and marketing messaging.
No thanks, I don't want to receive any marketing emails from IoT ONE.
Submit

Thank you for your message!
We will contact you soon.