HIFI's Transition from BigQuery to ClickHouse for Enhanced Music Royalty Data Management
Technology Category
- Application Infrastructure & Middleware - Database Management & Storage
- Application Infrastructure & Middleware - Middleware, SDKs & Libraries
Applicable Industries
- Buildings
- Cement
Applicable Functions
- Procurement
Use Cases
- Inventory Management
- Retail Store Automation
Services
- Cloud Planning, Design & Implementation Services
- System Integration
About The Customer
HIFI is a company that is building innovative technologies to provide critical financial and business insights for its members, primarily music creators. The company offers a product called HIFI Enterprise, which brings intelligent revenue processing to sophisticated businesses handling music royalty data at scale. This includes companies and funds acquiring music rights and business managers who work with the most notable artists in the world. HIFI ingests a lot of royalty data, with a single HIFI Enterprise account having over half a gigabyte of associated royalty data representing over 25 million rows of streaming and other transaction data. The company is backed by top VCs and chart-topping creators across the globe.
The Challenge
HIFI, a company providing financial and business insights to music creators, was facing challenges with its data management system. The company ingests a significant amount of royalty data, with a single HIFI Enterprise account having over half a gigabyte of associated royalty data representing over 25 million rows of streaming and other transaction data. This data needs to load into the user interface as soon as a customer logs in, and there can be multiple customers logging in simultaneously. Previously, it could take up to 30 seconds to load the data, and sometimes it would not load at all due to timeouts. HIFI was using Google Cloud's BigQuery (BQ) to store royalty data, but the pricing structure of BQ was a major challenge. It discouraged data usage and contradicted HIFI's data-driven values. Google's solution to purchase BQ slots ahead of time was not feasible for HIFI as a startup, as usage patterns could change dramatically week to week.
The Solution
HIFI decided to switch to ClickHouse, a fast open-source column-oriented database management system that allows for generating analytical data reports in real-time using SQL queries. ClickHouse's performance exceeded all other column-oriented database management systems, processing billions of rows and tens of gigabytes of data per server per second. With ClickHouse, even the largest datasets loaded in under a few seconds, improving the user experience significantly. ClickHouse also proved beneficial for calculating internal metrics such as total royalties in the system and indicators like total royalties paid out in the last year. However, the speed of ClickHouse came with some surprises, like operations like joins not working as expected. To overcome this, HIFI used a JOIN engine table to update records more efficiently. HIFI also connected ClickHouse to PostgreSQL databases for non-royalty data such as customer account data and metadata.
Operational Impact
Quantitative Benefit
Case Study missing?
Start adding your own!
Register with your work email and create a new case study profile for your business.
Related Case Studies.
Case Study
Energy Saving & Power Monitoring System
Recently a university in Taiwan was experiencing dramatic power usage increases due to its growing number of campus buildings and students. Aiming to analyze their power consumption and increase their power efficiency across 52 buildings, the university wanted to build a power management system utilizing web-based hardware and software. With these goals in mind, they contacted Advantech to help them develop their system and provide them with the means to save energy in the years to come.
Case Study
System 800xA at Indian Cement Plants
Chettinad Cement recognized that further efficiencies could be achieved in its cement manufacturing process. It looked to investing in comprehensive operational and control technologies to manage and derive productivity and energy efficiency gains from the assets on Line 2, their second plant in India.
Case Study
Intelligent Building Automation System and Energy Saving Solution
One of the most difficult problems facing the world is conserving energy in buildings. However, it is not easy to have a cost-effective solution to reduce energy usage in a building. One solution for saving energy is to implement an intelligent building automation system (BAS) which can be controlled according to its schedule. In Indonesia a large university with a five floor building and 22 classrooms wanted to save the amount of energy being used.
Case Study
Powering Smart Home Automation solutions with IoT for Energy conservation
Many industry leaders that offer Smart Energy Management products & solutions face challenges including:How to build a scalable platform that can automatically scale-up to on-board ‘n’ number of Smart home devicesData security, solution availability, and reliability are the other critical factors to deal withHow to create a robust common IoT platform that handles any kind of smart devicesHow to enable data management capabilities that would help in intelligent decision-making
Case Study
Commercial Building Automation Boosts Energy Efficiency
One of the challenges to building automation is the multitude of non-interoperable communications protocols that have evolved over the years. Buildings have several islands of automation. Bridging the islands of different automation without losing the considerable investment in each specialized control network is the main focus in this solution.
