ClickHouse > Case Studies > Network Traffic Monitoring and Optimization for Telcos: A Case Study on BENOCS and ClickHouse

Network Traffic Monitoring and Optimization for Telcos: A Case Study on BENOCS and ClickHouse

Technology Category

Cybersecurity & Privacy - Network Security
Sensors - Dimension & Displacement Sensors

Applicable Industries

Buildings
Telecommunications

Use Cases

Time Sensitive Networking
Traffic Monitoring

About The Customer

BENOCS' customers are some of the biggest telecommunications providers in the world. These customers need to monitor and analyze huge amounts of traffic, requiring a substantial databank behind it. The data these customers deal with is not static but constantly moving through cyberspace, necessitating the need to factor in time as an additional dimension. BENOCS Flow Analytics users need to investigate incidents that occurred in specific time frames, making fast access to specific time ranges while ignoring the rest of the data a basic requirement. These customers also operate in diverse environments with asynchronous data feeds, adding to the complexity of the data analysis and monitoring process.

The Challenge

BENOCS, a company that provides network traffic optimization and monitoring for some of the world's largest telecommunications providers, faced the challenge of monitoring and analyzing massive amounts of data traffic. The data was not static but constantly moving through cyberspace, requiring the company to factor in time as an additional dimension. BENOCS Flow Analytics users needed to investigate incidents that occurred in specific time frames, necessitating fast access to specific time ranges while ignoring irrelevant data. The company also had to deal with the challenge of analyzing network traffic at high complexity and speeds, especially in diverse environments with asynchronous data feeds. Across different network setups, BENOCS had to unify the data sources and correlate the incoming network information.

The Solution

BENOCS chose to build its architecture with ClickHouse at its core for several reasons, primarily for its fast indexing and fuzzy matching on data streams. ClickHouse's MergeTree table design allowed for extremely fast lookups on specific dimensions, enabling BENOCS to skip vast amounts of irrelevant data in a matter of seconds based on the primary key. For BENOCS, this dimension was time. In the ClickHouse pipeline, lookups based upon time were the first step towards any job being scheduled. ClickHouse's ability to skip data based on time made the table sizes almost irrelevant, as it could zoom in on the needed data efficiently. This made the processing time for a single time range independent of the actual table size as well as the position in the data. ClickHouse also offered a feature called ASOF join, which allowed BENOCS to find the nearest match instead of the exact match using a join. This feature, combined with well-selected WHERE clauses, expedited and simplified queries massively.

Operational Impact

By utilizing ClickHouse's powerful engine, BENOCS was able to process vast amounts of data, zero in on the needed data, and ignore irrelevant data. This allowed the company to build a self-healing data pipeline that could handle unreliable and volatile data feeds, turning them into a stable analysis for its customers. The fast indexing and fuzzy matching capabilities of ClickHouse enabled BENOCS to efficiently manage and analyze network traffic at high complexity and speeds, even in diverse environments with asynchronous data feeds. The solution also simplified queries and expedited the data analysis process, improving the overall efficiency and effectiveness of BENOCS' operations.

Quantitative Benefit

BENOCS processes and correlates data feeds of dozens of terabytes each day.
ClickHouse allows for skipping vast amounts of data in a matter of seconds based on the primary key.
By running DISTINCT SELECT queries on the primary indexing column, terabytes of data can be searched through in a matter of seconds.