InMarket Enhances Data Platform with ML-Powered Solution for Improved Efficiency and ROI

InMarket is an omnichannel marketing platform that assists Fortune 500 brands in identifying new prospects and customers, driving store visits, and increasing sales using AI- and data-driven consumer intelligence. The company collects and processes large volumes of real-time location data to extract actionable insights from consumers' behavior in the real world. However, InMarket's legacy data platform was inefficient and unable to accommodate the growing amount of data, leading to operational inefficiencies and developmental delays that impacted its ability to attract marquee brands and slowed down revenue growth.

InMarket, an omnichannel marketing platform, was grappling with an inefficient legacy data platform that was unable to handle the growing volume of real-time location data collected from multiple sources. The platform, built using 50 AWS nodes and 400 bare metal nodes managed by Apache Mesos, was processing over 5 billion events daily. However, it was plagued with delays, bottlenecks, and inefficiencies. The platform's job success rate was a mere 40%, with 60% of Apache Spark jobs being randomly aborted in the system. This led to developmental delays, inaccurate timeline projections, and a significant reduction in InMarket's ability to attract marquee brands, thereby slowing down revenue growth. The time taken to hand off a data pipeline from data scientists to data engineers and then to operations for deployment in production was estimated to be up to twelve months, which was unacceptable given InMarket's business model.

InMarket partnered with Provectus to design and build a robust, modern ML- and data-driven platform capable of processing over 5 billion events per day and supporting 10 specific data products. The ML-powered data & analytics platform was designed to scale dozens of thousands of analytics operations and to rapidly and efficiently build, test, deploy, and monitor predictive algorithms and models. The platform's data pipeline was implemented using Apache Spark, managed by Amazon EMR, with Amazon S3 used to store RTB Logs from partners. AWS Lambda was implemented to process Amazon S3 events and to notify the Kinesis producers into SQS for initial Kinesis processing. Data landed in Amazon Kinesis streams, triggering Spark Streaming jobs to perform required data transformations and aggregation, and also to clean the data. The data was then uploaded to the output S3 bucket and loaded into the Snowflake DWH for BI & Analytics. The solution was deployed across four main clusters, each serving a specific purpose.

• The implementation of the ML-powered data & analytics platform significantly elevated InMarket's data analytics capabilities. The completion rate for Apache Spark jobs increased from 40% to 99%, resulting in a 50% boost in the productivity of the company’s data team. This led to a $1M yearly increase in ROI. The platform also demonstrated a 10x increase in cluster throughput, optimizing resources and saving up to $100K per month. The platform's ability to cost-efficiently collect and process billions of events daily and deliver actionable insights in real time improved InMarket’s appeal to marquee brands and strengthened the company’s leading position among omnichannel marketing platforms for Fortune 500 brands.

• 99% completion rate of data processing events

• 50% increase in team’s productivity

• $1M increase in ROI per year