Fast-tracking onboarding of new customers and providing Data-as-a-Service
Challenges
Poor Platform Configuration and Rising Overhead
Founded in 2014, Juvo is an international financial services company that analyzes mobile network data to build financial identities for the 68% of adults worldwide with no formal credit history. Historically, Juvo leveraged Amazon Redshift as their data warehousing solution; of which, their business was plagued with rising platform costs and limited ROI.
In remediation, Juvo migrated to Snowflake, which promised a more robust, intuitive solution for their rapidly growing business. Unfortunately, poor platform configuration led to a sharp increase in operating costs, preventing Juvo from realizing the full value of the Snowflake platform. The goal now was set at cost optimization within Snowflake, along with reducing operational overhead without enforcing any significant downtime.
Key Metrics
Reduced operational spend, increased goodwill, and accelerated platform adoption
Solution
1. Status Quo Analysis
kipi.ai began the engagement with an analysis of Juvo’s existing Snowflake architecture. As Juvo’s future-state targeted overall cost optimization, the team’s analysis revolved around the major cost factor of data computing.
An investigation of the query history helped segregate the queries in terms of complexity, data utilization, and compute time. Based on those factors, varying degrees of compute requirements were identified and mapped with virtual warehouses of various sizes.
Previously, Juvo’s status quo was to have a fixed warehouse size (2xl) for all workloads, irrespective of the computational requirements. With best practice guidelines in mind, kipi.ai recommended a
solution to assign appropriately sized warehouses to respective operations.
Further analysis identified a select number of large tables queried to fetch records across multiple views – consuming credits more frequently and contributing to rising overhead.
The problem here was identified to be two-fold: first, the concentration of large volumes of data in a single table; second, the consumption of views, which were in essence stored queries executed as
frequently as the rate of view access.
As a result, certain data sets were being accessed repetitively. These repetitive data queries greatly increased the time taken to pull data from the underlying tables, adding to the overall compute costs of the Snowflake platform.
2. Clustering
Addressing the former, clustering was performed on the large tables. This exercise posed another problem for Juvo – what is the most effective way to identify the appropriate columns for clustering? The solution? Juvo’s query history! An analysis of the queries helped identify the nodes or column values which were frequent sites for filtering. Once identified, the values were then registered within Snowflake as cluster nodes.
3. Incremental Materialization
To solve the challenge of consumption, the kipi.ai team looked to improve the materialization of dbt models. Juvo’s current process was to materialize these models as tables; the advantage being faster table querying. However, by materializing the models as tables, Juvo created a new optimization-barrier: rebuilding these tables required significant time investment, especially if the table needed to be built on top of multiple complex transformations.
Additionally, new data records from the source were not automatically reflected in the destination tables. To resolve this issue, Juvo’s materialization approach was switched from table materialization to incremental materialization. As a result, dbt is allowed to insert new records or update records from the source into the destination table, thereby reducing the turn- around-time (TAT) for incremental data load.
4. Airflow Configuration Optmization
Throughout kipi’s analysis, it was also discovered that the tools Juvo currently leveraged for orchestration of the pre-existing workflows had been mis-configured, adding to operational overhead. While Airflow was configured to utilize a single Snowflake warehouse across all tasks, this was modified using the Python Connector in Airflow, which allowed assigning individual warehouses to individual tasks as per the complexity of execution.
5. Environment Segregation
To avoid downtime, kipi.ai needed to set up a separate developer (DEV) environment where all changes were to be made. An experimental run by the kipi.ai team confirmed the projected reduction in operating costs, following which, the changes were migrated to Juvo’s production (PROD) environment.
Results
Improved Performance & Cost Optimization
With the solutions delivered by the kipi.ai team — guided by platform best practices and extensive architecture analysis — Juvo was able to decrease their overall Snowflake consumption and reduce monthly operating expenses by 75-80%!
This additional liquidity frees up important cash flow Juvo can now leverage to continue the growth of their digital-first business.
Accelerated Snowflake Buy-In
kipi.ai’s delivery team boosted Juvo’s confidence in the Snowflake platform, and have since worked with Juvo to increase Snowflake adoption across additional applications an platform capabilities.
Technology Stack
- Snowflake
- dbt
- Airflow
- Redshift
- Spark
- Redis
- Datadog
- Postgres
About kipi.ai
kipi.ai helps businesses overcome data gaps and deliver rapid insights at scale.
With Snowflake at our core, we believe good data has the power to enable innovation without limits, helping you say goodbye to complex data solutions and hello to the modern world of cloud elasticity.
Kipi earned Snowflake’s Americas System Integrator Growth Partner of the Year Award at the 2023 Snowflake Summit and holds 7 industry competency badges.
Kipi is committed to pioneering world-class data solutions for Snowflake customers including 50+ Accelerators, Enablers, Solutions, and Native Apps to boost performance within Snowflake.
Let kipi.ai become your trusted partner for data and analytics and we’ll empower your teams to access data-driven insights and unlock new revenue streams at scale.
