Background
Republic National Distribution Company (RNDC), a leading wine and spirits distributor operating across 38 states and the District of Columbia, needed a sophisticated data management solution to break down data silos and enhance decision-making capabilities.
Challenge
RNDC faced challenges managing its data due to a mix of disparate on-premises and cloud databases, leading to data silos and making a unified view of organizational insights nearly impossible. Operating 12 different ERP systems further complicated data integration. To remain competitive and improve decision-making, RNDC required a scalable solution to consolidate its ERPs into a centralized platform and sought to decommission costly, inefficient legacy systems.
Solution
To address these needs, RNDC partnered with kipi.ai, a Snowflake implementation specialist. The strategy included migrating existing ERP systems to Snowflake, with Apache Airflow orchestrating data workflows. Snowflake was chosen for its scalability, performance, and advanced data-sharing capabilities.
For efficient data migration, Fivetran automated the extraction and loading of data from Google MySQL databases into Snowflake, reducing errors and downtime. Additionally, Snowflake’s powerful COPY command facilitated manual data loads from various ERPs, ensuring accurate and secure data transfer to the new platform.
A modern data stack with real-time ingestion and event-based triggers enabled continuous data flow into Snowflake, keeping the warehouse up-to-date. The transition from dbt (Data Build Tool) to stored procedures improved processing performance, aligning with RNDC’s need for robust data handling.
Impact
The migration to Snowflake produced transformative results for RNDC:
- Enhanced Decision-Making: Real-time data migration enabled critical stakeholders to access up-to-date information, allowing faster responses to market and operational changes.
- Performance Boost with Stored Procedures: Transitioning from dbt to stored procedures improved data processing performance, enabling quicker and more accurate data transformations.
- Secure Data Sharing: Snowflake’s data-sharing features allowed secure, granular access to critical data across departments, maintaining data security and compliance while fostering better collaboration.
- Unified Data Platform: Integrating multiple data sources into a scalable data warehouse eliminated data silos, enabling RNDC to access and analyze data from all ERPs in one place, giving a comprehensive view of the business.
Results
With the Snowflake migration and optimizations powered by Apache Airflow, RNDC saw substantial improvements:
- Database Size: The new data warehouse reached 143.9 TB, providing ample storage capacity for detailed analysis and reporting.
- Data Ingestion Across Markets: Apache Airflow DAGs automated data ingestion across 73 markets, covering both regular and control states. Daily data loading across 800+ tables ensured timely updates for business operations.
- Optimized Credit Consumption: Average credit consumption was balanced at approximately 6,000 credits over six months, achieving high-speed processing while controlling costs.
- Fact and Aggregate Tables: New fact tables and aggregates for inventory, sales, and revenue provided quick insights and supported trend analysis, enhancing reporting.
- Daily ETL Pipelines: With 670 DAGs running daily, RNDC processes data across raw, refined, and trusted layers, ensuring consistency and accuracy for business insights.
- Auditing Framework: A robust auditing framework with tables across 13 schemas and 11 ERP combinations ensured data accuracy and traceability, supported by a generic audit log and Lambda audit logging for exception handling.
- Automated Alerts: Real-time alerting promptly flagged failures via email and SNOW incidents, minimizing downtime by enabling quick resolution.
- Storage & Compute Optimization: Downstream environment cloning and paused non-production pipelines yielded significant cost savings while optimizing resources.
The migration to Snowflake created a scalable, cost-effective, high-performing data infrastructure, positioning RNDC for future innovation and growth.
Conclusion
Implementing Snowflake, alongside tools like Apache Airflow, Fivetran, and Snowflake’s COPY command, transformed RNDC’s data management. The project addressed the immediate challenges of data silos and legacy inefficiencies, equipping RNDC for future growth. With real-time data availability and a modern data stack, RNDC can now focus on leveraging insights for strategic initiatives, maintaining its competitive edge.
kipi.ai’s partnership was essential in navigating migration complexities and ensuring a seamless transition to an efficient and scalable data platform. This case study demonstrates the vital role of modern data solutions in helping organizations overcome data management challenges and unlock their data’s potential.
About kipi.ai
Kipi.ai is a leader in modern data solutions, helping organizations transform their data infrastructure and achieve scalable business results. As an Elite Snowflake Partner, kipi.ai specializes in AI-driven data centralization, automation, and advanced analytics. Through strategic partnerships and innovative technologies, kipi.ai empowers businesses to thrive in an AI-powered world. For more information, visit www.kipi.ai.
