Introduction
Today’s businesses have come to understand that ‘Data is an Asset’ and are making every effort to derive maximum value from their business data. In the market, there are numerous data warehousing, business intelligence, and analytical software technologies leading the way in extracting insights from data and creating value for businesses.
However, the rapid pace at which technology is changing makes it difficult for businesses to realize the return on their investments fully. Before a business can maximize the benefits of a technology, a newer and better platform becomes available. Businesses that transition to more robust technologies gain a competitive advantage over their competitors.
Businesses that struggle to adopt modern, cutting-edge technologies often find themselves stuck with legacy systems that impede their progress. As a result, they experience reduced productivity, diminished innovation, and lose their competitive edge to their competitors. Now, additionally see below some of the reasons why businesses stay with legacy systems:
- Unable to maximize the results on investments made in legacy systems & technology.
- Business logic built in legacy systems & technology cannot be extracted.
- Metadata in legacy systems are not synchronized with organizational changes.
So, how do businesses invest in new technologies and be ready to go to market in a shorter period of time and minimal investments…
Metadata Driven Development Approach has been around for a few years and has now started to make a bigger impact on technology implementation projects. Let’s answer some of the questions that linger upon reading this article-
- What is Metadata?
- What is Metadata Driven Development Approach (MDDA)?
- What are the advantages of MDDA?
- Why do we need MDDA for the DW & BI Development process?
What is Metadata?
Metadata is “data about data” stored in a Database. However, Metadata is also business logic that structures the Metadata, relationships between, classification etc. You can see Technical Metadata and Business Metadata in the below picture. The picture is the data, the post on social media with Photo Credit is Business Metadata and the image’s data is Technical Metadata.
Similarly, Metadata in implementing a technology/application gives the developer a picture of what the application should look like and work like for the business. A developer captures the business logic and technical details, he builds an application based on that. The business logic goes into the application as Ownership, Relationships between data, Access, etc., and the technical metadata provides information on how to store, organize, and extract data from the application.
What is Metadata Driven Development Approach? (MDDA)
The MDDA encompasses these 4 stages :
- Stage 1 – Requirement Gathering: Business Logic & Technical Metadata are documented and stored
- Stage 2 – Pre-Development Governance: Business Logic & Technical Metadata are governed and approved
- Stage 3 – Development: Business Logic and Technical Metadata are the only input in the development of applications
- Stage 4 – Post-Development Governance: Applications are Developed and Governed
Note: Newer development projects reuse MDDA stored/governed data when it’s time to move on to newer better technology.
Let’s break down this further in a realtime project with a client
Requirement Gathering: The requirement gathering phase is dedicated to capturing the requirements for the development of the Data Warehouse (Snowflake) and Business Intelligence (Tableau, or any other data visualization BI Platform) reporting project. We employ a uniquely developed ‘Data Governance Template’ to capture and store metadata in Collibra or any other Data Governance Metadata repository tool.
- The business logic discussion between the Business Analyst (BA) and the Business Leader provides details about the structure of the Dashboard/Report. The BI team further models this structure, breaking down information to schemas and column levels. Additionally, Data Protection and Access Controls are essential for developing Tables or Views in Snowflake, as are definitions of Key Metrics/KPIs and business logic. All of this constitutes the required metadata for application development.
- The technical discussion involving Business Analysts (BAs), Data Modeller, BI Developer provides details about technical aspects such as data source names, tables, columns, data types, etc., required for developing tables or views in Snowflake. Additionally, it encompasses details regarding arithmetic calculations, dimensions, measures, periods, calculated fields, data lineage, and visualization.
The images below are actual Collibra Metadata Governance assets for Business Metadata (picture 1) and Technical Metadata (picture 2) (Tableau report & Snowflake table & columns respectively)
Pre-Development Governance: Pre-development Governance is the process in which a team gathers the metadata for development containing both business logic and technical details. This BRD/Metadata goes through an automated approval workflow, enabling validation by Subject Matter Experts (SMEs) in the Client’s business or the stakeholders themselves. This governance process facilitates the validation and sharing of approved metadata for development. This data can be preserved and enriched over the years for future development as well, saving a lot of time and money for an organization to do the repetitive work for technology upgrades.
The image below is the actual Collibra approval workflow that validates the Business and Technical (Snowflake & Tableau) Metadata by the roles performed by SMEs
Development: The development phase is about developing the application through agile methodology with development teams (DW & BI). The developers on both platforms DW and BI access the approved Metadata with Business Logic and Technical data to develop the Objects in Snowflake and Dashboards in Tableau/Power BI or any similar tool. These are the only input that goes into the development of Snowflake Objects (Schema, Tables, Columns, Views) and BI Dashboard (Eg- Tableau Workbook, Tableau Dashboard, Tableau Worksheet, Tableau Data Attributes [Calculated fields, Dimensions, Measure]). The development process will continue to access the metadata throughout the process to complete the deployment of a reporting project.
Post Development Governance: In the post-development Governance the developed Snowflake Objects and Tableau Dashboards are validated against the initial metadata captured and approved for Business use. The steps involved here are:
- Extraction of Metadata from Snowflake and Tableau (using APIs for metadata extraction like Catalog and Lineage Harvesters).
- Organize and store this Metadata in Collibra as a Data and BI Catalog.
- Data Lineage and Business Lineage explain the data lineage to visualization and business context to the Visualization.
- Validation and approval (Certification Process) by roles performed by Business SME for use of Client’s employees.
The image below is actual Collibra Data (picture 1) & BI Catalog (picture 2) available for Business Use
What are the advantages of Metadata Driven Development Approach?
- Transparency: Business analysts and functional users have complete visibility into metadata, including business logic and technical details, throughout the DW and BI development process. Including the roles and responsibilities in a centralized DG tool.
- Control: Empowers stakeholders in DW and BI development without requiring technical coding/scripting expertise.
- Streamlined Analytics Development: Business analysts and developers benefit from standardized and simplified processes of development that are continuously aligned to business changes.
- Data Quality: Governance over metadata ensures that the CRUD operations on the DW objects, data models, and BI reports are accurate. Only relevant and usable data is present in the DW & BI, avoiding duplicates and multiple versions of the same data.
Why do we need an MDDA for the DW & BI development process?
Metadata Driven Development Approach on DW and BI transformations projects removes the high dependency on IT teams (to ‘know how’) and puts the power in the hands of Data Owner.
Kipi.ai’s “Metadata Data Governance” provides end-to-end solutions enabling businesses to design Logical Layer for
- Data Warehouse Governance: Objects, Schema, Table, and Columns
- Governance in the data pipelines: SQL views, EDW & Analytics Layers,, Data sets, PII
- BI Governance: BI Analytics, KPI, Key Metrics, Reports
- Automated Workflow: Custom workflow for metadata approval from stakeholders in DG Tool.
Take control of your DW and BI development through our “Metadata driven development solutions”.
Kipi.ai can help you automate & accelerate DW & BI development process by providing a governance framework to manage, govern & visualize all metadata on a single platform. For a detailed understanding of our metadata-driven development Approach’s features and functionalities, contact us and find out the value we can bring to your DW & BI project.
