Modern Data Architecture: Building a Foundation for the Data Driven Enterprise

This module looks at what data architecture is, why companies need one and what current issues are with respect to data architecture today.

This module looks at how to assess your existing data architecture to understand how well it is serving your business needs from both an operational and analytical perspective. This includes documenting the current problems and identifying standalone projects that currently bypass your architecture. The module will cover assessing your existing:

• Data sources you connect to
• Data ingestion techniques
• Data stores in use and their purpose
• Data stores on the cloud Vs on-premises
• Operational applications, processes and data flows
• Analytical data capabilities e.g. data warehouses, data lakes, feature stores
• Data integration data flows for analytics
• Tools used e.g. data catalog, data quality, data governance, ETL, self-service data preparation, data science
• Identifying and documenting issues with your data architecture e.g.
• Data complexity
• Siloed analytical systems
• Too many copies of data
• Data that is too big to move
• Brittle data engineering pipelines
• Unmanaged self-service data engineering
• Managing data in a distributed and multi-cloud hybrid computing environment
• Missing data & analytical capabilities
• Integration complexity
• Rapid growth in APIs
• How well does existing data architecture meet your business needs?
• E.g. Inability to respond in a timely way
• Are people bypassing your data architecture and creating their own?

Having understood data architecture capabilities, technologies and assessed your existing architecture, this module looks at multiple new data architectures. It also looks at how the emergence of open table formats, advances in SQL, universal data lake APIs, federated query, data catalogs, and data automation are changing data architecture to provide a data foundation for an AI-driven enterprise.

• Data architecture options for operational processing
• Data Integration hub
• Message bus
• Data architecture options for analytical processing and their pros and cons
• Zoned Data lake
• Cloud Data warehouse
• Lakehouse
• Cloud Data Warehouse and Data Lake Integration
• Logical Data Warehouse
• Data Fabric and Data Mesh
• Customer Data Platforms
• Data architecture options for both operational and analytical process
• Dataware zero copy integration for data collaboration
• Centralized versus Federated architectures
• Next-generation analytical data architecture
• The impact of Open table formats
• The OneTable initiative
• Merging of data warehouse, lakehouse and streaming on multipurpose open tables
• Separation of storage from compute
• Multiple query and data engineering engines on shared data integration
• Universal administration of multiple engines
• Advances in SQL to support graph analytics
• Federated query across open tables on multiple clouds and on-premises storage
• Technology options and their pros and cons
• Single vendor data and AI stacks
• Data fabric including data catalog, collaborative data engineering, data automation – using a data catalog and generative AI to auto-generate pipelines, virtual views, APIs and architecture, DataOps and Data Observability, and Data virtualization for on-demand data integration
• Data platform
• AI platform
• End-to-end data and AI governance
• Modern data stack
• Best of breed tools

This module focuses on what you need to think about with respect to designing a data architecture and how you operate and manage it:

• Ensuring a common approach to integrating transaction processing applications
• Introducing continuous change and change data capture in transaction systems to lower data latency
• Ensuring tools are integrated to share metadata and enable the reuse of business terms, data models, data transformations, data quality rules and data governance policies
• Data warehouse modernization
• Siloed analytical systems
• Modernizing ETL processing
• Data Warehouse automation
• Migration to the cloud
• Virtual data marts
• Where does data modeling fit in?
• Data engineering
• Supporting citizen data engineers, IT professional data engineers and code in data engineering pipelines
• Establishing a universal approach to data ingestion
• Incorporating DataOps, CI/CD, data orchestration and data observability into data engineering pipelines
• Dealing with sensitive data to govern data privacy
• Establishing a data marketplace to share data

• Ability to process any type of data
• Support hybrid multi-cloud as standard
• Documented sharable metadata rules on cleansing, transformation, matching and integration with full metadata lineage
• Governed data ingestion, staging and transformation
• Data automation to accelerate development
• Reusable high-quality data products available as a service
• Shared data products to reduce data redundancy
• Data contracts to govern data sharing
• Consuming data products via APIs
• Integrated data governance
• Integrated analytics e.g. BI and ML, ML and streaming, graph analysis and BI, graph and ML, streaming and graph analysis
• Scalability to handle data volumes, data velocity and an increasing number of concurrent users using tools with natural language Generative AI user interfaces
• History of changes to transaction data and/or data products used in transaction systems

In this module, we focus on how to design a modern data architecture and merge workloads to create a multi-purpose platform supporting multiple analytical workloads.

• Data architecture design principles
• Defining the key data capabilities needed in your data architecture
• Alignment of data capabilities needed to meet business strategy objectives and priorities
• Designing a data concept model for your business
• Designing your data architecture and data flows to support:
• Operational processing
• Analytical processing
• Knowledge management
• Merging your data warehouse, data lake, lakehouse, streaming and graph to create a new multi-purpose hybrid analytical data platform
• Defining the core data concepts needed
• Steps to migrating your data warehouse to the cloud
• Adoption of open tables formats
• Using data fabric catalog metadata and generative AI to generate pipelines to build data products in open tables
• Supporting streaming data in open tables
• Consuming data products in your cloud data warehouse
• Building virtual data marts from data products
• Building virtual feature stores for data science from data products
• Supporting graph analysis in data warehouses

This module looks at what you have to do to get started with a new data architecture initiative. In particular, it looks at:

• Creating an action plan to bring it to life
• Change Vs rebuild?
• What order do you do this in?
• How do you minimise impact on the business while you re-architect?
• How to you deal with a backlog of change when you are also trying to re-architect?
• Pros and cons of build Vs automating development
• What new skills are needed?
• Delivering new business value whilst re-architecting
• How do you involve business professionals in the re-architecting effort?