Data Architecture Layers Pattern with Shane Gibson

Key Aspects

• Introduction of the Architecture Template: Shane created an open-source template to clarify data architecture terminology and governance in the industry.
• Critique of the Medallion Architecture: The template serves as a rigorous alternative to the superficial “medallion architecture” diagram.
• Template Structure and Functionality: The tool in Google Sheets serves as a checklist for data layer principles and policies.
• AI Integration and Interrogation: The template was analysed using Claude, successfully addressing complex structural queries on field renaming.
• Evaluating Complex Architectural Parameters: The live AI testing assessed latency, data persistence, metrics, and mapping new business requirements.
• Identifying Documentation Deficiencies: The model revealed missing SLA synchronisation, freshness targets, and role definitions for service account permissions.
• The Necessity of Human Oversight: Pairing architectural templates with AI reduces cognitive load, but humans must prevent AI hallucinations.

Webinar Details

Title: Data Architecture Layers Pattern with Shane Gibson
Date: 2026-03-18
Presenter: Shane Gibson
Meetup Group: DAMA SA User Group Meeting
Write-up Author: Howard Diesel

What is the Purpose of Shane Gibson’s Interactive Architectural Model Platform?

In this session, Shane Gibson introduces an open-source architectural model platform dedicated to data architecture. He acknowledges the prevalent confusion surrounding architectural terminology, specifically regarding orchestration and pipelines. Rather than delivering a conventional slide-based presentation, Shane intends to adopt an interactive approach to stimulate discussion among participants. By open sourcing the template, the objective is to invite extensive community contribution and peer review, thereby enhancing the platform’s overall utility and robustness.

Figure 1 An Agile Data Guide Discussion – on the Data Layer Architecture Checklist Pattern Template

What is the Problem Space and Open-source Architecture Template?

To address structural ambiguity in data environments, Shane developed an open-source template structured as a comprehensive checklist for data layer architectures. This initiative was largely a critical response to the widespread adoption of the “medallion architecture”. While acknowledging its success as a marketing concept, Shane argues that the medallion model functions primarily as a superficial diagram rather than a detailed architectural framework.

Frequently, essential architectural rules remain undocumented, existing solely within code, disorganised wikis, or individual institutional knowledge. The proposed template rectifies this deficiency by meticulously codifying principles, policies, and patterns to define permissible actions within each architectural layer.

Figure 2 Building Reliable, Performant Data Pipelines with ‘Data Lake’

Figure 3 Reasons for Creating a Pattern Template

Figure 4 The Challenge of Finding Documented Rules

Figure 5 Data Architecture Layers Pattern Checklist

Figure 6 Data Architecture Layers Pattern Checklist

Figure 7 Access to the Template

What is Claude, and How is the Template Structured?

During the demonstration, Shane utilises the artificial intelligence assistant Claude to analyse an exported version of the Google Sheets template. Architecture is conceptually defined as a blueprint comprising distinct locations governed by specific rules. The template incorporates a blank checklist, an AI-assisted dictionary defining available options, lookup values for dropdown menus, and a practical “Agile Data” example. This Agile Data example rigorously documents the internal rules of the speaker’s data platform, explicitly establishing policies for managing personally identifiable information (PII) and for data persistence across various layers.

Figure 8 Prompting Claude: “What is this?”

Figure 9 Terms and Conditions of Use

Figure 10 Empty Template

Figure 11 Dictionary Terms

How do Business Rules Apply in your Organisation?

To evaluate the AI’s comprehension of the framework, the system is queried regarding the permissible architectural layer for renaming a technical field to a business-relevant term. Claude successfully identifies the applicable row by processing the structural mapping.

The AI correctly ascertains that renaming fields for business context is strictly confined to the “designed” layer. Within preceding layers, such as the landing or history layers, renaming is only authorised if dictated by physical storage limitations. This test effectively demonstrates the template’s capacity to operationalise complex business logic rules.

Figure 12 AgileData Example

Figure 13 Prompting Claude: “In AgileData, what layer can I change Column Names to make them business-related?”

Figure 14 Prompting Claude: “I need to change Column Cus_d into Customer ID in my data”

How Should We Effectively Communicate with Management When Sourcing Information?

Addressing inquiries about initial data acquisition, Shane clarifies that the current template requires manual input from domain experts, such as data architects or lead engineers. A significant operational challenge is that management teams frequently resist strict architectural constraints to preserve strategic flexibility.

If an organisation’s methodology lacks defined boundaries, the template can be configured to allow all selections, accurately documenting the absence of restrictions. However, Shane strongly advocates for establishing explicit rules to prevent an unstructured data environment.

What Tools are Used for Testing Latency and Persistence?

Further evaluation examines the architecture’s latency and persistence parameters. Claude accurately identifies a “micro-batch” processing pattern, which Shane defines as intervals of 15 minutes or less. However, the interrogation reveals a deficiency: the template currently lacks explicit definitions for Service Level Agreement (SLA) data synchronisation requirements or precise freshness targets.

Regarding persistence, the framework specifies that “landing” and “history” layers are immutable and permanently retained. Conversely, subsequent layers permit destructive rebuilds and virtualisation. The AI appropriately concludes that the architecture is unsuitable for sub-minute data streaming.

Figure 15 Prompting Claude: “I would like to understand the Latency Capabilities”

Figure 16 Dictionary Terms

Figure 17 Prompting Claude: “I have a need to have data from landing to last mile in less than one minute. What do I need to change in my architecture?”

Figure 18 Prompting Claude: “Where did you find the five to 15 minutes for Micro Batch?”

What Challenges are Associated with Live Testing of Business Logic?

In analysing business logic, the AI correctly locates calculated metrics within the designed and consumed layers. The framework employs precise terminology: a “fact” denotes an immutable physical value, a “measure” involves basic calculations, and a “metric” entails complex mathematical formulas.

An Attendee suggested enhancing the template by incorporating synonyms or a comprehensive semantic model to align with diverse industry vocabularies. This testing methodology illustrates how large language models can expose ambiguities, enabling architects to systematically refine their templates until the AI produces precise, unambiguous interpretations.

Figure 19 Prompting Claude: “Is there any ambiguity in where business logic should be created?”

What Challenges Might Arise during Live Testing of Complex Scenarios?

To test complex requirements, an “information product canvas” is introduced to evaluate the AI’s capacity to map business needs directly to the architecture. The AI successfully identifies requirements for derived data and exposes architectural gaps, notably the absence of designated operational systems within the “last mile” layer.

The evaluation also reveals a limitation in the security documentation, as the template fails to distinguish between a human persona reading data and a production service account writing data. Furthermore, Claude accurately identifies data duplication between the Google Cloud Storage landing layer and the BigQuery history layer.

Figure 20 Prompting Claude: “What Architecture Changes would I need to do to Support this Information Product?”

Figure 21 Prompting Claude: “Where would I do fine-grained security to mask Data from a Persona?”

Figure 22 Prompting Claude: “What Data is Duplicated Between Each Layer?”

Figure 23 Prompting Claude: “Where am I allowed to Aggregate Data?”

Figure 24 Prompting Claude: “I wonder if you could use this for Capacity Planning?”

Figure 25 Prompting Claude: “What about estimating where the big bucket of costs is?”

Figure 26 Prompting Claude: “Are there any gaps, issues, or risks regarding this architecture? What are the top three actions to address?”

How can Humans Strengthen the Effectiveness of AI Systems?

Shane concludes that integrating an architectural template with a large language model substantially diminishes the cognitive load required to comprehend complex data environments. By analysing the AI’s “reasoning path,” architects can identify persistent inquiries and recognise missing contextual information.

Despite the analytical benefits of AI, maintaining a “human in the loop” remains an absolute necessity for setting boundaries and mitigating hallucinations. Ultimately, the community is encouraged to utilise and refine this open-source tool, reinforcing the principle that robust layered architectures require rigorous, detailed documentation.

Figure 27 Prompting Claude: “Are there any gaps, issues, or risks regarding this architecture? What are the top three actions to address?” – Continued

Figure 28 Prompting Claude: “According to GDPR, PII is too specific, and personal data is in fact much broader, so is your 2nd action accurate?”

Figure 29 Prompting Claude: “Q: Where did I go wrong?” “A: GDPR Personal Data is not broader than PII”