Data Product Genesis Engine

The Proposition

Expertise Makes Agentic Tools Trustworthy

Any team can point an agentic tool at a legacy codebase. The output will be fast, it will look complete, and it will be wrong in ways that are not immediately visible. The missing ingredient is always the same: domain expertise that knows where the code is dynamic, where tools silently fail, and what a correct lineage edge actually looks like in a financial services context.

①

Impetus Designs

12 years of financial services legacy migration patterns encoded into extraction playbooks. Every dynamic code case — parameterised table names, macro chains, subgraph recursion — has a resolution strategy authored by people who have seen it fail in production.

②

Agentic Tool Executes

Devin — or any capable agentic system — runs the Impetus playbooks at scale: 4 parallel extraction streams, nightly re-index via DeepWiki, continuous ODCS contract regeneration. The execution is autonomous. The intelligence behind it is not.

③

Impetus Validates

At every phase gate, Impetus reviews extraction accuracy against known patterns before the next phase begins. No unreviewed AI output enters the lineage catalog. The HITL checkpoint is what converts agentic speed into auditable, regulatory-grade quality.

Why this is risk-free for Barclays

Risk	How HITL eliminates it
AI produces plausible but incorrect lineage	Impetus phase gate validates against known patterns before catalog write
Dynamic code patterns silently missed	Impetus playbooks pre-empt every known dynamic failure mode
Regulatory audit surfaces a gap	Lineage diff is timestamped, git-linked, and Impetus-verified at each sprint
New vendor dependency (Devin)	Playbooks are tool-agnostic — any capable agentic system can execute them
No institutional knowledge at rolloff	Playbooks, pattern library, and ODCS contracts remain with Barclays

Already proven at Barclays

Live · May 2026 Wave 1

Netezza / Informatica / AutoSys → Databricks migration. HITL model applied across full delivery lifecycle.

In Flight Wave 2 — GCA Lineage

11,896 files · 97% BTEQ accuracy · Impetus-designed extraction at scale. This engagement extends what is already working.

Proposed Wave 3 — Data Product Genesis

Same HITL model. Agentic execution added as the scale multiplier. 4 structured outputs per attribute across all 8 LOBs.

Why HITL Matters — Representative Dynamic Patterns

Legacy Code Dynamism — The Reason Generic Agentic Approaches Fail

The patterns below are a representative sample — not an exhaustive list — of the dynamic cases that silently drop lineage edges in financial services legacy codebases. Each one causes an agentic system without domain expertise to produce lineage that looks complete and is not. Impetus's role as HITL is built specifically around navigating patterns like these — knowing they exist, knowing where they appear, and having the resolution strategy ready before extraction begins.

Stack	Dynamic Pattern	What Breaks Without Domain Expertise	Impetus Resolution Approach
BTEQ / SQL	`KSH variable construction`	`"${ENV}_${REGION}_POSITIONS"` — table name invisible to any static scanner	Shell pre-processor resolves env vars before BTEQ parse
	`Teradata CALL macros`	Macro body lives in `DBC.TVM`, not the script — static tools see a CALL but cannot follow	Local macro registry built from `.btq` definitions
	`Volatile tables`	`CREATE VOLATILE TABLE` — session-scoped, no DDL in any persistent catalog; lineage breaks silently	Session-scoped volatile tracker maintained within BTEQ script scope
	`FastLoad shell args`	`${TARGET_DB}.${TARGET_TABLE}` passed as shell arguments — target invisible to BTEQ parser	Shell variable resolver in Phase 0 pre-processing pass
Ab Initio	`.pset non-resolution`	GDE XML shows `$(LOOKUP_TABLE)` token — lineage records a parameter reference, not a table name	Step 1 of 7-step playbook: .pset injection before any graph parse
	`Built-in components (Join / Rollup / Sort)`	No `.xfr` file exists for these components — field-level lineage is null without semantic knowledge	Semantic rules library: component type + input/output port schema inference
	`Subgraph recursion`	Depth unknown at parse time — naive parsers stop at the top-level graph	Recursive `.mp` traversal with cycle detection in Step 3
	`m_switch conditional flows`	Branch selection on runtime record values — static tools either inflate or lose lineage	All branches enumerated; confidence flag added per edge
SAS	`CALL SYMPUT / SYMGET`	Target table set from a data value mid-execution — unresolvable by any static parser	Pass 1 macro simulator resolves SYMPUT/SYMGET chains before extraction
	`%INCLUDE with macro paths`	`%include "&SASPROG_ROOT./load_&ENTITY..sas"` — included file path is itself a macro variable	Include chain resolver in Pass 1 before Pass 2 extraction
	`ARRAY indirect references`	Loop-driven `SET &tbl_refs{i}` — field lineage unresolvable without loop unrolling	Loop unroller: expands ARRAY iterations to explicit column references
	`%macro + &param substitution`	Table names computed at call time and passed as parameters — invisible in the calling script	Full macro simulation pass; all call sites resolved before lineage extraction
Informatica	`SQL Override on Source Qualifier`	Override query joins multiple tables — tools using repository metadata miss all override joins	SQL Override extracted and parsed as a standalone SQL block
	`Java Transformation`	JDBC calls inside Java are opaque — no input-to-output field mapping available to any static tool	Flagged for port-level manual review; never silently dropped from the lineage chain
	`Mapplet nesting`	Tools that flatten mapplets drop intermediate transformations silently	Mapplet registry with recursive port binding resolution across mapping XMLs
	`Router default group`	Unmatched rows route to default group — tools that only parse named conditions silently lose this lineage	All groups including default parsed explicitly; no silent drops
Cross-Stack	`Shell as integration layer`	Data contract between Ab Initio and BTEQ lives in JIL/KSH — neither stack's parser sees the other side	AutoSys JIL parser in Phase 0; shell wrapper resolver links cross-stack boundaries
	`File-based handoffs`	`positions_${BIZDATE}.dat` — writer and reader linked only by shell convention; invisible to any tool	File convention pattern-matching rules in the stitching layer
	`AutoSys global variable bus`	`${TARGET_DB}` injected at job start — the stack parser has no visibility of the AutoSys source	JIL + KSH + BTEQ parsed in a single resolution pass

Stack	Without Domain Expertise	With Impetus Playbooks
BTEQ	~65%	97% proven · GCA Wave 2
Ab Initio	Table-level only	Column-level post feasibility spike
SAS	~55% (macros as black boxes)	~95%
Informatica	~75% (surface XML only)	~95%+

Devin executes. Impetus defines what gets executed.

A Devin session running generic playbooks produces generic accuracy. The resolution strategies above — the .pset injector, the macro simulator, the shell variable resolver, the semantic rules library — are encoded as Impetus playbooks. That is the IP Devin executes at scale.

Agentic Execution — Within the HITL Framework

Devin as the Execution Layer

Devin (Cognition AI) is an autonomous software engineering agent deployed inside the Impetus HITL framework. It does not determine what to extract or how to resolve dynamic code — Impetus does. Devin executes those decisions at a scale and speed no engineering team can match manually. The capabilities below map directly to phases in the 5-phase plan, each operating under an Impetus-authored playbook.

🗺

DeepWiki — Codebase Indexing

Phase 0 / Phase 1 gate

Auto-indexes all 4 stacks every few hours → architecture diagrams with code links. Used to produce the naming convention map — Ab Initio output table names vs. BTEQ source table names — before the stitching layer is built.

⚡

Managed Devins — Parallel Workers

Phase 1 — all 4 streams

4 Managed Devin sessions run in isolated VMs simultaneously. Each carries its own stack-specific playbook. An orchestrator session monitors progress and merges the 4 unified CSV outputs. No stream waits for another.

📋

Playbooks — Reusable Task Templates

Phase 1 — per-stack playbooks

The 7-step Ab Initio extraction architecture, the SAS two-pass approach, and the Informatica XML parser are each encoded as a Devin playbook with structured output schema. Applied consistently across all plans within that stack.

{ }

Structured JSON Output

Phase 3 — attribute resolution

Schema defined in the orchestrator prompt. Devin updates its JSON notebook on every hop it traces — partial results are available mid-session. The unified lineage edge schema is enforced across all workers consistently.

🔄

Automations — Nightly Re-index

Phase 4 — catalog maintenance

Devin Automation triggers on a cron schedule (or on a GitHub push event via webhook). Re-runs extraction on changed files, updates the lineage graph delta, regenerates affected ODCS contracts and business summaries.

🧠

Knowledge Base — Codebase Context

All phases — per-LOB instances

Ab Initio DML patterns, component semantic rules, naming convention map, and LOB-specific conventions stored as Knowledge items. Per-LOB Devin instances keep each instance within the 300-item knowledge cap.

DeepWiki Deep Dive

Continuous Lineage with DeepWiki

DeepWiki is the intelligence layer that makes lineage continuous, not a one-time exercise. It auto-indexes all 4 stacks every few hours — the moment a BTEQ script or Ab Initio plan changes in source control, DeepWiki detects the delta and triggers the downstream pipeline automatically.

Continuous Lineage Lifecycle

🔀

Code Change

git push / DDL alter

▶

🗺

DeepWiki

Delta Index

▶

⚡

Devin Automation

Triggered

▶

🔁

Delta Extraction

changed scripts only

▶

📊

Lineage Diff

lineage_diff.json

▶

📋

ODCS Contracts

regenerated + validated

▶

📂

Catalog

published

Cycle triggered automatically on every code change · stewards notified on ODCS contract updates

DeepWiki's 4 Roles Across the Lineage Lifecycle

Role	When It Fires	What Barclays Gets	Phase
Initial Index	Sprint 0 — first run against the full codebase	Naming convention map across all 4 stacks · dead code filter · LOB attribution · architecture diagrams with live code links	Phase 0
Change Detection	Every few hours — continuous background indexing	Delta file list: which scripts changed since last run. Triggers Devin Automation for affected scripts only — no full re-extraction needed	Phase 1 onward
Impact Analysis	Pre-commit / on every PR against the code repo	"This BTEQ change touches 3 downstream ODCS contracts." Lineage validation as a CI gate — PR fails if an upstream hop disappears from a live contract	Phase 3–4
Knowledge Base Feed	Continuous — after each index cycle	New Ab Initio component patterns, new SAS macro variants, new BTEQ table naming patterns auto-captured into Devin Knowledge Base — workers stay current without manual updates	All phases

Lineage Diff — The BCBS 239 Audit Trail

Every time DeepWiki detects a code change and the delta extraction runs, it produces a lineage_diff.json. This records exactly which edges were added, removed, or modified — timestamped against the git commit that caused the change.

This is the provable audit trail Barclays BCBS 239 wave 3 needs: lineage changed only because code changed, with a git-linked timestamp.

                + edge added: GCA_STG.WT_CUST.BAL_AMT

                +   → GCA_CLEAN.CLEAN_CUST.BALANCE

                +   via: BTEQ 50123 | ROUND transform

                - edge removed: GCA_STG.WT_CUST.OLD_BAL

                -   → GCA_CLEAN.CLEAN_CUST.BALANCE

                commit: a4f9c2d · 2026-10-19 · author: dev-team

PR / CI Gate — Lineage as Code Quality

When a developer raises a PR against the BTEQ or Ab Initio codebase, a Devin Automation session fires against the changed files before merge:

→Run delta extraction on PR diff only
→Compare new lineage edges against live ODCS contracts
→Flag broken upstream hops → fail the PR check
→Auto-update ODCS contracts if change is intentional + approved

Result: Lineage integrity is enforced at the point of code change — not discovered weeks later during a regulatory audit.

Human-Facing Wiki — For Barclays Data Stewards & CDOs

DeepWiki's indexed understanding is browsable by non-engineers. Data stewards and CDOs can navigate the codebase wiki without reading SQL or Ab Initio DML — understanding what each script does, what tables it populates, and what business process it serves. This is the human layer of continuous lineage: the wiki is the lineage in plain language.

Data Stewards

Browse by LOB · see which scripts feed each report attribute · confirm business names without engineering support

CDO / Governance

BCBS 239 lineage diff report per sprint · attestation that lineage reflects production code · no manual documentation

Engineering Leads

Architecture diagrams with live code links · identify orphaned scripts before decommission · cross-stack dependency view

Regulators / Audit

Point-in-time lineage snapshot per regulatory date · git-linked timestamps · automated, not manually maintained

Technical Solution

The 5-Phase Approach

Each phase gates the next. No phase commits to engineering until its inputs are confirmed. Click any phase to expand its detail.

0

Active Inventory Discovery

AutoSys JIL → active script inventory per LOB per stack

Gate phase

▼

47–57k base attributes is too large to process cold. Legacy codebases carry 30–40% dead weight — orphaned scripts, tables no longer populated, columns no longer read. AutoSys scopes the work to what is live in production.

Process Steps

Parse JIL files → extract all job definitions
Map each job → script + tech stack
Build job dependency graph (which job depends on which)
Tag each job with LOB from job name conventions
Mark active vs. inactive (scheduled vs. orphaned)

Output

Active script inventory per LOB per stack
Active base attributes (47–57k scoped to live-only)
LOB-level code size breakdown per stack
Job dependency map (used for Phase 1 ordering)

Devin: JIL parser playbook Devin: DeepWiki index

1

Multi-Stack Lineage Extraction

4 parallel streams — BTEQ · Informatica · SAS · Ab Initio → unified lineage CSV

Streams A/B: Low risk Stream C: Medium Stream D: High ⚠

▼

All 4 streams produce the same unified CSV schema: source_schema · source_table · source_column · target_schema · target_table · target_column · transform_type · transform_expr · filter_logic · stack · script. No stream waits for another.

Streams

A — BTEQ/SQL: LeapLogic + LLM · proven at GCA Wave 2 · 97% accuracy
B — Informatica XML: CONNECTOR + TRANSFORMFIELD parser · .par resolution
C — SAS .sas: Two-pass: macro expansion → PROC SQL + DATA step + LLM for ARRAY
D — Ab Initio: 7-step playbook · .pset → .dml → .mp recursive → .xfr hybrid DML

Ab Initio Feasibility Spike (required before Stream D)

Run on 5–10 sample plans first

Validate XFR coverage per component type
Confirm .pset PROD values available
Assess DML complexity distribution
Go/No-go: column-level vs. table-level lineage

Devin: 4 managed worker sessions Devin: per-stack playbooks

2

Unified Cross-Stack Lineage Graph

Stitch all 4 stack CSVs at Teradata boundaries → single queryable graph

Requires naming convention audit

▼

Graph Structure

Node: SCHEMA.TABLE.COLUMN (Teradata-normalised)
Edge: source_col → target_col + {stack · script · transform · filter}
Storage: flat CSV + JSON — lineage_graph_query.py as query engine
Upgrade path: Neo4j if query complexity demands it (not for MVP)

Stitching Steps

Build naming convention map (Phase 0 audit)
Resolve $(PARAM) table names via .pset / .par
Resolve Teradata views to base tables
Build alias registry for known naming mismatches
Validate on 10 known end-to-end chains before batch

Devin: orchestrator merges 4 worker outputs Devin: DeepWiki naming map

3

Attribute Resolution Engine

Input: PCIDB.AB.BALANCE → 4 data product outputs per attribute

Requires: confirmed attribute list

▼

A Barclays user provides one report attribute. The engine runs a backward BFS traversal across the unified graph and generates all 4 outputs in one session. Runs in batch mode across all 8–10k report attributes.

Traversal Logic

BFS backward from target attribute
Traverse all stacks — cross-stack edges stitched in Phase 2
Collect: hop count · branch count · active flag · schedule
lineage_graph_query.py as the traversal engine (580 edges, 334 nodes proven on test data)

4 Outputs Per Attribute

A — Lineage JSON: full hop chain, machine-readable
B — Business Summary: plain-English markdown for data stewards
C — Graphical View: Mermaid diagram (renders in GitHub / Confluence)
D — ODCS Contract: YAML v3.1.0, validated by data-contract-cli

Devin: structured JSON output schema Devin: orchestrator per attribute

4

Data Product Catalog & Query Interface

File-based catalog · CLI query · platform-neutral · migrates to any target

Platform-neutral MVP

▼

Catalog Structure

catalog/<lob>/dp_<attr>.json — Lineage JSON
catalog/<lob>/dp_<attr>.md — Business Summary
catalog/<lob>/dp_<attr>.mmd — Mermaid source
catalog/<lob>/dp_<attr>.yaml — ODCS contract

CLI Query Examples

query.py --attribute "PCIDB.AB.BALANCE"
query.py --lob PBWM --list
query.py --attribute "..." --upstream
query.py --attribute "..." --impact

Devin: CLI written from playbook Devin: Automation nightly re-index

Future State (once target platform confirmed)

Databricks

YAML → Unity Catalog · SQL → Delta CTAS

Snowflake

YAML → Data Product Registry · SQL → Snowflake CTAS

AWS Native

YAML → Glue Data Catalog · SQL → Athena CREATE TABLE AS

Phase 1 Detail

Per-Stack Extraction

Each stack has a dedicated extraction strategy. Click each tab for parsing approach, input files, key challenges, and Devin's specific role.

BTEQ / Teradata SQL

Risk: LOW ✓ Proven at GCA Wave 2

Input Files

.sql · .bteq · .ksh (Korn shell scripts with dynamic SQL)

Parsing Strategy

LeapLogic static analysis for table dependencies + LLM column-level extraction for complex transformations. Proven on 11,896 files at 97% accuracy.

Challenges

Teradata-specific syntax (QUALIFY, RANKID, macros); dynamic SQL in KSH string concatenation; macro definitions split across files.

Mitigations

Teradata macro resolver pre-processing pass; flag dynamic SQL as "partial lineage" rather than fail; cross-file macro resolver.

Extraction Flow

Identify SQL statements→ Parse SELECT/INSERT/CTAS→ Expand Teradata macros→ Extract column mappings→ LLM for complex transforms→ Output unified CSV

Informatica PowerCenter XML

Risk: LOW ✓ Most structured format

Input Files

PowerCenter workflow XML files (.xml); .par parameter files for table name resolution.

Key XML Elements

CONNECTOR (explicit field-to-field mappings) · TRANSFORMFIELD (transformation expressions) · SOURCE/TARGET (Teradata table names, direct) · FILTER/LOOKUP (filter and lookup dataset refs).

Challenges

.par parameter files must be resolved; reusable mapplets create cross-XML references; Java/SQL override transforms need LLM extraction.

Mitigations

Parameter resolver (same pattern as .pset); cross-file resolver across mapping XMLs; LLM extraction routed only for Java/SQL override blocks.

Extraction Flow

Export repository → workflow XML→ Resolve .par parameters→ Parse CONNECTOR elements→ Parse TRANSFORMFIELD→ Route overrides to LLM→ Output unified CSV

SAS Programs

Risk: MEDIUM Two-pass required

Input Files

.sas files containing PROC SQL, DATA step code, and SAS macro definitions (%let · %macro · %if · %do).

Three Constructs, Three Strategies

PROC SQL: direct SQL parser (same as BTEQ). DATA Step: assignment parser + LLM for ARRAY loops and MERGE. Macros: must be expanded before extraction.

Pass 1 — Macro Expansion

Resolve all %let and %macro definitions. Substitute macro variables into code. Expand %if / %do conditional blocks. Output: flat, macro-free .sas code per file.

Pass 2 — Lineage Extraction

Parse PROC SQL → column edges. Parse DATA step → assignment expressions. LLM for ARRAY-derived columns. Flag ARRAY columns as "array-generated" in unified CSV.

Ab Initio

Risk: HIGH ⚠ Spike required

5 File Types — None Alone Is Sufficient

.mp — graph plans (data flow topology, source/target tables)
.xfr ★ — transforms (field-to-field DML expressions, the core lineage)
.pset — parameters (table names, env flags, runtime values)
.dml — port schemas (field names and types, reveals passthroughs)
Subgraph .mp — nested calls (recursive traversal required)

7-Step Extraction Playbook

Step 1: Parse .pset → parameter resolution map
Step 2: Parse .dml → port schema registry
Step 3: Recursive .mp traversal (depth-first, cycle detection)
Step 4: .xfr DML extraction (rule-based + LLM hybrid)
Step 5: Built-in component semantic rules (join/rollup/scan/merge)
Step 6: Per-plan lineage assembly
Step 7: Cross-stack boundary stitching

DML Parsing Approach — Hybrid (rule-based + LLM)

Rule-based: simple direct assignments out.X = in.Y — covers ~70–80% of cases

LLM-aided: conditionals, lookup functions, date/string transforms — covers residual ~20–30%

Combined accuracy: ~95%+ with human spot-check on a sample. Bootstrap with Ab Initio EME export where available.

Phase 3 Outputs

4 Data Product Outputs Per Attribute

One attribute in → four structured outputs out. All platform-neutral. No target platform decision required before delivery.

{ }

Lineage JSON

Full hop chain, machine-readable, one file per attribute. Powers all other outputs.

How Devin generates it: The resolution engine uses Devin's structured JSON output API — schema defined in the playbook prompt. Devin updates its JSON notebook on every hop it traces, so partial results are available mid-session.

{
  "attribute": "PCIDB.AB.BALANCE",
  "lob": "PBWM",
  "depth": 4,
  "active": true,
  "schedule": "daily 02:30 UTC",
  "hops": [
    {
      "hop": 1,
      "source_schema": "RAW_SYSTEMS",
      "source_table":  "CUSTOMER_MASTER",
      "source_column": "BALANCE",
      "target_schema": "GCA_STG",
      "target_table":  "WT_CUST",
      "target_column": "BAL_AMT",
      "stack":         "ABINITIO",
      "script":        "CUST_EXTRACT.mp",
      "transform_type":"CAST",
      "transform_expr":"CAST(in.BALANCE AS DECIMAL(18,2))"
    },
    {
      "hop": 2,
      "source_schema": "GCA_STG",
      "source_table":  "WT_CUST",
      "source_column": "BAL_AMT",
      "target_schema": "GCA_CLEAN",
      "target_table":  "CLEAN_CUST",
      "target_column": "BALANCE",
      "stack":         "BTEQ",
      "script":        "50123_ACCT_LOAD.SQL",
      "transform_type":"DERIVED",
      "transform_expr":"ROUND(BAL_AMT, 2)"
    },
    {
      "hop": 3,
      "source_schema": "GCA_CLEAN",
      "source_table":  "CLEAN_CUST",
      "source_column": "BALANCE",
      "target_schema": "GCA_AGG",
      "target_table":  "AGG_ACCT",
      "target_column": "TOTAL_BALANCE",
      "stack":         "BTEQ",
      "script":        "50267_DEALS_STAGE.SQL",
      "transform_type":"AGGREGATE",
      "transform_expr":"SUM(BALANCE) GROUP BY CUST_ID"
    },
    {
      "hop": 4,
      "source_schema": "GCA_AGG",
      "source_table":  "AGG_ACCT",
      "source_column": "TOTAL_BALANCE",
      "target_schema": "PCIDB",
      "target_table":  "AB",
      "target_column": "BALANCE",
      "stack":         "BTEQ",
      "script":        "50286_SUMMARY_LOAD.SQL",
      "transform_type":"DIRECT",
      "transform_expr":"TOTAL_BALANCE"
    }
  ]
}

📄

Business Summary

Plain-English markdown for data stewards, business analysts, and CDOs. No SQL, no UUIDs.

How Devin generates it: The orchestrator session feeds the Lineage JSON into our validated prompt (LINEAGE_TO_BUSINESS_SUMMARY_PROMPT.md). Devin writes the markdown to catalog/pbwm/dp_pcidb_ab_balance.md. A document owner note flags what requires human confirmation.

# Business Summary — Account Balance (BALANCE)

**Field:** PCIDB.AB.BALANCE
**Business Name:** Account Balance
**LOB:** PBWM — Private Banking & Wealth Management
**Version:** Auto-generated · lineage-grounded · no inferred claims

---

## 1. What This Field Represents

BALANCE holds the total account balance aggregated per client,
expressed in the reporting currency, as at the AutoSys run date
(daily 02:30 UTC). It is calculated by summing individual balance
amounts across all active Loan IQ accounts per client identifier.

> Note to document owner: Regulatory use cases and downstream
> consumer teams should be confirmed before sharing with stakeholders.

## 2. Where the Data Ultimately Comes From

| Priority | Source System   | Business Role                             |
|----------|-----------------|-------------------------------------------|
| Primary  | Customer Master | Authoritative client reference database   |

## 3. How the Value Is Determined

Step 1 — Stage raw balance from Customer Master via Ab Initio
         (type-casted to DECIMAL 18,2)
Step 2 — Clean and round to 2 decimal places (BTEQ 50123)
Step 3 — Aggregate per client identifier: SUM(BALANCE) (BTEQ 50267)
Step 4 — Write to PBWM reporting table (BTEQ 50286)

## 6. Lineage Snapshot

  Source systems → Customer Master (Ab Initio pipeline)
  Staging layers → 3 intermediate tables
  Target report  → PCIDB.AB.BALANCE
  Depth          : 4 hops  |  Stack mix: Ab Initio + BTEQ
  Active         : Yes  |  Schedule: daily 02:30 UTC

◈

Graphical View — Live Lineage Diagram

Mermaid diagram auto-generated from Lineage JSON. Renders in GitHub, Confluence, Notion — no build step.

How Devin generates it: Devin writes a json_to_mermaid.py converter as part of the Phase 1 playbook. For each attribute, it reads the Lineage JSON and emits a .mmd file. Phase 2 upgrade: Devin builds a react-flow interactive portal over the catalog.

flowchart LR
    A["RAW_SYSTEMS\nCUSTOMER_MASTER\n.BALANCE"] -->|"Ab Initio\nCUST_EXTRACT.mp\nCAST DECIMAL(18,2)"| B["GCA_STG\nWT_CUST\n.BAL_AMT"]
    B -->|"BTEQ 50123\nACCT_LOAD.SQL\nROUND 2dp"| C["GCA_CLEAN\nCLEAN_CUST\n.BALANCE"]
    C -->|"BTEQ 50267\nDEALS_STAGE.SQL\nSUM GROUP BY CUST_ID"| D["GCA_AGG\nAGG_ACCT\n.TOTAL_BALANCE"]
    D -->|"BTEQ 50286\nSUMMARY_LOAD.SQL\nDIRECT"| E["PCIDB.AB\n.BALANCE"]

Source: catalog/pbwm/dp_pcidb_ab_balance.mmd · Generated by json_to_mermaid.py from Lineage JSON

📋

ODCS Data Contract

YAML v3.1.0 (Open Data Contract Standard · LF AI & Data Foundation). Validated by data-contract-cli before writing to catalog.

How Devin generates it: After Lineage JSON is complete, the Devin orchestrator fills the ODCS v3.1.0 template — every field grounded in the lineage graph (no hallucination risk). Then runs data-contract-cli validate in the same session before writing to catalog.

apiVersion: v3.1.0
kind: DataContract

id: dp_pcidb_ab_balance
info:
  title: Account Balance
  status: active
  owner: PBWM Data Office
  domain: pbwm
  lob: Private Banking & Wealth Management

servers:
  teradata_prod:
    type: teradata
    database: PCIDB
    dataset: AB

schema:
  - name: AB
    fields:
      - name: BALANCE
        type: decimal(18,2)
        description: >
          Total account balance per client, aggregated daily.
          Sourced from Customer Master via Ab Initio + 3 BTEQ stages.
        lineage:
          depth: 4
          stacks: [ABINITIO, BTEQ]
          active: true
          schedule: "daily 02:30 UTC"

quality:
  - type: notNull
    field: AB.BALANCE
    description: "Balance must not be NULL for active accounts"
  - type: custom
    description: "balance_date must equal AutoSys run date"

custom:
  impetus_generated: true
  pipeline_version: "genesis-engine-v1"
  catalog_path: "catalog/pbwm/dp_pcidb_ab_balance.yaml"

# Validated: data-contract-cli validate dp_pcidb_ab_balance.yaml
# Result: ✓ Valid against ODCS v3.1.0 JSON Schema

Risk Register

Top Risks & Mitigations

All risks are known and mitigated. Risks 1, 2 and 3 have explicit gates — decisions made before engineering commits prevent schedule risk.

HIGH ⚠

Ab Initio XFR Coverage Gaps

Built-in components (Join, Rollup, Scan) have no XFR file — column logic must be inferred from component type + port schema. If coverage is below 60%, column-level lineage is not viable.

Mitigation: 1-week feasibility spike on 5–10 sample plans before committing. Fallback: table-level lineage for Ab Initio, column-level for all other stacks.

HIGH ⚠

Cross-Stack Naming Mismatch

Ab Initio output table names may not match BTEQ source table names after parameter resolution — different teams, different conventions. A single mismatch breaks the stitching edge silently.

Mitigation: Naming convention audit in first sprint using DeepWiki output. Build alias registry for known mismatches before stitching layer.

HIGH ⚠

Barclays Attribute List Not Available

Phase 3 cannot run without the 8–10k report attribute list. If not provided before Phase 3 starts, the MVP cannot demonstrate end-to-end resolution.

Mitigation: Escalate as a named gate dependency. Barclays provides a list of 20–30 priority attributes for MVP before Phase 3 begins.

MEDIUM

AutoSys JIL Export Incomplete

Incomplete JIL export means the active inventory misses live code — Phase 1 processes dead scripts and the graph has incorrect coverage.

Mitigation: Validate JIL coverage against known job counts per LOB. Cross-check with LOB teams before Phase 1 starts.

MEDIUM

Devin Session Cost at Scale

Batch run across 8–10k attributes = thousands of Devin sessions. At Devin's current pricing, this may exceed budget for the full production run.

Mitigation: Use Devin for MVP (20–30 attributes). Full batch uses direct Python pipeline with LLM API calls — same quality, fraction of the cost.

LOW ✓

Target Platform Not Confirmed

Databricks / Snowflake / AWS Native decision is still pending. This could delay catalog publication if outputs are platform-specific.

Mitigation: Simplified SQL is ANSI. ODCS YAML is platform-neutral. Catalog is file-based. MVP delivery is not blocked by platform decision.

Where to Start

Next Steps

Three actions before engineering commits. All run before the first line of extraction code is written.

01

Barclays Gates — 5 inputs needed

Nominated MVP LOB · code for that LOB (all 4 stacks) · AutoSys JIL for nominated LOB · list of 20–30 priority attributes · PROD .pset files for Ab Initio plans in scope. No engineering starts without these.

02

First Sprint — 4 parallel tracks on Day 1

AutoSys JIL parse → active inventory · Ab Initio feasibility spike (5–10 sample plans) · Informatica XML parser (fastest win) · Naming convention audit via DeepWiki. All run simultaneously.

03

First Milestone — Prove the Pipeline End-to-End

One report attribute resolved across 2+ stacks, producing all 4 outputs: Lineage JSON · Business Summary · Mermaid graph · ODCS YAML. Validated by Barclays against a known source-to-target chain.