The Landscape at a Glance

Genie Ontology — Two-Context Model

┌─────────────────────────────────────────────────────────┐
│                    GENIE ONTOLOGY                        │
│                                                          │
│  ┌─────────────────────┐   ┌──────────────────────────┐ │
│  │   MODELED CONTEXT   │   │    INFERRED CONTEXT      │ │
│  │ (you define & certify)│   │ (Genie extracts & ranks) │ │
│  │                     │   │                          │ │
│  │ · UC Metric Views   │   │ · Dashboards             │ │
│  │ · Domains           │   │ · SQL query history      │ │
│  │ · Genie Agent Pages │   │ · Genie Agents           │ │
│  └────────┬────────────┘   └─────────────┬────────────┘ │
│           └──────────────┬───────────────┘              │
│                          ▼                               │
│                     OntoRank                             │
│         (authority × usage × linkage × recency)         │
│                          ↓                               │
│             Permission gate (UC policies)                │
│                          ↓                               │
│              Snippets injected into context window       │
└─────────────────────────────────────────────────────────┘

OntoRank Scoring Signals

UC Metric Views — Technical Spec GA Apr 2026

YAML-based semantic layer inside Unity Catalog. Define once, query from SQL, BI tools, REST APIs, agents. Advanced features include multi-fact relationships, LOD calculations, parameterised metrics, window measures, measure-on-measure composition.

metric_view:
  name: order_metrics
  source: catalog.schema.orders

  measures:
    - name: total_revenue
      synonyms: ["revenue", "sales", "GMV"]
      aggregate: SUM(order_value)
      filter: WHERE status != 'RETURNED'

    - name: avg_order_value
      aggregate: total_revenue / order_count  # measure-on-measure

  dimensions:
    - name: region
    - name: channel

  materialization:
    enabled: true
    refresh_schedule: "0 6 * * *"

Open Semantic Interchange → Apache Ossie

Databricks joined four months after Snowflake originated it — a defensive move to ensure UC Metric Views' features (window functions, materialization) are represented in custom_extensions rather than lost. No native UI for OSI import/export yet on Databricks.

Ontos (Labs) — Business Catalog v1.0.0 · Jun 2026

A Databricks App that adds data mesh semantics UC doesn't natively provide: data products, data contracts (ODCS v3.1.0), domain ownership. React + FastAPI + OWL TTL entity model stored in Lakebase (Postgres).

OntoBricks (Labs) — Formal Knowledge Graph v0.9.0

LLM-assisted OWL 2 ontology generation from UC table metadata → R2RML mappings → RDF triples in Lakebase / Delta / Neo4j → GraphQL API + MCP server for agents.

The Full Platform Stack

Capability Overlaps — Where Boundaries Are Blurry

UC Metric Views vs Genie Ontology

They don't compete — they're designed to layer. Design intent: define the 20% of concepts that cannot be wrong as Metric Views. Let Genie Ontology handle the long tail (80% of business terms that would never justify a formal YAML definition).

Ontos (Labs) vs UC Business Glossary + Governance Hub

Ontos is the preview of what UC Business Semantics will look like in 12–18 months. Customers who need data contracts now use Ontos; those who can wait should watch Governance Hub GA.

How Context Flows Through the Stack

DEFINITION TIME
  1. Data engineer writes UC Metric View for "revenue" (YAML, versioned)
  2. Domain steward adds Business Glossary definition → linked to Metric View
  3. Genie Ontology extracts: certified Metric View + dashboard tiles + SQL history
     → OntoRank scores certified Metric View highest

QUERY TIME
  4. User asks Genie: "What was our revenue last quarter?"
  5. Genie Ontology retrieves top-ranked snippets → UC permission gate
  6. Snippets injected into context window
  7. Agent generates SQL using certified definition + UC lineage
  8. Result returned with source attribution

Without UC Metric Views, step 7 relies on inferred snippets — which can drift or conflict over time.

Lakewatch — Agentic SIEM GA

Databricks' entry into cybersecurity. Agentic SIEM powered by Claude (Anthropic), built on Unity Catalog + Panther (detection-as-code). Runs entirely on the lakehouse — security data lives where business data already lives.

Uses: Lakeflow ingest (OCSF-aligned) → Delta Lake → UC governance (NIS2/DORA) → Agent Bricks agents for detection, triage, threat hunting → Unity AI Gateway traces as security signals.

MemCurrent — Persistent Agent Memory Beta

Three access patterns: Plugins (automatic log/load without agent action), Tools (on-demand explicit query), Dreaming (async consolidation from episodic to distilled). Storage on Lakebase (Postgres) in customer's cloud account — never shared with Databricks.

Unity AI Gateway — Runtime AI Governance GA Aug 4 2026

Sits between all consumers (users, apps, agents) and all LLM endpoints (internal + external). Four capability clusters:

Agent Bricks Execution Architecture

┌─────────────────────────────────────────────┐
│  AGENT RUNTIME                               │
│  Manages hosting, identity, scaling on Apps  │
│               ↓                              │
│  DurableAgentServer                          │
│  Wraps any framework (LangGraph, CrewAI,     │
│  Claude Code SDK, vanilla Python)            │
│  API modes: sync · streaming · background    │
│  Runtime Store → crash recovery (Lakebase)  │
│               ↓                              │
│  Databricks Sandbox                          │
│  Isolated code execution, scoped UC access  │
│  Persistent home directory across calls     │
└─────────────────────────────────────────────┘

Future Evolution Signals

How Vertical Solutions Are Composed

┌──────────────────────────────────────────────────┐
│  Branded App (Genpact FinanceOne, CitiusTech ClinicalIQ)
├──────────────────────────────────────────────────┤
│  Databricks Apps (FastAPI + React, in workspace) │
├──────────────────────────────────────────────────┤
│  Genie Spaces  │  Agent Bricks  │  Lakewatch SIEM│
├──────────────────────────────────────────────────┤
│  Unity AI Gateway — routing, budgets, audit      │
├──────────────────────────────────────────────────┤
│  Genie Ontology / UC Metric Views (pre-populated)│
├──────────────────────────────────────────────────┤
│  Lakebase (operational)  ·  Lakehouse//RT (ms)   │
├──────────────────────────────────────────────────┤
│  Unity Catalog — governance, lineage, RBAC       │
├──────────────────────────────────────────────────┤
│  Delta Lake / Lakeflow Zerobus                   │
└──────────────────────────────────────────────────┘

Vertical IP is what partners pre-build on top: domain Metric Views, pre-populated Genie Ontology snippets, pre-wired Agent Bricks workflows, regulatory governance templates, industry-shaped Apps.

BFSI Composition Patterns

Real-Time Fraud Detection Reference Architecture

Raw card transactions
  └─ Lakeflow Zerobus (gRPC, sub-5s, no broker)
       └─ Delta Lake (raw + feature tables)
            ├─ Spark Real-Time Mode (~5ms e2e)
            │    └─ Lakebase customer risk profiles (<300ms reads)
            ├─ MLflow Model Registry + feature serving
            └─ Databricks App (FastAPI + React fraud dashboard)
                 └─ Unity AI Gateway (audit, spend controls)

Key design: Lakebase replaces external Redis/DynamoDB as the online feature store. No separate operational database. UC governs data and models on one platform for BCBS 239-style lineage.

Named BFSI Partners

HLS Composition Patterns

Databricks maintains an open-source hls-skills repo (databricks-industry-solutions, last updated Oct 19, 2026) with PHI detection, genomics cohort queries, RWE cohorts, HEDIS measures, and an evaluation harness.

Apache Ossie — Semantic Portability

Vendor-neutral YAML-based interchange format for semantic definitions. Think of it as Parquet for semantic models. Hub-and-spoke model: each platform has a converter to/from Ossie YAML as the hub.

version: "0.1.1"
semantic_model:
  - name: finance_model
    ai_context: "Use for P&L, revenue, and cost analysis"
    datasets:
      - name: orders
        source: prod_catalog.finance.orders
    metrics:
      - name: total_revenue
        expr:
          ANSI_SQL: "SUM(order_amount)"
          DATABRICKS: "SUM(order_amount)"
          SNOWFLAKE: "SUM(order_amount)"
        ai_context: "Primary top-line revenue metric"

Hub-and-spoke ecosystem: dbt MetricFlow → Ossie YAML (reference) · Snowflake native SYSTEM$ functions (GA Aug 2026) · Databricks UC Metric View converter (PR #152, bidirectional) · GoodData, Salesforce, Apache Polaris also listed.

SI Opportunity Map

The Strategic Shift

UC started as a governance layer (control access to tables and files). The 2024–2026 arc is a deliberate expansion into business semantics: making UC the system that knows what things mean.

Traditional data catalogs (Collibra, Alation, Atlan) were built for data stewards — humans who curate metadata. UC's new semantic layers are explicitly designed for AI agents — machines that query business context at inference time. This is a fundamentally different design target.

The Five-Layer UC Semantic Stack

Each layer is independently queryable and additive — you don't need Layer 5 before Layer 3 delivers value.

Layer Highlights

Business Glossary Preview

AI-native design: Genie Code drafts new Glossary pages from query patterns, flags definitions that drift from actual usage, surfaces definitions as context when agents query any linked asset. Key differentiator from Collibra/Alation: the glossary is actively maintained by the platform's AI, not just human stewards.

UC Pages GA

Human-authored markdown context attached to any UC asset. Free-form narrative: "what this dataset is, how to use it, known quirks." UC Pages feed the modeled context side of Genie Ontology — the human voice in the knowledge graph.

Domains GA Oct 8 2026

Business-aligned groupings of UC assets. Agents scoped to a Domain only retrieve context from that domain's assets — bounded, relevant context instead of the whole catalog. Subdomains supported (Finance > Treasury > Liquidity).

Unity Catalog MCP Servers

The Grounding Chain

Agent question: "What is our Q3 NPS by region?"
         ↓
Genie Ontology query (OntoRank retrieval)
  → certified UC Metric View: "net_promoter_score"
  → Domain: "Customer Experience"
  → Glossary term: "NPS" → definition + linked table
  → UC Page: "Survey data — NPS calculation caveats"
         ↓
Permission check (Unity Catalog)
  → user has read on nps_survey table? Yes
  → column mask on respondent_id? Yes → masked at runtime
         ↓
Compute: Metric View SQL executed
         ↓
Answer with source citations from UC lineage

Every agent answer inherits the governance and business meaning that data stewards built into UC.

UC vs. Standalone Semantic Tools

vs. Traditional Data Catalogs (Collibra, Alation, Atlan)

Collibra/Alation's defense: For enterprises with heterogeneous stacks (Snowflake + Databricks + Oracle + Salesforce), UC only governs the Databricks portion. Catalog Federation is narrowing this gap for some external catalogs.

vs. dbt MetricFlow / Cube

The coexistence reality: Most Databricks customers have dbt. UC Metric Views and dbt MetricFlow are not mutually exclusive — dbt builds and tests the tables; UC Metric Views define business metrics on top of those tables. Apache Ossie (PR #152) is the portability bridge.

GA vs. Preview Status (Oct 2026)

The "AgentOS" Framing

"AgentOS" is not an official Databricks product name. It is an industry concept describing the idea that production AI agents need an OS-equivalent: something that handles identity, execution boundaries, memory, permissions, scheduling, and observability.

Databricks framing

"Most teams focus on the core agent loop, but that is only 1% of what it actually takes to run agents in production."

The remaining 99% — deployment, security, context grounding, evaluation, memory, cost control, governance, observability — is what Agent Bricks + Omnigent collectively provide.

Four-Layer Agent Infrastructure Stack

Key design principle: every agent action is governed by Unity Catalog. A layer-3 policy that denies a git push is enforced before the action reaches the filesystem.

Omnigent — YAML-Declared Agent Apache 2.0

name: code-reviewer
harness: claude-code       # swap to: codex, pi, openai-agents (zero other changes)
model: claude-opus-5-5

prompt: |
  You are a careful code reviewer. Focus on correctness and security.

tools:
  - type: mcp
    server: uc-functions    # Unity Catalog functions as MCP tools
  - type: mcp
    server: genie-space
    space: risk-metrics

sub_agents:
  - name: test-writer
    harness: codex
  - name: security-scanner
    harness: claude-code
    skills: [databricks/security-review]

policies:
  spend:
    warn_at: "$3"
    hard_cap: "$5"
  actions:
    - pattern: "git push"
      decision: ask          # pause for human approval
    - pattern: "DROP TABLE *"
      decision: deny         # non-negotiable block

The Policy Engine — The Real Governance Innovation

Traditional guard rails live in the prompt — which can be circumvented by clever multi-step plans. Omnigent intercepts every action before execution:

Policies defined at three levels: agent YAML, session override, and platform-wide (Unity AI Gateway, workspace admins). This is the mechanism that lets enterprises say "AI at the helm, humans retain control over irreversible actions" — exactly what regulators in BFSI and HLS need.

The "Polly" Multi-Agent Pattern

Polly (orchestrator — plans, delegates, writes NO code)
  ├─ Sub-agent A in git worktree 1 (Claude Code) → implementation
  ├─ Sub-agent B in git worktree 2 (Codex)       → tests
  ├─ Sub-agent C (different vendor than writer)  → code review
  └─ Policy gates applied before any merge/push

Key principles: orchestrator never writes code (avoids context window exhaustion on large tasks) · parallel isolation in git worktrees · cross-vendor review (reviewer uses different model family — correlated blind spots avoided) · spend checks and HITL before final output.

DurableAgentServer vs. Legacy MLflow

Agent Bricks CLI + agent.toml

[agent]
name = "risk-analyst"
description = "Answers risk questions over UC Metric Views"

[[tools.mcp]]
type = "managed"
server = "uc-functions"

[[tools.mcp]]
type = "managed"
server = "genie"
space = "risk-metrics"

[memory]
type = "lakebase"          # MemCurrent-backed

[sandbox]
enabled = true
catalog = "prod_catalog"   # scoped UC access (only this catalog)

[tracing]
enabled = true             # all tool calls → UC system tables

Three CLI commands: agentbricks init (scaffold) · agentbricks dev (local against live Databricks) · agentbricks deploy (to Agent Runtime on Databricks Apps)

Full Agent Lifecycle on Databricks

Each step is on-platform. No separate tool needed for evaluation, secret store, observability, or memory database.

vs. Standalone Agent Frameworks

Why These Patterns Now

Every major data platform in 2026 — Databricks, Snowflake, Microsoft Fabric — is building what they each call a "semantic layer." Beneath the marketing, they are converging on the same set of structural design patterns. Naming them gives architects and SIs a vocabulary to evaluate platform maturity, design engagements, and talk to customers without resorting to product brochure language.

These patterns are not product features. They are recurring architectural decisions that appear independently across all three platforms.

The Five Patterns

Maturity Across All Three Platforms

Cross-Platform Pattern Comparison

SI Service Offerings from the Five Patterns