The Agentic Data Layer

Transformation logic is executable business policy. The Agentic Data Layer gives it a first-class architectural home.

An Agentic Data Layer is a governed operational layer that interprets transformation intent and applies versioned, testable, observable data policies between business systems.

Executive summary

Enterprise integration is usually described as a connectivity problem: one system exposes an API, another accepts data, and a pipeline or piece of middleware moves information between them. That description is incomplete. Connectivity is usually the simplest part of an integration. The durable difficulty lies in deciding what the data means, how it should change, which rules apply, and how those decisions evolve as the business changes.

A source system may represent a customer as an account, a contact, a household, or a legal entity. A destination may require a different structure. Fields that appear equivalent carry different semantics. Required values may not exist in the source, and defaults vary by region, product, and contractual relationship. The resulting mappings are not merely technical configuration — they are executable business policy.

Traditional integration tools can manage these concerns, and most are effective within their intended scope. The problem is architectural placement. Transformation logic ends up distributed across API handlers, ETL jobs, iPaaS workflows, stored procedures, event consumers, and application code. Each connection becomes independently responsible for understanding business meaning, the same customer or product is interpreted differently by each pipeline, and small policy changes require updates across many implementations.

The Agentic Data Layer is a reusable pattern that addresses this. Agentic does not mean software is permitted to make uncontrolled production changes. It means the layer uses contextual reasoning — often assisted by AI — to translate human intent into explicit transformation behavior. The behavior remains governed: humans approve changes, policies are versioned and tested, runtime execution is deterministic or explicitly constrained, and every material decision can be inspected.

The layer's purpose is deliberately narrow:

Preserve the meaning and reliability of operational data while allowing transformation rules to evolve without repeatedly rebuilding integrations.

Applications change. Schemas change. Business rules change. The data layer between them must be designed for change as a normal operating condition.

The problem: why integration keeps failing

Integration rarely fails because bytes cannot move between systems. It fails because meaning changes.

A technical integration begins as field mapping:

source.first_name  ->  destination.FirstName
source.last_name   ->  destination.LastName
source.email       ->  destination.Email

Soon, exceptions appear. Personal and business customers follow different rules. Some regions require tax identifiers. Duplicate accounts must be reconciled before insertion. Legacy records use older status values. The destination requires values the source does not store. The integration quietly changes from field mapping into business interpretation:

IF customer.type = "producer"
AND customer.country = "Panama"
AND customer.certification is active
THEN:
    create Account of RecordType "Certified Producer"
    normalize farm identifiers
    derive regional ownership
    attach current certification status
    preserve original source identifiers
ELSE:
    apply standard customer policy

This is business logic. Implementing it inside an integration tool — or behind a visual mapping interface — does not change its nature.

Schemas are temporary

Schemas are often treated as stable contracts. In practice, they are temporary representations of a system's current needs. Applications upgrade, vendors revise APIs, acquisitions introduce new models, and optional attributes become mandatory. The dangerous failures are semantic rather than syntactic — a field keeps its name while its meaning changes:

status = "active"

May mean:
  the account is commercially active
  the subscription is active
  the person logged in recently
  the record is not archived
  the supplier is approved for transactions

Syntactic compatibility does not guarantee semantic compatibility.

Rules change more often than interfaces

Interfaces may remain stable for years while transformation logic changes every quarter: territory ownership, customer segmentation, eligibility criteria, consent handling, product taxonomy, duplicate resolution. Each looks like a "small mapping update." Architecturally, each is a modification to executable policy — and when policy is embedded in individual integrations, every change triggers a search across implementations to find which pipelines, workflows, and reports depend on the old behavior.

Mappings become duplicated authority

Suppose five systems consume customer data, and each integration carries a version of the same rule: if the country is in the EU, apply the European customer category. Over time the copies diverge:

Integration A: EU member-state list from 2024
Integration B: hard-coded region field
Integration C: country prefix lookup
Integration D: manually maintained table
Integration E: no rule; defaults all customers to Global

The problem is not duplicated code. It is duplicated authority — no one can confidently say which rule represents the business's current intent.

Pipelines show the same pattern. A linear Extract → Transform → Load accumulates regional branches, retries, lookup tables, and dead-letter queues until a change to one rule ripples across several downstream paths. The pipeline is not brittle because it is poorly written. It is brittle because it carries more business responsibility than its abstraction was designed to express.

Local correctness does not guarantee global coherence.

Existing approaches

The Agentic Data Layer does not replace existing integration technology. Each approach below is effective within its intended scope. What they share is a gap: none of them, by default, provides a governed home for transformation policy.

Any of these can host parts of an Agentic Data Layer — the distinction is architectural, not technological.

A collection of integration workflows becomes a data layer only when transformation policy is intentionally centralized, governed, versioned, testable, and treated as a reusable capability.

Defining the layer

Every word of the definition is deliberate:

Core principles

Intent over implementation

Business rules should be expressed first as intended outcomes, then derived into endpoint-specific implementation.

The implementation remains necessary — but it is derived from intent rather than confused with it.

Declarative where it helps

Where practical, policies should describe desired behavior rather than execution mechanics:

policy: certified-producer-account
when:
  source.type: farm
  source.certification.status: active

produce:
  account.category: producer
  account.certification_status: certified

route:
  owner: regional_assignment(source.country)

Declarative rules are not automatically superior to code. They earn their place when they improve reviewability, reuse, and governance; complex logic may still require programmatic extensions.

Context is explicit

A field rarely has meaning in isolation. Transformation may depend on the source and destination system, entity type, jurisdiction, effective date, customer segment, prior state, and policy version. Context must be modeled explicitly rather than hidden in ad hoc conditional logic.

AI assists, humans govern

AI can infer candidate mappings, explain schema differences, draft transformations and validation rules, generate test cases, and summarize failures. It does not remove engineering control. Material changes move through a lifecycle:

Intent
  ↓
Generated draft
  ↓
Technical review
  ↓
Business approval
  ↓
Automated testing
  ↓
Controlled release

No model should silently alter production behavior.

Reversible where possible

Not every transformation can be reversed, but changes should preserve enough lineage to reconstruct what happened. The original value, transformed value, policy version, timestamp, and decision reason should be retained when the business risk justifies it. And a policy without tests is an undocumented assumption.

Architecture

The layer sits between business applications and the systems that create, store, or consume operational data:

+------------------------------------------------------+
|                 Experience Channels                  |
| Web | Mobile | Portals | Assistants | Internal Tools |
+--------------------------+---------------------------+
                           |
                           v
+------------------------------------------------------+
|                Business Applications                 |
| CRM | ERP | Commerce | Support | Industry Platforms  |
+--------------------------+---------------------------+
                           |
                           v
+======================================================+
|                 AGENTIC DATA LAYER                   |
|                                                      |
|  Intent          Policy         Transformation       |
|  Interpretation  Registry       Runtime              |
|                                                      |
|  Validation | Normalization | Enrichment | Routing   |
|  Decisioning | Lineage | Testing | Observability     |
+======================================================+
                           |
          +----------------+----------------+
          |                                 |
          v                                 v
+-----------------------+       +-----------------------+
| Operational Systems   |       | Integration Runtime   |
| Systems of record     |       | APIs | Events | Jobs  |
| External platforms    |       | Brokers | Connectors  |
+-----------+-----------+       +-----------+-----------+
            |                               |
            +---------------+---------------+
                            |
                            v
+------------------------------------------------------+
|               Analytical Data Systems                |
| Warehouses | Lakes | BI | ML Platforms | Archives    |
+------------------------------------------------------+

This is not a mandatory deployment topology. The pattern may be implemented as a dedicated platform, domain-aligned services, an iPaaS with shared governance, custom middleware, or a policy engine combined with existing integration infrastructure. The defining characteristics are the responsibilities and controls — not the product category.

Transport still belongs to the integration runtime: APIs, events, brokers, and connectors continue to move data. The layer defines how data is interpreted and adapted during that movement. Analytical systems consume operational data and may provide enrichment back — but they should not become the mediator for every operational exchange.

Responsibilities

Validation outcomes should be richer than pass/fail:

Reject
Quarantine
Warn
Default
Request human review
Continue with reduced confidence

What it is not

Boundaries are often clearer when defined by exclusion. The layer mediates operational data; it must not quietly absorb its neighbors' jobs.

System of record:    owns the authoritative business state
Agentic Data Layer:  owns the policy for interpreting
                     and moving that state

Example: Cropster to Salesforce

A coffee business manages farm, producer, lot, quality, and certification data in Cropster and needs selected operational data available in Salesforce. A direct integration begins with a fixed mapping:

Cropster producer.name    ->  Salesforce Account.Name
Cropster producer.email   ->  Salesforce Contact.Email
Cropster farm.country     ->  Salesforce Account.BillingCountry

Then the business introduces rules. Farms and exporters use different record types. A producer may be associated with multiple farms. Certification data syncs only while active. Regional ownership depends on country and commercial relationship. Legacy records contain informal country names. Records missing required identifiers must be quarantined, and existing Salesforce accounts must be matched before creation. The architecture becomes:

+------------------+
|     Cropster     |
| Producers, Farms |
| Lots, Quality    |
+--------+---------+
         |
         v
+----------------------------------------------+
|            Agentic Data Layer                |
|                                              |
| 1. Validate source contract                  |
| 2. Normalize country and identifiers         |
| 3. Resolve producer and farm identities      |
| 4. Classify business entity                  |
| 5. Enrich regional ownership                 |
| 6. Apply certification policy                |
| 7. Build Salesforce-specific payloads        |
| 8. Record lineage and policy version         |
+----------------------+-----------------------+
                       |
                       v
+----------------------------------------------+
|                 Salesforce                   |
| Accounts | Contacts | Farms | Certifications |
+----------------------------------------------+

Initial policy

policy: cropster-producer-to-salesforce
version: 1.0

classify:
  when:
    source.entity_type: producer
  as:
    salesforce.record_type: Producer

normalize:
  source.country:
    using: iso-country-normalization

validate:
  required:
    - source.external_id
    - source.name

match:
  salesforce.account:
    by:
      - source.external_id
      - normalized_name_and_country

A business change arrives

Without a shared policy layer, this rule touches the extraction job, Salesforce automation, a middleware function, a territory mapping table, and reporting logic. With one, the intent becomes a new policy version:

policy: cropster-producer-to-salesforce
version: 1.1

ownership:
  preserve_existing_when:
    salesforce.owner_type: commercial

  otherwise:
    when:
      certification.type: organic
      certification.status: active
    assign:
      team: specialty-sourcing

The change still requires engineering review, and tests must confirm that existing commercial owners are preserved, active organic certifications trigger the new assignment, expired certifications do not, missing dates are handled explicitly, and historical records are not silently reassigned. But the Cropster connector is not rebuilt, and the Salesforce transport is not rewritten. The change occurs in the policy governing how data is interpreted between them.

That is the architectural value of the pattern.

AI's place in the layer

The layer is not agentic because a model autonomously controls production data. It is agentic because contextual software helps translate intent into controlled behavior — generating candidate transformations from schemas, example records, rejected payloads, and documentation, along with the tests that prove them.

Human intent:
  "Normalize Central American country values to
   ISO-3166 alpha-2, preserving the original value
   for audit."

AI-generated draft:
  - map recognized aliases to canonical codes
  - preserve source value in lineage metadata
  - reject ambiguous abbreviations
  - generate tests for Panama, Guatemala,
    Costa Rica, and unknown values

The draft is useful. It is not production authority.

Production transformations should generally be deterministic, policy-constrained, explainable, and attributable to a version. Where a model must participate at runtime — classifying unstructured text, for example — the system records the model and prompt version, relevant context, confidence, output, fallback behavior, and review threshold.

An AI system should not change mappings directly in production, alter validation thresholds silently, invent source-of-truth relationships, rewrite historical data outside a controlled migration, or hide uncertainty behind a confident output.

Agentic assistance should increase the speed of controlled change — not bypass control.

Policy lifecycle

Intent
  ↓
Draft policy
  ↓
Schema and contract validation
  ↓
Test generation and review
  ↓
Approval
  ↓
Staged execution
  ↓
Production release
  ↓
Observation
  ↓
Revision or retirement

A mature implementation supports draft environments, test fixtures, output diffing, canary releases, rollback, replay, and policy retirement. Shadow evaluation lets a candidate policy run against production inputs without affecting outputs:

Input record
   |
   +-> Current policy   -> Production output
   |
   +-> Candidate policy -> Comparison only

Responsibility matrix

The matrix is not absolute. It clarifies the expected center of responsibility.

Tradeoffs

The pattern introduces a new architectural capability — and new cost. Adopt it with eyes open:

When not to use it

The pattern is not a default requirement for every organization:

• Simple CRUD integrations with aligned schemas, minimal rules, clear ownership, and rare change.
• Small organizations with two or three systems — a managed connector is enough.
• Bounded one-off migrations. Test and document the script; don't build a platform for it.
• Highly stable estates where schemas and rules rarely move.
• Presentation-only formatting, which belongs in the application or experience layer.
• Domain behavior — order cancellation, claim approval, subscription renewal — which belongs in the systems that own those capabilities.

Decision guide

Does the integration contain meaningful
business interpretation?
                |
       +--------+--------+
       |                 |
      No                Yes
       |                 |
       v                 v
Use direct        Are rules repeated across
integration or    systems or changing often?
managed connector        |
                 +-------+-------+
                 |               |
                No              Yes
                 |               |
                 v               v
        Keep logic near   Consider an Agentic
        the owning        Data Layer
        integration

Implementation patterns

One implementation of this pattern is DataExpandr, which provides mechanisms for describing, governing, and executing operational data transformations across systems. Evaluate it — or any product — as an implementation of the pattern, not the definition of it. The architectural test is always the same: Is transformation intent explicit? Are policies reusable, governed, versioned, and testable? Is execution observable? Are application and data responsibilities kept separate?

Key takeaways

• Integration difficulty is primarily semantic, not connective. Moving data is easier than preserving its meaning.
• Mappings become executable business policy and should be governed with the same care as other business rules.
• The Agentic Data Layer is a pattern, not a product category. It can be implemented with several technologies.
• Its canonical responsibility is operational interpretation: validating, normalizing, enriching, routing, and transforming data between business systems.
• Separate intent from endpoint-specific implementation so policy can evolve without rebuilding transport.
• AI assists policy creation; it does not bypass governance. Humans approve material changes.
• Preserve architectural boundaries. The layer is not a CRM, warehouse, gateway, broker, analytics system, or workflow engine.
• Observability must include decisions, not only infrastructure. You should be able to explain why a record was transformed or routed.
• The pattern has real costs — complexity, latency, governance, platform risk, and ownership must be addressed explicitly.
• Use it where change is persistent and transformation is shared. Avoid it for simple, stable, or temporary integrations.

Related knowledge

Within SKF-001, the Agentic Data Layer occupies the Operational Data layer and supports the Automation, Applications, and Experience layers above it:

Experience
    |
Applications
    |
Automation
    |
Operational Data   <-  Agentic Data Layer
    |
Business Capability

• SKF-001 — Summit Architecture Pyramid (framework)
• SKM-001 — The Summit Architecture Pyramid (module)
• SKM-003 — Operational Data vs Analytical Data (planned)
• SKM-005 — Integration Is Not Architecture (planned)

Closing perspective

Architecture is durable when it gives change an appropriate place to occur. In brittle integration estates, changing business intent has no clear home — it appears simultaneously in mappings, workflows, application code, warehouse models, and manual operating procedures. The Agentic Data Layer creates an explicit home for one important class of change: how operational data should be interpreted and adapted as it moves between systems.

It does not eliminate integration complexity; it organizes it. It does not replace business applications, integration infrastructure, or analytical platforms; it gives each of them a more reliable contract.

It does not make transformation autonomous. It makes transformation governable.

The enduring idea is not the name. It is the recognition that transformation logic deserves to be treated as a first-class architectural capability.

Changelog

0.5.0  2026-07-17  Initial handbook draft — restructured from the
                   v0.1 manuscript for clarity; corrected artifact
                   ID (SKM-008 → SKM-002); approach survey and
                   tradeoffs converted to tables
0.1.0  —           Original manuscript