Reference Architecture: Data Pipelines

Status: Accepted | Date: 2026-07-11 | Review: 2027-07-11

Applicability and Non-Goals

Use this reference architecture for governed movement and transformation of data across WA agency cloud, SaaS, and legacy environments, including:

• Scheduled extracts, file transfers, database replication, and batch ETL or ELT
• Event and change-data-capture pipelines where the business needs lower latency than a batch service can provide
• Curated datasets for business intelligence (BI), regulatory reporting, controlled sharing, data science, or downstream applications
• Warehouse or lakehouse implementations selected against explicit workload requirements

This is not an architecture for transactional request processing, an operational API, master-data ownership, records-management policy, or ad hoc movement of data without source and sharing authority. It does not mandate a cloud, pipeline product, transformation framework, query engine, table format, catalogue, notebook, or reporting tool.

Ibis, DuckDB, DuckLake, Amazon S3 Tables, Trino, and Quarto are optional implementation examples only. They are not standards established by this reference architecture.

How to Use This Reference Architecture

This reference architecture composes existing ADRs; it does not create a new technology decision. Apply requirements from Accepted ADRs. A Proposed ADR is a dependency under consideration, not an accepted standard: obtain the project’s required design and risk approval before relying on it, and record any different project decision.

Agency obligations for privacy, information sharing, classification, records, procurement, and offshoring still apply even where no repository ADR covers them.

Assumptions and Prerequisites

Before choosing products or creating production data movement, confirm:

• A business owner, source-system owner, data custodian, pipeline owner, and each consuming owner are named.
• The authoritative source and the authority to collect, use, transform, and share each dataset and attribute are documented. Technical access is not sharing authority.
• Purpose, consumers, classification, privacy obligations, retention and disposal authority, data location, and approved use restrictions are known.
• Source interfaces can provide a stable snapshot, change marker, event identifier, or other mechanism that supports complete and repeatable reads.
• Consumers can agree contracts, freshness and quality thresholds, planned outage handling, and breaking-change notification.
• Network paths, workload identities, encryption, key management, logging, and support coverage are available in every participating cloud and legacy environment.
• Business-approved recovery time objectives (RTOs), recovery point objectives (RPOs), maximum data latency, expected volume, concurrency, and cost envelope exist before service selection.

If a prerequisite is unavailable, record the gap, interim control, owner, and resolution date rather than allowing a tool default to become the decision.

Architecture and Capability Model

flowchart LR
    sources[Authoritative Sources]
    ingest[Ingestion and Event Buffer]
    process[Orchestration and Processing]
    stores[Controlled Data Stores]
    serve[BI, Sharing, APIs, and Analysis]
    control[Contracts, Quality, Lineage, Security, and Operations]

    sources -->|snapshot, change, or event| ingest
    ingest -->|validated input| process
    process -->|versioned output| stores
    stores -->|governed access| serve
    control -.-> ingest
    control -.-> process
    control -.-> stores
    control -.-> serve

Keep the following logical capabilities identifiable even when one managed product implements several of them:

Do not infer that a product catalogue entry, replication setting, or source-code graph supplies all of these capabilities. Validate the runtime evidence.

Variants

Minimum Batch Variant

Use for bounded datasets where an approved schedule meets the maximum data latency and the source supports repeatable extraction.

• Use one managed scheduler or existing supported enterprise scheduler, a fit-for-purpose execution service, encrypted landing and curated storage, and a governed serving interface.
• Capture a source snapshot identifier or high-water mark and a unique run ID. Make writes idempotent through stable business keys, partition replacement, merge semantics, or another tested method.
• Define safe rerun and backfill windows. A failed run must not silently append duplicates, publish partial output, or advance the source checkpoint.
• Apply contracts, blocking and warning quality thresholds, runtime lineage, quarantine, replay, and source-to-output reconciliation under Proposed ADR 015 or an approved project-equivalent decision.
• Prefer private connectivity and workload identity. Use an approved managed file-transfer gateway when a legacy endpoint supports only files or SFTP.

This is the minimum production shape, not a relaxation of classification, privacy, backup, or operational controls.

Higher-Assurance Batch Variant

Use when data sensitivity, public impact, statutory reporting, volume, or recovery objectives justify additional controls.

• Separate source landing, processing, quarantine, curated publication, and backup administration according to ADR 001 trust boundaries.
• Preserve immutable source extracts or equivalent reproducible source snapshots for the approved replay period.
• Require independent control totals or record-level reconciliation and approval before publication for high-consequence outputs.
• Run representative performance, restore, corruption, partial-source, schema-change, and full-backfill tests against RTO, RPO, and cost limits.
• Use protected release promotion and explicit data-version rollback rather than replacing a known-good publication in place.

Streaming Variant

Choose streaming only when a documented business response time cannot be met economically and safely by micro-batch or scheduled batch processing. Confirm that the source produces durable events or change records and that consumers can handle the selected delivery semantics.

• Define event identity, partition key, ordering scope, event and processing time, late-arrival policy, retention, replay horizon, and schema evolution.
• State and test whether delivery is at-most-once, at-least-once, or effectively once at the business outcome. Provider claims do not remove the need for idempotent consumers and reconciliation.
• Use a durable event log or buffer between source and processing when replay, burst absorption, or failure isolation is required.
• Monitor end-to-end lag, backlog, dropped and duplicate events, poison events, watermark movement, state growth, and sink publication.
• Quarantine poison events with reason codes and controlled replay. Reconcile source offsets and business totals to sink outcomes.
• Design state checkpoint recovery and stream-processor upgrades so that rollback does not lose or duplicate an unbounded range of events.

Streaming is not automatically the higher-assurance choice. It usually adds state, ordering, support, and cost complexity.

Higher-Assurance Streaming Variant

For critical or high-volume event services, add isolated failure domains, capacity headroom, tested broker or regional failover, protected schema compatibility gates, independent event or source backups where required, and regular replay exercises. Measure recovery from the retained source through to reconciled consumer state, not only broker availability.

Workload Selection Boundaries

Do not create a lakehouse solely because object storage is available. Do not use a BI semantic model as the authoritative integration contract. A solution may compose a lakehouse for durable analytical data with a warehouse or BI serving layer when the duplicated cost, lineage, and reconciliation are owned.

Implementation Examples

The following official product links illustrate possible building blocks, not standards, endorsements, or equivalent service sets. Feature, region, support, identity, networking, recovery, export, and pricing differences must be tested.

For a lightweight bounded workload, Ibis, DuckDB, or DuckLake may be evaluated as optional transformation or query examples. S3 Tables may be evaluated as an AWS-specific managed table option, Trino as a distributed query option, and Quarto as a bounded publication option. Their selection needs the same support, security, interoperability, performance, recovery, and exit evidence as any other product.

Provision infrastructure and supported platform configuration with OpenTofu or Terraform under ADR 010. Provider-native frameworks are only the documented bootstrap or exception options allowed by that ADR. Use versioned configuration management alongside them for legacy hosts and appliances.

Project Kickoff Artifacts

Create these artifacts before implementation selection is final:

• A one-page service context showing sources, consumers, trust boundaries, data flows, environments, regions, and responsible organisations
• A dataset register recording owner, custodian, authoritative source, collection and sharing authority, purpose, classification, privacy status, retention, location, offshoring decision, consumers, and restrictions
• Versioned input and output contracts with compatibility policy, quality thresholds, freshness, keys, volume envelope, and consumer sign-off
• A selection record comparing batch, micro-batch, and streaming and comparing warehouse, BI, and lakehouse needs against latency, scale, support, recovery, portability, total cost, and team capability
• An operational design covering run identity, runtime lineage, checkpoints, idempotency, rerun, backfill, quarantine, replay, reconciliation, alerts, dashboards, and support escalation
• Approved RTO and RPO, failure-mode analysis, backup inventory, restore and replay test plan, and continuity dependencies
• A threat model, privacy assessment, access model, records plan, offshoring and supplier assessment, and logging profile
• An OpenTofu or Terraform repository and state design, environment mapping, CI/CD and release plan, and any ADR 010 exception record
• A cost and performance model with representative test data, budgets, quotas, growth assumptions, unit-cost measures, and cost-alert owners
• A migration and exit runbook covering dual-run, reconciliation, cutover, rollback, format and catalogue export, retention, and decommissioning

Information Governance

Apply the WA Information Classification Policy, Privacy and Responsible Information Sharing, and WA Data Offshoring Governance as applicable. Record agency legal, privacy, information-management, cyber-security, procurement, and data-owner approvals rather than treating a cloud region selection as approval.

• Classify source data, curated outputs, quarantine records, lineage, quality samples, logs, backups, and catalogue metadata. Derived and joined data may require a higher classification or create new privacy risk.
• Minimise collection and sharing to the approved purpose. Apply retention, disposal, legal hold, access review, and disclosure controls to every copy.
• Assess storage, processing, support access, subprocessors, telemetry, metadata, backups, and disaster recovery locations for offshoring. Include SaaS control planes and vendor support, not only the primary data region.
• Keep sensitive record content out of ordinary logs and alerts. Restrict and audit access to quarantine, samples, and data-quality evidence.
• Confirm lawful and responsible sharing before onboarding each consumer and when purpose, claims, linkage, or destination changes. Involve Aboriginal people where sharing affects Aboriginal people and communities as required by applicable WA arrangements.

Ownership and Operations

The operating model must assign named teams, support hours, delegates, and escalations for:

Maintain runbooks for delayed or missing sources, schema drift, threshold failure, partial publication, duplicate output, poison events, credential or key failure, capacity exhaustion, regional or site outage, replay, rollback, and data correction. Review access, contracts, thresholds, costs, capacity, dependencies, and unused datasets on an agreed schedule and after material change.

Resilience and Recovery

• Derive pipeline and dataset RTO and RPO from business impact. Include source re-extraction time, event-retention window, full backfill duration, catalogue and key recovery, validation, and consumer reconciliation.
• Size retention, checkpoints, throughput, quotas, and recovery capacity to meet the objectives under representative backlog and source throttling.
• Keep at least one tested, logically and administratively independent backup where ADR 014 requires it. Replication, object versioning, table snapshots, and multi-zone storage are not independent backups by themselves.
• Back up or reproducibly recover contracts, code, deployment artifacts, infrastructure state, catalogue metadata, access configuration, keys or key procedures, source positions, and required data, not only curated tables.
• Test restore into an isolated environment and then replay and reconcile to a usable publication. Measure achieved RTO and RPO and remediate gaps.
• Design degraded operation explicitly: hold publication, serve a labelled last-known-good dataset, or switch to an approved alternate path. Never hide stale or partial data from consumers.

Migration and Exit

Inventory source semantics, transformations, schedules, contracts, history, formats, catalogue metadata, lineage, quality rules, identities, consumers, retention, performance, and cost before migration.

1. Export representative data in documented, readable formats and export catalogue schemas, ownership, classifications, contracts, lineage mappings, quality rules, and access configuration through supported APIs or files.
2. Prove the destination can read historical and incremental data, preserve required type and timestamp semantics, run the same controls, and restore without the source provider.
3. Dual-run old and new pipelines from a common snapshot or event position for an approved period. Compare record counts, control totals, quality results, lineage, latency, consumer queries, performance, and cost.
4. Define source freeze or checkpoint, cutover authority, consumer validation, rollback trigger, maximum rollback window, and handling for writes or events received during rollback.
5. Keep the old path and required recovery material protected until acceptance criteria and the rollback window are complete. Then revoke access and dispose of copies under approved retention and disposal authority.

A file export alone is not an exit plan if table semantics, catalogue metadata, identity mappings, contracts, or operational history are needed to use it.

Acceptance Checks

• Accepted ADR requirements are implemented; every Proposed dependency is identified and covered by project approval or an explicit alternative
• Source authority and sharing authority are recorded for every dataset and consumer
• Classification, privacy, records, location, supplier, and offshoring assessments cover data, metadata, logs, quarantine, backups, and support
• Batch, streaming, BI, warehouse, and lakehouse choices are supported by measured latency, scale, concurrency, cost, support, and portability needs
• Versioned contracts and approved quality thresholds define warning, quarantine, stop-publication, and consumer-notification behaviour
• Production runs emit queryable runtime lineage with run, code, contract, parameter, timestamp, and source or snapshot identifiers
• Quarantine, replay, reconciliation, idempotency, rerun, and backfill have passed failure and duplicate tests
• Security, freshness, lag, quality, volume, cost, and capacity alerts reach named responders without leaking sensitive records
• Performance and unit cost meet the approved envelope at expected load and tested growth or backlog
• RTO and RPO are approved and demonstrated by restore, replay, and reconciliation; independent backup is not claimed from replication alone
• OpenTofu or Terraform plans, state controls, drift detection, and recovery evidence meet ADR 010
• Dual-run migration, portable data and catalogue export, rollback, consumer acceptance, and legacy decommissioning are tested and approved