Building a fast, independent platform on a system you do not own
Many organizations run their core business on a vendor's system: a database they can read, whose schema, release cycle and API belong to someone else. Everything else grows around it. Tools query it directly, keep private copies that drift, and carry the vendor's quirks into every one of them.
This paper describes ORCA (Operational store, Read models, Change capture, Anti-corruption layer), an architectural pattern for that situation: own a translated, live copy for reads, and send every write back through the vendor. ORCA is assembled from known ideas. What it adds is how they are held together when the source is someone else's, and an exact account of which guarantees survive. It closes with what one implementation learned in production.
The source here is a vendor's database: a practice-management, ERP or line-of-business product the business runs on. You can read it; its schema, release cycle and API belong to someone else. Everything built after it grows around it.
This paper makes three contributions:
It closes with what one implementation learned in production (10).
Four answers come up first; none of them fits.
Table 1. The usual answers, against what a platform on someone else's system needs.
| Approach | The reads you need | Fast | Your own model | Vendor's schema untouched | In scope |
|---|---|---|---|---|---|
| Rebuild on the vendor's APIIt lacks the reads you need. | ✕ | ✕ | · | · | · |
| Replace the vendorOut of scope. | · | · | · | · | ✕ |
| Turn on change trackingThat alters a schema you do not own. | · | · | · | ✕ | · |
| Mirror the databaseYou get speed, but every app still speaks the vendor's model. | · | ✓ | ✕ | · | · |
| ORCAA translated, live copy for reads; every write back through the vendor. | ✓ | ✓ | ✓ | ✓ | ✓ |
✓ Met✕ Why it fails· Not the deciding issue
ORCA is the shape that remains: own a translated, live copy for reads, and send every write back through the vendor. It isn't a stack. Any language, database or message bus can implement it.
A low-latency, best-effort signal of which records changed, taken from outside the vendor's schema.
Capture tells the store where to look; nothing more. It's taken from outside the vendor's schema: a database trace session, a transaction-log reader, or network capture of the vendor's client traffic. Nothing in the vendor's schema is added or changed to produce it.
Capture is allowed to be wrong; it's never trusted for values. A record can arrive before its change commits, describe a change that rolls back, or not arrive at all. Everything downstream is built to survive.
Invariants
C1 Resumable. A consumer reconnecting from a position receives the records after it, or a gap.
C2 Honest gaps. If records may have been lost, the consumer is told; no gap means nothing was lost.
These are the only guarantees capture gives. Reconciliation (3.2) covers what capture misses.
A record names where a change happened; the store decides what to do with it. There are three shapes, with one response each:
Records naming tables the store doesn't copy (temporary tables and the like) are skipped.
loop: batch, gap ← capture.read_after(position) if gap: request reconcile(all) for record in batch: if record.keys: reread(record.table, record.keys) elif record.table: request reconcile(record.table) position ← batch.last
The database server is the one place every writer passes through. A trace session there sees the vendor's desktop clients, its API, its web portal and its scheduled jobs alike; capture taken from the API or from client traffic only sees the writers that use it.
The capturing service runs next to the database, writes to bounded rolling files, and serves records over a long poll with a resume position. A consumer that falls behind the files' retention gets a gap rather than a silent hole.
Your own current and full copy of the vendor's data.
A copy of every vendor database table. Anything downstream reads from it; it converges on what the source has committed without needing any trust in what the capture is. Its four parts are: a paced first copy, a re-read on hint, reconciliation by key range, and a raw layer.
Invariants
S1 Convergence. Every committed source row eventually appears in the store with its committed values.
S2 No rolled-back residue. A rolled-back value does not survive past the next re-read or reconciliation of its row.
S3 Ordered application. An older source read is never written over a newer one.
Each table is copied in primary-key order, in paced chunks; a chunk's rows and its last key commit together, making a restart resume after the last chunk is written. A failing chunk is retried at half the size, down to one row; the size grows back after each success. Capture runs alongside the copy, and the copy never overwrites a row capture has already written. A table counts as correct only after its first reconciliation.
A hint names only rows; it never supplies values. The store re-reads the named rows by primary key, with a committed-only read, and replaces its copy with the result.
on hint(table, keys): if keys is empty: request reconcile(table) return rows ← source.read_committed(table, keys) with lock(table): store.replace(table, keys, rows)
A hint that arrives before its transaction ends will wait; the read returns the new values if it commits and the original values if it rolls back (S2). The capture position is saved after each batch is applied; a restart reapplies the last batch, and re-reads are idempotent.
The hint only decides where to look; the source decides the truth.3.2 Operational store
Both sides compute a row count and a checksum sum per range; only ranges that differ are re-copied, with a committed-only read.
reconcile(table):
for range in key_ranges(table):
if summary(source, range) ≠ summary(store, range):
rows ← source.read_committed(table, range)
with lock(table):
store.replace_range(table, range, rows)
It runs after every gap, after each table's first copy, and on a slow schedule; it always beats the copy and the stream. A change capture missed lies in a range whose summary differs, up to checksum collisions; the next pass copies it (S1).
A store row missing from a range is deleted only after a lookup by primary key confirms it's gone; two engines rarely order or compare text the same way, and a range boundary drawn by one can exclude a row in the other. A re-read and a reconciliation of the same table each hold a lock on it from source read to commit; an older read never lands after a newer one (S3). Large tables reconcile off-peak; small tables, and tables a hint named, at any time.
The vendor's tables, kept exactly with a checksum per row. Nothing downstream reads them; they let the translation layer (3.3) and the read models be rebuilt without touching the vendor. A table schema that changes is copied again.
A translation library that turns the vendor's model into yours.
The one place the vendor's model is understood. It takes rows and returns your shapes; it does no I/O. The store feeds it from the raw layer, and a client's fallback feeds it straight from the source (4.2). Both get the same answer.
Invariants
A1 One home. Nothing outside the library reads the vendor's model.
A2 Pure. A shape is a function of the rows it's given; the same rows always produce the same shape.
Each shape lists the tables and columns it reads. That list decides which raw changes rebuild it (3.4) and what the fallback has to fetch.
build(entity, source):
rows ← source.fetch(declared_tables, entity) # store or vendor
return translate(rows, lookups)
A translation that's wrong is wrong everywhere at once, so it's checked. Every night, random entities are built from the store and compared field by field with the same entities built by independent queries straight from the source. A difference is read again a moment later and counts only if it persists.
Screen-shaped documents built by the translation layer from the raw layer.
What clients actually read. They're versioned, so a consumer always knows what it has and what it's missing.
Invariants
R1 Commit order. Version order equals commit order.
R2 Whole versions. A consumer never holds part of a version.
R3 No skips. A consumer reading after version v never misses a committed version.
A re-read or reconciliation that changes a raw row a shape depends on marks its entity dirty, in the same transaction. The shaper rebuilds dirty entities. A shape that comes out identical isn't written and takes no version. A changed lookup rebuilds everything that uses it.
A plain sequence isn't enough; sequence values are handed out before commit, so a later commit can carry an earlier number. Each shaping transaction takes a single counter row, increments it, and holds its lock until commit; versions commit in order (R1). The same transaction writes one outbox row per changed shape.
rebuild(entity):
begin
v ← counter.increment() # row lock held until commit
for doc in build(entity, store):
if doc ≠ stored(doc):
write(doc, v)
outbox.append(v, doc)
commit
Consumers read the outbox by version, never by row, and a batch always ends on a version boundary. A consumer applies a whole version and only then saves it as its position (R2, R3). A shape that no longer exists keeps its row with an empty document, so its deletion is a change like any other.
A fan-out layer (a pub/sub cache, a websocket service) that carries committed versions from the outbox to everyone listening.
It's delivery only, never the record. Anything it loses is recovered from the outbox by position; losing a message costs latency, never correctness. It never splits a version (R2).
A client can keep its own embedded copy of the read models it needs (typically open records plus a recent window), fed by delivery and resumed by version. A client that reconnects after its position has expired resynchronizes rather than replaying.
When the platform is down, a client may run the same translation library straight against the source; A2 makes the answer the same. The fallback is narrow: the one record on screen, rate-limited, and clearly marked as live. An outage of the platform must never become a flood of reads on the vendor.
Nothing writes to the vendor's database directly; every write goes through the vendor's own API.
Invariants
W1 One claimer. Each intent has at most one claimer at a time, in order per record.
W2 Never resend blind. An intent whose send is uncertain is resolved by observation, not by sending again.
W3 Evidence before done. A write is reported done only with evidence; a write with side effects only when confirmed.
A client records the user's intent in a durable local log and shows it at once; a short undo window is optional. A queue (e.g. FOR UPDATE SKIP LOCKED) gives each intent one claimer, in order per record (W1). That isn't exactly-once delivery. A worker can send and then crash before recording it.
A write is judged by what comes back through capture, not by the API's response alone.
judge(intent, observed):
if intent.created_id and observed.id = intent.created_id:
return confirmed
if observed holds intent.values and observed.changed_after(intent.sent_at):
return state_reached
return undetermined
Anything with side effects (creating records, sending, billing) counts only when confirmed (W3).
Leases with fencing tokens coordinate your own writers. They're checked when an intent is queued and again before it's sent. They can't recall a request already in flight, and they mean nothing to people working directly in the vendor's application; conflicts with those people are detected afterwards through capture, not prevented. The vendor's own record locks, if visible, are mirrored in as a warning.
What the whole pattern promises, assembled from the invariants of its parts.
Each failure, what happens, and the invariant that holds through it.
Table 2. What happens when each part fails.
| Situation | What happens | Holds |
|---|---|---|
| Capture misses a change (trigger writes, unparsable statements) | Reconciliation finds the differing range and re-copies it | S1 |
| Capture reports a gap | The consumer requests reconciliation of everything the gap could cover | S1, C2 |
| A change rolls back after its hint | The committed-only re-read returns the original values | S2 |
| The first copy races a change | Each table counts as correct only after its first reconciliation | S1 |
| A worker crashes after sending a write | The intent is resolved by observation; it is never resent blind | W2 |
| A client reconnects after its position has expired | Full resync of its read models, not a replay | R3 |
| The platform is down | Clients fall back narrowly to the source, visibly | A2 |
| The vendor changes its schema | The raw layer shows it; the table is copied again and translations are rebuilt | S1, A1 |
One order, followed through the system. "Order 42" is any record in the vendor's database. First the ordinary path, then the same order when things go wrong.
The user sees the change at once, from their own intent log. The vendor's API applies it. Capture says only "orders, key 42". The store re-reads that one row with a committed-only read, rebuilds the read models for order 42 in one transaction that takes version 1001 and writes it to the outbox, and every client receives version 1001 as a whole. The intent compares what came back with what it sent: the values match and arrived after the intent was sent, so it is judged "state reached". For a create, the API's returned ID would make it "confirmed".
Nothing arrives for order 42. The intent waits; after its timeout the store re-reads order 42 directly, and the next checksum sweep would find the range anyway. The intent then resolves on what it sees, never by sending again.
Capture still reports "orders, key 42", because the statement ran. The committed-only re-read returns the old values, nothing changes, and no version is issued. The intent sees its values never arrived and ends "undetermined", shown to the user.
Another worker claims the intent. Its outcome is uncertain, so it checks the store before doing anything. Order 42 already shows the change, so the intent is judged on that, and the vendor is never called twice.
Their change arrives through capture like any other. If it set the same values, the intent can only say "state reached", because nothing proves which request caused it. If it set different values, the intent ends "undetermined" and the user sees the conflict.
It reconnects from its last version. If the outbox still holds everything after that version, it receives whole versions in order; if not, it resynchronizes its read models instead of replaying.
Figure 2. Six scenarios, step by step: the ordinary path, a lost event, a rollback, a crashed sender, a concurrent edit in the vendor's application, and an offline client.
One implementation, against a vendor's practice-management system on SQL Server. It replaced direct reads from the office's tools; every app, dashboard and mobile tool reads from it, and almost every write goes through it. The stack is incidental. Any equivalent works: log-based capture such as Debezium, another database, Kafka for delivery, or SQLite on clients.
An Extended Events session on the database server, filtered to insert, update and delete, writes to a few bounded rolling files. A small service on the same server reads them and serves records over HTTPS as a long poll: a consumer asks for everything after its position and waits up to 25 seconds. Each response carries the next position and a gap flag; a consumer that falls behind the files gets the flag. Because it's taken at the server, it sees every writer: the vendor's desktop clients, its API, its web portal and its scheduled jobs.
A record names a table and the keys it has, or only an entity (an order), or nothing. An entity-only record re-reads that entity in every table keyed by it. A hint on a table whose key column isn't indexed is turned into a reconciliation of that table, so a hint never scans a large table.
A Rust service keeps every source table in PostgreSQL, in its own schema: the same table names, columns and primary keys, plus one row checksum column. Types map directly, with two losses worth knowing: datetime2(7) keeps six fractional digits, and a NUL inside a string is dropped. One column of stored documents, nearly 200 GB, isn't copied; it's fetched from the vendor when someone opens a document.
Re-reads use READCOMMITTEDLOCK, so a rolled-back write is never copied. Range checksums are computed on both sides with dirty reads to keep them cheap. A summary that's wrong at worst copies a range for nothing or leaves it for the next pass; the copy itself is committed-only. A table whose key includes a uniqueidentifier is reconciled as one range, because the two engines order those differently. A table with no primary key is copied and checked whole.
The source never sees more than three connections: one for re-reads on hints, one for the first copy and reconciliation, and one for the nightly check. Every statement has a timeout. Every ten minutes the service compares the source's tables, columns, types and keys with what it copied; on a change it exits, a supervisor restarts it, and changed tables are copied again.
Requests are numbered from a sequence. A table is covered up to the number that was current when its pass began, so a request that arrives during a pass gets another one. Large tables wait for an evening window; a pass still running when the window closes stops between ranges and resumes there the next night. Tables due go oldest attempt first, so one that keeps failing can't hold up the rest.
Shapes are built from the raw schema only, 500 entities at a time on a full build. Lookups are re-read every minute; a changed customer or contact rebuilds that customer's entities, and any other changed lookup rebuilds everything. Redis on a separate host fans versions out and keeps a stream per audience. Desktop clients keep open records plus the last year in LMDB, with a fuzzy index for search, and resume over a websocket by version; when the platform is down they run the same translation crate against SQL Server.
Each intent goes into an append-only, checksummed local log and shows on screen at once, with a five-second undo. A PostgreSQL queue claims intents with FOR UPDATE SKIP LOCKED. Leases take fencing tokens from a PostgreSQL sequence, and the vendor's own record locks are mirrored into the same leases. Mobile apps keep their own offline queue; the service that forwards their writes checks the record's lease first.
Table 4. Settings in this implementation.
| Setting | Value |
|---|---|
| First-copy chunk | 2,000 rows, 250 ms pause |
| Reconciliation range | 20,000 rows |
| Statement timeout | 60 s; 4 h for a whole-table read |
| Baseline reconciliation | every 24 h per table |
| Off-peak window | 19:00 to 02:00 |
| Reconciled at any hour | under 200,000 rows, or named by a hint |
| Schema check | every 10 min |
| Nightly check | 200 random entities, re-checked after 30 s |
| Freshness logged | every 15 min, p50 and p95 |
ORCA treats the vendor's system as the truth and everything else as a signal. Capture says where to look; the source decides the truth. Reconciliation catches what capture misses, and writes go through the vendor and are judged by what comes back.
The combination is the point. A CDC pipeline usually assumes trustworthy capture and owned targets; CQRS usually assumes you own the write side. ORCA assumes neither, and says exactly which guarantees survive.
Curtis, N. (2026). ORCA: Operational store, Read models, Change capture, Anti-corruption layer (Version 1.0). https://doi.org/10.5281/zenodo.23493408
@techreport{curtis2026orca,
author = {Curtis, Nathan},
title = {{ORCA}: Operational store, Read models, Change capture, Anti-corruption layer},
year = {2026},
month = oct,
note = {Version 1.0},
doi = {10.5281/zenodo.23493408},
url = {https://orcapattern.org}
}