SCROLL

Synchronized Change Replication Over Line Logs

Because we can do better than webhooks as a synchronization mechanism

draft-scroll-protocol-00 · request for comments · no implementations in the wild, one document

TL;DR

Webhooks have well-known pitfalls when used as a synchronization mechanism. Instead, you open an HTTP connection and receive objects as NDJSON (JSON objects separated by newlines), with a few clever conventions: every event carries a cursor, so resuming from where you left off is trivial, and every provider follows the same rules, so one client can consume many providers. SCROLL says nothing about how your data should be shaped — it is a delivery mechanism.

Press play on each lane to watch the same six events, the same outage, delivered both ways. The consumer goes down for a moment in each lane; only one of them notices anything went wrong — and it's the wrong one that doesn't.

WEBHOOKS · provider pushes to your endpoint

SCROLL · consumer reads the provider's log

speed ×0.5

What it is

A provider exposes each collection (customers, subscriptions, …) as an ordered, cursor-addressed change log of full-state events, served as NDJSON over HTTPS with the same credential as the rest of its API. A consumer with no cursor reads that log from the beginning, which is how the replica gets built; a consumer with a cursor resumes from it. Same URL, same request. The consumer's entire sync state is one saved cursor.

idle — no connection

GET /scroll/feed/customers
Authorization: Bearer <same key as your REST calls>

200 OK
Content-Type: application/x-ndjson

{"cursor":"01J9XP1A","operation":"upsert","object":{"id":"cus_120","plan":"pro"}}
{"cursor":"01J9XQ4R","operation":"upsert","object":{"id":"cus_123","plan":"free"}}

GET /scroll/feed/customers?scroll-cursor=01J9XQ4R
Prefer: stream

200 OK
Preference-Applied: stream

{"cursor":"01J9XR2M","operation":"upsert","object":{"id":"cus_123","plan":"pro"}}

your database ▸ saved cursor: — · 0 objects · bootstrap needed

With Prefer: stream the response never ends and each change arrives as it commits. Without it, the same URL returns a bounded page you can poll on your own schedule — same events, same cursors, one parser, one code path. Applying an event is a blind upsert keyed by object id (or a delete, for tombstones), so duplicates and replays are harmless by construction.

Nothing ever calls your server. Every connection is one you open, so there is no inbound endpoint, no signing secret, no dead-letter queue — and it works behind NAT, on localhost, and from a cron job.

Versus webhooks

Webhook-fed replicas fail in well-known ways, and none of them are bugs in anyone's implementation — they are what happens when a consumer must reconstruct an ordered log from a lossy notification stream. SCROLL has the provider serve the log itself, and most of the machinery on both sides — delivery queues, retry schedules, dedup stores, reconciliation crons — simply has no reason to exist. For both sides the diff is mostly deletions.

	Webhooks	SCROLL
Missed events	Retry schedules, dead-letter queues, replay tooling — and what still slips through is a silent gap.	Nothing is delivered; the consumer resumes reading from its saved cursor.
Order & duplicates	At-least-once and unordered; every consumer builds a dedup store and a reorder buffer.	The log is ordered and events are full-state upserts, so replays and reordering are harmless by construction.
Bootstrap	No backfill; an initial import races live deliveries.	Read the same feed with no cursor. The read that builds the replica is the read that keeps it current.
Downtime	A slow or down consumer trips failure thresholds; the provider disables the endpoint and the gap is silent.	Not an error. The consumer is merely behind, and catches up.
Deletes	Often never arrive.	Tombstones are first-class events in the log.
Drift	Reconciliation crons crawling list APIs, or a support ticket when someone notices.	The feed carries a count and checksum your replica checks itself against.
Security	A public inbound endpoint plus a second credential system: HMAC secrets to issue, verify, and rotate.	No inbound endpoint exists; the same bearer credential as the rest of the API.
Local development	A tunnel to fake a public HTTPS endpoint on your laptop.	Works anywhere outbound HTTPS works: laptops, containers, serverless, behind NAT.

ARCHETYPES.md walks through what this looks like for directory sync, billing, and transactional email.

How it works

1 · BOOTSTRAP

Read from nothing

Ask the feed without a cursor and it starts at the beginning of the log, which for a compacted log is current state. Ask for a stream at the same time and that first request is also your last: it fills the replica and then carries on into live changes. No second endpoint, no handoff, and no import-vs-live race to lose.

2 · SYNC

Tail

Hold a stream for commit-time latency, or poll bounded pages from a cron. Apply, save the cursor, repeat. A severed stream is a reconnect, not an incident; downtime just means you're behind.

3 · TRUST

Verify

At the end of the log the feed tells you what should be there: a count and a checksum at the cursor you now hold. Compare and you know. Nothing to schedule, no extra request, and a mismatch is repaired by the read you already implemented for step 1.

The consumer is one table and one loop; the provider is one table and one query. Appendix B of the spec gives each in ~40 lines of pseudocode, and scroll-examples/ is a runnable client and server with a test harness measuring latency and correctness against a webhook baseline.

The hard part: adoption

The biggest problem with SCROLL is not technical. It is adoption: a delivery protocol is worth nothing until providers ship it, and providers are the party with the least reason to move. Their webhook infrastructure is already built and paid for, and its ongoing costs — the dedup, the reconciliation, the drift — fall mostly on consumers. Asking providers to add an endpoint so that other people's code gets simpler is a hard sell, and this document does not pretend otherwise.

The one structural advantage: a SCROLL feed can be built on top of a provider's existing webhooks and list APIs by a shim adapter, while the reverse is impossible. So consumers can start without waiting — run a shim against providers that have never heard of SCROLL — and each shim is both a working feed and a demonstration to the provider of exactly what to ship natively. Whether that path actually gets walked is an open question, and honest arguments that it won't are as welcome as implementations.

Scope, honestly

One-way replication only: provider → consumer. Writes stay ordinary API calls; there is no merge, and therefore no conflict.
Not event sourcing: the log is a delivery buffer, never the query surface. You query your own tables with your own indexes.
Not a replacement for Slack-style incoming webhooks — those are writes. Write with an API call, learn with the log.
Requires the provider to sequence changes per collection. A provider that records no change ordering cannot conform, and the spec says so plainly.
The guarantees are transport-independent; NDJSON over HTTPS is the one binding every implementation must speak, so a consumer written once works everywhere.

Comments, requested

This is a request for comments in the original sense of the phrase: every normative statement in the spec is open to challenge. Section 13 lists the decisions where the authors' confidence is lowest — start there if you want to disagree efficiently. File an issue, or annotate the spec and send it back marked up — CONTRIBUTING.md says which route fits what. If you run a provider, tell us why you wouldn't ship this. Disagreement is the desired response; silence is the failure mode.