Event and discovery sync

Intent

This pattern assumes another system you rely on, such as a CI/CD platform, an issue tracker or a SaaS admin console, which can call a webhook when something changes, and has an API that lists what exists. It keeps your own copy of that system's state using two feeds. Events from the webhook keep the copy fresh, a scheduled sweep through the API keeps it complete, and idempotent writes, which give the same result however many times they run, let the two overlap without conflict. This enables three outcomes:

1. You can answer questions the other system's own screens and reports can't.
2. Changes show up within minutes, and anything an event missed turns up by the next sweep.
3. A record looks the same whichever feed wrote it, so the copy can be trusted.

Motivation

The systems a team relies on answer the questions their makers anticipated, through their own screens and reports. Anything else, such as which projects have stalled, or how long changes take to reach production across every team, needs a copy of their state that you can query yourself. Kept up to date by only one feed, that copy fails in one of two ways:

1. The copy is always out of date. Polling the API for everything is slow, runs into the system's rate limits, and leaves the copy only as fresh as the last poll.
2. The copy is never complete. Listening only to webhooks misses anything that hasn't changed since the subscription began, because it never sends an event, such as a project nobody has touched in months.

This pattern uses both feeds, and lets each do what it does best:

1. Events keep the copy fresh. Each webhook event records a change as it happens, so the copy is never more than minutes behind.
2. A scheduled sweep keeps the copy complete. It lists everything the API holds, including whatever hasn't changed since the subscription began, and runs outside busy hours, because it's the heaviest user of the API.

Three properties keep the two feeds consistent with each other:

1. Each event is treated as an event notification: it says what changed, and everything else is fetched from the API, so a record never depends on what one event's payload happened to include.
2. Every record is keyed by the other system's own IDs, and every write is idempotent, so retries, repeated events and the overlap between the feeds change nothing.
3. Facts that can't be worked out later are captured as each event arrives, from the related systems, and stored with the record. These are facts the context moves on from, such as who owned an item, or which version of something was current, when the change happened.

Applicability

Use this pattern when:

1. You need your own queryable copy of another system's state, because its own screens and reports can't answer your questions.
2. The system can call a webhook when something changes, and has an API that lists what exists.
3. Changes should show up within minutes, and anything missed should turn up by the next scheduled sweep.

Do not use this pattern when:

1. The system offers a change feed, a stream of every change it makes, or a bulk export you can read directly.
2. You only need its current state, which you can fetch from its API when you need it.

Structure

Participants

Constraints and trade-offs

1. The sweep restores what exists, not what happened. An event lost while the subscription was down is gone for good, and history starts when you subscribe, unless you backfill it through the API, as far back as the system keeps it.
2. Each event needs at least one call to the system's API, and those calls count against its rate limits, which matters most at the busiest times.
3. A full sweep grows with the system. Once it's too slow or too costly to run, it has to list only what changed since the last sweep, which needs an API that can filter by date.
4. Both feeds need an administrator on the external system, often in another team, to create the service identity and the webhook subscription.

Part of how I deliver

When the tools a team relies on can't answer the questions its leaders ask, this is how I give them a copy that can. It's one of many patterns I bring through contract delivery, and it works with whichever systems your teams already use.

Related