Sync cursors expire, and so does the empty window

The incremental sync worked for weeks, then quietly started re-scanning the same empty window on every cycle. Nothing errored. It just did the same nothing over and over, faster than I noticed. For anyone relying on it, new data simply stopped arriving while every status light stayed green, which is the worst kind of broken: the kind nobody spots until the gap is embarrassingly large.

The cursor was a history id from an external API. Store the last one you saw, ask the provider for everything since, save the new cursor, repeat. Simple, and wrong in two ways that only show up under real conditions.

An old cursor is a 404, not a network blip#

The first failure is age. Providers do not keep history forever. If your stored cursor is old enough, the provider has expired it, and asking for changes since a cursor it no longer knows returns a 404.

A 404 looks like a transient error if you squint, and transient errors get retried. So a naive sync retries an expired cursor forever, getting 404 forever, and the integration is stuck while looking busy.

The fix is to name the condition instead of lumping it in with network noise. Map that specific 404 to a typed sentinel, a distinct error the caller can match on:

CursorExpired  -> caller falls back to a full resync
network error  -> caller retries

When the sync sees CursorExpired, it stops trying to be incremental, does one full resync to rebuild state, and gets a fresh valid cursor out the other side. The recovery is deliberate because the error is specific. A generic failure could not tell the difference between “come back later” and “start over”.

Incremental cursors are not permanent. Design for expiry with a typed sentinel that triggers a full resync, so an old cursor is a known state and not a mystery retry.

An empty poll can still move the cursor#

The second failure is the one that had me re-scanning forever, and it is sneakier.

A polling cycle can find no new records and still receive a newer cursor from the provider. That is not a contradiction. The provider’s history advanced past events that do not concern you, so there is nothing to hand back, but the position has moved on regardless.

Here is the trap. If you only persist the cursor when you got records, an empty poll leaves the old cursor in place. Next cycle you ask from the same position, get the same empty result and the same newer cursor, discard it again, and repeat. You have built an infinite loop out of doing nothing.

The window never closes because the thing that closes it, the cursor, only gets saved on the branch that never runs.

I fixed it by making the cursor advance even when there is no data. A synthetic “cursor-only” record goes through the same pipeline, carrying no payload, existing purely to persist the newer cursor:

records present -> process records, persist cursor
records empty   -> push a cursor-only record, persist cursor anyway

Now an empty poll still moves the position forward, the empty window closes, and the next cycle starts from somewhere new. The pipeline has one job, advancing state, and an empty result is still a state change worth recording.

While you are here, renew the webhooks#

One more expiry to plan for. Push and webhook registrations do not live forever either. They come with a lifetime, and if you let one lapse the provider stops notifying you and your sync goes silent without a single error.

Renew inside a buffer, before expiry, not on it. Treat the registration as a lease you keep extending, because a webhook that has quietly expired looks exactly like a system with nothing to report.

The lesson#

Everything about a cursor tempts you to treat it as permanent, and it is not. Cursors age out, so give expiry a typed sentinel that forces a clean full resync. Cursors advance on empty results, so persist them even when there is no data, or “no new data” becomes an infinite reprocessing loop. And the registrations that feed the whole thing expire too, so renew them early. The failures here are all quiet, and quiet is the part that costs you.