Designing Idempotent Writebacks: Lessons from Status Acknowledgment
A writeback creates a patient in the PMS. The network drops before you record that it succeeded. Do you retry and risk a duplicate, or skip and lose the record? Neither. Here is how to make writebacks safe to retry.
A writeback is a command, not a read
Reading from a PMS is easy to make reliable: if the call fails, you call again, and the worst case is you burned a little bandwidth. A writeback is different. It is a command that mutates the source system — create a patient, book an appointment, attach a note, post a procedure. The side effect is real and it happens on someone else’s database. You cannot retry it blindly, because the second attempt might create a second patient.
That single asymmetry — reads are safe to repeat, writes are not — is the whole design problem. Everything below is about closing the gap between “I sent a command” and “I know for certain what that command did,” across a network that will drop the connection at the worst possible moment.
This is the writeback counterpart to our post on idempotency in webhook receivers. There, CRMBridge is the sender and your receiver must dedupe our retries. Here the roles flip: you send the command, and the on-prem worker that touches the PMS must be the idempotent one.
The two failure modes to avoid
Every naive writeback implementation lands in one of two ditches. Both are common. Both are avoidable.
Duplicate — you retried and created two patients.
The PMS write succeeded, but the acknowledgment got lost on the way back. Your code assumes failure, retries, and now there are two Jane Smiths in the chart. Someone has to notice, merge them, and hope no appointment got booked against the wrong one in the meantime.
Lost — you gave up and the record never landed.
The connection dropped, your code decided retrying was too risky, and the patient was never created at all. The front desk swears they entered it. The chart says otherwise. This one is worse, because nothing failed loudly — the record just quietly does not exist.
The instinct is to pick your poison: retry aggressively and eat the duplicates, or retry timidly and eat the losses. The right answer is to refuse the trade entirely. You make the operation safe to retry, and then you retry freely.
The queue-and-status model
CRMBridge does not expose a synchronous “create this patient and block until it’s done” call, precisely because that call is impossible to make reliable across a flaky on-prem link. Instead a writeback is enqueued and immediately returns a WritebackID. The command is now durable on our side; the network can drop and nothing is lost.
You then poll GetWritebackStatus with that ID. It returns a Status of "Completed", "InProgress", or "Failed", along with a human-readable Message. For create operations, a completed response also carries a CreatedEntityID — the new PMS-side PatientId or AppointmentId that the source system assigned.
The WritebackID is the idempotency handle.
Poll it — do not re-fire the command. If your status check times out, you call GetWritebackStatus again with the same ID; you never submit a second create. The ID is the one stable thing that ties “the command I sent” to “the outcome that happened,” and it survives every retry on either side.
This is the key move. Submitting a writeback and checking its status are two different operations, and only the first one mutates anything. Re-checking status is a read, and reads are always safe.
The pipeline is at-least-once, so the worker must be idempotent
Behind that clean API is a queue, an on-prem worker, and the PMS itself — three hops, each of which can fail and be retried. The whole pipeline is at-least-once: the same enqueued command can reach the worker more than once, because the alternative (dropping it on the first hiccup) loses writes. Given that, the worker at the PMS boundary carries the burden of making a repeated command harmless. Here is how it earns that.
1. Dedup on a stable key.
Before executing any PMS side effect, the worker asks: “Have I already processed this WritebackID?” If a record of that ID already exists with a terminal outcome, it skips the write and returns the stored result. The WritebackID is the dedupe key, and it is decided before any mutation happens.
2. Acknowledge before you finalize.
The worker only marks its local record done after the cloud confirms it recorded the status. If the worker crashes mid-flight — PMS write done, cloud not yet told — the command is left in a state that gets retried, not silently dropped. Ordering the acknowledgment before the finalize is what turns “lost” into “retried.”
3. Recover orphans on a later pass.
If the worker dies between “PMS write succeeded” and “status saved,” a later reconciliation pass detects the orphan — a claimed command with no recorded outcome — and resolves it. It reports the writeback as failed and lets the next cycle retry, rather than silently finalizing it, which could duplicate the write.
4. Re-notify on lost acknowledgments.
If the status update to the cloud itself failed — say the acknowledgment call returned a 500 — the worker keeps the stored result and re-sends it on the next cycle. Otherwise the cloud keeps retrying a writeback that has already succeeded in the PMS. Re-notifying is how the sender learns it can stop.
5. Force-clean stuck orphans after a max age.
A dependency that is permanently unreachable — a PMS box that got decommissioned mid-flight — cannot be allowed to wedge the queue forever. After a maximum age, the worker force-cleans stranded orphans and, critically, logs exactly what it dropped so the record is auditable rather than vanished.
The honest caveat: there is no exactly-once
It is worth saying plainly, because a lot of writeback code is built on a fantasy: perfect exactly-once delivery across a process boundary is impossible. The instant a command crosses from your process into another — a queue, a worker, a PMS — there is a window where the sender cannot distinguish “it succeeded and the ack was lost” from “it never ran.” No protocol closes that window; it can only be pushed around.
What you can build is effectively-once: the observable outcome is as if the command ran exactly once, achieved by combining three things — an idempotency key that makes repeats detectable, acknowledgment ordering that makes crashes retriable instead of lossy, and reconciliation that cleans up the in-between states. None of those three alone is sufficient. Together they get you a system where retries are boring.
The worker loop, in pseudocode
Put the pieces in order and the worker’s main loop looks like this. The ordering is the design — dedupe gates the mutation, the acknowledgment precedes the finalize, and the orphan branch handles the crash-in-the-middle case.
loop:
cmd = queue.claim() // lease a writeback command
if cmd == null: continue
// 1. Dedup gate: never mutate twice for the same WritebackID
if alreadyProcessed(cmd.WritebackID):
result = loadStoredResult(cmd.WritebackID)
acknowledgeCloud(cmd.WritebackID, result) // re-notify if needed
queue.finalize(cmd)
continue
// 2. Execute the real PMS side effect (create / book / note / post)
mark(cmd.WritebackID, status = "InProgress")
try:
entityId = pms.execute(cmd) // the one irreversible step
catch (err):
saveStatus(cmd.WritebackID, "Failed", message = err)
acknowledgeCloud(cmd.WritebackID, ...) // ack BEFORE finalize
queue.finalize(cmd)
continue
// 3. Persist the outcome locally, THEN tell the cloud, THEN finalize
saveStatus(cmd.WritebackID, "Completed", CreatedEntityID = entityId)
acknowledgeCloud(cmd.WritebackID, "Completed", entityId) // must precede finalize
queue.finalize(cmd) // last: safe to lose the lease now
// --- separate reconciliation pass, runs on a timer ---
reconcile:
for orphan in findClaimedButUnresolved(): // crashed between execute and save
if age(orphan) > MAX_ORPHAN_AGE:
forceClean(orphan) // log what was dropped
else:
saveStatus(orphan.WritebackID, "Failed", message = "orphan; retrying")
// next queue cycle re-claims and the dedup gate keeps it safe
Notice what is not in the loop: nowhere does the worker re-submit a command. Retries are driven by the queue re-leasing an unfinalized command, and the dedup gate at the top guarantees the second run through does not touch the PMS again. The only irreversible line is pms.execute(cmd), and everything else is arranged to make sure it happens once.
Idempotency is the contract, not a feature
It is tempting to treat all of this as a hardening pass — something you bolt on after the happy path works. It is the opposite. Idempotency is the contract that lets everything upstream retry freely. The queue can re-deliver, the cloud can re-poll, your integration can re-submit after a timeout, and none of it produces a duplicate patient or a lost record, because the writeback was designed to be safe to repeat from the start.
Get the WritebackID handle, the acknowledgment ordering, and the reconciliation pass right, and every retry in the system above you becomes a non-event. That is the whole point: retries should be boring.
Retries should be boring.
CRMBridge gives you safe, idempotent writebacks to every connected dental and veterinary PMS — enqueue a command, get a WritebackID, poll for status, and let the pipeline absorb every retry without ever creating a duplicate or losing a record.