Data synchronization requires agreement about which state is authoritative, how changes are ordered, and how a destination recovers when its copy diverges. Moving records between applications is only the transport step. A connection can deliver every message successfully and still produce the wrong final state if duplicates, ordering, deletions, or concurrent edits are handled incorrectly.
For an integration architect connecting a service platform to a reporting and customer portal layer, the key decision is what consistency the consumers need and which synchronization protocol can provide it. A read-only projection of an authoritative source is simpler than allowing several systems to edit the same fields. Both can be valid designs, but they require different rules.
Start by stating the desired invariant: for example, the portal should eventually reflect the latest accepted service status for every included work order, with a visible freshness limit and no resurrection of deleted records. That statement is more useful than “keep the systems in sync” because it identifies what must be tested.
Decide what a change actually means
A change message can contain a complete state snapshot, a partial field update, or an instruction to apply a difference. These are not interchangeable. A snapshot says what the state is at a specified version. A delta says how to modify a prior state. A command asks the receiver to perform an action under its own rules.
Specify which form the contract uses. Include a stable business identifier, the relevant source, a source version or ordering reference where required, and the scope of the fields represented. If an omitted field means “unchanged” in one message and “empty” in another, the destination needs to distinguish those cases explicitly.
Separate event identity from entity version. An event identifier supports recognizing the same message again; a version helps determine its relationship to other changes for the same entity. CloudEvents 1.0.2 defines uniqueness through the combination of source and event ID. It does not provide a universal business-version or conflict-resolution algorithm. CloudEvents Specification 1.0.2, Required Attributes
The distinction matters when one business update creates several events, or when a corrected event describes the same entity. A destination cannot safely infer business order merely from a globally unique message identifier.
A hypothetical out-of-order counter
Consider a hypothetical synchronization of an operational counter. At version eight, the authoritative value is twenty. Version nine subtracts five, giving fifteen. Version ten adds two, giving seventeen. The destination begins at version eight but receives version ten before version nine.
If the messages are complete snapshots, version ten explicitly carries the value seventeen. A destination using a properly defined, monotonic version for that one authoritative entity can accept seventeen and ignore a later-arriving version-nine snapshot of fifteen. That approach relies on the version and snapshot contract being valid; it is not a rule for unrelated entities or arbitrary timestamps.
If the messages are deltas, version ten contains only “add two.” Applying it to twenty produces twenty-two. If the receiver then discards version nine because it is older than the highest version seen, it never subtracts five. The final value is wrong even though both messages arrived.
For deltas, the consumer needs an appropriate strategy: enforce or restore the required sequence, track gaps and wait, obtain a trustworthy current snapshot, or use a mathematically valid merge design suited to the operation. Addition can be commutative in a deliberately designed counter, but general business state changes are not. A method appropriate for counts does not automatically apply to cancellations, ownership changes, or permissions.
The example shows why “ignore older versions” must be tied to message semantics. A reliable transport cannot repair a consumer algorithm that interprets a delta as if it were a complete state.
Plan for duplicate delivery
A producer or consumer can fail after work occurs but before acknowledgment is recorded. Retrying may therefore deliver a change again. The destination should have a documented duplicate-handling strategy that matches its effect.
Replacing a projection with the same full state may be naturally repeatable if the version checks are correct. Incrementing a counter or issuing a notification is different: applying the message twice changes the result or repeats an external action. Use a stable event identity and a durable record of applied effects where the design requires deduplication.
The deduplication record and the state update need a consistent commit strategy. If the receiver records “processed” before changing state and then crashes, the update can be lost. If it changes state and crashes before recording the event, a retry can repeat the effect. Where both records share a transactional store, committing them together can address that local gap. Effects in other systems require additional coordination or recovery.
PostgreSQL’s version-15 logical-decoding documentation explicitly notes that a crash can cause recent changes to be sent again and assigns clients responsibility for avoiding harmful repeated processing. This is a product-specific example of why a change feed should not be assumed duplicate-free. PostgreSQL 15, Logical Decoding Concepts, Replication Slots
Define how long duplicate identities are retained. A retention period shorter than the supported replay window can allow an old event to be applied again. The choice depends on the effect, storage, source behavior, and recovery requirements; there is no universal duration.
Join the initial copy to the continuing changes
A new destination usually needs an initial population before it can consume ongoing changes. Copying all records and then starting a feed can lose changes made between those steps. Starting the feed first and applying it carelessly during the copy can overwrite newer data with older snapshot values.
Use a source-supported consistency mechanism or a designed reconciliation protocol to connect the snapshot and change stream. The integration should know which change position corresponds to the snapshot and how to process later changes without gaps or harmful duplication.
PostgreSQL 15 describes exported snapshots associated with logical replication slot creation so an initial database state can be related to the following change stream. This is a specific mechanism, not a capability to assume in every application API. PostgreSQL 15, Logical Decoding Concepts, Exported Snapshots
Where the provider lacks such a mechanism, document the limitations and design a compensating reconciliation. A timestamp filter may miss changes with equal timestamps, delayed updates, or coarse precision. An overlapping extraction window can reduce some gaps but requires duplicate-safe processing and still needs testing against the source’s actual behavior.
Treat deletion and correction as first-class changes
If a record disappears from a source extract, the destination needs to know whether it was deleted, excluded by a changed filter, inaccessible due to permission, or simply missing from an incomplete run. Blindly deleting everything absent from one file can turn an extraction failure into data loss.
Use explicit deletion events or a verified complete-snapshot comparison where appropriate. Preserve enough deletion information to prevent an older update from recreating the record. Such a marker is often called a tombstone, but its behavior and retention must be defined for the particular system.
A source correction may affect historical as well as current views. Decide whether the destination maintains only current state, a history of changes, or business-effective history. A corrected service date and a new service event are different facts. Keeping only the latest value may be sufficient for a portal and insufficient for a reconciliation or audit use case.
Be deliberate about what is replicated. The destination should receive only the fields required for its purpose and permitted access. Synchronizing a full source record merely because a connector supports it can create unnecessary copies of sensitive information and complicate deletion obligations.
Avoid accidental multi-writer conflict
When both applications can change the same fact, define the authority and conflict rule before turning on bidirectional synchronization. “Last write wins” requires a meaningful definition of last and can silently discard a valid update. Wall-clock timestamps from different systems may not represent a reliable business order.
Prefer a single authoritative writer for a field or a clear partition of responsibility when the business permits it. If concurrent edits are necessary, the contract must define conflict detection, merge rules, and escalation. Some fields can be merged mechanically; others require a business decision.
Do not confuse a technically resolved conflict with an acceptable business outcome. A portal address update overwriting a verified dispatch address may satisfy a timestamp rule while violating an operational requirement. Synchronization must respect the lifecycle and authority of the fact being changed.
Verify convergence rather than only delivery
Monitor whether the destination reaches the intended state within the agreed conditions. Compare representative records and complete populations where feasible, using versions, counts, identifiers, and relevant totals. A successful message count does not establish correct state.
Test a duplicate, an out-of-order update, a missing version, a source correction, a deletion followed by an old update, and a restart during initial loading. Record the expected outcome and recovery owner for each. If the design cannot explain those cases, it is not ready to promise synchronization.
Begin with one authoritative entity and one consumer. Specify the message semantics, version rules, snapshot boundary, deletion behavior, and reconciliation method. Expand only after the destination can demonstrate convergence under realistic failures. The difficult part of synchronization is keeping meaning and state coherent when execution stops being orderly.
