Choose an integration cadence from the latest time at which information remains useful for a specific action. “Real time” is not a sufficient requirement, and batch is not automatically too slow. The architecture should meet an explicit freshness objective with acceptable correctness, recovery, and operating cost.
For an operations-platform owner integrating shipment status into customer service, the decision is whether scheduled transfers meet the service team’s response needs or whether event-driven updates justify their additional obligations. A different interaction in the same process, such as accepting a delivery-address change before dispatch, may need an immediate authoritative response. One label should not govern every flow.
Separate three questions: how quickly the source records the fact, how quickly the fact reaches the destination, and when the destination can safely act on it. Optimizing transport while ignoring the other two can produce a fast pipeline carrying late or unusable information.
Translate urgency into a measurable freshness objective
State the business event, the destination action, the required freshness, and the consequence of exceeding it. For example, a service team might need most shipment-state changes visible within ten minutes so it can answer routine calls without contacting the warehouse. That is a hypothetical operating requirement, not a recommended target for every business.
Define the measurement boundaries. Does elapsed time start when the package physically leaves, when an operator scans it, or when the source commits the scan? Does it end when a message is received or when the service screen displays the usable status? These are different measures and can lead to different designs.
Use a distribution and an exception policy rather than only an average. A low mean delay can coexist with a small but consequential population of very stale records. Identify what happens when the freshness target is missed: label the status stale, use an authorized alternative source, notify an owner, or restrict a dependent action.
Account for completeness as well as freshness. A destination showing the latest message from one carrier can look current while another carrier’s feed has stopped. Measure the expected population and source coverage so a healthy connection is not mistaken for a complete business view.
Distinguish cadence from interaction style
Batch processing groups work for a scheduled or bounded run. Streaming processes continuing arrivals, often in small units or micro-batches. Synchronous request-response makes the caller wait for a result. Asynchronous messaging separates submission from later processing. These properties can be combined rather than treated as mutually exclusive product categories.
Polling also does not necessarily mean a slow nightly batch. The Polling Consumer pattern describes a consumer explicitly requesting a message when ready. Its timing depends on the implementation and usage. An event-based architecture may itself contain polling consumers. Enterprise Integration Patterns, Polling Consumer
A scheduled five-minute extract can satisfy a ten-minute reporting need. A low-latency event notification may still require a slower authoritative fetch before a decision. A synchronous command can confirm acceptance immediately while later completion is reported asynchronously. Discuss the actual behavior instead of comparing slogans.
For actions requiring an immediate authoritative decision, ask whether a delayed copy is safe. Shipment status for an informational screen and permission to change a delivery address are different. A recent status copy may not establish that dispatch has not already occurred. The change request should go to the system that owns that decision under the agreed rules.
A hypothetical freshness calculation
Assume shipment-state changes arrive uniformly across a fifteen-minute extraction interval. Assume each batch then takes exactly two minutes to transfer and become visible, with no queue, failures, or source delay. These deliberately simplified assumptions make the scheduling effect clear.
Waiting for the next run ranges from almost zero to fifteen minutes. Average wait is seven and a half minutes, so average end-to-end delay under the model is nine and a half minutes. The ninety-fifth-percentile wait is fourteen and a quarter minutes; adding the fixed processing time gives sixteen and a quarter minutes. The limiting worst-case delay approaches seventeen minutes.
A requirement that ninety-five percent of changes appear within ten minutes would not be met by this idealized fifteen-minute schedule. An average below ten minutes would give the wrong answer to that requirement.
Now consider a five-minute interval with the same fixed two-minute processing assumption. Average delay becomes four and a half minutes, and the ninety-fifth-percentile delay becomes six and three-quarter minutes. The limiting worst case approaches seven minutes. Under this simplified model, scheduled integration could meet the stated requirement without per-event delivery.
The conclusion is conditional. More frequent extracts may increase source load; processing may not remain fixed; bursts can create queues; source events may be recorded late; failures can miss runs. The pilot must measure those effects and leave reasonable operating margin. The calculation identifies a candidate design, not a performance guarantee.
Evaluate the cost of making information faster
Faster updates are valuable when they change a decision or reduce a meaningful period of uncertainty. Ask the business owner to identify the action enabled by moving from hours to minutes or from minutes to seconds. If the organization cannot respond sooner, the added integration complexity may produce little practical benefit.
Event-driven updates introduce obligations around durable capture, duplicate handling, ordering, consumer lag, replay, and event-contract evolution. Batch introduces obligations around complete extracts, restartable processing, cutoff consistency, and the duration of a failed or missed run. Compare the full operating models rather than treating one as sophisticated and the other as simple.
A batch can be easier to reconcile because it has an explicit population and cutoff, but only if the extract is coherent and completeness is checked. A stream can reduce waiting, but only if events are durably captured and consumers keep up. Neither approach inherently guarantees correct or complete data.
Consider the source’s supported access mechanisms. An event feed may omit relevant changes or provide only notifications. Frequent polling may exceed a provider’s limits or increase load on a transactional application. Use the actual contract and observed behavior rather than assuming a connector’s name settles the issue.
Decide how much old information is still useful
Different messages age differently. A historical shipment event can remain useful for an audit trail after it is too old to trigger a timely customer notification. A request to change an address may become invalid once dispatch occurs. Distinguish retaining a fact from executing a stale instruction.
The Message Expiration pattern addresses messages whose usefulness is time-limited. Applying an expiry rule requires a business decision about what should happen after that period, including whether the item is discarded, recorded for investigation, or replaced by a current-state query. Enterprise Integration Patterns, Message Expiration
Do not silently drop an expired business obligation. If a time-sensitive instruction can no longer be executed, the process may need an exception and an authorized alternative. The transport’s expiration mechanism does not determine the correct customer or operational response.
Define freshness at the destination too. Cached views should expose the relevant effective time and source status. A page refresh does not make its underlying data current. If the display combines sources with different cadences, show the differences where they affect interpretation.
Test degraded operation before choosing the cadence
Run the candidate design with representative bursts, a missed extraction, a slow destination, a source correction, and a restart. Observe the age of the oldest relevant unprocessed item as well as throughput. A system processing many new messages can still leave one important population stuck.
Measure catch-up capacity. After an outage, the consumer must handle both new arrivals and accumulated work. If its sustainable rate merely equals the normal arrival rate, the backlog will not shrink. The required margin depends on the intended recovery time and source limits; it should be tested rather than assumed.
For scheduled processing, specify whether a failed run restarts from a checkpoint, reprocesses the whole population safely, or requires manual intervention. For a stream, specify how consumer position and applied state are reconciled during replay. In both cases, verify the business result after recovery.
Include operating hours and human response. A pipeline with a demanding freshness target needs an appropriate alerting and support arrangement. A target that cannot be supported during the hours it matters is an incomplete business commitment.
Choose a mixed design when the decisions differ
The shipment example may justify immediate commands for consequential changes, frequent scheduled updates for service visibility, and a daily reconciled snapshot for reporting. These can coexist if their meanings and relationships are clear. The daily snapshot can help detect divergence without being mistaken for the real-time authority.
Document the chosen cadence, the evidence supporting it, the failure policy, and the trigger for reassessment. A new customer promise, increased volume, or changed source capability may justify a different design later. Preserve the ability to change cadence without redefining the business meaning of the data.
Before approving “real-time integration,” ask for one measured freshness objective and a comparison of at least two feasible designs against it. Include degraded operation and recovery. The right architecture is the one that delivers usable information in time for the authorized action, with obligations the organization can sustain.
