How to fix: Race condition between events in an EDA system

🚨 Understanding the Error

In an Event-Driven Architecture (EDA), a race condition occurs when the system’s correctness depends on the timing or sequence of events that are processed asynchronously. Because distributed systems often prioritize horizontal scaling and high availability, events related to the same entity (e.g., an Order or a User) may be processed by different consumer instances or threads.

If an OrderUpdated event is processed before an OrderCreated event—or if two concurrent updates attempt to modify the same state simultaneously—the system ends up in an inconsistent or “stale” state. This is a violation of Linearizability within the aggregate.

🔍 Root Cause

Cause	Description
Network Jitter	Latency variations cause Event B (sent later) to arrive at the consumer before Event A (sent earlier).
Partition Rebalancing	In systems like Kafka, a consumer group rebalance can cause events to be re-read or shifted across workers.
Concurrent Execution	Multiple consumer threads fetching from the same queue without a shared lock on the resource ID.
Lack of Sequencing	The producer does not attach a version number or timestamp, making it impossible to determine the “source of truth.”
Retry Storms	A failed early event retries and eventually executes after a newer event has already successfully updated the state.

🛠️ Step-by-Step Solutions

1. Implement Optimistic Locking (Versioning)

The most robust way to handle out-of-order events is to include a version or sequence_number in every event. The consumer should only update the database if new_version == current_version + 1.

# Using SQLAlchemy for Optimistic Locking
from sqlalchemy.orm import Session
from models import Order

def process_order_update(db: Session, event_data: dict):
    order_id = event_data['id']
    incoming_version = event_data['version']

    # Fetch the current aggregate root
    order = db.query(Order).filter(Order.id == order_id).first()

    if not order:
        # If OrderCreated hasn't arrived, we may need to retry or DLQ
        raise DependencyNotMetError(f"Order {order_id} not found.")

    # Logic: Ignore stale versions
    if incoming_version <= order.version:
        print(f"Discarding stale event: {incoming_version}")
        return

    # Update logic
    order.status = event_data['status']
    order.version = incoming_version # Incrementing the state version
    db.commit()

2. Distributed Locking with Redis

If you must ensure that only one worker processes a specific aggregate ID at any given time, use a distributed lock (Redlock).

import redis
from potentials import LockTimeout

r = redis.Redis(host='localhost', port=6379, db=0)

def handle_event(event):
    aggregate_id = event['order_id']
    # Acquire a lock for 5 seconds specific to this Order ID
    lock = r.lock(f"lock:order:{aggregate_id}", timeout=5)

    if lock.acquire(blocking=True, timeout=2):
        try:
            # Process event safely
            update_order_status(aggregate_id, event['payload'])
        finally:
            lock.release()
    else:
        # Failed to acquire lock, re-queue the event
        raise Exception("Could not acquire lock, concurrent process active.")

3. State Machine Validation

Prevent illegal transitions (e.g., moving from SHIPPED back to PENDING) regardless of event order.

class OrderStatus:
    PENDING = "PENDING"
    PAID = "PAID"
    SHIPPED = "SHIPPED"

# Mapping allowed transitions
ALLOWED_TRANSITIONS = {
    OrderStatus.PENDING: [OrderStatus.PAID],
    OrderStatus.PAID: [OrderStatus.SHIPPED],
    OrderStatus.SHIPPED: []
}

def update_status(current_status, new_status):
    if new_status not in ALLOWED_TRANSITIONS.get(current_status, []):
        print(f"Invalid state transition: {current_status} -> {new_status}")
        return False # Drop the event or handle as out-of-order
    return True

🛡️ Prevention and Best Practices

1. Event Keying (Partitioning): Ensure all events for a specific Entity ID are sent to the same partition. In Kafka, use the order_id as the message key. This ensures the same consumer thread processes them sequentially.
2. Idempotency Keys: Every event should have a unique UUID. Store processed IDs in a Bloom Filter or a Redis SET to ensure you never process the same event twice.
3. Deterministic Retries: If an event arrives out of order (e.g., Update before Create), move it to a “Retry” topic with an exponential backoff rather than failing immediately.
4. OpenTelemetry Tracing: Use distributed tracing to visualize the flow of events. If a race occurs, you can trace the exact timestamps of when each service produced/consumed the message.
5. Side Effects First: Always update your local database/state before emitting downstream events to ensure that if the DB commit fails, the downstream system isn’t triggered with “ghost” data.