#Cross-Datacenter Replication

#Overview

The core functions of the system cover the entire replication link, including data interception, command parsing, operation log (Oplog) persistence, full and incremental data synchronization, and log slicing. The workflow begins with intercepting all write operations at the command execution entry point. Subsequently, the system deeply parses the commands and converts them into an idempotent, structured Oplog format. These logs are efficiently persisted to the local disk, forming a complete and reliable history of data changes, which provides a solid foundation for subsequent data recovery and failover.

In terms of data synchronization, the system has designed a synchronization mechanism that is both efficient and flexible. It supports full synchronization, which quickly builds a complete data copy on the downstream side by generating a Redis Database (RDB) snapshot and combining it with the starting position of the Oplog. More importantly, we have innovatively introduced an Offset mechanism that is tightly integrated with the Oplog, building an incremental synchronization capability with an ultra-large window period. Unlike Redis's native replication, which relies on a limited memory backlog, our Oplog is persisted on disk, which means that the catch-up range of incremental synchronization is limited only by disk space. Therefore, even if a network interruption of several hours or even days occurs between the upstream and downstream instances, once the connection is restored, the downstream side can still request incremental data from the last recorded Offset point, achieving efficient data catch-up and consistency recovery, which greatly improves the reliability and robustness of the architecture.

In addition, in order to ensure the long-term stable operation of the system and controllable resources, we have designed a refined log slicing function. This function will automatically scroll and segment the local Oplog file according to the preset size policy. This mechanism not only effectively prevents a single log file from growing indefinitely and exhausting storage resources, but also greatly facilitates the archiving, cleaning, and management of logs, which is a key part of ensuring the continuous and stable operation of the service in a production environment.

Finally, the system provides comprehensive and powerful observability. By comparing the latest write position of the upstream Oplog with the synchronized Offset of the downstream in real time, the system can accurately calculate the data synchronization delay between the upstream and downstream instances with extremely low overhead. These key performance indicators (KPIs) can be seamlessly connected to mainstream monitoring panels (such as Prometheus/Grafana) and alarm systems. This allows the operation and maintenance team to grasp the system health status, synchronization link status, and data differences in real time and intuitively, so as to intervene in a timely manner before potential risks evolve into actual failures, which effectively guarantees business continuity and data security.

#Core Advantages

• High reliability: Based on the persistent Oplog and the large-window incremental synchronization mechanism, it can calmly deal with long-term network partitions and delays, ensuring accurate recording and reliable recovery of data changes.
• High performance: As a lightweight Redis Module, the core logic is highly optimized and executed directly in the Redis I/O thread, minimizing the impact on the core performance of Redis.
• High stability: The intelligent log slicing mechanism automatically manages storage resources, effectively preventing disk space exhaustion caused by log accumulation, and ensuring the continuous and stable operation of the service.
• Strong observability: It provides delay calculation accurate to the operation level and rich monitoring indicators, making the system status completely transparent, which is convenient for integration and automated operation and maintenance, and realizes active risk prevention and control.

#Replication modes

Starting with Alauda Cache Service E1 v5.1.0, cross-datacenter replication offers two modes, selected per instance through spec.activeRedis.mode on the Redis resource. Both modes run the replication engine described above — the same Oplog, the same full and incremental synchronization, the same proxy layer. What differs is how members find each other, and which of them accept application writes.

Each mode has its own pages: Disaster Recovery and Active-Active. Command Support and Limitations and Risks apply to both.

#Choosing a mode

Choose Disaster Recovery when one datacenter owns the workload and the others exist to take it over if that datacenter is lost. It is the default, it is available on both module generations, and its consistency story is the simple one: a single writer, so no write has to be reconciled against a concurrent write somewhere else. The cost is that recovery is a deliberate manual operation rather than an automatic one — see Failover.

Choose Active-Active when applications in more than one datacenter have to write to their local instance — for local write latency, or to keep serving through the loss of any single datacenter without a promotion step. The cost is that every member is a writer, so concurrent writes to the same key are reconciled by rules that differ from standalone Redis, and those rules are decided by wall-clock time. Two obligations follow from that, and neither is optional:

• Read Command Support before pointing an application at an Active-Active group. It states, command by command, which writes are refused, which stay local, and how each replicated write resolves a conflict.
• Keep the system clocks of every member synchronized. The module measures and alarms on clock skew, but it never blocks a write because of it — see Clock synchronization.

Active-Active is alpha and available on Redis 7.2 only. An instance that is already replicating in Disaster Recovery mode should not be converted in place; build an Active-Active group from instances that are not yet wired into a Disaster Recovery group.

#Module generations and version applicability

The replication module ships in two generations. Which one an instance loads is decided by its Redis version, and the generation determines the feature set, the authentication model, and the peer transport.

#Architecture Description

The components described in this section — the proxy layer, its two ports, and the Service ID — are common to both modes.

To simplify the complexity of accessing the replication system for different Redis architectures, we have designed a dedicated proxy layer. As the core component of the system, the proxy layer provides a unified status detection and data synchronization entry point for Sentinel and cluster modes. Regardless of the deployment form of the back-end Redis, peers interact through this unified proxy interface. The proxy layer is responsible for encapsulating the internal logic of communication with the specific architecture (Sentinel, cluster), thereby decoupling the replication system from the implementation details of the back-end Redis, which greatly simplifies the access and management of instances.

A proxy is deployed for both module generations. Enabling cross-datacenter replication creates a proxy Deployment and a proxy Service named activeredis-proxy-<instance-name>.

For Sentinel instances, the instance name is prefixed with rfr-.

#Ports

#Instance ID: Service ID

To uniquely identify and manage each Redis instance in the replication system, we have introduced the concept of service_id. This ID is the core credential for routing and data synchronization in the entire system.

• Unique identifier: Each Redis instance, whether it is an upstream or a downstream, must be configured with a globally unique service_id. The system uses this ID to establish a mapping between members.
• Value range: The valid range of service_id is [0-15].
• Topology limitation: Due to the ID range, a single upstream can be associated with up to 15 downstream instances, supporting a "one-upstream-multiple-downstream" topology.

#Terminology

The replication direction is described as upstream (the instance that produces changes) and downstream (the instance that consumes them). These are the terms used by the ActiveRedis resource status fields upstreamPeer, downstreamPeers, and downstreamPeerCount.

The Web Console presents the same concepts as Source side and Target side when you configure a Disaster Recovery instance. A source is an upstream; a target is a downstream. In Disaster Recovery mode the role of an instance is not fixed — a target may itself be promoted to a source during a failover.

The pair names a direction of flow, not a permanent role. In Active-Active mode every member is at once an upstream for the changes it originates and a downstream for the changes it receives from each of its peers.

#Command Support

Not every Redis command can cross a datacenter boundary. Some are refused with an error, some execute locally and never replicate, and the replicated write commands resolve concurrent conflicts by rules that differ from standalone Redis. Command Support is the complete reference for both modes and both module generations. Read it before pointing an application at a replication group.

#Limitations and Risks

For the risks and limitations of cross-datacenter replication, please refer to the document Limitations and Risks.