<Mr. Dev/>
All notes

03 # EXPERIMENTING

Replication · High Availability · Failover

TECHNICAL NOTE

Set recovery objectives before adding replicas.

Treat the primary database as a system to recover, not an appliance to trust blindly. Replication helps only after recovery time, acceptable data loss, monitoring, and ownership are clear.

COMMON USES

Read pressure

Read replicas only after the primary has a measured bottleneck and the application can tolerate replica lag.

Critical writes

A single writable authority with durable backups, tested restores, and carefully designed transactions.

Service continuity

Automated failover where the team can monitor it, fence the old primary, and rehearse the recovery path.

QUESTIONS

Choices depend on context

01 # Replication mode

Asynchronous or synchronous replication?

Asynchronous

The primary confirms a write before every replica has it.

  • Lower write latency.
  • Useful for geographic distance and read scaling.
  • A sudden primary loss can lose recent acknowledged writes.

Synchronous

The primary waits for a replica acknowledgement before confirming.

  • Reduces acknowledged-data loss.
  • Adds latency and can reduce availability.
  • Requires clear quorum and failure behavior.

ConclusionBegin with asynchronous replicas for read scale and disaster recovery. Accept the latency cost of synchronous acknowledgement only when a transaction has a strict loss budget.

02 # Failover ownership

Manual or automated failover?

Manual runbook

An operator confirms the incident and promotes a healthy replica.

  • Simpler early on.
  • Human judgment can prevent a bad promotion.
  • Recovery time depends on availability and practice.

Automated orchestration

Health checks elect and promote a new primary.

  • Faster recovery for well-understood failures.
  • Needs robust fencing to prevent split brain.
  • Adds operational dependencies that must also be tested.

ConclusionAutomate only the failure modes that are observable and rehearsed. Until then, a clear runbook and restore drills are safer than automation nobody trusts.

TAKEAWAY

High availability is a promise about failure. The proof is a practiced recovery, not the number of replicas on a dashboard.

POINTS TO KEEP IN MIND

  • Test restores regularly; a backup that has never been restored is only a hope.
  • Monitor replication lag, storage, connection saturation, error rates, and backup freshness.
  • Fence the old primary before promotion to avoid two writers.
  • Document RPO and RTO in product terms, not only infrastructure terms.