AWS Re:cap 06: Amazon Aurora HA and DR Design Patterns for Global Resilience (DAT442)

Video: https://www.youtube.com/watch?v=TNURufOhu_A

Aurora's Approach to Performance

Adding replicas for performance purposes

● Offloads read traffic to replicas, reducing impact on the primary writer.

● Allows separation of noisy neighbors by dedicating replicas to specific workloads (e.g., analytics).

● Up to 15 replicas can be added for fine-grained performance management.

● Storage Scaling:

● Storage capacity and performance scale automatically in Aurora, requiring no manual provisioning.

● Instance Scaling:

● Instances can be added in various sizes (e.g., 16XL, 8XL, 4XL) for failover and performance management.

● DB instances are elastic, scaling from 0 to 256 Aurora capacity units.

● Failover tiers are defined to manage instance failover order based on size and capacity.

● Elastic instances help manage spikes in workload without long-term cost implications.

Challenge: Connection Management in Aurora

● Managing connections to multiple instances, especially identifying the writer for transactions.

● Techniques:

● Pooling and limiting connections per DB instance, avoiding reconnections.

● Driver Frameworks:

● Standard practices like Spring and JDBC may not handle clustered databases effectively.

Aurora Endpoints

Writer Endpoint

● Always points to the current writer node, facilitating consistent transaction routing.

● Reader Endpoint:

● Points to a pool of instances for read operations using DNS round robin for coarse load balancing.

● Custom Endpoints:

● Option to create custom endpoints for specific use cases (e.g., analytics).

AWS Advanced Wrapper Drivers

Rapid Drivers

● Enhanced drivers with intimate knowledge of cluster state, facilitating quicker failover.

● RDS Proxy:

● Additional tool for managing connections and reducing failover time.

● Plugins:

● Enhanced failover, rewrite splitting, blue-green deployments.

Multi-Region Resilience

● Challenge: Ensuring business continuity if a region fails or is cut off from the network.

● Solution: Maintaining a copy in another region, keeping it up-to-date with minimal cost, and utilizing it for various purposes.

● Multi-Region Resilience with Aurora Global Database:

● Aurora Global Database allows up to 10 secondary regions for AMS and APG.

● The internal replication server handles asynchronous physical replication between regions.

● Head nodes are not involved, avoiding performance associated with logical replication.

● Configuration and Benefits:

● Region A: Primary region with replicas and storage.

● Region B: Secondary region with storage initially, can add read-only instances for local reads.

● Asymmetric configurations are possible; instances in secondary regions do not need to match primary.

● RPO lag is typically about 1 second, depending on region pair distance.

● Global Endpoint and Failover:

● Global endpoint manages primary region identification.

● In case of failure in Region A, failover to Region B is managed, with potential data loss due to asynchronous replication.

● Global endpoint is repointed using Route 53 data planes for quick redirection.

● Switchover:

● Switchover is performed when both regions are healthy.

● Region A is told to stop being primary, and Region B becomes primary.

● Allow quick transition between regions without disruption.

● Switchover Process:

● Inserts a special log record (marker log record) into the log record stream.

● The other side recognizes the marker and becomes the primary at the specified log time.

● Region A stops being primary at the same time, ensuring a clean handoff.

● Results in zero RPO and very low RTO (about 30 seconds).

Maintenance and Upgrades

● Version upgrades (e.g., PostgreSQL 16 to PostgreSQL 17) and OS patches are necessary.

● Self-managed systems require downtime, snapshots for rollback, compatibility testing, and performance testing.

● Aurora provides a fully managed and automated minor version and patch upgrade experience.

● Rolling operating system upgrades are now available, increasing availability.

● Maintenance actions can be defined within a specified window or automatically managed by Aurora.

● AWS Health Dashboard provides maintenance notifications.

● Opt-in automation for maintenance is strongly recommended.

● AWS Organization's Upgrade Roll-out Policy:

● New feature to manage upgrades across multiple DB clusters in development, QA, and production environments.

● Simplifies the automation of the lifecycle management for fleets of DB clusters.

● Allows setting upgrade policies based on tags across multiple resources in an AWS organization.

● Example:

● Resources tagged Prod are upgraded last; dev resources are upgraded first.

● Default behavior: Resources without recognized tags are upgraded second.

● Upgrade rollout involves assigning the policy to resources, determining upgrade order, and proceeding in waves within maintenance windows.

● Provides flexibility to halt upgrades if issues are detected during the maintenance window.

● In-Place Upgrades:

● For major upgrades incompatible with open source, in-place upgrades are performed.

● Involves creating a clone of the database cluster, upgrading the clone, and testing with the application.

● Once validated, the clone is discarded, and the upgrade is applied to the production cluster.

● Blue-Green Deployments:

● More flexible approach for major upgrades, schema changes, or static parameter changes.

● Involves creating a new environment ( Green ) while keeping the old environment ( Blue ) in sync using logical replication.

● Allows testing the upgrade in the green environment before promoting it to production.

● Switchover renames resources to maintain consistency, completing in as little as a minute.

● Blue-green deployments are now available for global databases across multiple regions.

● Summary of Aurora APG and AMS Features:

● Addition of replicas to increase availability and enable failover.

● Use of AWS Advanced Rapid Drivers for smoother failover processes.

● Global Database for durability and availability across up to 10 secondary regions.

● Low latency local region reads and simplified endpoint management with global endpoints.

● Managed and automated cluster-wide upgrades, fast clones, and blue-green deployments for maintenance.

Introduction to Aurora DSQL

● Amazon Aurora DSQL is the fastest serverless distributed SQL database for always available applications. Aurora DSQL offers the fastest multi-Region reads and writes. It makes it effortless for customers to scale to meet any workload demand with zero infrastructure management and zero downtime maintenance.

● Combines the best of relational databases, distributed database architectures, and serverless services.

● Designed to run complex queries, scale out, and offer pay-per-use pricing.

● DSQL Architecture Overview:

● Comparison with self-managed PostgreSQL running on EC2.

● DSQL uses a query processor per connection, similar to PostgreSQL backend processes.

● Read queries are sent directly to the DSQL storage engine.

● Writes are buffered in memory on the query processor until commit.

● Upon commit, transactions are sent to the Adjudicator for concurrency control and then to the Journal for durability, replicating to at least 2 AZs.

● The Journal is used to update storage, similar to traditional PostgreSQL.

● DSQL is an active-active service, allowing any connection to read or write data at any time.

● Query processors (QPs) can run on different hosts or availability zones.

● Availability in DSQL:

● If a database node fails, the Query processors (QPs) detect the failure and shift traffic to a healthy replica.

● Replicas can be in the same or different availability zones.

● In-flight operations may experience a small increase in latency due to retries.

● DSQL continuously backs up data and automatically replaces failed nodes.

● Replacement nodes connect to the Journal to catch up with the latest data.

● Traffic is shifted back to the local replica for optimal performance.

● Concurrency Control and Durability:

● The Adjudicator handles concurrency control and conflict resolution.

● A single Adjudicator allows for quick conflict resolution without network coordination.

● Standby adjudicators are kept in other availability zones for failover.

● In case of failure, standby adjudicators run a leadership protocol to elect a new leader.

● Query processors retry in-flight commits against the new leader adjudicator.

● The process is seamless and requires no action from the application.

High Availability in DSQL

● DSQL is designed with no single point of failure.

● Active-active architecture provides high availability out of the box.

● DSQL is designed to scale automatically to handle increasing load.

● Concurrency Control Protocol:

● DSQL uses optimistic concurrency control, assuming transactions seldom conflict.

● Conflicts result in a serialization failure error and require the client to retry the transaction.

● Applications should handle retries and optimistic concurrency control issues.

● Recommended to build helper functions or incorporate handling into the lowest application layer.

● Avoiding Unnecessary Conflicts:

● Design applications to minimize unnecessary conflicts to reduce retries.

● Transactions updating different rows can commit simultaneously without conflict.

● Internal Sharding and Scaling:

● DSQL automatically shards itself internally to handle increased load on the adjudicator and journal.

● Query processors direct commits to the appropriate shard.

● Optimized two-phase commit ensures atomic transactions across multiple shards.

● DSQL automatically scales out by bringing new nodes online and load balancing across replicas.

● Replicas in DSQL are always strongly consistent, enabling seamless scaling.

● Consistency in DSQL:

● DSQL ensures strong consistency through a time-based synchronization protocol.

● Uses EC2 Time Sync Service for microsecond-accurate timestamps and Clockbound library to correct measurement errors.

● Guarantees a linearizable timestamp that is always after any previously committed timestamp.

● Storage waits for updates from the journal before returning rows to ensure consistency.

Two-Phase Commit (2PC) Protocol Overview

● 2PC is a distributed system protocol ensuring all participants in a transaction either commit or abort together.

● It consists of two phases: Prepare and Commit/Abort.

● Phase 1: Prepare Phase:

● A coordinator sends a "prepare" request to all participating nodes.

● Each participant performs the transaction's work, acquires necessary locks, and writes a "prepared" record to its transaction log.

● Participants send a "yes" or "no" vote back to the coordinator.

Phase 2: Commit/Abort Phase

If all participants voted "yes"

● The coordinator writes a "commit" record to its log and sends a "commit" message to all participants.

● Participants commit the transaction, release locks, and send an acknowledgement to the coordinator.

● If any participant voted "no":

● The coordinator writes an "abort" record to its log and sends an "abort" message to all participants.

● Participants roll back the transaction and release any locks.

● Key Considerations: Atomicity:

● 2PC guarantees that a transaction is atomic, either complete on all nodes or a complete failure on all nodes.

● Consistency:

● Ensures data consistency across multiple nodes, but may impact availability due to blocking and waiting for a decision.

● Failure Handling:

● Includes mechanisms to recover from failures; a coordinator failure can lead to an "in-doubt" state requiring manual intervention.

Connection Management in DSQL

● Connections to DSQL are managed by the session routing layer.

● Ensures quick availability of connections, even during network events causing fluctuations.

● Maintains a warm pool of query processors (QPs) ready for immediate use.

● When a connection is requested, DSQL assigns an available QP from the pool.

● Handling Connection Failures:

● In the event of a connection failure, the application must detect the failure and retry.

● The retry handler from the optimistic concurrency control section is useful for this purpose.

● Applications should implement retry logic for connection errors to maintain availability.

● Unlike storage and adjudicator failures, applications must manually reconnect failed connections.

● DSQL enforces a 1-hour maximum lifetime for connections to encourage proper connection management.

● Best Practices for Connection Management:

● Use client-side connection pooling libraries (e.g., Java C3PO) with health checks and maximum connection age.

● DSQL inherently includes connection pooling, making server-side pooling (e.g., PG Bouncer, RDS Proxy) unnecessary.

● Recommended to connect to QPs in the same availability zone for lower latency.

● DSQL automatically fails over to a healthy zone if the current zone is unavailable.

Multi-Region Clusters in DSQL

● DSQL supports multi-region clusters by creating clusters in two regions and designating a third region as the witness region.

● The witness region holds a copy of the journal to facilitate quorum-based protocol for regional failures.

● Applications can remain active across regions, with both regions capable of handling reads and writes.

● A quorum-based protocol

● is a method used in distributed systems to ensure data consistency, fault tolerance, and reliability by requiring a minimum number of participating nodes (a quorum) to agree on an operation before it is considered valid.

● S3 Quorum-based Protocol:

● In distributed systems, "Quorum" means "a minimum number of members required to constitute a meeting."

● The Quorum mechanism is a voting algorithm for ensuring data redundancy and consistency in distributed storage systems like Amazon S3.

● Core Concepts and Terminology:

● Read Quorum (Vr): Minimum number of read votes

● Write Quorum (Vw): Minimum number of write votes

● Total Votes (V)

● Working Principle:

● Assume there are V data replicas in a distributed system.

● To ensure data consistency, read and write operations must obtain a "quorum" of node confirmations.

● Core rules:

● Vr + Vw > V: Ensures the same data is not read and written simultaneously, avoiding read-write conflicts.

● Vw > V / 2: Ensures serialized data modifications, preventing two write operations from modifying data simultaneously.

● Adjusting Vr and Vw allows the system to balance between read and write performance.

● About Amazon S3:

● Amazon S3 announced in 2020 that all GET, PUT, and LIST operations have strong consistency.

● Data written can be immediately read as the latest version, simplifying big data workloads.

● S3 guarantees high durability and availability through its architecture, which stores data replicas across at least three availability zones.

Handling Regional Failures

● In the event of a regional failure (e.g., region B fails), the witness region helps disambiguate the failure and votes to exclude the failed region from the quorum.

● DSQL automatically reconfigures the journal to exclude the failed region, ensuring the cluster remains available in the healthy region (region A).

● There is no data loss, and transactions continue to be replicated to at least one other AWS region.

● When the failed region (region B) comes back online, DSQL detects it, updates it to the latest state, and restores the multi-region configuration.

● Recommended pattern: Deploy applications alongside clusters in both regions for active redundancy.

● Benefits of Active Redundancy:

● Deploying applications alongside clusters in multiple regions provides low latency and continuous validation of application functionality.

● Use a global endpoint (e.g., Route 53 DNS record) to direct traffic to the closest available healthy region.

● Handling Regional Failures with Global Endpoint:

● In the event of a regional failure, data is not lost, and DNS automatically directs traffic to the healthy region.

● Continuous validation ensures the healthy region is operational.

● Maintenance in DSQL:

● DSQL is a fully managed service with no maintenance required from customers.

● Query processes are periodically replaced with updated versions, ensuring the latest OS, security patches, performance improvements, and features.

● Customers do not need to perform any maintenance tasks.

● Takeaways:

● Aurora offers three engines: PostgreSQL, MySQL, and DSQL.

● All engines provide 3 AZ durability out of the box.

● Scaling options vary: APG and AMS scale up for writes and out for reads; DSQL offers hands-free, automatic scaling.

● Multi-region considerations: APG and AMS allow testing failover without data loss; DSQL supports active-active across regions.

● All engines offer easy maintenance with automated security updates and minor version upgrades.