← Financial Cloud Cloud Cloud Club · AWS Re:cap

Vertex Macro | Financial Cloud Cloud · AWS Re:cap

AWS Re:cap 04: FSI Meetup 2025 Q4 - Brex Database Disaster Recovery

Speaker: Fabiano Honorato, Michelle Koo, Stephen Brandon

Session: 04

Session
Summit Dev Lounge2026 Re:cap
01 Encode Architecture as Steering for AI Agents
02 Agent Harness Is the Real Engineering Moat
03 Ask Observability Data in Plain Language
04 Serverless AR Game with Bedrock AgentCore
05 Multi-Agent Quant Backtesting on AgentCore
06 Blog to Slides in Three Minutes with Kiro
AWS Community Day Hong Kong 2025 Re:cap
02 AWS Compliance with Terraform
03 Beginner to Builder An AWeSome Cloud Journey
04 Team-First Serverless Engineering with Laravel & Bref
05 Event Opening - AWS Community Day Hong Kong 2025
06 Agent-to-Agent: Building Interoperable AI on AWS
07 Utilize another telemetry data for faster improvement with AI agent
08 Graduating from Vibe Coding: Spec-Driven Development with Kiro
09 Automated Testing using MCP & AI Agents
10 Modernizing Telecom Security ML Powered Approach
11 Rethinking GenAI Agent: RAG & MCP
12 Disaster and Emergency Response with TAK and AWS
13 Rethinking Serverless Application Workflows from a Testing Perspective
14 Practical AWS FinOps for Cloud Success
15 AI-Powered Global Pure-Alpha Macro Trades on AWS: Revolutionizing Risk-Adjusted Asset Returns
FSI Recap
01 Modern Trade Lifecycle: Trading to Settlement
02 Goldman Sachs: Fast Track your applications onto Cloud - AWS Re:cap Q1/2023
03 Zurich Insurance Group: Building an Effective Log Management Solution on AWS
04 FSI Meetup 2025 Q4 - Brex Database Disaster Recovery
05 FSI Meetup 2025 Q4 - A Graviton Migration Success Story
06 FSI Meetup 2025 Q4 - Stifel Modern Data Platform
07 FSI Meetup 2025 Q4 - Financial Transaction Data Reconciler PayPal
08 FSI Meetup 2025 Q4 - Scaling Resilience
09 Maximizing AI Inference Cost Efficiency: Strategic Adoption of AWS GPU Instances
10 Advanced Agentic AI Design Patterns
11 Build New Modern Apps on AWS
AWS re:Invent 2025
01 Coinbase re:Invent Recap (IND3312)
02 Building the Future Trading Platform Leveraging AI and AWS
03 Trading Innovation: Jefferies' AI Assistant on Amazon Bedrock (IND3315)
04 How FSI Revolutionized HFT Analytics with Agentic AI (GBL302)
05 Improving Distributed Systems with Amazon Time Sync Featuring Nasdaq
06 Amazon Aurora HA and DR Design Patterns for Global Resilience (DAT442)
07 Building Agentic AI: Amazon Nova Act and Strands Agents in Practice (DEV327)
08 Deep Dive into Amazon Aurora and Its Innovations (DAT441)
09 Deep dive on Amazon S3 (STG407)
10 Nasdaq: Build Resilient Infrastructure for Global Financial Services (HMC327)
11 What's New with AWS Lambda (CNS376)
12 Spec-Driven Development with Kiro (DEV314)
13 Amazon's finops: Cloud cost lessons from a global e-commerce giant (AMZ308)
14 Tick to trade latency trading platforms on aws
Government data
01 The AI Era: The Boundary Between Development and Design Is Disappearing
02 On-Device Multimodal AI and Smart-City Practice
03 Large-Model Capability Evaluation and a Method for Landing AI Projects
04 Controlled End-to-End Automation of Government Development with Cloud Agents
05 A New Software Ecosystem for the Agent Era, Seen Through Multi-Agent Systems
06 AI-Driven Macro Quantitative Research and Smart Governance
07 Authorized Operation of Public Data and Smart-Government Practice
08 Putting Data Assetization into Practice: Rights, Compliance, Engineering Governance, and Digital-Government Cases
09 AI for Mental-Health Public Welfare: Governance, Architecture, and Practice of a Trustworthy Platform
Amarathon 2025 Recap
01 A Developer’s Roadmap to Architecting for Agents
02 Amazon Bedrock Data Automation
03 Multi-Agent on AgentCore
04 Building Agentic AI Nova Act and Strands Agents in Practice
04 Accelerating Migration Projects with Kiro using Spec-Driven Development
06 From Matching to Understanding: Personalized AI Search Practice Driven by AgentCore Memory
07 Observe to Optimize – LLM Observability to AIOps Turning real-time insights into intelligent automation
08 Deploying TEAM and Building the Best Engineering Team
09 Five Hard Lessons from Five Years of So-Called Serverless Databases
14 What if AI does my job How Q Developer CLI and Kiro have changed my daily routine
16 Velocity with Vigilance: Security Essentials for Amazon Bedrock Agent Development
26 Run OSS LLMs on a Single H100 Smarter, Cheaper, Faster
28 A Modern Unified Metadata Architecture: New Approaches to Breaking Down Data Silos
29 Serverless MediaOps: Automating Video Workflows with AI on Amazon Web Services
30 Architecting for Efficiency and Reliability with Performance Testing at Scale
31 Connecting the World Through Open Source: Practical Journey of Technology, Community and Global Developer Relations
33 Building Streaming Iceberg Tables for Real-Time Logistics Analytics
34 Accelerating Large-Scale Robot Strategy Training: An Automated Closed-Loop Architecture Based on Kiro, Trainium, and EKS
35 From Vibe to Viable with spec driven development
36 Making Cloud Cost Analysis Smarter: Building FinOps Intelligent Agents with Strands and AgentCore
37 Transform Conversational Agentic AIOps for K8s Using CNCF Kagent, K8sGPT, and Nova Sonic

Introduction to Brex

● Financial operating system platform for managing expenses, travel, credit.

● Engineering manager and team members discuss leveraging Amazon Aurora for resiliency and international expansion

Brex services

● Corporate cards, expense management, travel, bill pay, and banking

● Aim to help clients spend wisely and smartly


Importance of preparing infrastructure for disaster scenarios

● Focus on the data layer, primarily using PostgreSQL with PG bouncer and replicas for applications and analytical purposes

● Merge smaller databases into a single database instance

● Past disaster recovery process was manual and time-consuming

Goals for disaster recovery solution

● Warm disaster recovery solution to decrease Recovery Time Objective (RTO) and Recovery Point Objective (RPO)

● RTO: maximum time to recover normal operations after disaster

● RPO: maximum amount of data tolerable to lose

Determining RPO and RTO

● Analyze metrics, assess current capabilities, and conduct extensive testing

● Understand how applications will handle additional latency and data loss

Choice of Amazon Aurora Global Database

● Provides necessary features without significant changes to the current setup

● Allows use of a secondary region when needed

Current implementation caveats

● Creating custom DNS endpoint for read applications used for both applications and analytical purposes

Migration challenges and approach

● Difficulty in migrating from PostgreSQL to Aurora due to potential application downtime

● Focus on automation to minimize manual handling

● Built a temporal workflow for running automated jobs to validate migration steps and prepare the environment

● Performed the switch over to Aurora Global after ensuring automation validated database status

Downtime management during migration

● AWS provides a small window of downtime (2-3 minutes) for migration

● Utilized this window to adjust endpoints and applications consuming the database

● Leveraged the short downtime period for a smooth transition


Using temporal workflows for automation

Current state before migration

● Application connected to PG bouncer, which connected to PostgreSQL instance and replica instance

Migration process

● Created Aurora read replica through AWS with zero downtime

● Workflow promoted Aurora read replica and created Aurora global cluster

● Application connected to PG bouncer, which then connected to Aurora global cluster using global writer endpoint

● Possibility to create another cluster for multi-region setup

Flux:

● Tool for keeping Kubernetes clusters in sync through git repositories

● Workflow generated Flux git pull requests ahead of time

● Workflow automatically merged pull requests after manual verification

● Confirmation signal sent to workflow to proceed with downtime and promote Aurora global cluster

Automatically reviewing Flux using AI

● Identified errors or issues in pull requests and provided comments for review

Dry run migration flag

● Allowed testing of migration without causing destructive actions or downtime

● Created Flux git pull requests ahead of time for review, but did not merge or promote cluster


Additional tools and processes used in the migration

● Terraform: created a template for managing databases as a new global cluster

● Added Terraform for each database after migration workflow completion

● Managed reader instances in the global cluster through Terraform

Internal command line tool:

● Added commands for teams to self-service switch over or failover for their Aurora global clusters

● Failover: used to recover from unplanned outages by switching to a different region if one region is down

● Switchover: used for controlled scenarios like operational maintenance or planned procedures with no data loss


Iterative journey of improving workflow performance

● Initial workflow took around 15 minutes for end-to-end automation

● Downtime for promoting Aurora global cluster and creating Flux git pull requests

● Sequential process with no parallelization

Addition of parallelization reduced workflow time to 10 minutes

● Updated workflow to perform steps in parallel, including fetching credentials and creating Flux pull requests ahead of time

● Introduced dry run flag for non-destructive testing of migration

Final restructuring of workflow achieved 3-minute performance time

● Created Flux pull requests ahead of time, allowing workflow to pause until downtime window

● Reduced git add operations to minimize costs

● Added signal command for controlled initiation of downtime

Lessons learned during the process

● Thorough testing and deliberate deployment are crucial before formal migration

● Start with staging environment, resolve issues, and then proceed to production

● Automation reduces human error and enables easy replication for multiple databases

● Simulate migration using the dry run option to test workflow without causing downtime (A dry run or practice run, a trial exercise or rehearsal, is a software testing process used to make sure that a system works correctly and will not result in severe failure.)

● Iterative improvement by migrating a few databases each week