← Financial Cloud Cloud Cloud Club · AWS Re:cap

Vertex Macro | Financial Cloud Cloud · AWS Re:cap

AWS Re:cap 08: FSI Meetup 2025 Q4 - Scaling Resilience

Speaker: Gregory St. Cyr

Session: 08

Session
Summit Dev Lounge2026 Re:cap
01 Encode Architecture as Steering for AI Agents
02 Agent Harness Is the Real Engineering Moat
03 Ask Observability Data in Plain Language
04 Serverless AR Game with Bedrock AgentCore
05 Multi-Agent Quant Backtesting on AgentCore
06 Blog to Slides in Three Minutes with Kiro
AWS Community Day Hong Kong 2025 Re:cap
02 AWS Compliance with Terraform
03 Beginner to Builder An AWeSome Cloud Journey
04 Team-First Serverless Engineering with Laravel & Bref
05 Event Opening - AWS Community Day Hong Kong 2025
06 Agent-to-Agent: Building Interoperable AI on AWS
07 Utilize another telemetry data for faster improvement with AI agent
08 Graduating from Vibe Coding: Spec-Driven Development with Kiro
09 Automated Testing using MCP & AI Agents
10 Modernizing Telecom Security ML Powered Approach
11 Rethinking GenAI Agent: RAG & MCP
12 Disaster and Emergency Response with TAK and AWS
13 Rethinking Serverless Application Workflows from a Testing Perspective
14 Practical AWS FinOps for Cloud Success
15 AI-Powered Global Pure-Alpha Macro Trades on AWS: Revolutionizing Risk-Adjusted Asset Returns
FSI Recap
01 Modern Trade Lifecycle: Trading to Settlement
02 Goldman Sachs: Fast Track your applications onto Cloud - AWS Re:cap Q1/2023
03 Zurich Insurance Group: Building an Effective Log Management Solution on AWS
04 FSI Meetup 2025 Q4 - Brex Database Disaster Recovery
05 FSI Meetup 2025 Q4 - A Graviton Migration Success Story
06 FSI Meetup 2025 Q4 - Stifel Modern Data Platform
07 FSI Meetup 2025 Q4 - Financial Transaction Data Reconciler PayPal
08 FSI Meetup 2025 Q4 - Scaling Resilience
09 Maximizing AI Inference Cost Efficiency: Strategic Adoption of AWS GPU Instances
10 Advanced Agentic AI Design Patterns
11 Build New Modern Apps on AWS
AWS re:Invent 2025
01 Coinbase re:Invent Recap (IND3312)
02 Building the Future Trading Platform Leveraging AI and AWS
03 Trading Innovation: Jefferies' AI Assistant on Amazon Bedrock (IND3315)
04 How FSI Revolutionized HFT Analytics with Agentic AI (GBL302)
05 Improving Distributed Systems with Amazon Time Sync Featuring Nasdaq
06 Amazon Aurora HA and DR Design Patterns for Global Resilience (DAT442)
07 Building Agentic AI: Amazon Nova Act and Strands Agents in Practice (DEV327)
08 Deep Dive into Amazon Aurora and Its Innovations (DAT441)
09 Deep dive on Amazon S3 (STG407)
10 Nasdaq: Build Resilient Infrastructure for Global Financial Services (HMC327)
11 What's New with AWS Lambda (CNS376)
12 Spec-Driven Development with Kiro (DEV314)
13 Amazon's finops: Cloud cost lessons from a global e-commerce giant (AMZ308)
14 Tick to trade latency trading platforms on aws
Government data
01 The AI Era: The Boundary Between Development and Design Is Disappearing
02 On-Device Multimodal AI and Smart-City Practice
03 Large-Model Capability Evaluation and a Method for Landing AI Projects
04 Controlled End-to-End Automation of Government Development with Cloud Agents
05 A New Software Ecosystem for the Agent Era, Seen Through Multi-Agent Systems
06 AI-Driven Macro Quantitative Research and Smart Governance
07 Authorized Operation of Public Data and Smart-Government Practice
08 Putting Data Assetization into Practice: Rights, Compliance, Engineering Governance, and Digital-Government Cases
09 AI for Mental-Health Public Welfare: Governance, Architecture, and Practice of a Trustworthy Platform
Amarathon 2025 Recap
01 A Developer’s Roadmap to Architecting for Agents
02 Amazon Bedrock Data Automation
03 Multi-Agent on AgentCore
04 Building Agentic AI Nova Act and Strands Agents in Practice
04 Accelerating Migration Projects with Kiro using Spec-Driven Development
06 From Matching to Understanding: Personalized AI Search Practice Driven by AgentCore Memory
07 Observe to Optimize – LLM Observability to AIOps Turning real-time insights into intelligent automation
08 Deploying TEAM and Building the Best Engineering Team
09 Five Hard Lessons from Five Years of So-Called Serverless Databases
14 What if AI does my job How Q Developer CLI and Kiro have changed my daily routine
16 Velocity with Vigilance: Security Essentials for Amazon Bedrock Agent Development
26 Run OSS LLMs on a Single H100 Smarter, Cheaper, Faster
28 A Modern Unified Metadata Architecture: New Approaches to Breaking Down Data Silos
29 Serverless MediaOps: Automating Video Workflows with AI on Amazon Web Services
30 Architecting for Efficiency and Reliability with Performance Testing at Scale
31 Connecting the World Through Open Source: Practical Journey of Technology, Community and Global Developer Relations
33 Building Streaming Iceberg Tables for Real-Time Logistics Analytics
34 Accelerating Large-Scale Robot Strategy Training: An Automated Closed-Loop Architecture Based on Kiro, Trainium, and EKS
35 From Vibe to Viable with spec driven development
36 Making Cloud Cost Analysis Smarter: Building FinOps Intelligent Agents with Strands and AgentCore
37 Transform Conversational Agentic AIOps for K8s Using CNCF Kagent, K8sGPT, and Nova Sonic

What is USAA?

● USAA is a financial services company.

● Offers a wide variety of products across banking, insurance, wealth management.

● Emphasizes customer service as a key product.

● Serves 14 million members, primarily service members and their families.

● Focuses on availability and resiliency due to the unique needs of its members.

USAA’s Journey to Cloud

● Began cloud deployment in 2019 with initial deployments in US East 1.

● Gradual increase in cloud traffic and momentum over the next few years.

● By June 2023, had dozens of deployments.

● Adopted a “never let an outage go to waste” mentality.

● Instead of repatriating, used outages as opportunities to mature the platform.

● Enabled a secondary region by deploying applications to US West 2.

● By 2024, had hundreds of applications deployed to AWS and dozens to US West 2.

● Viewed the October outage as another learning and growth opportunity.

Multi-Region Deployment Strategy

● Decided to adopt a multi-region deployment strategy.

● Focused on three failover patterns: backup restore, pilot light, and warm standby.

● Backup and restore is critical, especially for data availability.

● Cannot fail over to the secondary region without data availability.

● Must have a process to get data to the secondary region, even if it involves recovering from snapshots.

● Pilot light and warm standby are used for high and critical applications.

● Goal is to improve Recovery Time Objective (RTO) and Recovery Point Objective (RPO).

● Aim is to enhance resiliency and customer experience for military members.


Three Phases for Application Resilience:

Architectural Design:

● Assess each application and identify dependencies.

● Understand necessary actions to make the application resilient.

● Consider downstream dependencies in the critical path.

● Focus on instant response and recovery requirements.

● Define necessary Recovery Time Objective (RTO).

● Incorporate RTO as a requirement in architectural design.

● Select appropriate failover pattern (backup restore, pilot light, warm standby) for the use case.

Operational Readiness:

● Configure necessary settings for deployment.

● Develop playbooks and monitoring tools.

● Ensure availability of resources to meet RTO and Recovery Point Objective (RPO) metrics and Service Level Objectives (SLOs).

Testing and Feedback:

● Conduct resiliency testing and plan for outages.

● Learn from outages and gather continuous feedback.

● Perform retrospective analysis to ensure SLO objectives for RTO and RPO are met.

● Improve platform maturity through continuous learning and adaptation.


Process and Tooling:

Centralized Resilience:

● Organize and centralize resilience standards and frameworks.

● Ensure a common set of standards across the organization.

Distributed DevOps Platform:

● Build a DevOps platform that meets application engineers where they are.

● Ensure consistency across all development teams.

● Provide necessary tools and support for DevOps maturity.

Automation:

● Look for opportunities to automate resiliency processes.

● Implement self-service blueprints integrated into SDLC processes.

● Provide infrastructure as code templates and quick start guides.

● Ensure app teams can quickly start development with resiliency requirements built-in.


Specific Examples of USAA’s Resiliency Process

Custom Well-Architected Reviews:

● Started with the Well-Architected Framework and applied it to all USAA applications.

● Recognized consistent elements across applications due to company structure (centralization, DevOps platform).

● Built a custom lens on top of the Well-Architected Review for more meaningful, automated scoring.

● Identifies and highlights components each application can control to improve resiliency.

Shifting Left:

● "Shift left" is a strategy in software development and IT service management that involves moving critical activities, such as testing, security, and quality assurance, to an earlier stage in the development or support lifecycle. This approach aims to find and fix issues sooner, which reduces costs, improves quality, and speeds up delivery.

● Early validation of RTO to reduce churn and improve resiliency in application building.

● Ensured resiliency understanding is critical during the solutioning of business use cases.

Tooling Integration in CI/CD Pipeline:

● Use GitLab in CI/CD pipeline.

● Integrated AWS Resilience Hub and Resource Groups for insights.

● Application teams receive immediate feedback on assessments during pipeline runs.

● Evidence-based, real-time reporting on resilience before production implementation.

● Lower environment stages also provide feedback.

Canary Deployment Strategy:

● Rolling out new software versions by gradually shifting a small percentage of user traffic to the new version.

● Automated canary deployments through the pipeline.

● Particularly useful for serverless and container-based applications.

● Weight routing to slowly roll out changes and test resiliency.

● Allows learning from changes and understanding application resiliency as it rolls out to members.


Pattern Library Components

Quick Start Guides:

● Allow developers to start quickly while ensuring consistency in development practices at USAA.

● Include non-functional requirements related to resiliency.

Serverless Patterns:

● Examples include async API, choreography pattern with event bridge, real-time API with orchestration using Step Functions.

● Provide quick starts for serverless applications.

Container Patterns:

● Use Helm charts (packages that define, install, and upgrade applications in Kubernetes) as templates for deploying container-based applications based on specific needs.

Archival Data Patterns:

● Moving inactive data from active systems to cheaper, long-term storage to improve performance and reduce costs, using tiered storage like on-premises, cloud, or hybrid models, and implementing strategies like data partitioning for efficient movement.

● Include archival patterns for moving data from application VPC to data VPC for archival.

● Automated snapshots and restore patterns for recovery from archival data VPC or local snapshots.

● Ensures safe data movement and resiliency within or across regions based on application needs.

Selection Criteria:

● Used to identify the best pattern for a use case based on reliability, performance, latency, and sensitivity.

● Helps implement consistency in applications and choose the best resiliency architecture.


Example: Origination System for Deposits Application

Overview:

● Critical system for opening new deposit accounts (checking, savings) at USAA.

● Part of the new member onboarding process.

● Complex multi-service architecture relying on downstream systems.

Before Multi-Region Resiliency Implementation:

● Assessment showed poor performance in disaster recovery, durability, and observability.

● Not resilient or available enough for members.

After Implementing Warm Standby Pattern:

● Reduced RTO from 4 hours to 30 minutes.

● Reduced RPO from 1 hour to 5 minutes.

● Enabled failover within 30 minutes during the US East 1 outage, ensuring seamless member experience.

● Significant business impact with improved recovery and data handling.

Architectural Design:

● Platform account handles network traffic and routing.

● Application deployed in US East 1 and US West 2 VPCs (warm standby implementation).

● Use of DynamoDB global tables for near real-time replication to the secondary region.


Additional Considerations for Multi-Region Deployment

Idempotency:

● Concerns whether a transaction can be posted more than once without adverse effects.

● Critical in banking space where monetary transactions must not be duplicated.

● Considered from a within-region resiliency perspective.

Retries and Transaction IDs:

● Necessary to manage retries and maintain transaction IDs in DynamoDB.

● Essential for managing state within Step Functions.

State Management During Failover:

● Challenge of maintaining state when failing over during a Step Function execution.

● State is not maintained when moving to a secondary region.

● Must design around this consideration to ensure resiliency during failover.

Architectural Design (Phase One):

● Incorporates considerations like Idempotency and state management.

● Ensures resiliency is designed into the failover procedure.

Operational Runbooks (Phase Two):

● Describes procedures and considerations for operational resilience.

● Includes handling retries, transaction IDs, and state management.

Testing and Continuous Feedback (Phase Three):

● Learn from errors and discrepancies during testing and real-world scenarios.

● Addresses issues like Idempotency, performance degradation, cold starts, and cache hydration in the secondary region.

Key to Building Resilient Systems:

● Understanding and accounting in design.

● Ensuring resiliency not just within one region but across multiple regions.

● Critical for providing resilient services to USAA members.


USAA’s Resiliency Framework and Success Factors

100% New Applications Through Resiliency Framework:

● All new applications at USAA now go through the resiliency framework.

● Reassessing existing applications deployed to AWS to ensure they meet resiliency needs.

Shift to Conservative Multi-Region Strategy:

● Moving more applications to a multi-region architecture based on learned best practices.

● Example: improved origination system RTO and RPO significantly.

Improved Resilience and Architecture Reviews:

● Resilience reviews and architecture reviews have become faster with increased learning and feedback.

● Understanding trade-offs has accelerated the process.

● Reduction in resilience-related incidents by 66% during recent outages.


Organizational Success Factors for Resiliency

Culture:

● Shared responsibility among technologists, IT, and business partners.

● Understanding the meaning of resiliency and defining Service Level Objectives (SLOs).

Architecture Leadership:

● Executive sponsorship and dedicated funding for resiliency as a primary objective.

● Using members’ needs as the mission to drive investment in resiliency.

Approach:

● Started small with a single application deploying multi-region.

● Learned from initial deployments and scaled with short feedback loops.

● Improved platform maturity and consistency in resiliency implementation.

Tooling:

● Investment in automation and self-service capabilities.

● Blueprints within the SDLC and templates for Infrastructure as Code (IaC).

● Ensuring developers can implement resilient practices without churn or toil.

● Key to meeting business objectives while providing excellent member service.


Key Takeaways from the Session

Standardization Accelerates Adoption:

● Use blueprints and templates to reduce complexity.

● Simplify understanding of resiliency for both developers and business stakeholders.

Shift Left:

● Move resiliency considerations earlier in the project ideation phase.

● Foster partnerships with business partners to align resiliency strategy with business needs.

Continuous Testing:

● Testing must be ongoing and iterative.

● Learn from outages and resilience tests; don’t let any learning go to waste.

● Build confidence through continuous testing and feedback.

Culture and Technology Must Evolve Together:

● Adopt a resiliency mindset organization-wide.

● Use this mindset to influence technology development.

● Ensure that culture and technology for sustained success.