Vertex Macro | Financial Cloud Cloud · AWS Re:cap
AWS Re:cap 03: Zurich Insurance Group: Building an Effective Log Management Solution on AWS
Agenda
Start with the basics
● What is log management and why is it critical for financial sector organizations
● Its role in compliance, security, and operational efficiency
Explore unique challenges faced by financial institutions in log management
● Handling large data volumes
● Meeting regulatory requirements
● Managing costs
Discuss Zurich's specific goals for a log management solution
● Understand the approach chosen
Dive into the technical details of the solution
● Outline key AWS services used
● Explain design principles for scalability, cost-effectiveness, and efficiency
Review the outcomes and benefits achieved
● Apply lessons to other financial sector organizations
Wrap up with key takeaways
● Provide actionable insights for organizations using or exploring AWS for log management
Practical and relatable, focusing on real-world strategies and inspiration for the audience's journey
Begin with the fundamentals:
● Why is log management important in the finance sector?
The basics of log management
● Log sources: Logs come from servers, appliances, network devices, cloud services, etc.
● Log collection: Centralizing log collection is essential for visibility and compliance
● Processing and analysis: Filtering noise, adding context, and identifying patterns or red flags
● Storage and retention: Secure, organized, and long-term storage (e.g., S3 tiers) is crucial for compliance
● Insights and actions: Leveraging data for actionable insights and responses
Log management is crucial for financial services due to three main reasons:
● Compliance: Regulators require proof of due diligence, and logs serve as evidence.
● Security: Logs are essential for early threat detection and mitigation.
● Operational efficiency: Smoothly running systems lead to happy customers, employees, and regulators.
● Log management is not just about data collection but about transforming data into useful insights for system protection, regulation compliance, and operational efficiency.
Key pain points:
Specific challenges Zurich faced in log management:
● Centralized logging can become expensive quickly.
● Approach treated all logs equally, storing everything in high-cost storage and processing all logs as critical, leading to rapid cost escalation.
Data volume:
● Financial services generate vast amounts of log data from various sources (firewall rules, access attempts, vulnerability scans, etc.).
● Security systems, due to sensitive data handling, generate terabytes of logs daily.
● Not all security logs are equally important; some can be analyzed later, while critical alerts require immediate attention.
Compliance regulations:
● Regulations like PCI DSS and GDPR mandate log retention (e.g., seven years for audits) and detailed access records.
● Ensuring logs are secure, tamper-proof, and accessible without excessive cost is challenging.
Cost efficiency:
● Balancing the need for comprehensive log management with cost-effective solutions is crucial.
Treating all logs equally leads to resource waste:
● Not all logs require high-cost, real-time processing.
● Example: One-time debug logs vs. API access logs containing security-critical information.
● Need to store and retain logs based on their importance and potential use.
Complexity and integration challenges:
● Logs come from various sources (legacy systems, cloud services, apps, devices) with different formats.
● Translating between systems can lead to delays and blind spots.
● Example: Vulnerability management system and CASB tool (Cloud Access Security Broke) using different formats.
Security and real-time analysis:
● SIME (Security information and event management) logs are critical for threat detection and response.
● Need to analyze logs quickly and efficiently to detect patterns (e.g., failed login attempts indicating a brute force attack).
● Trust is crucial in the financial sector, so timely threat detection is essential.
Summary of challenges:
● Overwhelming data volumes
● Strict compliance requirements
● High cost of logging
● Messy integrations
● Need for real-time security
Solution goals:
● Modernize log management approach:
● Move away from treating every log equally.
● Use scalable tools to prioritize data collection based on importance.
Reduce cost:
● Lower cost per gigabyte ingested without compromising performance and compliance.
● Manage data smarter by filtering out unnecessary logs before ingestion.
Decommission old SIME (Security information and event management) infrastructure:
● Retire legacy systems to reduce technical debt, high maintenance costs, slow performance, and limited scalability.
● Free up resources for more modern and efficient solutions.
Improve analytics performance:
● Reduce latency and log search time for faster insights and real-time decision-making.
● Focus on delivering better performance without unnecessary delays.
Objectives summary:
● Achieve smarter log management that reduces costs, simplifies the environment, and delivers better performance.
● Focus on what matters most.
Solution architecture:
Prioritize logs based on importance:
● High-priority logs (e.g., security events) are routed for immediate analysis.
● Lower-priority logs (e.g., compliance-related) are stored in cost-effective archives like S3.
● Ensures resources are focused on critical data while controlling costs.
ETL pipeline:
● Acts as a traffic controller for logs, filtering out unnecessary data to reduce ingestion costs.
● Enriches data with additional metadata to make logs more actionable.
● Routes data to the appropriate destination (real-time analytics or long-term storage).
● Critical for reducing cost per gigabyte ingested and ensuring valuable data is available where needed.
Route logs into SIME and AWS infrastructure:
● After prioritization and processing, logs are directed into the SIME (Security information and event management) and AWS infrastructure for further analysis and storage.
AWS infrastructure components and their roles:
Amazon OpenSearch Service:
● Enables real-time analytics with fast log searches.
● Provides dashboards and monitoring tools for DevOps and security teams.
● Helps identify root causes quickly (e.g., sudden spike in failed login attempts).
Amazon S3:
● Stores all logs for compliance purposes.
● Scalable, secure, and cost-effective.
● Perfect for compliance needs.
AWS Glue and Data Catalog:
● Organizes and makes logs searchable.
● Reduces time spent searching through raw data.
Amazon Athena:
● Allows querying data directly in S3 without data movement.
● Cost-effective for deep dive investigations and ad hoc analysis.
● Useful for investigating potential compliance issues.
Combining data prioritization, ETL capabilities, and AWS services:
● Modernized log management approach.
● Reduced costs by storing and processing only the most important logs.
● Simplified infrastructure by retiring legacy SIME (Security information and event management) systems.
● Improved performance with faster log searches and better analytics.
Benefits and outcomes of the solution:
Cost savings:
● Reduced SIME (Security information and event management) ingestion from 5 TB per day to 500 GB per day.
● Achieved by deploying Cribl with version control and automatic backups.
● Unnecessary logs were filtered out before ingestion.
● Scales rapidly using Terraform.
Performance improvements:
● Sample query for counting firewall events per firewall reduced from 93 seconds to 2 seconds.
● OpenSearch proved to be fast and easy to manage.
● One-click deployment and upgrades.
● Consistent and reliable rollout using version-controlled Terraform.
Clean house and technical debt reduction:
● Retired old SIME (Security information and event management) infrastructure, eliminating technical debt.
● Transitioned to a sleek, modern architecture.
Scalability:
● Utilized AWS services like S3 for tiered data storage (frequent access, infrequent access, and Glacier).
● Achieved rapid scaling using Terraform for Cribl and other components.
● The system grows with organizational needs.
Compliance and security:
● Logs securely stored in S3 and easily searchable in OpenSearch.
● Made regulatory reporting and security monitoring faster, simpler, and more reliable.
Key lessons learned from Journey:
Prioritize the right data:
● Not all logs are equal; treat them based on importance.
● Zurich achieved over 50% reduction in ingestion costs by prioritizing critical logs.
Invest in modern tools:
● Use tools designed for current challenges (e.g., Cribl, AWS OpenSearch, S3).
● These tools scale seamlessly with the organization.
Automate for consistency:
● Utilize automation (e.g., Terraform) for deployments, upgrades, and scaling.
● Reduces human error and frees teams for strategic tasks.
Think beyond compliance:
● Modern log management offers faster searches, real-time insights, and better security monitoring.
● Focus on making data work for the organization, not just meeting regulatory requirements.
Summary:
● A smarter approach to log management is transformative.
● Prioritize what matters, invest in modern tools, automate processes, and think beyond compliance.
Future plans for log management:
Continue focusing on cost savings:
● Refine S3 tiering strategies to optimize storage efficiency.
● Shift logs between frequent, infrequent, and Glacier tiers based on usage.
Expand automation:
● Further streamline scaling and configuration updates using Terraform.
● Aim for infrastructure that practically runs itself.
Integrate advanced analytics:
● Utilize OpenSearch's built-in machine learning capabilities like anomaly detection.
● Catch unusual patterns in log data before they escalate.
Import and optimize SIME correlation searches and alerts:
● Continue transitioning from old legacy SIME (Security information and event management) by importing and optimizing these into OpenSearch.
● Maintain high monitoring standards while embracing the modern platform.
Strengthen data governance:
● Enhance metadata management with tools like AWS Glue.
● Keep logs organized and stay ahead of compliance requirements.
Explore generative AI with OpenSearch:
● Use OpenSearch as a vector database to power retrieval augmented generation (RAG) for AI use cases.
● Example: Train AI models using logs to predict system issues before they happen.