← Financial Cloud Cloud Cloud Club · Amarathon 2025 Recap

Vertex Macro|Financial Cloud Cloud · AWS Amarathon 2025

Vertex Macro|Amarathon 2025 Recap 28:A Modern Unified Metadata Architecture: New Approaches to Breaking Down Data Silos

Speaker: Shaofeng Shi

Session: 28

Session
Summit Dev Lounge2026 Re:cap
01 Encode Architecture as Steering for AI Agents
02 Agent Harness Is the Real Engineering Moat
03 Ask Observability Data in Plain Language
04 Serverless AR Game with Bedrock AgentCore
05 Multi-Agent Quant Backtesting on AgentCore
06 Blog to Slides in Three Minutes with Kiro
AWS Community Day Hong Kong 2025 Re:cap
02 AWS Compliance with Terraform
03 Beginner to Builder An AWeSome Cloud Journey
04 Team-First Serverless Engineering with Laravel & Bref
05 Event Opening - AWS Community Day Hong Kong 2025
06 Agent-to-Agent: Building Interoperable AI on AWS
07 Utilize another telemetry data for faster improvement with AI agent
08 Graduating from Vibe Coding: Spec-Driven Development with Kiro
09 Automated Testing using MCP & AI Agents
10 Modernizing Telecom Security ML Powered Approach
11 Rethinking GenAI Agent: RAG & MCP
12 Disaster and Emergency Response with TAK and AWS
13 Rethinking Serverless Application Workflows from a Testing Perspective
14 Practical AWS FinOps for Cloud Success
15 AI-Powered Global Pure-Alpha Macro Trades on AWS: Revolutionizing Risk-Adjusted Asset Returns
FSI Recap
01 Modern Trade Lifecycle: Trading to Settlement
02 Goldman Sachs: Fast Track your applications onto Cloud - AWS Re:cap Q1/2023
03 Zurich Insurance Group: Building an Effective Log Management Solution on AWS
04 FSI Meetup 2025 Q4 - Brex Database Disaster Recovery
05 FSI Meetup 2025 Q4 - A Graviton Migration Success Story
06 FSI Meetup 2025 Q4 - Stifel Modern Data Platform
07 FSI Meetup 2025 Q4 - Financial Transaction Data Reconciler PayPal
08 FSI Meetup 2025 Q4 - Scaling Resilience
09 Maximizing AI Inference Cost Efficiency: Strategic Adoption of AWS GPU Instances
10 Advanced Agentic AI Design Patterns
11 Build New Modern Apps on AWS
AWS re:Invent 2025
01 Coinbase re:Invent Recap (IND3312)
02 Building the Future Trading Platform Leveraging AI and AWS
03 Trading Innovation: Jefferies' AI Assistant on Amazon Bedrock (IND3315)
04 How FSI Revolutionized HFT Analytics with Agentic AI (GBL302)
05 Improving Distributed Systems with Amazon Time Sync Featuring Nasdaq
06 Amazon Aurora HA and DR Design Patterns for Global Resilience (DAT442)
07 Building Agentic AI: Amazon Nova Act and Strands Agents in Practice (DEV327)
08 Deep Dive into Amazon Aurora and Its Innovations (DAT441)
09 Deep dive on Amazon S3 (STG407)
10 Nasdaq: Build Resilient Infrastructure for Global Financial Services (HMC327)
11 What's New with AWS Lambda (CNS376)
12 Spec-Driven Development with Kiro (DEV314)
13 Amazon's finops: Cloud cost lessons from a global e-commerce giant (AMZ308)
14 Tick to trade latency trading platforms on aws
Government data
01 The AI Era: The Boundary Between Development and Design Is Disappearing
02 On-Device Multimodal AI and Smart-City Practice
03 Large-Model Capability Evaluation and a Method for Landing AI Projects
04 Controlled End-to-End Automation of Government Development with Cloud Agents
05 A New Software Ecosystem for the Agent Era, Seen Through Multi-Agent Systems
06 AI-Driven Macro Quantitative Research and Smart Governance
07 Authorized Operation of Public Data and Smart-Government Practice
08 Putting Data Assetization into Practice: Rights, Compliance, Engineering Governance, and Digital-Government Cases
09 AI for Mental-Health Public Welfare: Governance, Architecture, and Practice of a Trustworthy Platform
Amarathon 2025 Recap
01 A Developer’s Roadmap to Architecting for Agents
02 Amazon Bedrock Data Automation
03 Multi-Agent on AgentCore
04 Building Agentic AI Nova Act and Strands Agents in Practice
04 Accelerating Migration Projects with Kiro using Spec-Driven Development
06 From Matching to Understanding: Personalized AI Search Practice Driven by AgentCore Memory
07 Observe to Optimize – LLM Observability to AIOps Turning real-time insights into intelligent automation
08 Deploying TEAM and Building the Best Engineering Team
09 Five Hard Lessons from Five Years of So-Called Serverless Databases
14 What if AI does my job How Q Developer CLI and Kiro have changed my daily routine
16 Velocity with Vigilance: Security Essentials for Amazon Bedrock Agent Development
26 Run OSS LLMs on a Single H100 Smarter, Cheaper, Faster
28 A Modern Unified Metadata Architecture: New Approaches to Breaking Down Data Silos
29 Serverless MediaOps: Automating Video Workflows with AI on Amazon Web Services
30 Architecting for Efficiency and Reliability with Performance Testing at Scale
31 Connecting the World Through Open Source: Practical Journey of Technology, Community and Global Developer Relations
33 Building Streaming Iceberg Tables for Real-Time Logistics Analytics
34 Accelerating Large-Scale Robot Strategy Training: An Automated Closed-Loop Architecture Based on Kiro, Trainium, and EKS
35 From Vibe to Viable with spec driven development
36 Making Cloud Cost Analysis Smarter: Building FinOps Intelligent Agents with Strands and AgentCore
37 Transform Conversational Agentic AIOps for K8s Using CNCF Kagent, K8sGPT, and Nova Sonic

A Brief History to Un-silo the Data

● LATE 1980'S: Data Warehouse

● 2011: Data Lake

● 2020: Lakehouse

Goal

● To achieve SSOT (Single Source of Truth)

● Full management of data

● Get rid of risks, such as data leak, compliance for a data-driven business.

● New Data Silos in Clouds & Regions

Nobody like vendor “lock-in”

● If data is deployed with different cloud vendors:

● Hard to Process together

● Expensive to Move

Nobody like geo-distributed data,

● But data goes with business to become international:

● Regulation requirement

● Cost for cross-ocean transfer

● More than "Data Access"

Data you see

● Technical & Business Data

● Legal Hold Data

Metadata you overlook

● 3rd Party Data

● PII & PI Data

● Credentials

● IP Data

Data Management Functions

● Data Connect: Connect to the Data That Matters Most.

● Data Right Automation: Automate end-to-end data rights requests and reporting.

● Metadata Enrichment: Enrich technical metadata with business and operational metadata for full visibility.

● Data Discovery: Automatically find, classify, and map all of your data - everywhere.

● Data Classification: Automatically classify more types of data in more places.

● Data Lifecycle Management: Simplify and automate data lifecycle management from collection to destruction.


What is Gravitino

● Next-gen unified data catalog for Data/AI

Integrations

● Trino

● Spark

● Flink

● Doris

● ClickHouse

● PyTorch

● TensorFlow

Metadata Lake Using Gravitino Components

● Hive Metastore

● Built-in Catalog

● Schema Registry

● Fileset Management

● Model Catalog

Data Sources

● Hadoop Data Lake

● Data Warehouse

● Streaming Processing

● Unstructured Data

● Machine Learning

Problems to solve

● Have a "Big Picture" of whole data

● Achieve SSOT of data while it is distributed and consumed in various ways

● Data governance in one place, secure and audit data everywhere

● Next-Gen Data Catalog is the Core in New Open Data Architecture.


Gravitino Architecture

Functionality Layer

● Unified Processing

● Unified Governing

Interface Layer

● Unified REST API's

● Iceberg REST API's

Core with Object Model

● Metalake

● Catalogs

● Schemas

● Object Types: Table, Fileset, Model, Topic

Connection Layer

● Connections

Metadata Storage

● Supported Data Types (Bottom Layer):

● Tabular

● Files

● Models

● Message Queue


Process Tabular and Non-tabular data with Gravitino

Tabular data (via connectors)

● Engines: Spark

● Operations: Create, Load, Alter, Drop

● API: Unified Tabular API

Schema (struct)

● name: string

● comment: string

● properties: map<string, string>

Table (struct)

● name: string

● columns: Column[]

● partitioning: Transform[]

● distribution: Distribution

● sortOrder: SortOrder[]

● indexes: Index[]

Related Definitions

● Transform, Distribution, SortOrder, Index, Type


Non-tabular data

● Engines: Spark, PyTorch, Ray, TensorFlow

● Filesystems: Gravitino Virtual FileSystem, Python FileSystem

● Operations: Create, Load, Alter, Drop

● API: Unified Non-tabular API

Schema (struct)

● name: string

● comment: string

● Properties: map<string, string>

Fileset (struct)

● name: string

● storageLocation: string

● type: Type

Storage Locations

● S3, HDFS, ADLS, GCS


Scenarios

Lakehouse Federation

● Multi-clouds, multi-engines and multi-formats

● An open solution for Lakehouse Federation

Platform Capabilities

● Analytics

● Machine Learning

● 360° View

● App

Query/Language Tools

● SQL

● Python

● R

Core Functionality

● Gravitino Data Connector

● Federated Query over multi-cloud, multi-formats and multi-engines.


Make Data and AI team to work seamlessly

Roles

● Data Engineer

● Data Scientist

● AI Engineer

Use Scenario

● Efficient collaborations between Data Engineers and Data Scientists or AI engineers

● Data Scientists get an unified definition of metadata for heterogeneous data sources

● Data engineers use metadata to process data

● Unified metadata for multiple AI frameworks

● Unified security control

Core Technology

● Gravitino

External Factors

● Technology

● Communication

● ETL

● Internet of things

● Automation

● Networking

Data & Tools

[ 1 ] Data Ingestion:

● Spark

● HDFS Client

● S3 SDK

[ 2 ] Model Training:

● Tensorflow

● Pytorch

● Ray

● Gravitino Python lib

[ 3 ] Data Types:

● Structured Data

● Unstructured Data

Gravitino Features

● Gravitino IO (Data read & write)

● Gravitino ACL (Access Control)


Gravitino Next - metadata-driven action system

● Catalog service

● APIs: Unified REST API, Iceberg REST API

● Components: Catalog, Schema, Table, Fileset, Model, Topic

● Connections: Connectors to various data sources (databases, files)

Gravitino Next

● Catalog service

● APIs: Unified REST API, Iceberg REST API

● Components: Catalog, Schema, Table, Fileset, Model, Topic, Policy

● Job system items: Job

Systems Included

● Policy system

● Statistics system

● Job system

● Action framework

Action framework items

● TTL Action

● Compaction Action

● Clustering Action