Vertex Macro｜Financial Cloud Cloud · AWS Amarathon 2025
Vertex Macro｜Amarathon 2025 Recap 28：A Modern Unified Metadata Architecture: New Approaches to Breaking Down Data Silos
A Brief History to Un-silo the Data
● LATE 1980'S: Data Warehouse
● 2011: Data Lake
● 2020: Lakehouse
Goal
● To achieve SSOT (Single Source of Truth)
● Full management of data
● Get rid of risks, such as data leak, compliance for a data-driven business.
● New Data Silos in Clouds & Regions
Nobody like vendor “lock-in”
● If data is deployed with different cloud vendors:
● Hard to Process together
● Expensive to Move
Nobody like geo-distributed data,
● But data goes with business to become international:
● Regulation requirement
● Cost for cross-ocean transfer
● More than "Data Access"
Data you see
● Technical & Business Data
● Legal Hold Data
Metadata you overlook
● 3rd Party Data
● PII & PI Data
● Credentials
● IP Data
Data Management Functions
● Data Connect: Connect to the Data That Matters Most.
● Data Right Automation: Automate end-to-end data rights requests and reporting.
● Metadata Enrichment: Enrich technical metadata with business and operational metadata for full visibility.
● Data Discovery: Automatically find, classify, and map all of your data - everywhere.
● Data Classification: Automatically classify more types of data in more places.
● Data Lifecycle Management: Simplify and automate data lifecycle management from collection to destruction.
What is Gravitino
● Next-gen unified data catalog for Data/AI
Integrations
● Trino
● Spark
● Flink
● Doris
● ClickHouse
● PyTorch
● TensorFlow
Metadata Lake Using Gravitino Components
● Hive Metastore
● Built-in Catalog
● Schema Registry
● Fileset Management
● Model Catalog
Data Sources
● Hadoop Data Lake
● Data Warehouse
● Streaming Processing
● Unstructured Data
● Machine Learning
Problems to solve
● Have a "Big Picture" of whole data
● Achieve SSOT of data while it is distributed and consumed in various ways
● Data governance in one place, secure and audit data everywhere
● Next-Gen Data Catalog is the Core in New Open Data Architecture.
Gravitino Architecture
Functionality Layer
● Unified Processing
● Unified Governing
Interface Layer
● Unified REST API's
● Iceberg REST API's
Core with Object Model
● Metalake
● Catalogs
● Schemas
● Object Types: Table, Fileset, Model, Topic
Connection Layer
● Connections
Metadata Storage
● Supported Data Types (Bottom Layer):
● Tabular
● Files
● Models
● Message Queue
Process Tabular and Non-tabular data with Gravitino
Tabular data (via connectors)
● Engines: Spark
● Operations: Create, Load, Alter, Drop
● API: Unified Tabular API
Schema (struct)
● name: string
● comment: string
● properties: map<string, string>
Table (struct)
● name: string
● columns: Column[]
● partitioning: Transform[]
● distribution: Distribution
● sortOrder: SortOrder[]
● indexes: Index[]
Related Definitions
● Transform, Distribution, SortOrder, Index, Type
Non-tabular data
● Engines: Spark, PyTorch, Ray, TensorFlow
● Filesystems: Gravitino Virtual FileSystem, Python FileSystem
● Operations: Create, Load, Alter, Drop
● API: Unified Non-tabular API
Schema (struct)
● name: string
● comment: string
● Properties: map<string, string>
Fileset (struct)
● name: string
● storageLocation: string
● type: Type
Storage Locations
● S3, HDFS, ADLS, GCS
Scenarios
Lakehouse Federation
● Multi-clouds, multi-engines and multi-formats
● An open solution for Lakehouse Federation
Platform Capabilities
● Analytics
● Machine Learning
● 360° View
● App
Query/Language Tools
● SQL
● Python
● R
Core Functionality
● Gravitino Data Connector
● Federated Query over multi-cloud, multi-formats and multi-engines.
Make Data and AI team to work seamlessly
Roles
● Data Engineer
● Data Scientist
● AI Engineer
Use Scenario
● Efficient collaborations between Data Engineers and Data Scientists or AI engineers
● Data Scientists get an unified definition of metadata for heterogeneous data sources
● Data engineers use metadata to process data
● Unified metadata for multiple AI frameworks
● Unified security control
Core Technology
● Gravitino
External Factors
● Technology
● Communication
● ETL
● Internet of things
● Automation
● Networking
Data & Tools
[ 1 ] Data Ingestion:
● Spark
● HDFS Client
● S3 SDK
[ 2 ] Model Training:
● Tensorflow
● Pytorch
● Ray
● Gravitino Python lib
[ 3 ] Data Types:
● Structured Data
● Unstructured Data
Gravitino Features
● Gravitino IO (Data read & write)
● Gravitino ACL (Access Control)
Gravitino Next - metadata-driven action system
● Catalog service
● APIs: Unified REST API, Iceberg REST API
● Components: Catalog, Schema, Table, Fileset, Model, Topic
● Connections: Connectors to various data sources (databases, files)
Gravitino Next
● Catalog service
● APIs: Unified REST API, Iceberg REST API
● Components: Catalog, Schema, Table, Fileset, Model, Topic, Policy
● Job system items: Job
Systems Included
● Policy system
● Statistics system
● Job system
● Action framework
Action framework items
● TTL Action
● Compaction Action
● Clustering Action