A letter to shareholders, clients, traders, engineers, regulators, and the people entrusted with keeping automated markets under control
For most of the history of ➔ That division solved real ➔ Databricks describes a dif ➔ The opportunity is substan ➔ Lakehouse//RT eliminates a
For most of the history of electronic trading, institutions divided operational truth from analytical truth. PostgreSQL and other transactional databases accepted instructions, preserved workflow state, and committed individual changes. Warehouses and lakes reconstructed those changes later for risk, surveillance, finance, and research. Between the two sat change-data-capture services, message buses, polling jobs, reconciliation processes, serving databases, caches, and an expanding collection of operational dependencies.
That division solved real engineering problems, but it also introduced a dangerous interval between what the trading robot had done and what the institution understood. An order could be committed in the operational database while the risk view remained behind. An intervention could be recorded while the supervisor dashboard still showed the prior state. A model could be operating from one position, the order gateway from another, and the post-trade system from a third. Every copy acquired its own clock, owner, failure mode, and definition of truth.
Databricks describes a different direction as Lake Transactional and Analytical Processing, or LTAP. Lakebase provides a managed, PostgreSQL-compatible transactional environment integrated with the Databricks platform. Lakehouse Real-Time, also written Lakehouse//RT, provides a low-latency and high-concurrency analytical serving tier over governed Delta Lake and Apache Iceberg tables. The Reyden engine powers this real-time query path. Unity Catalog supplies common governance. Lakeflow and Structured Streaming move and transform data when explicit propagation remains necessary. Databricks Apps and approved operational services expose those capabilities to users and machines.
The opportunity is substantial, but the language must remain exact. Compute-storage separation does not abolish data movement between every transactional and analytical representation. A row written to PostgreSQL is not automatically the same physical object as a Delta Lake or Apache Iceberg row merely because both systems use cloud object storage. Lakebase is built from stateless PostgreSQL compute, safekeepers, pageservers, and durable object storage. Databricks provides a governed change-data path to store PostgreSQL changes as Delta history and mechanisms to synchronize Unity Catalog data into Lakebase. These integrations reduce custom engineering and eliminate many external polling and copy pipelines, but they remain explicit data contracts with measurable lag, ordering, recovery, and cost.
Lakehouse//RT eliminates a different copy. It is designed to query governed Delta Lake and Apache Iceberg tables directly at low latency, avoiding the traditional need to export lakehouse data into a separate proprietary serving database. This is where the architectural claim is strongest. The institution can serve selective operational analytics to applications, dashboards, users, and agents without creating another permission system or another continuously synchronized analytical copy.
For an onsite trading robot, these distinctions are not academic. A robot can place thousands of orders before a vague architecture diagram is corrected. The low-latency execution path, pre-trade limits, exchange sessions, and kill switch must remain deterministic, locally resilient, and independently operable. LTAP should become the governed institutional nervous system around that path. It should preserve transactional state, provide fresh analytical context, support supervision, and reconstruct every consequential action. It should not become a marketing excuse to put a remote analytical database into the exchange gateway’s critical loop.
I approach this subject from the background of working in a music production house and performing live with a keyboard before leading a proprietary firm’s DevOps field-engineering team responsible for installing onsite trading robots. Live music teaches a useful truth. The audience hears one performance, but the performance depends on separate systems for sound, timing, monitoring, power, signal routing, and recovery. The keyboardist cannot wait for the studio to diagnose a failed monitor. The show needs local control, a clear cue, and a fallback. Automated trading is the same. The institution needs a unified score, but the robot also needs a local metronome and a physical stop.
The raised bar is therefore not a single database. It is a single accountable operating model across several latency and authority domains.