Databricks Apps for Governed Dark-Pool Oversight and Institutional AI Workflows

Agenda

  1. Speaker background
  2. Important dark-pool boundary
  3. Databricks Free Edition boundary
  4. Why Databricks Apps
  5. Step 1: Define the stakeholder experiences
  6. Step 2: Create the Databricks Free Edition workspace
  7. Step 3: Create the Unity Catalog data model
  8. Step 4: Create synthetic order and execution tables
  9. Step 5: Create case and annotation tables
  10. Step 6: Generate a safe synthetic dataset
  11. Step 7: Create execution-quality metrics
  12. Step 8: Define surveillance indicators
  13. Step 9: Apply Unity Catalog security
  14. Step 10: Create stakeholder-safe views
  15. Step 11: Create the Databricks App
  16. Step 12: Create the application files
  17. Step 13: Build the application connection layer
  18. Step 14: Build the role-aware navigation
  19. Step 15: Build the customer page
  20. Step 16: Build the trader page
  21. Step 17: Build the line-manager page
  22. Step 18: Build the surveillance and compliance pages
  23. Step 19: Add data correction and annotation forms
  24. Step 20: Add the governed knowledge assistant
  25. Step 21: Connect Model Serving safely
  26. Step 22: Build controlled what-if simulations
  27. Step 23: Add transactional write safety
  28. Step 24: Trigger data-quality and retraining workflows
  29. Step 25: Deploy and share the app
  30. Step 26: Monitor the application
  31. Step 27: Validate the stakeholder workflows
  32. Step 28: Move from prototype to an institutional deployment
  33. Lessons learned
  34. Important failure modes
  35. References

Key Takeaway

The speaker moved from in-house entertainment and media production into freelance application development for proprietary trading firms and small and medium-sized businesses using institutional-grade AI.

Speaker background

The speaker moved from in- ➔ Entertainment production a ➔ This guide creates a Datab ➔ The application supports c

The speaker moved from in-house entertainment and media production into freelance application development for proprietary trading firms and small and medium-sized businesses using institutional-grade AI.

Entertainment production and application development share a practical discipline. Both turn specialist work into an experience that other people can use. A data scientist can create a strong model, but the model produces little business value if a customer, trader, line manager, compliance officer, or operations analyst cannot use it safely through an understandable interface.

This guide creates a Databricks App for dark-pool oversight. The application does not operate a dark pool, route an order, reveal hidden liquidity, or help a user exploit non-public order information. It provides a governed control surface for customers, traders, proprietary-firm line managers, compliance, surveillance, operations, risk, and audit stakeholders.

The application supports controlled data access, participant inquiries, trading-quality review, venue analysis, alert triage, exception annotation, evidence preservation, and human-approved case management.

Important dark-pool boundary

A dark pool is generally a ➔ The prototype must not exp ➔ For a United States NMS St ➔ A firm operating in Hong K

A dark pool is generally an alternative trading system that does not publish its order book before execution. Dark pools are commonly used by institutional investors seeking to execute large orders with less public market impact. Executed trades remain subject to the applicable reporting framework.

The prototype must not expose another participant's unexecuted order, confidential trading intent, subscriber identity, hidden order size, proprietary matching logic, or information that could facilitate front-running, spoofing, wash trading, market manipulation, discriminatory access, or misuse of material non-public information.

For a United States NMS Stock ATS, Regulation ATS and Form ATS-N obligations can apply. The operating broker-dealer must maintain written safeguards and procedures protecting subscribers' confidential trading information. FINRA publishes delayed OTC activity reported by ATSs and member firms, but this delayed transparency data is not a live execution feed.

A firm operating in Hong Kong, Singapore, the European Union, the United Kingdom, or another jurisdiction must map the application to the relevant licensing, market-structure, transaction-reporting, best-execution, surveillance, recordkeeping, privacy, outsourcing, and cybersecurity requirements. This guide is a technical prototype and not a legal determination that a firm may operate an ATS, crossing network, systematic internaliser, or other non-displayed venue.

Databricks Free Edition boundary

Databricks Free Edition is ➔ Free Edition supports up t ➔ Use synthetic participants ➔ The Free Edition prototype

Databricks Free Edition is a perpetual, no-cost, serverless-only cloud environment for learning and prototyping. It provides no production service-level agreement and is governed by fair-use limits.

Free Edition supports up to three Databricks Apps per account. An app runs for up to 24 hours after it is started, updated, or redeployed, and then stops automatically. It can be restarted. Free Edition also includes one SQL warehouse limited to 2X-Small, one AI Search endpoint limited to one search unit, and restricted model-serving capabilities.

Use synthetic participants, accounts, parent orders, child orders, executions, market data, inquiries, alerts, policies, and annotations. Do not upload live subscriber orders, customer identities, production credentials, confidential venue logic, client allocations, real surveillance cases, or restricted market data.

The Free Edition prototype demonstrates application behavior and governance. It must not sit in the path of a production order, cancellation, execution, kill switch, trade report, allocation, or settlement instruction.

Why Databricks Apps

Traditional data applicati ➔ Databricks Apps runs conta ➔ Databricks handles applica ➔ Supported application patt ➔ The application reads gove

Traditional data applications often export data to an external database, deploy a separate web server, configure containers or virtual machines, establish network routes, implement authentication, manage load balancing, and duplicate permissions in another platform.

Databricks Apps runs containerized applications on Databricks serverless infrastructure. Each app has an isolated runtime, a unique URL, and a dedicated service principal. Applications can use Databricks SQL, Unity Catalog, AI Search, Lakebase, Model Serving, and other approved platform resources.

Databricks handles application hosting and platform authentication. Developers still need to design authorization, validate inputs, protect secrets, manage dependencies, monitor logs, review generated code, and control every write operation.

Supported application patterns include Streamlit, Dash, Gradio, Flask, FastAPI, and Node.js frameworks. This guide uses Streamlit because it supports a rapid controlled prototype with forms, data tables, filters, charts, and session-aware interaction.

The application reads governed Delta data through Databricks SQL and writes only to narrow action, annotation, and case tables. It must not grant the user broad modification rights over raw order or execution records.

Step 1: Define the stakeholder experiences

Create one application wit ➔ The customer or subscriber ➔ Submitted inquiry status. ➔ Executed-trade quality sum ➔ Approved venue disclosures

Create one application with role-aware pages, or separate apps if the security boundaries require stronger isolation.

The customer or subscriber page should provide:

  • Submitted inquiry status.
  • Executed-trade quality summary.
  • Approved venue disclosures.
  • Complaint and review submission.
  • Evidence references available to that customer.
  • No visibility into another subscriber's activity.

The trader page should provide:

  • The trader's own parent orders and completed executions.
  • Fill rate, execution price, benchmark comparison, venue allocation, and timing summary.
  • Open surveillance requests requiring explanation.
  • Approved response and annotation forms.
  • No access to confidential counterparty identity or other desks' hidden orders.

The proprietary-firm line-manager page should provide:

  • Desk-level order and execution summaries.
  • Concentration, information-leakage indicators, excessive re-routing, abnormal cancellation, and venue-quality alerts.
  • Human approval workflow for reducing access, pausing a strategy, or escalating a case.
  • No direct order-entry or cancellation control in the Free Edition app.

The compliance and surveillance page should provide:

  • Alert triage.
  • Potential wash-trading, spoofing, information-leakage, restricted-list, and best-execution indicators.
  • Evidence lineage and case history.
  • Escalation and disposition forms.

The operations page should provide:

  • Trade-reporting exceptions.
  • Missing execution identifiers.
  • Allocation and booking breaks.
  • Duplicate and late reports.
  • Reconciliation status.

The audit page should provide read-only access to approved evidence, lineage, action history, model version, application version, and policy version.

Step 2: Create the Databricks Free Edition workspace

Create a Free Edition work ➔ dark_pool_control_apps_fre ➔ Create these workspace fol ➔ 00_environment ➔ 01_synthetic_data

Create a Free Edition workspace named:

dark_pool_control_apps_free_nonprod

Create these workspace folders:

  • 00_environment
  • 01_synthetic_data
  • 02_governance
  • 03_surveillance_features
  • 04_ai_search
  • 05_model_serving
  • 06_app_source
  • 07_evaluation
  • 08_audit

Create a notebook named 00_environment_check and run:

SELECT
  current_user() AS current_user,
  current_catalog() AS current_catalog,
  current_schema() AS current_schema,
  current_timestamp() AS checked_at;

Record that the environment is synthetic, non-production, serverless-only, quota-limited, and disconnected from all execution venues.

Step 3: Create the Unity Catalog data model

Create the catalog and sch ➔ If catalog creation is res ➔ Create an approved partici ➔ Create an account-to-desk

Create the catalog and schemas:

CREATE CATALOG IF NOT EXISTS dark_pool_lab;

CREATE SCHEMA IF NOT EXISTS dark_pool_lab.raw;
CREATE SCHEMA IF NOT EXISTS dark_pool_lab.silver;
CREATE SCHEMA IF NOT EXISTS dark_pool_lab.gold;
CREATE SCHEMA IF NOT EXISTS dark_pool_lab.reference;
CREATE SCHEMA IF NOT EXISTS dark_pool_lab.actions;
CREATE SCHEMA IF NOT EXISTS dark_pool_lab.knowledge;
CREATE SCHEMA IF NOT EXISTS dark_pool_lab.governance;

If catalog creation is restricted, create the schemas in the default catalog.

Create an approved participant table:

CREATE TABLE IF NOT EXISTS dark_pool_lab.reference.participants (
  participant_id STRING,
  participant_name STRING,
  participant_type STRING,
  jurisdiction STRING,
  customer_segment STRING,
  onboarding_status STRING,
  permitted_products ARRAY<STRING>,
  permitted_venues ARRAY<STRING>,
  effective_from TIMESTAMP,
  effective_to TIMESTAMP,
  is_synthetic BOOLEAN
) USING DELTA;

Create an account-to-desk map:

CREATE TABLE IF NOT EXISTS dark_pool_lab.reference.accounts (
  account_id STRING,
  participant_id STRING,
  firm_id STRING,
  desk_id STRING,
  trader_id STRING,
  strategy_id STRING,
  jurisdiction STRING,
  approved BOOLEAN,
  effective_from TIMESTAMP,
  effective_to TIMESTAMP,
  is_synthetic BOOLEAN
) USING DELTA;

Step 4: Create synthetic order and execution tables

Create parent orders: ➔ Create venue-routed child ➔ Create executions: ➔ Do not store an opposing s

Create parent orders:

CREATE TABLE IF NOT EXISTS dark_pool_lab.raw.parent_orders (
  parent_order_id STRING,
  account_id STRING,
  trader_id STRING,
  created_at TIMESTAMP,
  instrument_id STRING,
  side STRING,
  order_quantity DECIMAL(24,8),
  limit_price DECIMAL(24,10),
  benchmark_type STRING,
  strategy_type STRING,
  restricted_flag BOOLEAN,
  is_synthetic BOOLEAN,
  ingested_at TIMESTAMP
) USING DELTA;

Create venue-routed child orders:

CREATE TABLE IF NOT EXISTS dark_pool_lab.raw.child_orders (
  child_order_id STRING,
  parent_order_id STRING,
  submitted_at TIMESTAMP,
  venue_id STRING,
  routing_algorithm STRING,
  side STRING,
  quantity DECIMAL(24,8),
  limit_price DECIMAL(24,10),
  minimum_quantity DECIMAL(24,8),
  time_in_force STRING,
  order_status STRING,
  cancelled_at TIMESTAMP,
  is_synthetic BOOLEAN,
  ingested_at TIMESTAMP
) USING DELTA;

Create executions:

CREATE TABLE IF NOT EXISTS dark_pool_lab.raw.executions (
  execution_id STRING,
  child_order_id STRING,
  parent_order_id STRING,
  execution_time TIMESTAMP,
  venue_id STRING,
  instrument_id STRING,
  side STRING,
  executed_quantity DECIMAL(24,8),
  executed_price DECIMAL(24,10),
  reference_bid DECIMAL(24,10),
  reference_ask DECIMAL(24,10),
  reference_mid DECIMAL(24,10),
  reporting_status STRING,
  is_synthetic BOOLEAN,
  ingested_at TIMESTAMP
) USING DELTA;

Do not store an opposing subscriber identity in the application-facing execution table. Where regulatory or control processing requires counterparty details, keep them in a separately restricted schema.

Step 5: Create case and annotation tables

Create a surveillance-aler ➔ Create an append-only case ➔ Create correction requests ➔ The app should append a co

Create a surveillance-alert table:

CREATE TABLE IF NOT EXISTS dark_pool_lab.gold.surveillance_alerts (
  alert_id STRING,
  detected_at TIMESTAMP,
  alert_type STRING,
  severity STRING,
  participant_id STRING,
  account_id STRING,
  trader_id STRING,
  instrument_id STRING,
  parent_order_id STRING,
  measured_value DECIMAL(24,8),
  threshold_value DECIMAL(24,8),
  evidence_reference STRING,
  model_version STRING,
  alert_status STRING,
  is_synthetic BOOLEAN
) USING DELTA;

Create an append-only case event ledger:

CREATE TABLE IF NOT EXISTS dark_pool_lab.actions.case_events (
  event_id STRING,
  case_id STRING,
  alert_id STRING,
  event_time TIMESTAMP,
  actor_identity STRING,
  actor_role STRING,
  action_type STRING,
  action_reason STRING,
  previous_state STRING,
  requested_state STRING,
  evidence_reference STRING,
  application_version STRING,
  is_synthetic BOOLEAN,
  recorded_at TIMESTAMP
) USING DELTA;

Create correction requests rather than allowing users to overwrite source records:

CREATE TABLE IF NOT EXISTS dark_pool_lab.actions.data_correction_requests (
  request_id STRING,
  source_table STRING,
  source_record_id STRING,
  requested_by STRING,
  requested_at TIMESTAMP,
  issue_type STRING,
  proposed_value STRING,
  reason STRING,
  evidence_reference STRING,
  approval_status STRING,
  approved_by STRING,
  approved_at TIMESTAMP,
  is_synthetic BOOLEAN
) USING DELTA;

The app should append a correction request. A controlled pipeline validates and applies an approved correction to a curated table. It must not change the immutable raw event.

Step 6: Generate a safe synthetic dataset

Generate at least: ➔ 20 synthetic participants. ➔ 100 accounts. ➔ 50 traders. ➔ 10,000 parent orders.

Generate at least:

  • 20 synthetic participants.
  • 100 accounts.
  • 50 traders.
  • 10,000 parent orders.
  • 30,000 child orders.
  • 15,000 executions.
  • 500 surveillance alerts.
  • 100 policy documents.

Include normal and abnormal scenarios:

  • Large institutional parent order split across venues.
  • Low fill rate.
  • Price improvement at midpoint.
  • Repeated rapid cancellations.
  • Excessive reuse of one venue.
  • Execution after apparent information leakage.
  • Potential self-match across commonly controlled accounts.
  • Late trade report.
  • Duplicate execution record.
  • Trader explanation and compliance disposition.

Mark every record with is_synthetic = true.

Do not create a simulation that predicts or reveals a hidden counterparty's current order. Use historical synthetic events to test controls.

Step 7: Create execution-quality metrics

Create a curated execution ➔ Create parent-order fill m ➔ Do not present midpoint im

Create a curated execution-quality view:

CREATE OR REPLACE VIEW dark_pool_lab.gold.execution_quality AS
SELECT
  e.execution_id,
  e.parent_order_id,
  e.execution_time,
  e.venue_id,
  e.instrument_id,
  e.side,
  e.executed_quantity,
  e.executed_price,
  e.reference_mid,
  CASE
    WHEN e.side = 'BUY'
      THEN (e.reference_mid - e.executed_price)
           / NULLIF(e.reference_mid, 0) * 10000
    WHEN e.side = 'SELL'
      THEN (e.executed_price - e.reference_mid)
           / NULLIF(e.reference_mid, 0) * 10000
  END AS midpoint_improvement_bps,
  e.reporting_status,
  e.is_synthetic
FROM dark_pool_lab.raw.executions e;

Create parent-order fill metrics:

CREATE OR REPLACE VIEW dark_pool_lab.gold.parent_order_summary AS
SELECT
  p.parent_order_id,
  p.account_id,
  p.trader_id,
  p.instrument_id,
  p.side,
  p.order_quantity,
  COALESCE(SUM(e.executed_quantity), 0) AS executed_quantity,
  COALESCE(SUM(e.executed_quantity), 0)
    / NULLIF(p.order_quantity, 0) AS fill_rate,
  SUM(e.executed_quantity * e.executed_price)
    / NULLIF(SUM(e.executed_quantity), 0) AS execution_vwap,
  COUNT(DISTINCT e.venue_id) AS executing_venue_count,
  p.is_synthetic
FROM dark_pool_lab.raw.parent_orders p
LEFT JOIN dark_pool_lab.raw.executions e
  ON p.parent_order_id = e.parent_order_id
GROUP BY
  p.parent_order_id,
  p.account_id,
  p.trader_id,
  p.instrument_id,
  p.side,
  p.order_quantity,
  p.is_synthetic;

Do not present midpoint improvement as complete best-execution evidence. Also review opportunity cost, adverse selection, fill probability, latency, market impact, fees, information leakage, routing alternatives, and order instructions.

Step 8: Define surveillance indicators

Use surveillance indicator ➔ Potential self-match or wa ➔ Common beneficial owner or ➔ Opposite-side trading in t ➔ Close execution times.

Use surveillance indicators to prioritize human review, not to make automatic misconduct findings.

Potential self-match or wash-trading indicator:

  • Common beneficial owner or firm control.
  • Opposite-side trading in the same or economically equivalent instrument.
  • Close execution times.
  • Small net risk change.
  • Repeated pattern.

Potential spoofing indicator:

  • Large non-bona-fide orders.
  • High cancellation rate.
  • Orders placed and cancelled around opposite-side fills.
  • Repeated market influence pattern.

Potential information-leakage indicator:

  • Material unfavorable price movement after routing to a venue.
  • Repeated venue-specific adverse movement.
  • Execution quality deterioration by order size or strategy.
  • Short time between routing and suspicious external activity.

Potential venue-control indicator:

  • Unapproved participant category.
  • Inconsistent access treatment.
  • Execution outside approved price or matching rules.
  • Missing report.
  • Confidential-data access outside authorized operations.

Every alert must retain input timestamps, feature values, thresholds, model or rule version, and evidence lineage.

Step 9: Apply Unity Catalog security

Create groups: ➔ dark_pool_customers ➔ dark_pool_traders ➔ prop_firm_line_managers ➔ market_surveillance

Create groups:

  • dark_pool_customers
  • dark_pool_traders
  • prop_firm_line_managers
  • market_surveillance
  • trade_operations
  • compliance_officers
  • independent_audit
  • app_developers

Grant the app access only to required views and action tables.

Use user authorization for pages requiring individual row-level access. Databricks Apps supports OAuth-based user authorization so the application can act with the interacting user's identity and permissions.

Use app authorization for shared background work such as loading approved configuration or writing application telemetry. Every app receives a dedicated service principal. Its permissions must be granted explicitly.

Do not rely on app authorization alone when different users must see different data. Under app authorization, users share the app service principal's permissions.

Apply row filters that restrict participants to their records, traders to their own activity, and line managers to their desks.

Apply column masks to customer names, account identifiers, employee identities, and restricted surveillance fields where users need only tokenized identifiers.

Treat the app's permission and the user's permission as two independent boundaries. Grant neither broad catalog ownership nor unrestricted MODIFY access.

Step 10: Create stakeholder-safe views

Create a customer view tha ➔ Create a trader view conta ➔ Create a line-manager view ➔ Create a compliance view w ➔ Create an audit view conta

Create a customer view that excludes confidential venue and subscriber details:

CREATE OR REPLACE VIEW dark_pool_lab.gold.customer_execution_view AS
SELECT
  p.parent_order_id,
  p.account_id,
  p.instrument_id,
  p.side,
  p.order_quantity,
  s.executed_quantity,
  s.fill_rate,
  s.execution_vwap,
  s.executing_venue_count,
  p.created_at,
  p.is_synthetic
FROM dark_pool_lab.raw.parent_orders p
JOIN dark_pool_lab.gold.parent_order_summary s
  ON p.parent_order_id = s.parent_order_id;

Create a trader view containing only the trader's own orders and alerts.

Create a line-manager view aggregated by desk, strategy, venue, and risk classification.

Create a compliance view with detailed evidence references but keep legal-privilege and investigation notes in a separately restricted table.

Create an audit view containing immutable event history, model version, app version, source lineage, and approval references.

Step 11: Create the Databricks App

Open Apps and select Creat ➔ Name the app: ➔ dark-pool-control-center ➔ Choose the Streamlit templ ➔ Copy the app's dedicated s

Open Apps and select Create App.

Name the app:

dark-pool-control-center

Choose the Streamlit template.

Copy the app's dedicated service-principal identity from the Authorization page.

Grant the app service principal:

  • CAN USE on the Free Edition SQL warehouse.
  • USE CATALOG on dark_pool_lab.
  • USE SCHEMA on required schemas.
  • SELECT on approved views.
  • MODIFY only on case_events and data_correction_requests if app authorization is used for controlled writes.

Do not grant MODIFY on raw orders, executions, participants, accounts, alerts, or policies.

Step 12: Create the application files

Create requirements.txt: ➔ Create app.yaml: ➔ Store the SQL warehouse re ➔ Use environment variables

Create requirements.txt:

streamlit
pandas
databricks-sdk
databricks-sql-connector

Create app.yaml:

command:
  - streamlit
  - run
  - app.py
  - --server.port
  - "8000"
  - --server.address
  - "0.0.0.0"

Store the SQL warehouse resource in the app configuration. Do not ask business users to type a warehouse HTTP path or table name.

Use environment variables supplied through the app resource configuration rather than hard-coded identifiers.

Step 13: Build the application connection layer

Create app.py with app-sco ➔ Use parameterized queries. ➔ Use an allowlist for page

Create app.py with app-scoped authentication:

import os
import uuid
import pandas as pd
import streamlit as st
from databricks import sql
from databricks.sdk.core import Config

cfg = Config()


def get_connection():
    host = cfg.host.replace("https://", "").replace("http://", "")
    return sql.connect(
        server_hostname=host,
        http_path=os.environ["DATABRICKS_WAREHOUSE_HTTP_PATH"],
        credentials_provider=lambda: cfg.authenticate,
        _use_arrow_native_complex_types=False,
    )


def run_query(statement, parameters=None):
    with get_connection() as connection:
        with connection.cursor() as cursor:
            cursor.execute(statement, parameters or [])
            if cursor.description:
                return cursor.fetchall_arrow().to_pandas()
    return pd.DataFrame()

Use parameterized queries. Never concatenate a user-provided table name, identity, date expression, status, or free text directly into SQL.

Use an allowlist for page names, sort columns, filter columns, and case actions.

Step 14: Build the role-aware navigation

Read the authenticated use ➔ Map the user's group membe ➔ Show only pages authorized ➔ Customer Summary. ➔ Trader Review.

Read the authenticated user context through the supported Databricks Apps user-authorization pattern.

Map the user's group membership to approved pages.

Show only pages authorized for the identity:

  • Customer Summary.
  • Trader Review.
  • Line Manager Control.
  • Surveillance Triage.
  • Operations Exceptions.
  • Compliance Case.
  • Audit Evidence.

Do not hide a page only in the browser. Every query must remain protected by Unity Catalog permissions, row filters, masks, and user authorization.

When the identity cannot be resolved or there is no authorized role, deny access and record the denied event without exposing data.

Step 15: Build the customer page

Allow an authorized custom ➔ Display fill rate, executi ➔ Do not display another par ➔ Provide a complaint form w ➔ Parent-order ID.

Allow an authorized customer to filter by date, instrument, and parent-order ID.

Display fill rate, execution VWAP, benchmark, price improvement, reporting status, and complaint status.

Do not display another participant's identity, order, size, or interest.

Provide a complaint form with:

  • Parent-order ID.
  • Issue category.
  • Customer description.
  • Requested review.
  • Optional approved evidence reference.

Write the submission to a customer-inquiry table. Record the authenticated user, timestamp, application version, and request ID.

Require confirmation before submission. After submission, show the generated request ID without revealing internal surveillance notes.

Step 16: Build the trader page

Show the trader's own pare ➔ Allow the trader to add an ➔ Example explanation fields ➔ Strategy context. ➔ Market condition.

Show the trader's own parent orders, fills, route summary, execution-quality metrics, and assigned surveillance requests.

Allow the trader to add an explanation to an alert. Do not allow the trader to close, delete, or downgrade the alert.

Example explanation fields:

  • Strategy context.
  • Market condition.
  • Client or portfolio instruction reference.
  • Algorithm configuration version.
  • Approval reference.
  • Evidence URI.

Append the explanation as an event. Preserve the original alert and every prior explanation.

Step 17: Build the line-manager page

Show desk-level metrics: ➔ Parent-order count. ➔ Fill rate. ➔ Venue concentration. ➔ Adverse-selection estimate

Show desk-level metrics:

  • Parent-order count.
  • Fill rate.
  • Venue concentration.
  • Adverse-selection estimate.
  • Cancellation rate.
  • Unresolved critical alerts.
  • Missing report count.
  • Pending data corrections.
  • Cases awaiting compliance review.

Allow the line manager to request these controlled actions:

  • Request trader explanation.
  • Freeze new dark-pool routing for a strategy.
  • Request venue removal from the approved route set.
  • Escalate to compliance.
  • Request production kill switch.
  • Request manual order review.

These are workflow requests only. The production order or risk platform must perform and confirm any actual control action.

Record requested_state separately from confirmed_state.

Step 18: Build the surveillance and compliance pages

Display the rule or model ➔ Allow a surveillance analy ➔ FALSE_POSITIVE ➔ DATA_QUALITY_ISSUE ➔ REQUIRES_TRADER_EXPLANATIO

Display the rule or model that generated each alert, input features, threshold, evidence records, and previous case events.

Allow a surveillance analyst to select:

  • FALSE_POSITIVE
  • DATA_QUALITY_ISSUE
  • REQUIRES_TRADER_EXPLANATION
  • REQUIRES_OPERATIONS_REVIEW
  • REQUIRES_COMPLIANCE_REVIEW
  • ESCALATE_FORMAL_INVESTIGATION
  • CLOSE_WITH_RATIONALE

Require a written rationale for every disposition.

Do not let an AI model autonomously close a market-abuse alert. AI may summarize evidence and identify missing documents, but a qualified human must decide the disposition.

Separate privileged legal notes from normal operational comments.

Step 19: Add data correction and annotation forms

Provide a correction form ➔ The form should include: ➔ Source table from an allow ➔ Source record ID. ➔ Issue type.

Provide a correction form for an authorized operations analyst.

The form should include:

  • Source table from an allowlist.
  • Source record ID.
  • Issue type.
  • Current displayed value.
  • Proposed value.
  • Reason.
  • Evidence reference.

Insert the request into data_correction_requests.

A downstream validation process should verify the record, data type, source ownership, evidence, and approval. It should write an approved correction into the curated layer or amend the source through its official workflow.

Never overwrite the raw order or execution event from the web form.

For training-data annotation, write labels to a separate table with annotator identity, label taxonomy version, timestamp, record reference, and review status. Retraining should start only after minimum sample, quality, and approval conditions are met.

Step 20: Add the governed knowledge assistant

Load approved venue disclo ➔ Create a Databricks AI Sea ➔ The source can be a Delta ➔ Use hybrid retrieval becau ➔ Apply metadata filters for

Load approved venue disclosures, participant procedures, order-handling policies, surveillance standards, escalation playbooks, support FAQs, and application instructions into a Delta table.

Create a Databricks AI Search index where the feature is available. Free Edition supports one AI Search endpoint with one search unit and does not support Direct Vector Access.

The source can be a Delta table, streaming table, materialized view, or supported managed Iceberg table. A standard index requires change data feed behavior; row-tracked tables provide compatible change tracking automatically.

Use hybrid retrieval because exact identifiers, rule names, field names, and policy-version numbers are often important alongside semantic similarity.

Apply metadata filters for jurisdiction, stakeholder role, document status, effective date, and confidentiality.

The assistant should answer only from approved retrieved evidence and cite the document title, version, effective date, and source reference.

Do not index confidential subscriber-order data into a general knowledge assistant.

Step 21: Connect Model Serving safely

Use Databricks Model Servi ➔ Model Serving provides a g ➔ Use small deterministic mo ➔ Do not ask a language mode ➔ Send only the minimum auth

Use Databricks Model Serving or an approved foundation-model service where available.

Model Serving provides a governed REST interface for custom MLflow models, Databricks-hosted foundation models, external foundation models, and supported agents. Free Edition serving limits apply, and some models or endpoint types are unavailable.

Use small deterministic models or rules for numerical surveillance features. Use a language model for summarization, document retrieval, multilingual explanation, and draft case notes.

Do not ask a language model to infer hidden liquidity, predict a subscriber's unexecuted order, generate manipulative strategies, or make a final market-abuse determination.

Send only the minimum authorized context. Remove customer identifiers and confidential order details not required for the task.

Store model name, endpoint, prompt-template version, retrieval references, response timestamp, and reviewer decision.

Step 22: Build controlled what-if simulations

Create a scenario page tha ➔ Parent-order size. ➔ Participation-rate cap. ➔ Minimum execution quantity ➔ Venue allocation percentag

Create a scenario page that allows approved users to change research parameters such as:

  • Parent-order size.
  • Participation-rate cap.
  • Minimum execution quantity.
  • Venue allocation percentage.
  • Maximum acceptable information-leakage score.
  • Maximum venue concentration.
  • Benchmark selection.
  • Market volatility assumption.

The simulator can estimate fill probability, expected cost, market impact, and adverse selection using synthetic or approved historical models.

Label every result as a simulation. It must not route or stage an order.

Clamp inputs to approved ranges. Reject negative quantities, unsupported instruments, unknown venues, and values exceeding the research limit.

Show the model version, data period, uncertainty, and limitations. A single numerical prediction must not be presented as guaranteed execution quality.

Step 23: Add transactional write safety

Use parameterized INSERT s ➔ Generate a unique event ID ➔ Use an idempotency key for ➔ Validate allowed state tra ➔ Do not implement client-si

Use parameterized INSERT statements for application writes.

Generate a unique event ID and include the authenticated actor, app version, and UTC timestamp.

Use an idempotency key for form submissions so a browser retry does not duplicate a case event.

Validate allowed state transitions. For example, a closed case cannot move directly to NO_ALERT; it must be reopened through an authorized event.

Do not implement client-side-only validation. Recheck every value on the server.

Separate write privileges by table. The app may append case events but must not update or delete the alert history.

For higher-frequency operational transactions in a production design, evaluate Lakebase as the application transaction store while syncing governed history to the lakehouse. Free Edition includes one Lakebase project with scale-to-zero compute, but it remains a prototype environment.

Step 24: Trigger data-quality and retraining workflows

After an approved correcti ➔ The job should: ➔ Validate the correction or ➔ Update the curated dataset ➔ Run data-quality checks.

After an approved correction or annotation reaches the required state, trigger a Databricks job rather than retraining inside the web request.

The job should:

  • Validate the correction or label.
  • Update the curated dataset.
  • Run data-quality checks.
  • Recompute affected features.
  • Evaluate model drift and label balance.
  • Train a candidate model only when the minimum data threshold is reached.
  • Compare the candidate against the current model.
  • Require human approval before deployment.

Free Edition allows a limited number of concurrent job tasks. Keep the prototype workflow small and sequential.

Do not retrain automatically from one disputed annotation. Require dual review for labels related to suspected manipulation or misconduct.

Step 25: Deploy and share the app

Deploy the application fro ➔ Databricks automatically c ➔ Grant CAN USE only to appr ➔ Test access with one ident ➔ Confirm that:

Deploy the application from the Apps interface.

Databricks automatically containerizes and runs the application in its serverless environment and assigns a unique app URL.

Grant CAN USE only to approved groups.

Test access with one identity from each role.

Confirm that:

  • A customer cannot see another customer.
  • A trader cannot see another desk.
  • A line manager sees only the approved desk.
  • Operations cannot see privileged compliance notes.
  • An auditor cannot change a case.
  • An app developer cannot access production data by default.

Free Edition apps stop after the documented runtime window. Restart the app when needed. Do not rely on it for continuous production surveillance.

Step 26: Monitor the application

Review application logs, a ➔ Create alerts for: ➔ Repeated denied access. ➔ Unusual export volume. ➔ High query volume from one

Review application logs, audit events, SQL history, failed authentication, denied authorization, data writes, model calls, and app telemetry.

Create alerts for:

  • Repeated denied access.
  • Unusual export volume.
  • High query volume from one user.
  • Correction requests against restricted tables.
  • Repeated failed case transitions.
  • Unknown model endpoint.
  • Missing evidence references.
  • Unusually large model prompts.
  • Application exceptions containing sensitive data.

Do not log raw confidential order payloads, secrets, access tokens, or unrestricted model prompts.

Version application code in Git and use a separate deployment identity for production.

Step 27: Validate the stakeholder workflows

Create a synthetic custome ➔ Create two traders on diff ➔ Create a line manager and ➔ Create a potential self-ma ➔ Create a missing-report ex

Create a synthetic customer with two parent orders and confirm that only those orders appear.

Create two traders on different desks and confirm that each sees only their own records.

Create a line manager and confirm access to desk aggregates without revealing another desk's activity.

Create a potential self-match alert and confirm that the trader can explain it but cannot close it.

Create a missing-report exception and confirm that operations can submit a correction request but cannot overwrite the execution.

Create a restricted surveillance case and confirm that ordinary customer-support users cannot retrieve it through AI Search.

Create a duplicated form submission and confirm that the idempotency key prevents a duplicate case event.

Create an app service principal with no raw-table write permission and verify that a direct raw update fails.

Step 28: Move from prototype to an institutional deployment

Create separate developmen ➔ Implement identity federat ➔ Use licensed market data a ➔ Integrate the app with the ➔ Keep deterministic pre-tra

Create separate development, test, surveillance-test, and production workspaces.

Implement identity federation, single sign-on, group lifecycle, service principals, private networking, secrets management, audit delivery, customer-managed encryption where required, and regional data-residency controls.

Use licensed market data and documented retention periods.

Integrate the app with the production order, execution, reporting, compliance, and case-management platforms through reviewed APIs and asynchronous workflows.

Keep deterministic pre-trade limits, order controls, and kill switches in approved execution infrastructure.

Perform legal mapping for every venue and jurisdiction. For a U.S. NMS Stock ATS, review Regulation ATS, Form ATS-N disclosures, subscriber-information safeguards, trade reporting, books and records, fair access where applicable, and broker-dealer supervisory obligations.

Benchmark concurrency, SQL latency, write contention, model latency, AI Search latency, and recovery behavior under realistic stakeholder load.

Complete independent application-security testing, threat modeling, code scanning, penetration testing, model-risk review, surveillance-model validation, privacy review, and business-continuity testing.

Lessons learned

Databricks Apps closes the ➔ The strongest design does ➔ A web form should not rewr ➔ A dark-pool control applic ➔ AI is useful for retrieval

Databricks Apps closes the operational gap between data science and business use. A governed model, SQL view, or detection rule can become an interactive application without exporting the entire dataset to a separate unmanaged web stack.

The strongest design does not give the app universal access. It combines user authorization, a narrow app identity, Unity Catalog policies, stakeholder-specific views, parameterized SQL, append-only action history, and external confirmation for consequential controls.

A web form should not rewrite immutable market events. It should record a correction request, annotation, or case action with identity and evidence.

A dark-pool control application should improve oversight without revealing the information that the dark venue is obligated to protect.

AI is useful for retrieval, summarization, explanation, multilingual assistance, and case preparation. Deterministic rules and validated models should calculate surveillance features. Qualified humans should decide legal, conduct, disciplinary, and market-access outcomes.

Important failure modes

An authorization failure o ➔ A hidden-client-order fail ➔ A false-control failure oc ➔ A raw-data overwrite failu ➔ A model-authority failure

An authorization failure occurs when every user inherits the app service principal's broad permissions. Use user authorization and Unity Catalog controls for individual data boundaries.

A hidden-client-order failure occurs when confidential interest is exposed in a dashboard, log, model prompt, search index, or exported file. Use minimum necessary data and restricted schemas.

A false-control failure occurs when a workflow request is shown as a completed kill-switch or venue-disable action. Keep requested and confirmed state separate.

A raw-data overwrite failure occurs when a form edits the original order or execution. Preserve immutable source events and use correction workflows.

A model-authority failure occurs when an LLM closes a surveillance alert or accuses a trader. Require human disposition and evidence.

A retrieval-leak failure occurs when AI Search indexes confidential documents without role or jurisdiction filters. Separate indexes and enforce ACLs and source permissions.

A Free Edition availability failure occurs when the app stops after 24 hours or compute quota is exhausted. Treat the environment as a demonstration, not continuous oversight.

A zero-latency claim failure occurs when direct lakehouse access is described as literally instantaneous. Databricks eliminates a separate data-copy pipeline, but actual latency still includes SQL execution, warehouse startup, network, model inference, and application rendering.

References

Databricks Apps ➔ Databricks Apps key concep ➔ Streamlit application tuto ➔ Databricks Apps authorizat