AWS Re:cap 04: Serverless AR Game with Bedrock AgentCore

AWS Hong Kong Summit 2026 · Developer Lounge Recap · Article 4 of 6

AWS Magazine · Field Report from Victoria Harbour

---

The Scene: Central-District Swagger Meets Serverless Sorcery

In my six decades of tracking technology shifts, I have stood on hundreds of conference floors, but nothing quite matches the electric pulse of the AWS Hong Kong Summit 2026 Developer Lounge.

Perched against the sweeping backdrop of Victoria Harbour, the Developer Lounge was a striking collision of Hong Kong luxury and AWS high-tech craft. Picture this: venture capitalists and startup founders closing seven-figure deals over artisanal drip coffee, international speakers trading architecture diagrams, and private developer meetups humming with high-stakes venture talk. Money was moving in and out, collaborations were forming on the fly, and amidst the neon shimmer and Central-district energy, builders were gathered around screens to witness pure serverless sorcery.

The standout attraction of the lounge? A demo by Cyrus Wong that brought together luxury entertainment and cloud engineering: The Domain Expansion AR Game.

(Not bad! Your reaction is almost as fast as me grabbing a limited-edition bag in Shibuya!)

Strike a Domain Expansion hand sign in front of a camera, and the screen erupts in 3D particle effects while a real (or simulated) humanoid robot adopts a kung-fu stance—all narrated live by an AI commentator speaking sassy, fashion-obsessed local Cantonese.

Here is the deep, hands-on architectural breakdown of how this fan-made application was constructed using AWS Bedrock AgentCore, AWS CDK, and serverless orchestrations.

---

The Star of the Show: AI Commentator

At the heart of the experience is a sassy AI commentator. Powered by multimodal AI vision, she roasts players' hand gestures, clothes, postures, and messy rooms in real-time.

Multilingual Support & Vision Intelligence

● Languages: Local Cantonese, Taiwanese Mandarin, Japanese, and English.

● Personality: Central-district high fashion meets competitive AR flair.

● Model Stack: moonshotai.kimi-k2.5 running inside a custom ECR container on Bedrock, paired with AWS Polly for high-fidelity neural speech synthesis.

System Architecture Summary

---

Step-by-Step Hands-On Execution Flow

+-----------------------------------------------------------------------------------+
END-TO-END GESTURE TO SPEECH FLOW

1. [ Browser Client ] -- MediaPipe 60FPS --> [ API Gateway ]
2. [ API Gateway ] -- Cognito JWT --> [ Monolithic Lambda ]
3. [ Monolithic Lambda ] -- XML multimodal payload --> [ AgentCore Runtime ]
4. [ AgentCore Runtime ] --> [ AWS Polly ] neural speech
5. [ AWS Polly ] --> [ S3 Audio Bucket ] --> browser playback
+-----------------------------------------------------------------------------------+

Step A: Client-Side Gesture Detection & API Signalling

The browser runs hand_tracker.js using Google’s MediaPipe Hands to analyze 21 3D landmarks at 60 FPS. When a Domain Expansion gesture is verified, battle.js sends a POST request with a Cognito Bearer JWT to Amazon API Gateway:

// Cognito JWT + recognized gesture → API Gateway → Polly audio URL
const response = await fetch("/api/trigger-technique", {
  method: "POST",
  headers: { Authorization: `Bearer ${await getCognitoAccessToken()}` },
  body: JSON.stringify({ technique: techniqueName }),
});
playCommentaryAudio((await response.json()).audioUrl);

Step B: Cognito Authorization & XML Base64 Bypass

When API Gateway validates the token, it forwards the request to a monolithic Flask Lambda (lambda_function.py).

To prevent enterprise network proxies from stripping binary multipart image data during transit, commentary.py embeds webcam snapshots inside custom XML tags in the text payload:

# Hide JPEG inside text so enterprise proxies can't strip multipart binaries
content_block += f"<p1_webcam_base64_jpeg>{image_base64}</p1_webcam_base64_jpeg>"
agent_client.invoke_agent_runtime(
    agentRuntimeArn=os.environ["AGENTCORE_RUNTIME_ARN"],
    payload=json.dumps({"prompt": content_block, "session_id": session_id}).encode(),
)

Step C: Serverless AgentCore Runtime Execution

The AWS Bedrock AgentCore Runtime routes the call via IAM SigV4 to the custom container (domain_commentator_agentcore). The container’s FastAPI service (commentator_agent.py) uses regex to extract the base64 XML block and reconstruct binary multimodal payloads for Bedrock:

# Unwrap XML → rebuild multimodal Bedrock payload
image_b64 = re.search(
    r"<p1_webcam_base64_jpeg>(.*?)</p1_webcam_base64_jpeg>", prompt_text, re.DOTALL
).group(1)
await strands_agent.invoke_async([
    {"image": {"format": "jpeg", "source": {"bytes": base64.b64decode(image_b64)}}},
    {"text": prompt_text},
])

Step D: Hardware Dispatch via AgentCore Tool Gateway

When the model triggers physical actions, the runtime calls the AgentCore Tool Gateway (bedrockagentcore.Gateway), which invokes robot_tool_lambda.py via IAM SigV4 to publish MQTT commands to AWS IoT Core:

# AgentCore tool name → IoT MQTT action
local_tool = event["tool_name"].split("___")[-1]  # "...___robot_wave"
iot_client.publish(topic="robot_1/topic", payload=json.dumps({"action": "wave_hand"}))

---

Infrastructure as Code: CDK Core Patterns

To keep infrastructure repeatable, modern, and clean, the entire system is modeled in AWS CDK.

1. Centralized Tool Gateway Construct (robot-tool-gateway.ts)

// Robot Lambda behind AgentCore Gateway — IAM SigV4 only
this.gateway = new bedrockagentcore.Gateway(this, "RobotToolGateway", {
  authorizerConfiguration: bedrockagentcore.GatewayAuthorizer.usingAwsIam(),
});
this.gateway.addLambdaTarget("RobotToolLambdaTarget", {
  lambdaFunction: this.robotToolFunction,
});

2. Serverless AgentCore Commentator Runtime (domain-expansion-serverless.ts)

// Cost-aware AgentCore lifecycle: idle sessions die fast
new agentcore.Runtime(this, "Runtime", {
  runtimeName: "domain_commentator_agentcore",
  lifecycleConfiguration: {
    idleRuntimeSessionTimeout: Duration.seconds(120), // $0 when idle
    maxLifetime: Duration.seconds(900),
  },
  environmentVariables: { BEDROCK_MODEL_ID: "moonshotai.kimi-k2.5" },
});

---

Split-Route API Gateway Security Model

The team implemented a split-route REST model to resolve a classic web conflict: standard browser elements (<img src="...">) cannot pass Bearer headers, but write operations cost LLM money.

// Reads are public (<img> can't send Bearer); LLM writes require Cognito
apiResource.addResource("get-snapshot").addMethod("GET", lambdaIntegration, {
  authorizationType: apigateway.AuthorizationType.NONE,
});
apiResource.addResource("trigger-technique").addMethod("POST", lambdaIntegration, {
  authorizationType: apigateway.AuthorizationType.COGNITO,
  authorizer: restAuthorizer,
});

---

Observability & Enterprise Tracing

Observability is handled via CloudWatch Vended Logs namespaces and AWS X-Ray:

// AgentCore → CloudWatch Vended Logs (X-Ray maps the full path)
logGroupName: `/aws/vendedlogs/bedrock-agentcore/gateway/APPLICATION_LOGS/${gateway.gatewayId}`

This ensures end-to-end distributed tracing on a single X-Ray map:

Browser Gesture → API Gateway → Lambda Router → AgentCore Container → Strands LLM → AWS Polly TTS.

---

Smart Client Guards & Character Engineering

1. Cost-Guarding Session Logic (auth-check.js)

To avoid billing leaks from idle browser tabs keeping AgentCore sockets open, the client checks the Cognito JWT every 30 seconds on-device:

Session Expired = currentTime >= jwt.exp

If expired, it closes active WebSockets immediately:

// Expire JWT locally → close sockets → no idle AgentCore bill
if (isTokenExpired(getCognitoToken())) webSocketConnection?.close();
setInterval(checkSessionGuard, 30000); // client-side only

2. Speech Text Sanitization (commentary_tts.py)

Before passing model outputs to AWS Polly, Markdown symbols (**bold**, #) are converted to clean plain text using BeautifulSoup to prevent Polly from reading syntax aloud:

# Stop Polly from reading **bold** / # aloud
BeautifulSoup(markdown.markdown(raw_text), "html.parser").get_text()

3. Soul as Code Configuration

Character identity is fully decoupled into standard Markdown files loaded at runtime:

● IDENTITY.md: Directives on sassy Cantonese styling and demeanor.

● SOUL.md: Combat lore rules (e.g., how to roast rival Domain Expansion users).

---

Architectural Economics & Pay-As-You-Go

---

Closing Reflection: The Developer Lounge Vibe

What Cyrus Wong and the team demonstrated at the AWS Hong Kong Summit 2026 Developer Lounge was more than a fun fan game. It was a masterclass in modern serverless design: high fashion, high technology, low latency, and zero idle cost.

As the sun set over Central, illuminating Victoria Harbour, developers were still clustered around the booth, scanning CDK repositories and testing hand signs. That is the real spirit of the AWS Developer Lounge—bringing world-class engineering, vibrant local culture, and serverless power together under one roof.