Shared Browser Architecture
Overview
The Web tab (in-app browser) renders the agent's shared Playwright page so you and the agent see and interact with the same live page. There are three rendering paths, chosen based on deployment environment:
| Path | When | Feel | How |
|---|---|---|---|
| Iframe proxy | Primary (local & Cloud Run) | Native cursor, text selection, scrolling | Server fetches Playwright page, injects bridge script, returns HTML to <iframe> |
| App Window | Local desktop only | Real Chrome window, zero latency | Launcher opens visible Chrome; server attaches over CDP |
| JPEG screencast | Fallback (future) | Click-to-interact via coordinates | CDP captures JPEG frames, streams over WebSocket |
Iframe proxy (current primary path)
The Web tab loads pages through GET /api/v1/browser/proxy?bs_id=X&url=Y&token=Z. This endpoint:
- Navigates Playwright to the target URL (keeping the agent in sync)
- Waits for the page to render (2s settle for JS)
- Injects a
<base>tag so relative resources load from the real origin - Injects a bridge script that intercepts link clicks and SPA navigation, reporting them to the parent via
postMessage - Returns the HTML to the iframe
Link clicks
When you click a link in the iframe, the bridge script prevents default navigation and sends the URL to the parent. The parent optimistically loads the new URL in the iframe AND tells the server to navigate Playwright — instant feedback, agent stays in sync.
Agent navigation
When the agent navigates (via browser_action), a 1-second polling loop in the WebSocket detects the URL change and pushes a nav event to the client, which loads the new page into the iframe. A URL deduplication guard prevents redundant reloads when the server echoes back a nav we already initiated.
Limitations
- Form filling and button clicks in the iframe are not forwarded to Playwright — only link navigation is synchronized
- SPA navigation is detected but the server re-navigates Playwright, which may cause full-page reload and state loss
- Authenticated pages that rely on Playwright's cookie jar won't see those cookies in the user's browser iframe
- Anti-iframe pages (X-Frame-Options, CSP frame-ancestors) will not render
Future
The App Window path for local desktop installs. The JPEG screencast as a fallback for pages that break in the iframe. The screencast code can be restored as a secondary path with a per-page toggle.