HyperAgent - currentState
# HyperAgent Current State Analysis
## Overview
HyperAgent is a browser automation SDK that uses LLM-powered agents to execute tasks on web pages. It provides both imperative page methods (`page.ai()`, `page.extract()`) and a programmatic task execution API.
---
## Core Architecture
### 1. Entry Points & Public API
#### **HyperAgent Class** ([src/agent/index.ts](src/agent/index.ts))
The main class that orchestrates everything:
```typescript
class HyperAgent<T extends BrowserProviders = "Local"> {
// Core methods
async executeTask(task: string, params?: TaskParams, initPage?: Page): Promise<TaskOutput>
async executeTaskAsync(task: string, params?: TaskParams, initPage?: Page): Promise<Task>
// Page management
async getCurrentPage(): Promise<Page>
async newPage(): Promise<HyperPage>
async getPages(): Promise<HyperPage[]>
// Browser lifecycle
async initBrowser(): Promise<Browser>
async closeAgent(): Promise<void>
}
```
#### **HyperPage Interface** ([src/agent/index.ts:567-605](src/agent/index.ts#L567-L605))
Enhanced Playwright `Page` with AI methods:
```typescript
interface HyperPage extends Page {
// Execute a task on this page
ai(task: string, params?: TaskParams): Promise<TaskOutput>
// Execute task asynchronously (non-blocking)
aiAsync(task: string, params?: TaskParams): Promise<Task>
// Extract structured data
extract<T>(
task?: string,
outputSchema?: z.AnyZodObject,
params?: TaskParams
): Promise<T | string>
}
```
**Key Implementation Details:**
- `page.ai()` → calls `agent.executeTask(task, params, page)` ([line 569-570](src/agent/index.ts#L569-L570))
- `page.extract()` → wraps `executeTask()` with extraction-specific prompts ([lines 573-603](src/agent/index.ts#L573-L603))
- Adds `maxSteps: 2` by default for extractions
- Prepends extraction instructions to the task
- Parses JSON output if outputSchema provided
---
## 2. Task Execution Flow
### **Main Task Loop** ([src/agent/tools/agent.ts:105-306](src/agent/tools/agent.ts#L105-L306))
```
runAgentTask()
├── 1. Get DOM State (getDom)
│ ├── Inject JavaScript into page
│ ├── Find interactive elements
│ ├── Draw numbered overlay (canvas)
│ └── Capture screenshot with overlay
│
├── 2. Build Agent Messages (buildAgentStepMessages)
│ ├── System prompt
│ ├── Task description
│ ├── Previous steps context
│ ├── DOM representation (text)
│ └── Screenshot (base64 image)
│
├── 3. Invoke LLM (llm.invokeStructured)
│ ├── Request structured output (Zod schema)
│ └── Get list of actions to execute
│
├── 4. Execute Actions (runAction)
│ ├── For each action in list
│ ├── Run action handler
│ └── Wait 2 seconds between actions
│
└── 5. Repeat until complete/cancelled/maxSteps
```
**Location:** [src/agent/tools/agent.ts:132-291](src/agent/tools/agent.ts#L132-L291)
---
## 3. DOM State Extraction
### **Current Implementation: Visual DOM with Canvas Overlay**
#### **Entry Point:** `getDom(page)` ([src/context-providers/dom/index.ts:5-18](src/context-providers/dom/index.ts#L5-L18))
```typescript
export const getDom = async (page: Page): Promise<DOMState | null> => {
const result = await page.evaluate(buildDomViewJs) as DOMStateRaw;
return {
elements: Map<number, InteractiveElement>,
domState: string, // Text representation
screenshot: string // Base64 PNG with overlays
};
};
```
#### **Build DOM View** ([src/context-providers/dom/build-dom-view.ts:54-130](src/context-providers/dom/build-dom-view.ts#L54-L130))
**Process:**
1. **Find Interactive Elements** ([find-interactive-elements.ts:4-63](src/context-providers/dom/find-interactive-elements.ts#L4-L63))
- Traverse entire DOM including Shadow DOM and iframes
- Check each element with `isInteractiveElem(element)`
- Returns `InteractiveElement[]` with metadata
2. **Render Highlights Offscreen** ([highlight.ts:105-222](src/context-providers/dom/highlight.ts#L105-L222))
- Create `OffscreenCanvas` with viewport dimensions
- Draw colored rectangles around each interactive element
- Draw numbered labels (1, 2, 3...) on each element
- Return `ImageBitmap`
3. **Composite Screenshot** ([agent.ts:33-42](src/agent/tools/agent.ts#L33-L42))
```typescript
const compositeScreenshot = async (page: Page, overlay: string) => {
const screenshot = await page.screenshot({ type: "png" });
// Overlay numbered boxes onto base screenshot using Jimp
baseImage.composite(overlayImage, 0, 0);
return buffer.toString("base64");
};
```
4. **Build Text Representation** ([build-dom-view.ts:78-123](src/context-providers/dom/build-dom-view.ts#L78-L123))
```
[1]<button id="submit" class="btn-primary">Submit Form</button>
[2]<input type="text" name="email" placeholder="Enter email">
Some text between elements
[3]<a href="/pricing">View Pricing</a>
```
**Output Structure:**
```typescript
interface DOMState {
elements: Map<number, InteractiveElement> // index → element mapping
domState: string // [idx]<tag>text</tag> format
screenshot: string // base64 PNG with overlays
}
```
---
## 4. Action System
### **Available Actions** ([src/agent/actions/](src/agent/actions/))
| Action | Purpose | Key Parameters | Location |
|--------|---------|----------------|----------|
| `clickElement` | Click an element | `index: number` | [click-element.ts](src/agent/actions/click-element.ts) |
| `inputText` | Fill input field | `index: number, text: string` | [input-text.ts](src/agent/actions/input-text.ts) |
| `extract` | Extract data | `objective: string` | [extract.ts](src/agent/actions/extract.ts) |
| `goToUrl` | Navigate to URL | `url: string` | [go-to-url.ts](src/agent/actions/go-to-url.ts) |
| `selectOption` | Select dropdown | `index: number, option: string` | [select-option.ts](src/agent/actions/select-option.ts) |
| `scroll` | Scroll page | `direction: "up"\|"down"` | [scroll.ts](src/agent/actions/scroll.ts) |
| `keyPress` | Press keyboard key | `key: string` | [key-press.ts](src/agent/actions/key-press.ts) |
| `complete` | End task | `output?: string` | [complete.ts](src/agent/actions/complete.ts) |
### **Action Execution** ([src/agent/tools/agent.ts:71-103](src/agent/tools/agent.ts#L71-L103))
#### **Click Element Example** ([click-element.ts:18-57](src/agent/actions/click-element.ts#L18-L57))
```typescript
run: async function (ctx: ActionContext, action: ClickElementActionType) {
const { index } = action;
const locator = getLocator(ctx, index); // Get element by index
await locator.scrollIntoViewIfNeeded({ timeout: 2500 });
await locator.waitFor({ state: "visible", timeout: 2500 });
await waitForElementToBeEnabled(locator, 2500);
await waitForElementToBeStable(locator, 2500);
await locator.click({ force: true });
return { success: true, message: `Clicked element with index ${index}` };
}
```
**Element Selection:** ([actions/utils.ts](src/agent/actions/utils.ts))
```typescript
export const getLocator = (ctx: ActionContext, index: number): Locator | null => {
const element = ctx.domState.elements.get(index);
if (!element) return null;
return ctx.page.locator(element.cssPath); // Use CSS path selector
};
```
---
## 5. Key Workflows
### **Workflow 1: `page.ai("click the login button")`**
1. User calls `page.ai("click the login button")`
2. → `agent.executeTask(task, params, page)` ([index.ts:569](src/agent/index.ts#L569))
3. → `runAgentTask()` starts task loop ([agent.ts:105](src/agent/tools/agent.ts#L105))
4. → `getDom(page)` extracts DOM + screenshot ([agent.ts:155](src/agent/tools/agent.ts#L155))
- Injects JS to find interactive elements
- Draws numbered overlays
- Composites screenshot
5. → `buildAgentStepMessages()` creates LLM prompt ([agent.ts:201](src/agent/tools/agent.ts#L201))
6. → `llm.invokeStructured()` gets action plan ([agent.ts:220](src/agent/tools/agent.ts#L220))
7. → Execute actions ([agent.ts:253-275](src/agent/tools/agent.ts#L253-L275))
- LLM returns: `{ type: "clickElement", params: { index: 5 } }`
- `runAction()` calls `ClickElementActionDefinition.run()`
- Gets locator for element 5
- Clicks element via Playwright
8. → Repeat loop or mark complete
### **Workflow 2: `page.extract("product prices", schema)`**
1. User calls `page.extract("product prices", PriceSchema)`
2. → Wraps task: "You have to perform an extraction on the current page..." ([index.ts:586-590](src/agent/index.ts#L586-L590))
3. → Sets `maxSteps: 2` (extractions are quick) ([index.ts:581](src/agent/index.ts#L581))
4. → Adds `outputSchema` to actions ([index.ts:584](src/agent/index.ts#L584))
5. → `executeTask()` runs normal agent loop
6. → LLM returns structured output matching schema
7. → Parse JSON and return typed result ([index.ts:592](src/agent/index.ts#L592))
### **Workflow 3: Extract Action (Internal)**
The `extract` action is **different** from `page.extract()`:
**Location:** [src/agent/actions/extract.ts](src/agent/actions/extract.ts)
```typescript
run: async (ctx: ActionContext, action: ExtractActionType) => {
// Get page HTML
const content = await ctx.page.content();
const markdown = await parseMarkdown(content);
// Take screenshot via CDP
const cdpSession = await ctx.page.context().newCDPSession(ctx.page);
const screenshot = await cdpSession.send("Page.captureScreenshot");
// Call LLM with markdown + screenshot
const response = await ctx.llm.invoke([{
role: "user",
content: [
{ type: "text", text: `Extract: "${objective}"\n\n${markdown}` },
{ type: "image", url: `data:image/png;base64,${screenshot.data}` }
]
}]);
return { success: true, message: `Extracted: ${content}` };
}
```
**This is an action the agent can choose** during task execution, not the page-level method.
---
## 6. DOM State Representation
### **Current Approach: Visual DOM + Numbered Overlay**
**Strengths:**
- ✅ Simple index-based selection (LLM just says "5")
- ✅ Visual feedback in screenshots
- ✅ Works well with vision models
**Weaknesses:**
- ❌ Screenshot required every step (slow)
- ❌ Screenshot → LLM → token cost is high
- ❌ Numbered overlay can occlude important UI
- ❌ Full DOM traversal every step (no caching)
- ❌ Large token counts (screenshot + DOM text)
**Performance:**
- ~8,000-15,000 tokens per step
- ~1,500-3,000ms per action
- No caching mechanism
---
## 7. Element Discovery
### **Interactive Element Detection** ([src/context-providers/dom/elem-interactive.ts](src/context-providers/dom/elem-interactive.ts))
**Current Rules:**
```typescript
isInteractiveElem(element: HTMLElement): { isInteractive: boolean, reason?: string }
```
**Checks (in order):**
1. Native interactive tags: `button`, `a[href]`, `input`, `select`, `textarea`
2. ARIA roles: `button`, `link`, `tab`, `checkbox`, `menuitem`
3. Event listeners: `data-has-interactive-listener="true"` (injected)
4. Contenteditable elements
5. Elements with `onclick` attribute
6. Cursor style: `cursor: pointer`
7. Custom detection for common patterns
**Ignored Elements:**
- Hidden elements (`display: none`, `visibility: hidden`)
- Zero-dimension elements
- Disabled elements
- Script and style tags
---
## 8. Message Building
### **Prompt Construction** ([src/agent/messages/builder.ts](src/agent/messages/builder.ts))
**Message Structure:**
```typescript
[
{ role: "system", content: SYSTEM_PROMPT },
{ role: "user", content: [
{ type: "text", text: "Task: click login button\n\nDOMState:\n[1]<button>..." },
{ type: "image", url: "data:image/png;base64,..." }
]},
{ role: "assistant", content: "..." }, // Previous step
{ role: "user", content: "..." }, // Previous action results
// ... more history ...
{ role: "user", content: [ // Current step
{ type: "text", text: "Current DOM:\n..." },
{ type: "image", url: "..." }
]}
]
```
---
## 9. Variable System
### **Variable Management** ([src/agent/index.ts:174-202](src/agent/index.ts#L174-L202))
```typescript
interface HyperVariable {
key: string;
value: string;
description?: string;
}
// API
agent.addVariable({ key: "email", value: "user@example.com" })
agent.getVariable("email")
agent.deleteVariable("email")
```
**Usage in Actions:**
```typescript
// In inputText action:
text = text.replace(`<<${variable.key}>>`, variable.value);
// Agent can use: inputText(5, "<<email>>") → "user@example.com"
```
---
## 10. Browser Provider Architecture
### **Supported Providers:**
1. **LocalBrowserProvider** (default)
- Uses `patchright` (Playwright fork with anti-detection)
- Runs locally
2. **HyperbrowserProvider**
- Cloud-based browser service
- Remote CDP connection
**Selection:** ([index.ts:85-94](src/agent/index.ts#L85-L94))
```typescript
new HyperAgent({
browserProvider: "Local" | "Hyperbrowser",
localConfig: { ... },
hyperbrowserConfig: { ... }
})
```
---
## 11. MCP Integration
### **Model Context Protocol Support** ([src/agent/mcp/](src/agent/mcp/))
**Purpose:** Connect external tools as custom actions
```typescript
await agent.initializeMCPClient({
servers: [{
id: "filesystem",
command: "npx",
args: ["-y", "@modelcontextprotocol/server-filesystem"]
}]
});
```
**How it works:**
1. MCP server exposes tools
2. Tools converted to `AgentActionDefinition`
3. Registered with agent
4. LLM can invoke MCP tools as actions
---
## 12. Debug Mode
### **Debug Output** ([src/agent/tools/agent.ts:112-148](src/agent/tools/agent.ts#L112-L148))
When `debug: true`:
```
debug/
└── {taskId}/
├── step-0/
│ ├── elems.txt # DOM text representation
│ ├── screenshot.png # Composite screenshot
│ ├── msgs.json # LLM messages
│ └── stepOutput.json # Action results
├── step-1/
└── taskOutput.json # Final output
```
---
## 13. Summary: Current vs Desired State
### **What Works Well:**
- ✅ Simple API (`page.ai()`, `page.extract()`)
- ✅ Flexible action system
- ✅ Multi-step task execution
- ✅ MCP integration
- ✅ Variable substitution
### **Performance Bottlenecks:**
- ❌ Screenshot required every step
- ❌ No DOM caching
- ❌ No action caching
- ❌ High token usage (8K-15K per step)
- ❌ Slow actions (1.5-3s each)
### **Accuracy Issues:**
- ❌ Numbered overlay can be occluded
- ❌ Full DOM may miss semantic meaning
- ❌ No accessibility tree
- ❌ No self-healing on failure
- ❌ Single-attempt actions (no retry logic)
---
## 14. File Reference Map
| Component | File Path | Key Lines |
|-----------|-----------|-----------|
| **Main Agent Class** | `src/agent/index.ts` | 37-606 |
| **Task Execution Loop** | `src/agent/tools/agent.ts` | 105-306 |
| **DOM Extraction** | `src/context-providers/dom/index.ts` | 5-18 |
| **Build DOM View** | `src/context-providers/dom/build-dom-view.ts` | 54-130 |
| **Find Elements** | `src/context-providers/dom/find-interactive-elements.ts` | 4-63 |
| **Canvas Overlay** | `src/context-providers/dom/highlight.ts` | 105-222 |
| **Click Action** | `src/agent/actions/click-element.ts` | 18-57 |
| **Input Text Action** | `src/agent/actions/input-text.ts` | 16-37 |
| **Extract Action** | `src/agent/actions/extract.ts` | 16-104 |
| **System Prompt** | `src/agent/messages/system-prompt.ts` | - |
| **Message Builder** | `src/agent/messages/builder.ts` | - |
---
## Next Steps: Performance & Accuracy Improvements
Based on Stagehand and Skyvern analysis, key opportunities:
1. **Adopt Accessibility Tree** (Stagehand approach)
- 3-4x token reduction
- Better semantic understanding
- No screenshot required for actions
2. **Implement Caching** (Stagehand approach)
- Action cache (instruction+URL → selector)
- LLM cache (prompt → response)
- 20-30x speed improvement for cached actions
3. **Hybrid Visual Approach** (Skyvern approach)
- DOM injection for element IDs (no overlay)
- Bounding boxes only when needed
- Keep visual feedback but reduce occlusion
4. **Self-Healing** (Stagehand approach)
- Re-observe on failure
- Multiple selector strategies
- Retry logic with different approaches
See `improvement-plan.md` for detailed implementation strategy.
Instructions flagged against the user
D4 · Tool/Action Safety
“await locator.click({ force: true });”
The system allows an LLM-powered agent to execute arbitrary browser actions including clicking elements, inputting text, navigating to URLs, and pressing keys with force flags and no validation or safety checks. Actions are executed with 'force: true' bypassing normal safety checks, and there is no mention of validating URLs, restricting dangerous actions, or requiring user confirmation for consequential operations. The MCP integration allows connecting arbitrary external tools as custom actions without apparent safety validation.