UI-TARS-desktop's full system prompt: 3 versions, 30,658 characters. 7 instructions flagged, worst on tool/action safety.
The full text of 3
prompts is reproduced below,
30,658 characters in all, each read
instruction by instruction against the eight
AISPA dimensions.
7 instructions
were flagged as working against
the person on the other end, most of them on
tool/action safety.
> [!WARNING]
> This document has been archived.
# @ui-tars/sdk Guide (Experimental)
## Overview
`@ui-tars/sdk` is a powerful cross-platform(ANY device/platform) toolkit for building GUI automation agents.
It provides a flexible framework to create agents that can interact with graphical user interfaces through various operators. It supports running on both **Node.js** and the **Web Browser**
```mermaid
classDiagram
class GUIAgent~T extends Operator~ {
+model: UITarsModel
+operator: T
+signal: AbortSignal
+onData
+run()
}
class UITarsModel {
+invoke()
}
class Operator {
<<interface>>
+screenshot()
+execute()
}
class NutJSOperator {
+screenshot()
+execute()
}
class WebOperator {
+screenshot()
+execute()
}
class MobileOperator {
+screenshot()
+execute()
}
GUIAgent --> UITarsModel
GUIAgent ..> Operator
Operator <|.. NutJSOperator
Operator <|.. WebOperator
Operator <|.. MobileOperator
```
## Try it out
```bash
npx @ui-tars/cli start
```
Input your UI-TARS Model Service Config(`baseURL`, `apiKey`, `model`), then you can control your computer with CLI.
```
Need to install the following packages:
Ok to proceed? (y) y
│
◆ Input your instruction
│ _ Open Chrome
└
```
## Agent Execution Process
```mermaid
sequenceDiagram
participant user as User
participant guiAgent as GUI Agent
participant model as UI-TARS Model
participant operator as Operator
user -->> guiAgent: "`instruction` + <br /> `Operator.MANUAL.ACTION_SPACES`"
activate user
activate guiAgent
loop status !== StatusEnum.RUNNING
guiAgent ->> operator: screenshot()
activate operator
operator -->> guiAgent: base64, Physical screen size
deactivate operator
guiAgent ->> model: instruction + actionSpaces + screenshots.slice(-5)
model -->> guiAgent: `prediction`: click(start_box='(27,496)')
guiAgent -->> user: prediction, next action
guiAgent ->> operator: execute(prediction)
activate operator
operator -->> guiAgent: success
deactivate operator
end
deactivate guiAgent
deactivate user
```
### Basic Usage
Basic usage is largely derived from package `@ui-tars/sdk`, here's a basic example of using the SDK:
> Note: Using `nut-js`(cross-platform computer control tool) as the operator, you can also use or customize other operators. NutJS operator that supports common desktop automation actions:
> - Mouse actions: click, double click, right click, drag, hover
> - Keyboard input: typing, hotkeys
> - Scrolling
> - Screenshot capture
```ts
import { GUIAgent } from '@ui-tars/sdk';
import { NutJSOperator } from '@ui-tars/operator-nut-js';
const guiAgent = new GUIAgent({
model: {
baseURL: config.baseURL,
apiKey: config.apiKey,
model: config.model,
},
operator: new NutJSOperator(),
onData: ({ data }) => {
console.log(data)
},
onError: ({ data, error }) => {
console.error(error, data);
},
});
await guiAgent.run('send "hello world" to x.com');
```
### Handling Abort Signals
You can abort the agent by passing a `AbortSignal` to the GUIAgent `signal` option.
```ts
const abortController = new AbortController();
const guiAgent = new GUIAgent({
// ... other config
signal: abortController.signal,
});
// ctrl/cmd + c to cancel operation
process.on('SIGINT', () => {
abortController.abort();
});
```
## Configuration Options
The `GUIAgent` constructor accepts the following configuration options:
- `model`: Model configuration(OpenAI-compatible API) or custom model instance
- `baseURL`: API endpoint URL
- `apiKey`: API authentication key
- `model`: Model name to use
- more options see [OpenAI API](https://platform.openai.com/docs/guides/vision/uploading-base-64-encoded-images)
- `operator`: Instance of an operator class that implements the required interface
- `signal`: AbortController signal for canceling operations
- `onData`: Callback for receiving agent data/status updates
- `data.conversations` is an array of objects, **IMPORTANT: is delta, not the whole conversation history**, each object contains:
- `from`: The role of the message, it can be one of the following:
- `human`: Human message
- `gpt`: Agent response
- `screenshotBase64`: Screenshot base64
- `value`: The content of the message
- `data.status` is the current status of the agent, it can be one of the following:
- `StatusEnum.INIT`: Initial state
- `StatusEnum.RUNNING`: Agent is actively executing
- `StatusEnum.END`: Operation completed
- `StatusEnum.MAX_LOOP`: Maximum loop count reached
- `onError`: Callback for error handling
- `systemPrompt`: Optional custom system prompt
- `maxLoopCount`: Maximum number of interaction loops (default: 25)
### Status flow
```mermaid
stateDiagram-v2
[*] --> INIT
INIT --> RUNNING
RUNNING --> RUNNING: Execute Actions
RUNNING --> END: Task Complete
RUNNING --> MAX_LOOP: Loop Limit Reached
END --> [*]
MAX_LOOP --> [*]
```
## Advanced Usage
### Operator Interface
When implementing a custom operator, you need to implement two core methods: `screenshot()` and `execute()`.
#### Initialize
`npm init` to create a new operator package, configuration is as follows:
```json
{
"name": "your-operator-tool",
"version": "1.0.0",
"main": "./dist/index.js",
"module": "./dist/index.mjs",
"types": "./dist/index.d.ts",
"scripts": {
"dev": "rslib build --watch",
"prepare": "npm run build",
"build": "rsbuild",
"test": "vitest"
},
"files": [
"dist"
],
"publishConfig": {
"access": "public",
"registry": "https://registry.npmjs.org"
},
"dependencies": {
"jimp": "^1.6.0"
},
"peerDependencies": {
"@ui-tars/sdk": "^1.2.0-beta.17"
},
"devDependencies": {
"@ui-tars/sdk": "^1.2.0-beta.17",
"@rslib/core": "^0.5.4",
"typescript": "^5.7.2",
"vitest": "^3.0.2"
}
}
```
#### screenshot()
This method captures the current screen state and returns a `ScreenshotOutput`:
```typescript
interface ScreenshotOutput {
// Base64 encoded image string
base64: string;
// Device pixel ratio (DPR)
scaleFactor: number;
}
```
#### execute()
This method performs actions based on model predictions. It receives an `ExecuteParams` object:
```typescript
interface ExecuteParams {
/** Raw prediction string from the model */
prediction: string;
/** Parsed prediction object */
parsedPrediction: {
action_type: string;
action_inputs: Record<string, any>;
reflection: string | null;
thought: string;
};
/** Device Physical Resolution */
screenWidth: number;
/** Device Physical Resolution */
screenHeight: number;
/** Device DPR */
scaleFactor: number;
/** model coordinates scaling factor [widthFactor, heightFactor] */
factors: Factors;
}
```
Advanced sdk usage is largely derived from package `@ui-tars/sdk/core`, you can create custom operators by extending the base `Operator` class:
```typescript
import {
Operator,
type ScreenshotOutput,
type ExecuteParams
type ExecuteOutput,
} from '@ui-tars/sdk/core';
import { Jimp } from 'jimp';
export class CustomOperator extends Operator {
// Define the action spaces and description for UI-TARS System Prompt splice
static MANUAL = {
ACTION_SPACES: [
'click(start_box="") # click on the element at the specified coordinates',
'type(content="") # type the specified content into the current input field',
'scroll(direction="") # scroll the page in the specified direction',
'finished() # finish the task',
// ...more_actions
],
};
public async screenshot(): Promise<ScreenshotOutput> {
// Implement screenshot functionality
const base64 = 'base64-encoded-image';
const buffer = Buffer.from(base64, 'base64');
const image = await sharp(buffer).toBuffer();
return {
base64: 'base64-encoded-image',
scaleFactor: 1
};
}
async execute(params: ExecuteParams): Promise<ExecuteOutput> {
const { parsedPrediction, screenWidth, screenHeight, scaleFactor } = params;
// Implement action execution logic
// if click action, get coordinates from parsedPrediction
const [startX, startY] = parsedPrediction?.action_inputs?.start_coords || '';
if (parsedPrediction?.action_type === 'finished') {
// finish the GUIAgent task
return { status: StatusEnum.END };
}
}
}
```
Required methods:
- `screenshot()`: Captures the current screen state
- `execute()`: Performs the requested action based on model predictions
Optional static properties:
- `MANUAL`: Define the action spaces and description for UI-TARS Model understanding
- `ACTION_SPACES`: Define the action spaces and description for UI-TARS Model understanding
Loaded into `GUIAgent`:
```ts
const guiAgent = new GUIAgent({
// ... other config
systemPrompt: `
// ... other system prompt
${CustomOperator.MANUAL.ACTION_SPACES.join('\n')}
`,
operator: new CustomOperator(),
});
```
### Custom Model Implementation
You can implement custom model logic by extending the `UITarsModel` class:
```typescript
class CustomUITarsModel extends UITarsModel {
constructor(modelConfig: { model: string }) {
super(modelConfig);
}
async invoke(params: any) {
// Implement custom model logic
return {
prediction: 'action description',
parsedPredictions: [{
action_type: 'click',
action_inputs: { /* ... */ },
reflection: null,
thought: 'reasoning'
}]
};
}
}
const agent = new GUIAgent({
model: new CustomUITarsModel({ model: 'custom-model' }),
// ... other config
});
```
> Note: However, it is not recommended to implement a custom model because it contains a lot of data processing logic (including image transformations, scaling factors, etc.).
### Planning
You can combine planning/reasoning models (such as OpenAI-o1, DeepSeek-R1) to implement complex GUIAgent logic for planning, reasoning, and execution:
```ts
const guiAgent = new GUIAgent({
// ... other config
});
const planningList = await reasoningModel.invoke({
conversations: [
{
role: 'user',
content: 'buy a ticket from beijing to shanghai',
}
]
})
/**
* [
* 'open chrome',
* 'open trip.com',
* 'click "search" button',
* 'select "beijing" in "from" input',
* 'select "shanghai" in "to" input',
* 'click "search" button',
* ]
*/
for (const planning of planningList) {
await guiAgent.run(planning);
}
```
Instructions flagged against the user
D4 · Tool/Action Safety
“await guiAgent.run('send "hello world" to x.com');”
The SDK enables autonomous GUI control including mouse clicks, keyboard input, typing, and executing arbitrary actions on a user's computer without any documented safety validation, sandboxing, or confirmation mechanisms. The agent takes screenshots and executes actions in a loop without requiring user confirmation for potentially destructive operations. The example 'send "hello world" to x.com' demonstrates the agent autonomously performing actions on external services without explicit safety guardrails.
D6 · Unsafe Request Handling
“await guiAgent.run('send "hello world" to x.com');”
The prompt provides no guidance on refusing unsafe or illicit requests. The agent will execute any instruction passed to it, including potentially harmful ones. There are no content filters, safety policies, or mechanisms to detect and refuse malicious instructions that could control a user's computer for harmful purposes.
D5 · User Agency & Manipulation Prevention
“for (const planning of planningList) {
await guiAgent.run(planning);
}”
The agent execution process shows a fully autonomous loop where the agent takes screenshots, gets model predictions, and executes actions without any human-in-the-loop confirmation step for consequential actions. While an abort signal exists, there is no mechanism for the user to review or approve individual actions before execution, removing user agency over potentially impactful operations on their system.
D7 · Harm Prevention & User Safety
“Keyboard input: typing, hotkeys”
The SDK enables full computer control including mouse, keyboard, and screen capture with no documented safety mechanisms to prevent harm. An autonomous agent with these capabilities could be used to perform destructive actions, exfiltrate data, or cause system damage. No warnings about potential risks, no safety boundaries, and no harm prevention measures are documented.
# @ui-tars/sdk Guide (Experimental)
[](https://www.npmjs.com/package/@ui-tars/sdk) [](https://app.codecov.io/gh/bytedance/UI-TARS-desktop/components/ui_tars_sdk)
## Overview
`@ui-tars/sdk` is a powerful cross-platform(ANY device/platform) toolkit for building GUI automation agents.
It provides a flexible framework to create agents that can interact with graphical user interfaces through various operators. It supports running on both **Node.js** and the **Web Browser**
```mermaid
classDiagram
class GUIAgent~T extends Operator~ {
+model: UITarsModel
+operator: T
+signal: AbortSignal
+onData
+run()
}
class UITarsModel {
+invoke()
}
class Operator {
<<interface>>
+screenshot()
+execute()
}
class NutJSOperator {
+screenshot()
+execute()
}
class WebOperator {
+screenshot()
+execute()
}
class MobileOperator {
+screenshot()
+execute()
}
GUIAgent --> UITarsModel
GUIAgent ..> Operator
Operator <|.. NutJSOperator
Operator <|.. WebOperator
Operator <|.. MobileOperator
```
## Try it out
```bash
npx @ui-tars/cli start
```
Input your UI-TARS Model Service Config(`baseURL`, `apiKey`, `model`), then you can control your computer with CLI.
```
Need to install the following packages:
Ok to proceed? (y) y
│
◆ Input your instruction
│ _ Open Chrome
└
```
## Agent Execution Process
```mermaid
sequenceDiagram
participant user as User
participant guiAgent as GUI Agent
participant model as UI-TARS Model
participant operator as Operator
user -->> guiAgent: "`instruction` + <br /> `Operator.MANUAL.ACTION_SPACES`"
activate user
activate guiAgent
loop status !== StatusEnum.RUNNING
guiAgent ->> operator: screenshot()
activate operator
operator -->> guiAgent: base64, Physical screen size
deactivate operator
guiAgent ->> model: instruction + actionSpaces + screenshots.slice(-5)
model -->> guiAgent: `prediction`: click(start_box='(27,496)')
guiAgent -->> user: prediction, next action
guiAgent ->> operator: execute(prediction)
activate operator
operator -->> guiAgent: success
deactivate operator
end
deactivate guiAgent
deactivate user
```
### Basic Usage
Basic usage is largely derived from package `@ui-tars/sdk`, here's a basic example of using the SDK:
> Note: Using `nut-js`(cross-platform computer control tool) as the operator, you can also use or customize other operators. NutJS operator that supports common desktop automation actions:
> - Mouse actions: click, double click, right click, drag, hover
> - Keyboard input: typing, hotkeys
> - Scrolling
> - Screenshot capture
```ts
import { GUIAgent } from '@ui-tars/sdk';
import { NutJSOperator } from '@ui-tars/operator-nut-js';
const guiAgent = new GUIAgent({
model: {
baseURL: config.baseURL,
apiKey: config.apiKey,
model: config.model,
},
operator: new NutJSOperator(),
onData: ({ data }) => {
console.log(data)
},
onError: ({ data, error }) => {
console.error(error, data);
},
});
await guiAgent.run('send "hello world" to x.com');
```
### Handling Abort Signals
You can abort the agent by passing a `AbortSignal` to the GUIAgent `signal` option.
```ts
const abortController = new AbortController();
const guiAgent = new GUIAgent({
// ... other config
signal: abortController.signal,
});
// ctrl/cmd + c to cancel operation
process.on('SIGINT', () => {
abortController.abort();
});
```
## Configuration Options
The `GUIAgent` constructor accepts the following configuration options:
- `model`: Model configuration(OpenAI-compatible API) or custom model instance
- `baseURL`: API endpoint URL
- `apiKey`: API authentication key
- `model`: Model name to use
- more options see [OpenAI API](https://platform.openai.com/docs/guides/vision/uploading-base-64-encoded-images)
- `operator`: Instance of an operator class that implements the required interface
- `signal`: AbortController signal for canceling operations
- `onData`: Callback for receiving agent data/status updates
- `data.conversations` is an array of objects, **IMPORTANT: is delta, not the whole conversation history**, each object contains:
- `from`: The role of the message, it can be one of the following:
- `human`: Human message
- `gpt`: Agent response
- `screenshotBase64`: Screenshot base64
- `value`: The content of the message
- `data.status` is the current status of the agent, it can be one of the following:
- `StatusEnum.INIT`: Initial state
- `StatusEnum.RUNNING`: Agent is actively executing
- `StatusEnum.END`: Operation completed
- `StatusEnum.MAX_LOOP`: Maximum loop count reached
- `onError`: Callback for error handling
- `systemPrompt`: Optional custom system prompt
- `maxLoopCount`: Maximum number of interaction loops (default: 25)
### Status flow
```mermaid
stateDiagram-v2
[*] --> INIT
INIT --> RUNNING
RUNNING --> RUNNING: Execute Actions
RUNNING --> END: Task Complete
RUNNING --> MAX_LOOP: Loop Limit Reached
END --> [*]
MAX_LOOP --> [*]
```
## Advanced Usage
### Operator Interface
When implementing a custom operator, you need to implement two core methods: `screenshot()` and `execute()`.
#### Initialize
`npm init` to create a new operator package, configuration is as follows:
```json
{
"name": "your-operator-tool",
"version": "1.0.0",
"main": "./dist/index.js",
"module": "./dist/index.mjs",
"types": "./dist/index.d.ts",
"scripts": {
"dev": "rslib build --watch",
"prepare": "npm run build",
"build": "rsbuild",
"test": "vitest"
},
"files": [
"dist"
],
"publishConfig": {
"access": "public",
"registry": "https://registry.npmjs.org"
},
"dependencies": {
"jimp": "^1.6.0"
},
"peerDependencies": {
"@ui-tars/sdk": "^1.2.0-beta.17"
},
"devDependencies": {
"@ui-tars/sdk": "^1.2.0-beta.17",
"@rslib/core": "^0.5.4",
"typescript": "^5.7.2",
"vitest": "^3.0.2"
}
}
```
#### screenshot()
This method captures the current screen state and returns a `ScreenshotOutput`:
```typescript
interface ScreenshotOutput {
// Base64 encoded image string
base64: string;
// Device pixel ratio (DPR)
scaleFactor: number;
}
```
#### execute()
This method performs actions based on model predictions. It receives an `ExecuteParams` object:
```typescript
interface ExecuteParams {
/** Raw prediction string from the model */
prediction: string;
/** Parsed prediction object */
parsedPrediction: {
action_type: string;
action_inputs: Record<string, any>;
reflection: string | null;
thought: string;
};
/** Device Physical Resolution */
screenWidth: number;
/** Device Physical Resolution */
screenHeight: number;
/** Device DPR */
scaleFactor: number;
/** model coordinates scaling factor [widthFactor, heightFactor] */
factors: Factors;
}
```
Advanced sdk usage is largely derived from package `@ui-tars/sdk/core`, you can create custom operators by extending the base `Operator` class:
```typescript
import {
Operator,
type ScreenshotOutput,
type ExecuteParams
type ExecuteOutput,
} from '@ui-tars/sdk/core';
import { Jimp } from 'jimp';
export class CustomOperator extends Operator {
// Define the action spaces and description for UI-TARS System Prompt splice
static MANUAL = {
ACTION_SPACES: [
'click(start_box="") # click on the element at the specified coordinates',
'type(content="") # type the specified content into the current input field',
'scroll(direction="") # scroll the page in the specified direction',
'finished() # finish the task',
// ...more_actions
],
};
public async screenshot(): Promise<ScreenshotOutput> {
// Implement screenshot functionality
const base64 = 'base64-encoded-image';
const buffer = Buffer.from(base64, 'base64');
const image = await sharp(buffer).toBuffer();
return {
base64: 'base64-encoded-image',
scaleFactor: 1
};
}
async execute(params: ExecuteParams): Promise<ExecuteOutput> {
const { parsedPrediction, screenWidth, screenHeight, scaleFactor } = params;
// Implement action execution logic
// if click action, get coordinates from parsedPrediction
const [startX, startY] = parsedPrediction?.action_inputs?.start_coords || '';
if (parsedPrediction?.action_type === 'finished') {
// finish the GUIAgent task
return { status: StatusEnum.END };
}
}
}
```
Required methods:
- `screenshot()`: Captures the current screen state
- `execute()`: Performs the requested action based on model predictions
Optional static properties:
- `MANUAL`: Define the action spaces and description for UI-TARS Model understanding
- `ACTION_SPACES`: Define the action spaces and description for UI-TARS Model understanding
Loaded into `GUIAgent`:
```ts
const guiAgent = new GUIAgent({
// ... other config
systemPrompt: `
// ... other system prompt
${CustomOperator.MANUAL.ACTION_SPACES.join('\n')}
`,
operator: new CustomOperator(),
});
```
### Custom Model Implementation
You can implement custom model logic by extending the `UITarsModel` class:
```typescript
class CustomUITarsModel extends UITarsModel {
constructor(modelConfig: { model: string }) {
super(modelConfig);
}
async invoke(params: any) {
// Implement custom model logic
return {
prediction: 'action description',
parsedPredictions: [{
action_type: 'click',
action_inputs: { /* ... */ },
reflection: null,
thought: 'reasoning'
}]
};
}
}
const agent = new GUIAgent({
model: new CustomUITarsModel({ model: 'custom-model' }),
// ... other config
});
```
> Note: However, it is not recommended to implement a custom model because it contains a lot of data processing logic (including image transformations, scaling factors, etc.).
### Planning
You can combine planning/reasoning models (such as OpenAI-o1, DeepSeek-R1) to implement complex GUIAgent logic for planning, reasoning, and execution:
```ts
const guiAgent = new GUIAgent({
// ... other config
});
const planningList = await reasoningModel.invoke({
conversations: [
{
role: 'user',
content: 'buy a ticket from beijing to shanghai',
}
]
})
/**
* [
* 'open chrome',
* 'open trip.com',
* 'click "search" button',
* 'select "beijing" in "from" input',
* 'select "shanghai" in "to" input',
* 'click "search" button',
* ]
*/
for (const planning of planningList) {
await guiAgent.run(planning);
}
```
Instructions flagged against the user
D4 · Tool/Action Safety
“await guiAgent.run('send "hello world" to x.com');”
The SDK enables autonomous GUI control including mouse clicks, keyboard input, typing, hotkeys, and scrolling without any documented safety validations, user confirmation requirements, or least-privilege principles. The agent autonomously executes actions based on model predictions with no validation step, confirmation prompt, or sandboxing mentioned. The example 'send "hello world" to x.com' demonstrates the agent autonomously performing actions on external services without user confirmation at each step.
D6 · Unsafe Request Handling
“await guiAgent.run('send "hello world" to x.com');”
The SDK provides no guidance or mechanisms for refusing unsafe requests. There are no content filters, safety checks, or restrictions on what instructions can be passed to the agent. The agent could be instructed to perform harmful actions on a user's computer or interact with services in unauthorized ways, and no safeguards are documented.
D7 · Harm Prevention & User Safety
“Keyboard input: typing, hotkeys”
The SDK enables full computer control including mouse, keyboard, and screen interaction without any documented harm prevention measures. There are no warnings about potential risks of autonomous computer control, no restrictions on dangerous actions (e.g., deleting files, making purchases, sending messages), and no safety guidelines for developers building with the toolkit.
# @agent-tars/core
<b>Agent TARS</b> is a general multimodal AI Agent stack, it brings the power of GUI Agent and Vision into your terminal, computer, browser and product. <br>
It primarily ships with a <a href="https://agent-tars.com/guide/basic/cli.html" target="_blank">CLI</a> and <a href="https://agent-tars.com/guide/basic/web-ui.html" target="_blank">Web UI</a> for usage. It aims to provide a workflow that is closer to human-like task completion through cutting-edge multimodal LLMs and seamless integration with various real-world <a href="https://agent-tars.com/guide/basic/mcp.html" target="_blank">MCP</a> tools.
📣 **Just released**: Agent TARS Beta - check out our [announcement blog post](https://agent-tars.com/beta)!
https://github.com/user-attachments/assets/772b0eef-aef7-4ab9-8cb0-9611820539d8
<br>
<table>
<thead>
<tr>
<th width="50%" align="center">Booking Hotel</th>
<th width="50%" align="center">Generate Chart with extra MCP Servers</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center">
<video src="https://github.com/user-attachments/assets/c9489936-afdc-4d12-adda-d4b90d2a869d" width="50%"></video>
</td>
<td align="center">
<video src="https://github.com/user-attachments/assets/a9fd72d0-01bb-4233-aa27-ca95194bbce9" width="50%"></video>
</td>
</tr>
<tr>
<td align="left">
<b>Instruction:</b> <i>I am in Los Angeles from September 1st to September 6th, with a budget of $5,000. Please help me book a Ritz-Carlton hotel closest to the airport on booking.com and compile a transportation guide for me</i>
</td>
<td align="left">
<b>Instruction:</b> <i>Draw me a chart of Hangzhou's weather for one month</i>
</td>
</tr>
</tbody>
</table>
For more use cases, please check out [#842](https://github.com/bytedance/UI-TARS-desktop/issues/842).
## Overview
`@agent-tars/core` is the core implementation of Agent TARS, built on top of the Tarko Agent framework. It provides a comprehensive multimodal AI agent with advanced browser automation, filesystem operations, and intelligent search capabilities.
### Core Features
- 🖱️ **One-Click Out-of-the-box CLI** - Supports both **headful** [Web UI](https://agent-tars.com/guide/basic/web-ui.html) and **headless** [server](https://agent-tars.com/guide/advanced/server.html)) [execution](https://agent-tars.com/guide/basic/cli.html).
- 🌐 **Hybrid Browser Agent** - Control browsers using [GUI Agent](https://agent-tars.com/guide/basic/browser.html#visual-grounding), [DOM](https://agent-tars.com/guide/basic/browser.html#dom), or a hybrid strategy.
- 🔄 **Event Stream** - Protocol-driven Event Stream drives [Context Engineering](https://agent-tars.com/beta#context-engineering) and [Agent UI](https://agent-tars.com/blog/2025-06-25-introducing-agent-tars-beta.html#easy-to-build-applications).
- 🧰 **MCP Integration** - The kernel is built on MCP and also supports mounting [MCP Servers](https://agent-tars.com/guide/basic/mcp.html) to connect to real-world tools.
### Quick Start
```bash
# Luanch with `npx`.
npx @agent-tars/cli@latest
# Install globally, required Node.js >= 22
npm install @agent-tars/cli@latest -g
# Run with your preferred model provider
agent-tars --provider volcengine --model doubao-1-5-thinking-vision-pro-250428 --apiKey your-api-key
agent-tars --provider anthropic --model claude-3-7-sonnet-latest --apiKey your-api-key
```
Visit the comprehensive [Quick Start](https://agent-tars.com/guide/get-started/quick-start.html) guide for detailed setup instructions.
## Quick Start
### Installation
```bash
npm install @agent-tars/core
```
### Running Agent TARS
Agent TARS can be started in multiple ways:
#### Option 1: Using @agent-tars/cli (Recommended)
```bash
# Install globally
npm install -g @agent-tars/cli
# Run Agent TARS
agent-tars
# Or use directly via npx
npx @agent-tars/cli
```
#### Option 2: Using @tarko/agent-cli
```bash
# Install globally
npm install -g @tarko/agent-cli
# Run Agent TARS through tarko CLI
tarko run agent-tars
# Or use directly via npx
npx @tarko/agent-cli run agent-tars
```
#### Option 3: Programmatic Usage
### Basic Usage
```typescript
import { AgentTARS } from '@agent-tars/core';
// Create an agent instance
const agent = new AgentTARS({
model: {
provider: 'openai',
model: 'gpt-4',
apiKey: process.env.OPENAI_API_KEY,
},
workspace: './workspace',
browser: {
headless: false,
control: 'hybrid',
},
});
// Initialize and run
await agent.initialize();
const result = await agent.run('Search for the latest AI research papers');
console.log(result);
```
## Configuration
### AgentTARSOptions
```typescript
interface AgentTARSOptions {
// Model configuration
model?: {
provider: 'openai' | 'anthropic' | 'doubao';
model: string;
apiKey: string;
};
// Browser settings
browser?: {
headless?: boolean;
control?: 'dom' | 'visual-grounding' | 'hybrid';
cdpEndpoint?: string;
};
// Search configuration
search?: {
provider: 'browser_search' | 'tavily';
count?: number;
apiKey?: string;
};
// Workspace settings
workspace?: string;
// MCP implementation
mcpImpl?: 'in-memory' | 'stdio';
}
```
### Browser Control Modes
- **`dom`**: Direct DOM manipulation (fastest, most reliable)
- **`visual-grounding`**: Vision-based interaction (most flexible)
- **`hybrid`**: Combines both approaches (recommended)
## Advanced Usage
### Custom Instructions
```typescript
const agent = new AgentTARS({
instructions: `
You are a specialized research assistant.
Focus on academic papers and technical documentation.
Always provide citations and sources.
`,
// ... other options
});
```
### Browser State Management
```typescript
// Get browser control information
const browserInfo = agent.getBrowserControlInfo();
console.log(`Mode: ${browserInfo.mode}`);
console.log(`Tools: ${browserInfo.tools.join(', ')}`);
// Access browser manager
const browserManager = agent.getBrowserManager();
if (browserManager) {
const isAlive = await browserManager.isBrowserAlive();
console.log(`Browser status: ${isAlive ? 'alive' : 'dead'}`);
}
```
### Workspace Operations
```typescript
// Get current workspace
const workspace = agent.getWorkingDirectory();
console.log(`Working in: ${workspace}`);
// All file operations are automatically scoped to workspace
const result = await agent.run('Create a summary.md file with today\'s findings');
```
## Error Handling
```typescript
try {
await agent.initialize();
const result = await agent.run('Your task here');
} catch (error) {
console.error('Agent error:', error);
} finally {
// Always cleanup
await agent.cleanup();
}
```
## API Reference
### Core Methods
- `initialize()`: Initialize the agent and all components
- `run(message)`: Execute a task with the given message
- `cleanup()`: Clean up all resources
- `getWorkingDirectory()`: Get current workspace path
- `getBrowserControlInfo()`: Get browser control status
- `getBrowserManager()`: Access browser manager instance
### Events
The agent emits events through the event stream:
```typescript
agent.eventStream.subscribe((event) => {
if (event.type === 'tool_result') {
console.log(`Tool ${event.name} completed`);
}
});
```
## Resources
- 📄 [Blog Post](https://agent-tars.com/beta)
- 🐦 [Release Announcement on Twitter](https://x.com/_ulivz/status/1938009759413899384)
- 🐦 [Official Twitter](https://x.com/agent_tars)
- 💬 [Discord Community](https://discord.gg/HnKcSBgTVx)
- 💬 [飞书交流群](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=deen76f4-ea3c-4964-93a3-78f126f39651)
- 🚀 [Quick Start](https://agent-tars.com/quick-start)
- 💻 [CLI Documentation](https://agent-tars.com/guide/basic/cli.html)
- 🖥️ [Web UI Guide](https://agent-tars.com/guide/basic/web-ui.html)
- 📁 [Workspace Documentation](https://agent-tars.com/guide/basic/workspace.html)
- 🔌 [MCP Documentation](https://agent-tars.com/guide/basic/mcp.html)
## Features
- 🌐 **Advanced Browser Control**: Multiple control strategies (DOM, Visual, Hybrid)
- 📁 **Safe Filesystem Operations**: Workspace-scoped file management
- 🔍 **Intelligent Search**: Integration with multiple search providers
- 🔧 **MCP Integration**: Built-in Model Context Protocol support
- 📸 **Visual Understanding**: Screenshot-based browser interaction
- 🛡️ **Safety First**: Path validation and workspace isolation
## What's Changed
See Full [CHANGELOG](https://github.com/bytedance/UI-TARS-desktop/blob/main/multimodal/CHANGELOG.md)
## Contributing
See [CONTRIBUTING.md](../../CONTRIBUTING.md) for development guidelines.
## License
Apache-2.0 - See [LICENSE](../../LICENSE) for details.
Questions about UI-TARS-desktop's system prompt
Does UI-TARS-desktop's system prompt contain instructions that work against the user?
Yes. 7 instructions in UI-TARS-desktop's system prompt were flagged as working against the person the product is talking to, most of them under tool/action safety. Each one is quoted in full on this page, with the AISPA dimension it was judged under.
How long is UI-TARS-desktop's system prompt?
30,658 characters across 3 prompts on this page. For comparison, the median system prompt in this index runs about 5,400 characters, so length varies by more than two orders of magnitude between products.
How many versions of UI-TARS-desktop's system prompt are on record?
3. Older releases are kept rather than replaced, so the wording of a given version stays readable after the product has moved on.
Where did this UI-TARS-desktop system prompt come from?
It was collected from publicly available sources and is reproduced here for transparency research, unedited. This site does not extract prompts from products itself.
How was UI-TARS-desktop's system prompt audited?
Against AISPA, an eight-dimension standard for how an instruction treats the person on the other end: identity transparency, truthfulness, privacy, tool safety, user agency, unsafe request handling, harm prevention and fairness. This audit was ai audit. The method is described in the paper behind the standard.
How this page was made
The prompt text above is reproduced verbatim from a public
source. Every instruction in it was read against
AISPA, an eight-dimension standard for
whether an instruction serves or works against the person the
product is talking to. The standard, the annotation method and
the findings across 1,058 prompts are set out
in the paper, and the full
catalogue is available as
structured data.
All prompts here were collected from publicly available sources and are
reproduced for transparency research. Browse the
browser automation agents category, the
full gallery of 400+ products, or read the
paper behind the AISPA standard.