Local Desktop Agent

Q: How do I connect Local Desktop Agent to my app?

Use Weavz Hosted Connect for custom authentication. Install the SDK, create a connection through the connect portal or API, then execute tools programmatically.

Q: Can Local Desktop Agent be used in stateful agent workflows?

Yes. Weavz lets agents combine Local Desktop Agent tools with Filesystem, State KV, Human Gates, input partials, and other workspace integrations while keeping execution scoped to the selected workspace or end user.

Q: What authentication does Local Desktop Agent use?

Local Desktop Agent uses custom authentication. Weavz stores credentials securely and uses them for authorized action execution.

Give agents a goal-driven desktop workspace for approved Mac tasks. Weavz observes selected apps, plans and verifies each step, asks for confirmation on sensitive actions, and keeps pause, scope, human takeover, and audit controls visible to the local user.

CustomExperimentalAI Developer Tools

31 tools

Tools

Quick Start

TypeScript

import { WeavzClient } from '@weavz-io/sdk'

const weavz = new WeavzClient({ apiKey: 'wvz_...' })

const result = await weavz.actions.execute('local-computer-control-agent', 'start_session', {
  workspaceId: '550e8400-e29b-41d4-a716-446655440000',
  integrationAlias: 'computer_agent',
  input: { /* ... */ },
})

API Reference TypeScript SDK Python SDK All Features Pricing

Related Integrations

1Password

9 tools

Ably

5 tools

Abstract API

5 tools

Adobe Express

5 tools

Works well with

Pike13

67 tools

ServiceM8

61 tools

protocols.io

55 tools

JobNimbus

54 tools

Frequently Asked Questions

How do I connect Local Desktop Agent to my app?

Use Weavz Hosted Connect for Custom authentication. Install the SDK, create a connection through the connect portal or API, then execute tools programmatically.

What tools does Local Desktop Agent support?

Local Desktop Agent supports 31 tools including Start Session, Ensure Connected, Session Status, Snapshot, Observe, and more.

Can I use Local Desktop Agent with AI agents?

Yes. Add Local Desktop Agent to an MCP server in Tool Mode or Code Mode. Code Mode provides 3 meta-tools with on-demand API discovery so agents can compose stateful workflows without loading every action schema up front.

Can Local Desktop Agent be used in stateful agent workflows?

Yes. Agents can combine Local Desktop Agent tools with Filesystem, State KV, Human Gates, input partials, and other workspace integrations while keeping execution scoped to the selected workspace or end user.

What authentication does Local Desktop Agent use?

Local Desktop Agent uses Custom. Weavz stores credentials securely and uses them for authorized action execution.

How much does the Local Desktop Agent integration cost?

The Local Desktop Agent integration is included on all Weavz plans including the free tier with 20,000 actions per month. No credit card required.

Connect Local Desktop Agent in minutes

Get 20,000 free action executions every month. No credit card required.

Get Started Free Browse All Integrations

Name	Description
`start_session` Start a user-approved local computer control session. The local Mac companion owns the allowed apps and scope. Prefer the signed-in Weavz Local Computer Control app; the returned command is a one-time fallback for setup or recovery. Call ensure_connected before control actions.	Start a user-approved local computer control session. The local Mac companion owns the allowed apps and scope. Prefer the signed-in Weavz Local Computer Control app; the returned command is a one-time fallback for setup or recovery. Call ensure_connected before control actions.
`ensure_connected` Preflight local computer readiness. If nextAction is grant_permissions, ask the local user to use the companion permission buttons and retry; if local_confirmation, wait for the local confirmation before acting.	Preflight local computer readiness. If nextAction is grant_permissions, ask the local user to use the companion permission buttons and retry; if local_confirmation, wait for the local confirmation before acting.
`session_status` Get the current local computer session state, permissions, capabilities, and companion status.	Get the current local computer session state, permissions, capabilities, and companion status.
`snapshot` Capture sanitized local target metadata and compact semantic element refs when available. Use observe for screenshot plus richer Mac UI semantics.	Capture sanitized local target metadata and compact semantic element refs when available. Use observe for screenshot plus richer Mac UI semantics.
`observe` Observe the selected Mac target with an agent-optimized screenshot, app/window metadata, focused element, cursor, display transform, and normalized accessibility element refs. Pass targetRef from list_targets to inspect an approved running app/window without activating it. Prefer observe before semantic ref actions.	Observe the selected Mac target with an agent-optimized screenshot, app/window metadata, focused element, cursor, display transform, and normalized accessibility element refs. Pass targetRef from list_targets to inspect an approved running app/window without activating it. Prefer observe before semantic ref actions.
`screenshot` Take an agent-optimized screenshot of the selected local target. App/window sessions capture the selected window; display sessions require local confirmation.	Take an agent-optimized screenshot of the selected local target. App/window sessions capture the selected window; display sessions require local confirmation.
`list_targets` List locally approved Mac app/window targets the agent can request. Use this before activate_target instead of Spotlight or Finder.	List locally approved Mac app/window targets the agent can request. Use this before activate_target instead of Spotlight or Finder.
`activate_target` Launch or focus a locally approved Mac app/window target returned by list_targets, then verify it is frontmost. Blocked targets require local user handling.	Launch or focus a locally approved Mac app/window target returned by list_targets, then verify it is frontmost. Blocked targets require local user handling.
`focus_ref` Focus an observed Mac UI element by ref. The companion re-resolves and verifies the element before acting.	Focus an observed Mac UI element by ref. The companion re-resolves and verifies the element before acting.
`press_ref` Press an observed Mac UI control by ref using Accessibility when possible, with bounded coordinate fallback only for the same verified element.	Press an observed Mac UI control by ref using Accessibility when possible, with bounded coordinate fallback only for the same verified element.
`set_text_ref` Set text in an observed editable, non-secure Mac UI field. Secure/password-like fields are rejected and should use request_human.	Set text in an observed editable, non-secure Mac UI field. Secure/password-like fields are rejected and should use request_human.
`select_ref` Select, toggle, expand, or collapse an observed Mac UI element by ref when supported.	Select, toggle, expand, or collapse an observed Mac UI element by ref when supported.
`scroll_ref` Scroll an observed Mac scroll area, list, table, or control by ref. Falls back to bounded wheel input over the verified element when needed.	Scroll an observed Mac scroll area, list, table, or control by ref. Falls back to bounded wheel input over the verified element when needed.
`move_mouse` Move the cursor to bounded coordinates inside the selected local target without clicking.	Move the cursor to bounded coordinates inside the selected local target without clicking.
`click` Click bounded coordinates inside the selected local target.	Click bounded coordinates inside the selected local target.
`drag` Drag the mouse along a bounded path of screenshot/display coordinates inside the selected local target.	Drag the mouse along a bounded path of screenshot/display coordinates inside the selected local target.
`type` Type text into the focused field. Secure fields and terminal/admin targets are rejected by the local companion.	Type text into the focused field. Secure fields and terminal/admin targets are rejected by the local companion.
`press_key` Press a key or key chord in the selected local target. Supports Enter, Escape, Tab, Space, arrows, Home, End, PageUp, PageDown, Delete, function keys F1-F12, letters, digits, and common punctuation.	Press a key or key chord in the selected local target. Supports Enter, Escape, Tab, Space, arrows, Home, End, PageUp, PageDown, Delete, function keys F1-F12, letters, digits, and common punctuation.
`scroll` Scroll inside the selected local target.	Scroll inside the selected local target.
`wait` Wait for a short amount of time or for the local companion to settle after a UI change.	Wait for a short amount of time or for the local companion to settle after a UI change.
`wait_for` Wait for a Mac UI condition such as app_active, element_exists, element_focused, text_present, window_title_contains, dialog_present, or screen_stable. Prefer this over blind sleeps.	Wait for a Mac UI condition such as app_active, element_exists, element_focused, text_present, window_title_contains, dialog_present, or screen_stable. Prefer this over blind sleeps.
`run_steps` Run up to 50 deterministic desktop operations in one session round trip. Supports semantic ref actions, per-step delayMs, and per-step waitUntil conditions. Returns per-step ok/result/error entries and can include a final observation or screenshot.	Run up to 50 deterministic desktop operations in one session round trip. Supports semantic ref actions, per-step delayMs, and per-step waitUntil conditions. Returns per-step ok/result/error entries and can include a final observation or screenshot.
`request_human` Hand control to the local user for login, MFA, sensitive prompts, or manual review.	Hand control to the local user for login, MFA, sensitive prompts, or manual review.
`resume` Request agent control again. If the local user reclaimed control, local confirmation is required.	Request agent control again. If the local user reclaimed control, local confirmation is required.
`end_session` End the local computer control session and disconnect the companion.	End the local computer control session and disconnect the companion.
`act` Use the configured model provider to complete an approved Mac task through the local desktop workspace. Prefer this when the caller wants the agent to plan, act, wait, verify, and stop for confirmation instead of chaining low-level mouse and keyboard steps. The companion enforces sign-in, permissions, selected apps, capability gates, pause/resume, and human confirmation.	Use the configured model provider to complete an approved Mac task through the local desktop workspace. Prefer this when the caller wants the agent to plan, act, wait, verify, and stop for confirmation instead of chaining low-level mouse and keyboard steps. The companion enforces sign-in, permissions, selected apps, capability gates, pause/resume, and human confirmation.
`plan` Inspect the current local Mac state and produce a safe execution plan without acting.	Inspect the current local Mac state and produce a safe execution plan without acting.
`observe_goal` Observe the local Mac and identify UI candidates relevant to an instruction without acting.	Observe the local Mac and identify UI candidates relevant to an instruction without acting.
`extract_from_screen` Extract structured information from the current local Mac observation without acting.	Extract structured information from the current local Mac observation without acting.
`run_recipe` Run a transparent deterministic local-computer recipe made of existing safe operations, waits, and confirmation points.	Run a transparent deterministic local-computer recipe made of existing safe operations, waits, and confirmation points.
`stop_task` Stop the current Local Desktop Agent task and return control to the local user.	Stop the current Local Desktop Agent task and return control to the local user.