Local Computer Control

Local Computer Control gives agents visible, user-approved access to a signed-in Mac when work must happen in a desktop app, on the user's device, or on the user's network.

For goal-based desktop work, use local-computer-control-agent with an alias such as computer_agent. The caller gives an instruction and Weavz chooses safe desktop steps with the configured model provider. Keep local-computer-control available when you need explicit, step-by-step desktop operations for debugging, recovery, or narrow workflows.

Local computer control is always consented and visible. The local user signs in to the companion, grants macOS permissions, turns on local access, chooses the desktop target, and can pause or turn off control from the companion UI at any time.

The npm launcher is available as the public command surface, but the signed macOS app artifact is private-preview only until the public app manifest marks it available. Only enable this integration for users who already have access to the signed Weavz Local Computer Control app.

When to use it

Use Local Computer Control for workflows that need:

• A desktop app or app window instead of a web page
• The user's device, network, certificates, or local app state
• A bounded semantic observe-act loop with screenshot and coordinate fallback
• A local user available to approve setup, sensitive steps, or resume after reclaiming control

Prefer hosted agent-browser when the workflow can run in a managed browser session. Prefer agent-local-browser-control when the workflow only needs a browser on the user's machine.

Setup flow

Add the deterministic integration to a workspace with a stable alias:

{
  "integrationName": "local-computer-control",
  "alias": "computer",
  "displayName": "Local Computer Control"
}

Add the managed agent integration when you want the act loop:

{
  "integrationName": "local-computer-control-agent",
  "alias": "computer_agent",
  "displayName": "Local Desktop Agent"
}

local-computer-control-agent requires a model-provider connection, the same way other Weavz AI agent tools do. The shared provider/model picker supports OpenAI, Anthropic, Google AI, and xAI.

The normal setup path is:

1. Install and open Weavz Local Computer Control on the Mac.
2. Sign in to the same Weavz workspace.
3. Grant Screen Recording and Accessibility from the companion.
4. Choose the allowed apps or full-computer display mode in the companion.
5. Turn on Allow Weavz to control this computer.
6. Have the agent call start_session, then ensure_connected.

When the signed-in app is online and local access is on, Weavz connects the pending session automatically. The start_session response may also include a one-time local runner command for setup or recovery:

npx -y @weavz-io/local-computer-control connect --url "<localRunnerUrl from start_session>"

If the signed app is installed, the launcher opens a one-time recovery session in Weavz Local Computer Control. This command-started session does not keep the Mac connected for future agent work. If the signed app is not installed yet, the launcher exits with the release page where download metadata will appear when the public app artifact is available.

After the companion is open, call ensure_connected. Continue only when it returns ready: true.

Sessions are scoped to the configured workspace integration and the execution end user when an end user is supplied. Backend-owned executions without an end user use the workspace scope, so only use that mode for a shared machine the workspace owner intentionally approves.

Action model

The deterministic integration exposes session controls and bounded desktop operations:

For agent loops, call ensure_connected, then observe. Prefer the returned element refs with focus_ref, press_ref, set_text_ref, select_ref, or scroll_ref, then call wait_for and observe again. Use list_targets and activate_target to focus locally allowed apps instead of driving Spotlight or Finder. Use coordinates only when no safe ref exists; app/window screenshots use window-local coordinates that the companion translates to the selected screen window, while display screenshots use screen coordinates. Use run_steps only for short deterministic sequences after the target state is known, with small delays or waitUntil conditions between UI-changing steps. Use request_human instead of controlling login, MFA, payment, admin, or sensitive prompts.

The managed local-computer-control-agent integration adds act, plan, observe_goal, extract_from_screen, run_recipe, and stop_task. Prefer act when the caller has a goal and wants Weavz to observe the Mac, choose the next safe operation, wait, verify, and stop when complete or when local human confirmation is required.

Safety behavior

• macOS is the only supported platform.
• The local companion owns the allowed-app list and full-computer display mode; agents cannot widen this from start_session.
• Display-wide control requires explicit local selection and confirmation.
• The session pauses if focus drifts outside the selected target unless the user selected display control.
• The local user can pause or turn off control from the companion UI and remove the device by signing out.
• If the local user reclaims control, resume reports requiresLocalConfirmation: true until the companion confirms agent control again.
• Desktop observations redact secure fields and password-like values.
• The deterministic control integration does not expose shell, filesystem, or clipboard actions.
• The managed AI companion can use broader local capabilities only when the local user explicitly enables them in the Mac app and approves risky actions locally.

Neither integration should be used for password extraction, Keychain access, passkey extraction, browser cookie export, microphone capture, camera capture, lock screen control, or admin prompt automation. Browser page-structure control remains a separate local browser workflow until the Mac companion exposes that capability.

MCP and SDK usage

Expose the workspace integration through MCP the same way you expose other built-ins. In Code Mode, configured aliases such as computer or computer_agent appear under weavz.<alias>.

For SDK calls, use generated integration action helpers after adding the workspace integration. Do not pass a connectionId for local-computer-control; local-computer-control-agent uses a model-provider connection.

Related docs

• Built-In Workspace Integrations
• Using Built-In Workspace Integrations
• Agent Browser
• End Users