Local Computer Control
Use Agent Local Computer Control to run a visible, user-approved macOS desktop session from a workspace integration.
Local Computer Control
agent-local-computer-control is an experimental built-in workspace integration for controlling a local Mac through a visible companion app. Use it when a workflow needs a real user device, local network access, or an already configured desktop app, and hosted browser automation is not enough.
Local computer control is always consented and visible. The local user starts the companion, chooses the desktop target, grants macOS permissions, and can pause or revoke control from the companion UI at any time.
The npm launcher is available as the public command surface, but the signed macOS app artifact is private-preview only until the public app manifest marks it available. Only enable this integration for users who already have access to the signed Weavz Local Computer Control app.
When to use it
Use Local Computer Control for workflows that need:
- A desktop app or app window instead of a web page
- The user's device, network, certificates, or local app state
- A bounded click/type/screenshot loop with human handoff
- A local user available to approve setup, sensitive steps, or resume after reclaiming control
Prefer hosted agent-browser when the workflow can run in a managed browser session. Prefer agent-local-browser-control when the workflow only needs a browser on the user's machine.
Setup flow
Add the integration to a workspace with a stable alias:
{
"integrationName": "agent-local-computer-control",
"alias": "local_mac",
"displayName": "Local Mac"
}Then call start_session. The response includes a local runner command:
npx -y @weavz-io/local-computer-control connect --url "<localRunnerUrl from start_session>"The local user runs the command on their Mac. If the signed app is installed, the launcher opens the Weavz Local Computer Control app, which guides the user through Screen Recording and Accessibility permission setup when needed. If the signed app is not installed yet, the launcher exits with the release page where download metadata will appear when the public app artifact is available.
After the companion is open, call ensure_connected. Continue only when it returns ready: true.
Sessions are scoped to the configured workspace integration and the execution end user when an end user is supplied. Backend-owned executions without an end user use the workspace scope, so only use that mode for a shared machine the workspace owner intentionally approves.
Action model
The integration exposes session controls and bounded desktop operations:
| Action | Purpose |
|---|---|
start_session | Start or replace the active local computer session for the configured workspace integration and user scope |
ensure_connected | Wait briefly for the companion and return setup or recovery instructions |
session_status | Read connection, permission, handoff, and device status |
snapshot | Capture sanitized target metadata; v1 does not expose stable accessibility element refs |
screenshot | Capture an agent-optimized screenshot of the selected app/window or locally confirmed display |
move_mouse, click, drag, type, press_key, scroll, wait | Run bounded desktop interactions with mouse and keyboard primitives |
run_steps | Execute up to 50 deterministic desktop steps in one call |
request_human, resume | Hand control to the local user and request agent control again |
end_session | Revoke the session and disconnect the companion |
For agent loops, call screenshot for visual observation, then use that screenshot coordinate space
for move_mouse, click, and drag. App/window screenshots use window-local coordinates that the
companion translates to the selected screen window; display screenshots use screen coordinates. Use
run_steps only for short deterministic sequences after the target state is known. Use
request_human instead of controlling login, MFA, payment, admin, or sensitive prompts.
Safety behavior
The first version is intentionally narrow:
- macOS is the only supported platform.
- The default target is the currently selected app/window scope; the companion rejects display-wide screenshots unless display scope is explicitly selected.
- Display-wide control requires explicit local selection.
- The session pauses if focus drifts outside the selected target unless the user selected display control.
- The local user can pause or revoke from the companion UI.
- If the local user reclaims control,
resumereportsrequiresLocalConfirmation: trueuntil the companion confirms agent control again. - Desktop snapshots redact secure fields and password-like values.
The integration does not provide Terminal or shell automation, filesystem browsing, clipboard access, password extraction, Keychain access, passkey extraction, browser cookie export, microphone capture, camera capture, system audio capture, lock screen control, or admin prompt automation.
MCP and SDK usage
Expose the workspace integration through MCP the same way you expose other built-ins. In Code Mode, a configured alias such as local_mac appears under weavz.local_mac.
For SDK calls, use generated integration action helpers after adding the workspace integration. Do not pass a connectionId; this built-in has no external account auth.