How Does Anthropic’s Computer Use Work and What Is Mechanistic Interpretability?

The OS-Level Breakthrough: Anthropic’s "Computer Use" API

In October 2024, Anthropic became the first frontier AI laboratory to release **Computer Use**—a capability allowing an AI model (Claude 3.5 Sonnet) to operate a standard computer desktop exactly as a human does: by looking at a screen, moving a cursor, clicking buttons, and typing text [1,2].

Prior automation systems required developers to build bespoke API integrations for every specific application [1,2]. Anthropic took the generalist approach: when provided access to a virtual containerized OS (such as Ubuntu with X11), Claude receives a continuous stream of raw desktop screenshots [1,2,3]. The model outputs structured JSON coordinate commands (e.g., `mouse_move(x=450, y=320)`, `left_click()`, `type(text="git status")`, `key_combination("Ctrl+C")`), enabling it to navigate arbitrary web browsers, legacy ERP software, and terminal environments out of the box [1,2,3].

"Rather than requiring custom API connectors, Claude looks at raw desktop screenshots and emits pixel-coordinate mouse clicks and keystrokes to operate any software."

Overcoming the Coordinate Precision and Latency Bottleneck

Direct pixel-level interaction poses extreme challenges for Vision-Language Models [1,3,4]. Standard transformers tend to blur spatial coordinate details when resizing high-resolution $1920 \times 1080$ displays [1,3].

Anthropic trained Claude with specialized spatial grounding objectives, allowing the model to accurately translate semantic goals (*"Click the blue Submit button in the top right"*) into exact pixel coordinates [1,3,4]. Furthermore, the model inspects subsequent screenshots to verify whether dropdowns opened or forms submitted, automatically retrying if a click missed its target or if a pop-up modal blocked execution [1,2,4].

Opening the Black Box: Mechanistic Interpretability & Sparse Autoencoders

Alongside agent capabilities, Anthropic has led the world in **Mechanistic Interpretability**—the science of reverse-engineering the internal weights and activations of neural networks into human-understandable algorithms [1,5,6].

For decades, neural networks were deemed uninterpretable "black boxes" because individual artificial neurons exhibited **superposition**: a single neuron activated for dozens of unrelated concepts (e.g., Arabic text, radioactive decay, and poetry) [1,5,6]. In 2024, Anthropic published *Scaling Monosemanticity*, using massive **Sparse Autoencoders (SAEs)** with millions of dictionary features to disentangle Claude 3 Sonnet’s internal activations [1,5,6].

"Sparse Autoencoders (SAEs) decompose the tangled activations of neural networks into millions of crisp "monosemantic features" representing distinct concepts."

Mapping Millions of Features: From the Golden Gate Bridge to Safety Alignment

The SAE dictionary extracted millions of high-level semantic features, including features corresponding to specific cities, scientific concepts, programming bugs, and the famous "Golden Gate Bridge" feature (clamping this feature caused Claude to become obsessed with the bridge in every response) [1,5,6].

Crucially, interpretability researchers identified safety-critical internal features: neurons that light up when the model is contemplating deception, power-seeking, generating bioweapon synthesis instructions, or encountering adversarial jailbreak attempts [1,5,7]. By monitoring and steering these features directly during inference, engineers can mathematically verify model safety before deploying autonomous agents with system-level computer access [1,5,7].