A robot should inspect its movement, explain what it expects, notice when reality disagrees, and deliberately try again.

Robots are usually controlled through layers of abstraction: planner, grasp detector, inverse kinematics, trajectory generator, and finally motor control.

This stack is useful. It also makes intelligence difficult to attribute. If a frontier vision model calls top_grasp(red_cube), did the model understand how to grasp the cube—or did the skill?

We are starting with the SO-101, a low-cost five-degree-of-freedom arm. The model receives camera images, joint state, and results of its previous actions. It returns bounded changes to the robot’s joints. Deterministic software handles timing and non-negotiable safety, but every meaningful movement decision belongs to the model.

The interface is not incidental. It is the research problem.

The hypothesis

1. Frontier models already contain useful task-level physical priors.
2. Given a legible action interface and action-conditioned visual feedback, they can discover how those priors map onto a particular robot body.

Physical intuition is not calibrated robot control. A model does not begin with knowledge of the SO-101’s joint signs, backlash, limits, camera geometry, or the interactions between joints.

Direct control does not ask a language model to generate raw motor current at hundreds of hertz. It asks the model to choose each meaningful joint-position action.

{
  "joint_delta_rad": {
    "shoulder_pan":  0.010,
    "shoulder_lift": -0.020,
    "elbow_flex":    0.015,
    "wrist_flex":    0.000,
    "wrist_roll":    0.005
  },
  "gripper_delta": -0.10,
  "duration_s": 0.50,
  "expected_effect":
    "lower the open gripper toward the cube"
}

A high-frequency controller interpolates between the measured configuration and the accepted target. While the model thinks, the arm holds position.

The safety layer may reject malformed actions, enforce joint and workspace limits, constrain velocity and acceleration, stop on stale observations or excessive load, and prevent known collisions. It may not run IK, infer a better grasp, substitute another trajectory, or complete the task for the model.

Every action produces an execution receipt: what was requested, accepted or rejected, actually executed, and subsequently observed.

Drawing before moving

The harness also includes a path-drawing interface where a trajectory can be sketched directly into the robot’s visual workspace. A path passes through an explicit lifecycle—draw, validate, request, approve, execute—and remains visually comparable with measured motion.

The sketch is the plan; the joints are the policy.

A ghost robot can preview proposed joint configurations, but that preview provides an external forward simulator. It is therefore an experimental variable, not an invisible default.

If the hypothesis begins with knowledge learned from video, the feedback cannot consist only of isolated screenshots.

After each action, the harness returns a synchronized motion replay: the frame before movement, several frames during it, the final frame, joint position and velocity, available motor-load telemetry, and the requested, mediated, and executed action.

Before execution, the model predicts what its action will do. Afterwards, it compares that prediction with the measured result. This separates success by chance from operational understanding.

“Does the model understand physics?” is too broad. The evaluation separates three capabilities and distinguishes cold-start performance from within-episode learning.

Task ladder

1. Move the gripper in a requested image-space direction.
2. Trace a visible three-dimensional path.
3. Touch a specified object.
4. Push an object in a requested direction.
5. Grasp and lift an object.
6. Place an object inside a target region.

Matched interfaces

For each task, the model, scene, seeds, action budget, and time budget remain constant while the interface changes:

1. RGB only → direct joint deltas.
2. RGB plus named numerical joint state.
3. RGB plus joint state and temporal action history.
4. VLM-drawn path plus direct joint control.
5. Ghost-configuration preview.
6. Short simulated rollout preview.
7. Direct Cartesian control.
8. A trained action model or VLA baseline.

Task success is only one metric. The harness also records valid-action rate, safety rejection and mediation rate, progress per action, path error, contacts, peak load, correction quality, intervention rate, wall-clock time, and inference cost.

Camera placement and calibration can then vary to reveal whether the model learned the embodiment or merely memorized a convenient view.

Direct joint control describes how the model acts. It does not describe how the model learns.

For learning, we are exploring EGGROLL, the optimisation method introduced in Evolution Strategies at the Hyperscale. EGGROLL stands for Evolution Guided GeneRal Optimisation via Low-rank Learning.

Each perturbed model observes the robot and proposes joint deltas. The simulator applies the same hard safety envelope and produces scalar fitness from task success, progress, path error, collisions, load, action efficiency, and interventions. EGGROLL shifts parameters toward perturbations that produced better rollouts.

The objective need not be differentiable. It may contain simulator events, programmatic checks, safety violations, visual judges, or outcomes from an external process. Low-rank perturbations make population-based optimisation substantially more practical at large population sizes.

Research progression

1. Measure what the unmodified frontier model can already do.
2. Measure how quickly it identifies the SO-101 embodiment within an episode.
3. Use EGGROLL to optimise direct-control performance across simulated episodes.
4. Test whether improvements transfer to changed scenes, cameras, and eventually hardware.

The harness separates staging from execution through immutable, single-use approvals. Revising a trajectory or changing observed state invalidates approval.

The executing process independently monitors observation freshness, tracking error, timeouts, motor state, and stop requests.

Raw torque or current control is a legitimate future research question, but it is a different regime with much stronger real-time and hardware-safety requirements. It is not part of the initial system.

Recent work suggests frontier models become capable robot controllers when given the right interfaces. These systems primarily ask models to operate Cartesian interfaces, compose skills, supervise policies, or write robot programs.

Our experiments ask something complementary: what happens when the foundation model must inhabit the robot’s joints itself?

Cartesian waypoints, visual probes, IK, learned skills, and VLAs remain valuable. Here, they become controlled comparison conditions rather than invisible sources of capability.

Learning from experience

Each trace contains the model’s observation, physical prediction, requested joint action, safety mediation, executed trajectory, motion video, measured outcome, and subsequent correction.

These traces can reveal systematic misunderstandings of embodiment. They may also train future agents to control joints more efficiently, predict failure sooner, and request human help when uncertainty is high.

Over time, the system may progress from slow action-by-action reasoning to adaptive action chunks, then to models trained specifically on their own interaction traces. But the first objective is simpler: construct an interface honest enough to reveal what frontier models can already do.