Skip to content

AI Agents (MCP)

stringman-headless can serve an MCP server, so an AI agent such as Claude Code can watch and drive your robot. It can also write new behaviors and load them into the running controller.

Quick start

Install the controller with the mcp extra, start it with the MCP server on, then add the server to your agent.

python3 -m virtualenv venv
source venv/bin/activate
pip install "nf_robot[host,mcp]"
stringman-headless --mcp --mcp_allow_code --maneuver_dir ./agent_maneuvers

python3.13.exe -m pip install "nf_robot[host,mcp]"
python3.13.exe -m nf_robot.host.observer --mcp --mcp_allow_code --maneuver_dir ./agent_maneuvers

python3.13 -m venv venv
source venv/bin/activate
pip install "nf_robot[host,mcp]"
stringman-headless --mcp --mcp_allow_code --maneuver_dir ./agent_maneuvers

In a second terminal, register the server with Claude Code and start a session:

claude mcp add --transport http stringman http://127.0.0.1:4250/mcp
claude

Then give it a first task, for example:

Read the stringman guide, then show me what the gripper camera sees.

Any other MCP client that supports streamable HTTP can connect to the same URL, http://127.0.0.1:4250/mcp.

Warning

--mcp_allow_code lets the agent run arbitrary Python inside the controller process. Leave it off if you only want the agent to watch and drive.

What the agent can do

The server runs inside the controller itself. The agent's tools call the same methods as the robot's built-in behaviors. Everything the agent does is subject to the same stop button, line tension monitor, marker monitor and parked check.

Purpose Tools
Read get_state, get_events, capabilities, api_docs, guide
See get_image, get_images, get_frames, record_clip (gripper camera, anchor cameras, top-down floor view)
Move move_to, nudge, drive, run_command, run_verb
Gripper grip, set_fingers, set_wrist, run_command('GRASP') for automatic grasping
Wait for a result wait, get_events
Stop stop
Popups answer_popup
Code (needs --mcp_allow_code) load_maneuver, unload_maneuver, get_maneuver_source, save_maneuver, run_script

The guide tool serves the agent its instructions, which cover safety rules, coordinates, how motion ends, gripping, and writing maneuvers. Agents are told to read it before anything else. You can read the same text in GUIDE.md. Practical lessons from earlier sessions are kept in TRICKS.md. The server does not serve that file, so point your agent at it if you want it to use them.

Moves return at once. The agent then calls wait to learn whether the move finished, failed, or was stopped by you or by a safety monitor.

Writing behaviors

With --mcp_allow_code, the agent can write a maneuver and load it into the running controller without restarting. It can then run the maneuver's verbs, watch what happened, edit the code, and load it again. Maneuvers it saves go in the --maneuver_dir folder, and are loaded from there every time the controller starts. You can also run a saved verb yourself from the debug box.

Safety

  • The agent is told to ask before its first move. Check that the room is clear and that nobody is under the gripper before you agree.
  • The stop button in the control panel always works, and so does asking the agent to stop. The agent is told to ask you why before retrying a task you stopped.
  • The agent won't unpark the robot, answer popups, or weaken a safety monitor on its own. It asks you first. Shutdown and firmware updates are refused through MCP.
  • Stay nearby the first few times. The agent can't see everything you can. Tell it about things it shouldn't touch, where things belong, and anything the cameras miss.

Options

Option Description
--mcp [PORT] Serve the MCP server at http://127.0.0.1:PORT/mcp. The port is 4250 if not given.
--mcp_allow_code Allow the code tools. Off unless given.
--maneuver_dir DIR Folder of maneuver modules (*.py) that are loaded at startup, and where the agent saves the maneuvers it writes.

The server only listens on 127.0.0.1. To use an agent on a different machine from the one running stringman-headless, forward the port over SSH from the agent's machine:

ssh -L 4250:127.0.0.1:4250 user@robot-host

Tips for good results

  • Give a goal and the context only you know. For example, "clear the floor; toys go in the two bins, trash in the can by the door, papers on the desk".
  • Expect it to learn as it goes. Early attempts are slow. Once the agent has a working routine, such as "drop what's held, then grasp the next thing", it speeds up a lot. In one playroom cleanup, about 70 items were cleared in about two hours.
  • Help it with things it can't fix itself. Move bins next to the pile, take away objects the gripper can't hold, and move things that distract automatic grasping, such as a book cover with a large printed eye.
  • Calibrate before a long task. Position estimates drift over many moves. A half calibration fixes this, and the agent knows to run one.