Success Metrics
4
Stringman robots deployediThis is the number of robots that have registered with Neufangled.com and been active in the last three months. It does not count bots that operate exclusively in LAN mode, so it is an underestimate.

Robot Evaluation

Installation Success

Starting from a fresh image, install the Stringman software, set up the robot, and run it following the same instructions a customer would. How many problems came up during install?

— problems during install

Positioning Accuracy

Freshly calibrate the robot. Pick ten random starting locations in the work area and command a move to a known location — e.g. placing the fingers around a marble on a stick exactly one meter above the origin. Measure the distance to target after the move.

— avg distance to target

Move Linearity

Sensitive to calibration errors in a cable robot. Command the robot to move in a horizontal line at constant speed across the work area. Measure the average deviation from the target altitude with the laser rangefinder, in a clear area with nothing on the floor.

— avg altitude deviation

Uptime

Average number of hours the robot can stay turned on before something makes it unusable. (Currently the limiter is overheating Raspberry Pis.)

— hours before failure

Model Evaluation

Grasping Success

From a collection of 20 objects that do not occur in the training data, challenge the robot to pick up each object starting from a close position. It succeeds if it lifts the object securely within 30 seconds. The metric is the fraction of the set that can be grasped.

— / 20 objects grasped

Target Identification

From an overhead picture of a room captured with the same kind of cameras, output video overlays pointing out the targets the model plans to pick up. The room contains 20 objects absent from the training data. The metric is targets positively identified minus false positives.

— identified − false positives

Room Cleanup Test

20 objects from each category the robot handles (currently laundry, toys, and trash) are thrown around the work area. The "pick and place" job is activated with a single button, as a non-technical customer would. The score is how many items end up in the correct bin after 30 minutes.

— items in correct bin (30 min)