Robot Evaluation
Installation Success
Starting from a fresh image, install the Stringman software, set up the robot, and run it following the same instructions a customer would. How many problems came up during install?
Positioning Accuracy
Freshly calibrate the robot. Pick ten random starting locations in the work area and command a move to a known location — e.g. placing the fingers around a marble on a stick exactly one meter above the origin. Measure the distance to target after the move.
Move Linearity
Sensitive to calibration errors in a cable robot. Command the robot to move in a horizontal line at constant speed across the work area. Measure the average deviation from the target altitude with the laser rangefinder, in a clear area with nothing on the floor.
Uptime
Average number of hours the robot can stay turned on before something makes it unusable. (Currently the limiter is overheating Raspberry Pis.)
Model Evaluation
Grasping Success
From a collection of 20 objects that do not occur in the training data, challenge the robot to pick up each object starting from a close position. It succeeds if it lifts the object securely within 30 seconds. The metric is the fraction of the set that can be grasped.
Target Identification
From an overhead picture of a room captured with the same kind of cameras, output video overlays pointing out the targets the model plans to pick up. The room contains 20 objects absent from the training data. The metric is targets positively identified minus false positives.
Room Cleanup Test
20 objects from each category the robot handles (currently laundry, toys, and trash) are thrown around the work area. The "pick and place" job is activated with a single button, as a non-technical customer would. The score is how many items end up in the correct bin after 30 minutes.