Reveal
The ball appears beside a cup—the memory cue.
NYCU AI CAPSTONE · 2026
ShellBench: A Parametric Shell Game Benchmark for Memory-Aware Manipulation
ShellBench turns the classic shell game into a controlled robotics benchmark for measuring long-horizon memory—not just dexterity.
01 / THE PROBLEM
Manipulation policies are good at reacting.
We test whether they can remember.
Once the ball is covered, every cup looks identical. The task becomes non-Markovian: the correct action depends on information that disappeared hundreds of frames ago. Short observation windows may learn how to lift a cup, but not which cup to lift.
02 / BENCHMARK
One task.
Four distinct phases.
Cover
The cup hides the ball and removes visual evidence.
Shuffle
Cups swap through a controlled sequence.
Act
The policy selects and lifts the remembered cup.
PARAMETRIC DIFFICULTY
Turn memory into a measurable variable.
3–5cups
0–5+shuffles
0.5–2×shuffle speed
03 / METHODS
Short-term generation
vs. persistent memory.
Diffusion Policy
Two camera views and robot state condition a denoising model that generates smooth action chunks. It replans effectively, but sees only a short recent window.
2 frames → ResNet-18 → 16 actions
LSTM Policy
A recurrent state persists from reveal, through every shuffle, until action—carrying information that is no longer visible.
230 steps → 2-layer LSTM → 8 actions
OBSERVEFront + wrist cameras9D robot state
→ENCODEVisual backboneResNet-18
→REMEMBERTemporal modelWindow or recurrent state
→ACTAction chunk7 joints + gripper
04 / EVALUATION
Separate knowing
from doing.
Decision Success Rate
Did the policy choose the cup hiding the ball?
Manipulation Success Rate
Did the robot successfully lift any cup?
Success Rate
Was the correct cup selected and lifted?
Cohen's Kappa
Decision accuracy corrected for random chance.
CURRENT RESEARCH INSIGHT
Good hands can hide a bad memory.
Early LSTM runs achieved perfect manipulation while repeatedly choosing one position. Action regression can average across valid trajectories and collapse to a positional preference.
ShellBench exposed this because DSR and MSR are independent. A cup-classification head now supervises the decision explicitly; validation is ongoing.
05 / CONTRIBUTIONS
Built for reproducible
memory research.
01
A non-Markovian manipulation benchmark with controllable difficulty.
02
A unified Isaac Sim and LeRobot pipeline for data, training, sweeps, and evaluation.
03
Metrics that disentangle memory decisions from manipulation skill.
04
Direct comparison of Diffusion Policy and episode-level LSTM memory.