diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/_index.md b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/_index.md new file mode 100644 index 0000000000..3ad1e8c855 --- /dev/null +++ b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/_index.md @@ -0,0 +1,75 @@ +--- +title: Build an Edge AI Reachy Mini App with Raspberry Pi, MediaPipe, and MuJoCo + +description: Run MediaPipe gesture inference on a Raspberry Pi 5, connect to a Reachy Mini MuJoCo simulation on a development machine, and use a browser dashboard to decide Reachy's fate with a thumbs-up or thumbs-down. + +minutes_to_complete: 60 + +who_is_this_for: This Learning Path is for developers interested in edge AI, robotics simulation, and physical AI applications. You can complete the main path without owning a physical Reachy Mini. + +learning_objectives: + - Understand why simulation environments can aid Edge AI and robotics development. + - Run a simulated Reachy Mini robot on a laptop or desktop. + - Use MediaPipe and TensorFlow Lite gesture recognition on Raspberry Pi 5. + - Connect an edge inference node to a robot daemon over the network. + - Display results over a web dashboard. + - (Optional) Extend the project toward a physical Reachy Mini, audio or multimodal interaction, or your own app. + +prerequisites: + - A Raspberry Pi 5, ideally with 16 GB RAM. + - A USB webcam connected to the Raspberry Pi. + - A macOS or Linux machine, or a Windows machine with WSL2, capable of running the Reachy Mini MuJoCo simulation. + - Basic Python and Bash terminal experience. + - (Optional) [Reachy Mini](https://huggingface.co/reachy-mini) + +author: Matt Cossins + +### Tags +skilllevels: Introductory +subjects: ML +armips: + - Cortex-A +tools_software_languages: + - Raspberry Pi + - Reachy Mini + - Python + - MediaPipe + - FastAPI + - MuJoCo +operatingsystems: + - Linux + - macOS + - Windows + +### Cross-platform metadata only +shared_path: true +shared_between: + - embedded-and-microcontrollers + - laptops-and-desktops + +further_reading: + - resource: + title: Make and publish your Reachy Mini app + link: https://huggingface.co/blog/pollen-robotics/make-and-publish-your-reachy-mini-apps + type: blog + - resource: + title: Reachy Mini Python SDK documentation + link: https://pollen-robotics.github.io/reachy-mini-sdk/ + type: documentation + - resource: + title: Reachy Mini project examples + link: https://github.com/pollen-robotics/reachy-mini-sdk/tree/main/examples + type: website + - resource: + title: MediaPipe Gesture Recognizer guide + link: https://ai.google.dev/edge/mediapipe/solutions/vision/gesture_recognizer + type: documentation + + + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 # _index.md always has weight of 1 to order correctly +layout: "learningpathall" # All files under learning paths have this same wrapper +learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. +--- diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/_next-steps.md b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/_next-steps.md new file mode 100644 index 0000000000..727b395ddd --- /dev/null +++ b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # The weight controls the order of the pages. _index.md always has weight 1. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/battle-cry.gif b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/battle-cry.gif new file mode 100644 index 0000000000..0822020054 Binary files /dev/null and b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/battle-cry.gif differ diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/extend.md b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/extend.md new file mode 100644 index 0000000000..a99cae4655 --- /dev/null +++ b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/extend.md @@ -0,0 +1,132 @@ +--- +title: (Optional) Extend the project +weight: 7 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Try focused changes + +You now have a working physical AI pattern: sensor input at the edge, local +inference, robot commands, and a dashboard for visibility. A good next step is +to make small changes where the effect is easy to observe: + +- Change `VERDICT_MIN_CONFIDENCE` in `main.py` to make thumbs detection + stricter or more forgiving. +- Change `MOVE_REPETITIONS` in `main.py` so each move plays more or fewer + times. +- Add a fifth move in `moves.py` and register it in `MOVE_CATALOGUE`. +- Try a different webcam with `REACHY_GLADIATOR_CAMERA_INDEX=1`. + +These changes are deliberately small. They help you learn which part of the +system owns each behavior before you make larger changes to perception, robot +motion, or packaging. + +## Change the classifier + +Trying adding new gesture controls. Start in `gesture.py`, where the app maps MediaPipe labels such as `Thumb_Up` and `Thumb_Down` to app labels. Then add matching robot behavior in `moves.py` and branch on the new label in `main.py`. + +Some game-themed ideas: + +- Map `Closed_Fist` to `challenge`, and make Reachy repeat the current move +- Map `Pointing_Up` to `reroll`, and make Reachy reject the current move and + choose another one. +- Map `Number of fingers` to choosing a specific move. + +## Add audio output + +The gladiator theme is a good fit for sound. Try adding audio cues such as: + +- a crowd cheer during victory, +- a dramatic sound during defeat, +- a short drum hit before each move, +- a spoken move name before Reachy performs it. + +Keep audio output separate from `moves.py` at first. For example, create an +`audio.py` helper and call it from `main.py` when the state changes. This keeps +robot motion and sound effects easy to change independently. + +## Replace thumbs with audio input + +The vision-based verdict is one edge AI input modality. You can replace +or complement it with audio - many webcams include microphones or you can use a USB microphone. + +- say "yes" for victory and "no" for defeat, +- clap once for victory and twice for defeat, + +A lightweight keyword-spotting model can map spoken commands to the same game states currently triggered by MediaPipe gestures. + +## Try the packaged app on a physical Reachy + +If you have a physical Reachy Mini, the quickest way to try the finished +experience is to install the packaged [Reachy Gladiator app](https://huggingface.co/spaces/cossinsmatthew/reachy_gladiator) through the [Reachy Mini Control app](https://github.com/pollen-robotics/reachy-mini-desktop-app). + +Simply install Reachy Mini Control on a supported machine, connected to your Reachy, and search for the reachy gladiator app. + +{{% notice Warning %}} +If using a physical Reachy Mini, use caution and ensure the robot is used in an area with appropriate space. The robot has moving parts and could be a health & safety risk. You are responsible for your safety and the safety of others around you when using physical robotic devices +{{% /notice %}} + +## Adapt this source project for physical Reachy + +The main learning path uses the Raspberry Pi USB webcam for perception and a +remote MuJoCo daemon for robot motion. A physical Reachy route changes two +things: + +- camera frames come from the Reachy daemon instead of the Pi USB webcam, +- the Pi app connects to the physical Reachy daemon instead of the simulation + daemon. + +The source project exposes these switches as environment variables, so you do +not need to edit the Python source: + +```bash +REACHY_GLADIATOR_MEDIA_BACKEND=reachy \ +REACHY_GLADIATOR_CAMERA=reachy \ +REACHY_GLADIATOR_DAEMON_PORT=8000 \ +./scripts/run_pi_app.sh localhost +``` + +Use `localhost` only when the physical daemon runs on the same Pi as the app. +If the daemon runs on another machine, replace `localhost` with that machine's +IP address and set `REACHY_GLADIATOR_DAEMON_PORT` to the daemon port. + +These variables map to the code in two places: + +- `REACHY_GLADIATOR_MEDIA_BACKEND=reachy` lets `ReachyMiniApp` request daemon + camera media. +- `REACHY_GLADIATOR_CAMERA=reachy` tells `camera.py` to use + `ReachyMediaFrameSource` instead of `OpenCVCameraFrameSource`. + +## Build your own Reachy Mini app + +A Reachy Mini app is a Python package with a class that inherits from `ReachyMiniApp`, implements `run()`, and exposes an entry point in `pyproject.toml`. + +To build a fresh app: + +1. Start with one `ReachyMiniApp` class. +2. Add one safe motion such as `neutral()`. +3. Add one input source such as a camera gesture, audio command, button, or web + endpoint. +4. Add a dashboard only after the core loop works. +5. Test in simulation before physical hardware. +6. Package the app when it is stable enough for repeated use. + +The Reachy Mini tooling can scaffold and validate a shareable app: + +```bash +reachy-mini-app-assistant create my_app ~/reachy_projects +reachy-mini-app-assistant check ~/reachy_projects/my_app +``` + +The [Reachy Mini app publishing guide](https://huggingface.co/blog/pollen-robotics/make-and-publish-your-reachy-mini-apps) +explains the packaging and publishing workflow in more detail. + +Use the Reachy Mini SDK documentation and examples to understand available +motion, media, and daemon APIs. If you use an AI coding agent, give it the +Pollen Robotics `AGENTS.md` instructions, provided by the [Reachy Mini project](https://github.com/pollen-robotics/reachy_mini) so it follows the expected app structure. + +## What you learned + +You explored options for extending from simulation to a physical Reachy, as well ideas for changing the project to include audio, new vision gestures, or different behaviours. diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/mujoco.png b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/mujoco.png new file mode 100644 index 0000000000..36fa80c99d Binary files /dev/null and b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/mujoco.png differ diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/reachy-app.md b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/reachy-app.md new file mode 100644 index 0000000000..70de17a25d --- /dev/null +++ b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/reachy-app.md @@ -0,0 +1,123 @@ +--- +title: Learn about Reachy and understand the application +weight: 2 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Reachy Gladiator - Decide Reachy's Fate + +In this Learning Path, Reachy is stepping out onto the sands of the Arena. Reachy has practiced their gladiator moves, but are they good enough? + +You will decide Reachy's fate in the classic Roman way - a 👍 for **Victory**, or a 👎 for **Defeat**. + +![Illustration of Reachy Mini in a gladiator style pose. The app uses this arena theme for its moves, verdicts, and dashboard.#center](reachy_gladiator.png "Reachy Mini Gladiator App Concept") + +## What is Reachy Mini? + +Reachy Mini is a small open robotics platform from Pollen Robotics. It is designed for expressive head, antenna, and body motion, and it can be controlled from Python with the Reachy Mini SDK. The Reachy Mini Wireless version includes an onboard Arm-powered Raspberry Pi 4 Compute Module, and the Lite version is operated with external compute (e.g., Raspberry Pi, DGX Spark, Mac/PC). + +Reachy can also be simulated using MuJoCo software. Most developers do not have a physical Reachy Mini robot on their desk, and it is often useful to develop software before hardware is available. Extrapolating from Reachy to more industrial robotics, it is also important to test applications in simulation in advance for safety. + +{{% notice Warning %}} +If using a physical Reachy Mini, use caution and ensure the robot is used in an area with appropriate space. The robot has moving parts and could be a health & safety risk. You are responsible for your safety and the safety of others around you when using physical robotic devices +{{% /notice %}} + +## What will you build? + +The workflow of this learning path is split across two machines: + +**Laptop/Desktop: macOS, Linux, or Windows with WSL2** + - Runs the Reachy Mini daemon + - Runs the MuJoCo simulation + - Displays simulated Reachy movement + - Displays the Pi-hosted dashboard at `http://:8042` + +**Raspberry Pi 5: Raspberry Pi OS** + - Captures frames from a USB webcam + - Runs the Edge AI application (local MediaPipe gesture recognition) + - Serves a dashboard on port 8042 + - Sends robot movement commands to the simulation host daemon + +This split is a common edge/physical AI pattern: + +- A small edge device handles sensors and inference close to the user. +- A robot API or daemon receives movement commands. +- A dashboard gives visibility into the live state of the system. +- Digital twin simulation reduces hardware access as a blocker and allows for safer development. + +This is similar to how larger industrial robotics systems are often built. +Keeping perception, robot control, and observability as separate pieces makes it easier to test, replace, and deploy parts of the system independently. + +## What does the app do? + +The app is called Reachy Gladiator. Reachy (in simulation or otherwise) performs a randomly-chosen scripted gladiator move. You provide a 👍 for **Victory**, or a 👎 for **Defeat**. **Victory** makes Reachy celebrate, **Defeat** makes Reachy react sadly. + +This learning path starts from the complete `reachy_gladiator_lp` project instead of asking you to create every file from scratch. The simulation host only needs a launcher script, but the Raspberry Pi will clone and run the full project. You will inspect the different parts of the system so you can recreate your own apps running on Reachy or in simulation. + +The Reachy Gladiator app runs a repeated loop: + +1. Start with a 10-second preparing countdown +2. Randomly pick one gladiator move +3. Repeat the selected move three times +4. Return Reachy to a neutral pose +5. Watch for a 👍 or a 👎 +6. Run a victory or defeat reaction +7. Repeat with another move + +There are four preset moves: + +- `Salute` +![Salute Move#center](salute.gif "Salute Move") +- `Sword Swing` +![Sword Swing Move#center](sword.gif "Sword Swing Move") +- `Shield Up` +![Shield Up Move#center](shield.gif "Shield Up Move") +- `Battle Cry` +![Battle Cry Move#center](battle-cry.gif "Battle Cry Move") + +The app shuffles all four moves and performs each once before any move repeats. When the bag is empty, it reshuffles and avoids repeating the same move at the shuffle boundary. + +## Use a compatible terminal + +The commands in this learning path use a Bash-style shell. They work directly on macOS and Linux. + +On Windows, use WSL2 with an Ubuntu distribution for the simulation host commands. This keeps the commands almost identical to the macOS and Linux flow: + +- Use the WSL terminal for `python3`, `source .venv/bin/activate`, and `./scripts/start_sim.sh`. +- Keep the browser on Windows if you prefer; open the Pi dashboard from any browser that can reach the Raspberry Pi. +- If the Pi cannot reach a daemon running inside WSL2, check Windows firewall and WSL networking. WSL2 uses virtualized networking, so inbound access from another device on your LAN may require Windows port forwarding. + +The rest of this learning path shows the common Bash commands and calls out the one IP-address command that differs by host operating system. + +## Project structure + +The simulation host does not need the full project checkout. In the next +section, you will download only the simulation launcher script on that machine. + +You will clone the full `reachy_gladiator_lp` project on the Raspberry Pi later on. + +The key files are: + +```text +reachy_gladiator_lp/ +├── pyproject.toml +├── scripts/ +│ ├── start_sim.sh +│ ├── setup_pi.sh +│ ├── run_pi_app.sh +│ └── check_pi_camera.sh +└── reachy_gladiator_lp/ + ├── main.py + ├── camera.py + ├── gesture.py + ├── moves.py + ├── assets/ + │ └── gesture_recognizer.task + └── static/ # dashboard HTML, CSS, JavaScript, and media +``` + +## What you learned and what is next + +You learned what Reachy Mini is, why simulation is useful for edge/physical AI development, and how the app splits work between a simulation host and a Raspberry Pi. You are now ready to start the simulation host with a lightweight launcher script. diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/reachy_gladiator.png b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/reachy_gladiator.png new file mode 100644 index 0000000000..35b4f097b8 Binary files /dev/null and b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/reachy_gladiator.png differ diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/reachy_gladiator_logo.png b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/reachy_gladiator_logo.png new file mode 100644 index 0000000000..275723b3ba Binary files /dev/null and b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/reachy_gladiator_logo.png differ diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/run-pi-app.md b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/run-pi-app.md new file mode 100644 index 0000000000..8b1b18ae4b --- /dev/null +++ b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/run-pi-app.md @@ -0,0 +1,188 @@ +--- +title: Setup Pi and run Edge AI app +weight: 4 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Prepare the Raspberry Pi + +The Raspberry Pi should be installed with Raspberry Pi OS Trixie, and be accessible over SSH. + +Connect to your Raspberry Pi over SSH using [VSCode Remote - SSH](https://code.visualstudio.com/docs/remote/ssh), or via terminal e.g., + +```bash +ssh @ +``` + +On your Pi, create a workspace and move into it: + +```bash +mkdir -p ~/reachy_projects +cd ~/reachy_projects +``` + +Install and enable Git LFS before cloning the app. The MediaPipe gesture model +is stored as a Git LFS asset, so a normal clone without LFS can leave a tiny +pointer file instead of the real model: + +```bash +sudo apt update +sudo apt install git-lfs +git lfs install +``` + +Clone the `reachy_gladiator_lp` project into this directory: + +```bash +git clone https://github.com/matt-cossins/reachy_gladiator_lp.git +cd reachy_gladiator_lp +git lfs pull +``` + +Check that the gesture model was downloaded: + +```bash +ls -lh reachy_gladiator_lp/assets/gesture_recognizer.task +``` + +The file should be about 8 MB. If it is only 132 bytes, run `git lfs pull` +again before continuing. + +## Install the Pi runtime + +Run the Pi setup script: + +```bash +./scripts/setup_pi.sh +``` + +The script installs the system packages, Python version, Python packages, and app entry point used by this Learning Path. + +Raspberry Pi OS Trixie can use Python 3.13 as the default `python3`, but the tested Pi environment for this app uses Python 3.12. The setup script installs Python 3.12.3 with `pyenv` so you do not need to modify the system Python. + +The script also handles the Pi-specific package versions: + +- `mediapipe==0.10.18`, because newer MediaPipe wheels are not available for the tested Pi environment. +- `numpy==2.4.4`, which is required by the Reachy Mini SDK and has been tested with the gesture worker. +- `reachy-mini==1.7.3`, installed without dependency resolution so pip does not reject the MediaPipe and NumPy combination. +- `git-lfs`, so the MediaPipe gesture model is downloaded as the real binary asset rather than an LFS pointer file. +- `v4l-utils`, so you can inspect connected camera devices with `v4l2-ctl`. + +When setup finishes, activate the virtual environment: + +```bash +source .venv/bin/activate +``` + +The setup script runs an import smoke test. It also runs `pip check`; a MediaPipe NumPy metadata warning is expected for this Pi setup. + +Test the MediaPipe gesture worker from a real Python file: + +```bash +cat > /tmp/test_gesture_worker.py <<'PY' +import numpy as np +from reachy_gladiator_lp.gesture import ThumbGestureDetector + +def main(): + print("creating detector") + detector = ThumbGestureDetector() + frame = np.zeros((480, 640, 3), dtype=np.uint8) + result = detector.detect(frame) + print(result) + detector.close() + print("gesture detector OK") + +if __name__ == "__main__": + main() +PY +python /tmp/test_gesture_worker.py +``` + +The output should show that the detector starts and returns a neutral result for the blank test frame: + +```output +creating detector +GestureResult(label=None, x_px=320, y_px=240, confidence=0.0) +gesture detector OK +``` + +MediaPipe may also print TensorFlow Lite or CPU delegate warnings before the result. The important line is `gesture detector OK`. + +## Check the USB webcam + +Plug the USB webcam into the Raspberry Pi and list video devices: + +```bash +ls /dev/video* +``` + +Run the included camera check: + +```bash +./scripts/check_pi_camera.sh +``` + +The output is similar to: + +```output +Video devices: +/dev/video0 +Camera index 0 OK: 1280x720 +``` + +If camera index `0` does not work, try index `1`: + +```bash +REACHY_GLADIATOR_CAMERA_INDEX=1 ./scripts/check_pi_camera.sh +``` + +If a different camera index works, use the same value when you run the app. For example, if camera index `1` works: + +```bash +REACHY_GLADIATOR_CAMERA_INDEX=1 \ +REACHY_GLADIATOR_DAEMON_PORT=18000 \ +./scripts/run_pi_app.sh +``` + +## Run the distributed app + +Keep the simulation terminal running on your simulation host. + +On the Raspberry Pi, run the app with the simulation host IP address and the simulation port. This learning path uses port `18000`: + +```bash +REACHY_GLADIATOR_DAEMON_PORT=18000 ./scripts/run_pi_app.sh +``` + +If you started the simulation on a different port, pass that same value in `REACHY_GLADIATOR_DAEMON_PORT`. + +## What is the script doing? + +The script sets the app configuration: + +```text +REACHY_GLADIATOR_DAEMON_HOST= +REACHY_GLADIATOR_DAEMON_PORT=18000 +REACHY_GLADIATOR_DASHBOARD_HOST=0.0.0.0 +REACHY_GLADIATOR_DASHBOARD_PORT=8042 +REACHY_GLADIATOR_MEDIA_BACKEND=no_media +REACHY_GLADIATOR_CAMERA=opencv +REACHY_GLADIATOR_CAMERA_INDEX=0 +``` + +The app then runs: + +```bash +python -m reachy_gladiator_lp.main +``` + +`REACHY_GLADIATOR_MEDIA_BACKEND=no_media` tells the Reachy SDK not to request +camera media from the daemon. This is the right default for the learning path +because the Pi owns the USB webcam. `REACHY_GLADIATOR_CAMERA=opencv` tells the +gesture recognizer to read frames from that local webcam. + +## What you learned and what is next + +You installed the Pi runtime with the project setup script, validated the MediaPipe gesture worker, checked the USB webcam, and started the edge AI app so it can run the application, perform inference on incoming frames, and send Reachy commands to the simulation host. Now you get to try out the app! diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/salute.gif b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/salute.gif new file mode 100644 index 0000000000..f1d4c22fc0 Binary files /dev/null and b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/salute.gif differ diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/shield.gif b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/shield.gif new file mode 100644 index 0000000000..d6305d78f4 Binary files /dev/null and b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/shield.gif differ diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/start-simulation.md b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/start-simulation.md new file mode 100644 index 0000000000..4ec27253dd --- /dev/null +++ b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/start-simulation.md @@ -0,0 +1,146 @@ +--- +title: Start the Reachy simulation +weight: 3 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## What is MuJoCo? + +MuJoCo is an open-source physics engine from Google DeepMind, used for simulating articulated bodies such as robots. It can model joints, contacts, gravity, and rigid-body dynamics, which makes it useful for robotics prototyping, control development, reinforcement learning, and safety testing before hardware is available. + +In this project, MuJoCo lets you see Reachy Mini move without owning a physical +Reachy. It is not a perfect replacement for hardware: real cameras, lighting, +network latency, calibration, mechanical tolerances, and safety constraints +still matter. It is best treated as a fast development and validation tool +before moving to physical testing. + +## Create or activate a simulation environment + +The simulation host does not need the full `reachy_gladiator_lp` project. It +only needs the Reachy Mini SDK with MuJoCo support and the `start_sim.sh` +launcher script. + +On the machine that will run the simulation, create a small workspace and a +Python virtual environment: + +```bash +mkdir -p ~/reachy_projects/reachy_sim/scripts +cd ~/reachy_projects/reachy_sim +python3 -m venv .venv +source .venv/bin/activate +python -m pip install --upgrade pip +``` + +Install the Reachy Mini SDK with the MuJoCo simulation dependencies: + +```bash +python -m pip install --upgrade "reachy-mini[mujoco]" +``` + +The `mujoco` extra is required for `--sim`. If you install only `reachy-mini`, the daemon can start but simulation fails with `MuJoCo is not installed`. + +If you already have a working Reachy Mini simulation environment, you can activate that environment instead. + +Download the simulation launcher script from the project repository: + +```bash +curl -L https://raw.githubusercontent.com/matt-cossins/reachy_gladiator_lp/main/scripts/start_sim.sh -o scripts/start_sim.sh +chmod +x scripts/start_sim.sh +``` + +`curl` is available by default on macOS and is commonly available on Linux and +WSL distributions. If your Linux environment does not include it, install it +with your package manager. + +## Start the Reachy Mini simulation + +Start the daemon and MuJoCo simulation from the simulation workspace: + +```bash +cd ~/reachy_projects/reachy_sim +source .venv/bin/activate +REACHY_SIM_PORT=18000 ./scripts/start_sim.sh +``` + +This can take several minutes to start up. Leave this terminal running after it completes and boots the simulation view. + +![MuJoCo Simulation#center](MuJoCo.png "MuJoCo Simulation") + + +The script starts the Reachy Mini daemon with simulation enabled, binds FastAPI to `0.0.0.0`, and disables localhost-only mode so the Raspberry Pi can connect. This Learning Path uses port `18000`. + +The daemon is the network boundary between the Pi app and the simulated robot. +The Pi does not run MuJoCo. It connects to this daemon and sends the same SDK +motion commands that it would send to a physical Reachy daemon. + +The script runs a command equivalent to: + +```bash +mjpython -m reachy_mini.daemon.app.main \ + --sim \ + --fastapi-host 0.0.0.0 \ + --fastapi-port 18000 \ + --no-localhost-only +``` + +On macOS, `mjpython` is often needed because MuJoCo opens a native graphics +window. On Linux, regular `python` is usually enough. The provided script picks +an appropriate runtime when it can. + +### Troubleshooting the simulation + +If the script reports that address `0.0.0.0:18000` is already in use, another daemon or server is already using port `18000`. + +Find the process: + +```bash +lsof -nP -iTCP:18000 -sTCP:LISTEN +``` + +If the process is an old Reachy daemon, stop it with `Ctrl+C` in its terminal. If you need to terminate it from the command line, replace `` with the process ID from `lsof`: + +```bash +kill +``` + +You can also run the simulation on a different port: + +```bash +REACHY_SIM_PORT=18001 ./scripts/start_sim.sh +``` + +If you change the simulation port, use the same port when configuring the Pi app later: + +```bash +REACHY_GLADIATOR_DAEMON_PORT=18001 ./scripts/run_pi_app.sh +``` + +## Find the simulation host IP address + +Use the tab for your simulation host operating system. + +{{% notice Note %}} +On macOS, `en0` is usually Wi-Fi. If you use Ethernet or another network interface, the device name might be different. +{{% /notice %}} + +{{< tabpane code=true >}} + {{< tab header="macOS" language="bash">}} +ipconfig getifaddr en0 + {{< /tab >}} + {{< tab header="Linux" language="bash">}} +hostname -I + {{< /tab >}} + {{< tab header="Windows with WSL2" language="bash">}} +hostname -I + {{< /tab >}} +{{< /tabpane >}} + +{{% notice Note %}} +If the Pi cannot connect later, check that the simulation machine and Raspberry Pi are on the same network and that the local firewall allows inbound connections to port `18000`. +{{% /notice %}} + +## What you learned and what is next + +You created a simulation environment, started the Reachy Mini MuJoCo daemon on a network-accessible port, and found the host IP address the Raspberry Pi needs when it connects to the simulated robot. You should now see a simulated Reachy robot in MuJoCo, and be ready to prepare the Raspberry Pi. diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/sword.gif b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/sword.gif new file mode 100644 index 0000000000..b6074e4f67 Binary files /dev/null and b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/sword.gif differ diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/thumbs-down.gif b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/thumbs-down.gif new file mode 100644 index 0000000000..c3bb734cfa Binary files /dev/null and b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/thumbs-down.gif differ diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/thumbs-up.gif b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/thumbs-up.gif new file mode 100644 index 0000000000..4607b43337 Binary files /dev/null and b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/thumbs-up.gif differ diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/understand-code.md b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/understand-code.md new file mode 100644 index 0000000000..d84c1b7f88 --- /dev/null +++ b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/understand-code.md @@ -0,0 +1,422 @@ +--- +title: Understand the app code +weight: 6 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Understand the code structure + +The project separates perception, app logic, robot motion, and dashboard rendering. + +The key files running on the Pi are: + +```text +reachy_gladiator_lp/ +├── main.py +├── camera.py +├── gesture.py +├── moves.py +├── assets/ +│ └── gesture_recognizer.task +└── static/ # dashboard HTML, CSS, JavaScript, and media +``` + +## Follow the app lifecycle - main.py + +`reachy_gladiator_lp/main.py` is the center of the application. It defines the `ReachyGladiatorLp` class, which inherits from `ReachyMiniApp`. + +Two settings are important for the distributed simulation route: + +```python +custom_app_url: str | None = "http://0.0.0.0:8042" +request_media_backend: str | None = "no_media" +``` + +`custom_app_url` tells the Reachy app framework where the dashboard is served. +`request_media_backend = "no_media"` tells the SDK not to request camera media +from the daemon, because the Pi uses its own USB webcam. For a physical Reachy +route, setting `REACHY_GLADIATOR_MEDIA_BACKEND=reachy` lets the SDK request the +daemon media backend. + +The useful part to inspect is `run()`, because that is where the app receives a +connected `ReachyMini` object and starts the edge AI loop: + +```python +def run(self, reachy_mini: ReachyMini, stop_event: threading.Event) -> None: + rng = random.Random() + move_queue: list[str] = [] + last_move: str | None = None + detector: ThumbGestureDetector | None = None + latest_frame: np.ndarray | None = None + + self._status = { + "state": "starting", + "round": 0, + "active_move": None, + "gesture": None, + "confidence": 0.0, + "camera_ready": False, + } +``` + +This state drives both sides of the app: robot behavior and dashboard display. +The `run()` method keeps track of: + +- `move_queue` stores the shuffled bag of moves. +- `last_move` prevents immediate repeats at a shuffle boundary. +- `detector` stores the MediaPipe gesture detector. +- `latest_frame` stores the newest webcam frame from the capture thread. +- `status` stores the data returned by the dashboard `/status` endpoint. + +The `_update_status()` helper updates dashboard state safely: + +```python +def _update_status(self, **values: Any) -> None: + with self._state_lock: + self._status.update(values) +``` + +The nested `capture_frames()` helper runs in the background. It reads frames from the selected camera source, saves the newest frame for gesture detection, and marks the dashboard camera as ready. + +The actual loop is intentionally small: + +```python +def capture_frames(frame_source: FrameSource) -> None: + while not stop_event.is_set() and not capture_stop.is_set(): + frame = frame_source.get_frame() + if frame is None: + time.sleep(LOOP_SLEEP_S) + continue + + self._update_frame(frame) + self._update_status(camera_ready=True) + time.sleep(LOOP_SLEEP_S) +``` + +`main.py` also registers FastAPI endpoints on `settings_app`: + +- `/status` returns the latest round, state, move, gesture, confidence, and camera status. +- `/video` streams resized JPEG frames as an MJPEG feed for the browser dashboard. + +Those routes are registered inside the app class: + +```python +@self.settings_app.get("/status") +def get_status() -> dict[str, Any]: + with self._state_lock: + return dict(self._status) + +@self.settings_app.get("/video") +def video() -> StreamingResponse: + return StreamingResponse( + self._video_stream(self._read_frame), + media_type="multipart/x-mixed-replace; boundary=frame", + ) +``` + +The main loop has four phases: + +1. Build the next move sequence with `_build_sequence()`. +2. Perform the selected move `MOVE_REPETITIONS` times. +3. Return to neutral and call `_await_verdict()`. +4. Run `victory()`, `defeat()`, or continue neutrally. + +The verdict code requires two consecutive confident classifications before it accepts a gesture. This debounce step helps avoid reacting to a single noisy frame. + +This is the core move-and-verdict flow: + +```python +sequence = self._build_sequence(rng, move_queue, last_move) +for name in sequence: + move_fn = gmoves.MOVE_CATALOGUE[name] + for repeat_idx in range(1, MOVE_REPETITIONS + 1): + self._update_status(active_move=name, current_repeat=repeat_idx) + move_fn(reachy_mini) + +gmoves.neutral(reachy_mini) +verdict, detector = self._await_verdict( + reachy_mini, + self._read_frame, + detector, + stop_event, + self._update_status, +) +``` + +The app is still a normal Reachy Mini app. It receives a connected +`ReachyMini` object in `run()`. That object is the SDK boundary, so the rest of +the app does not need to know whether commands go to MuJoCo simulation or to a +physical robot. + +## Capture camera frames - camera.py + +`reachy_gladiator_lp/camera.py` selects the frame source. A frame source is +anything that can return the next camera image as a NumPy array. + +For the default simulation-plus-Pi route, frames come from a USB webcam plugged +into the Raspberry Pi. The app uses OpenCV to open that webcam and read frames: + +```python +self._capture = cv2.VideoCapture(camera_index) +``` + +`camera_index` is the local video-device number on the Pi. Index `0` usually +means the first camera OpenCV can open; index `1` means the next one. If +`REACHY_GLADIATOR_CAMERA_INDEX=1` worked in the camera check, use the same +value when running the app. + +For the physical Reachy route, frames come from the Reachy daemon media +pipeline instead. In that case, `camera_index` is not used. + +The `REACHY_GLADIATOR_CAMERA` environment variable chooses the route: + +- `opencv` uses a local USB webcam through OpenCV. +- `reachy` uses camera frames from the Reachy daemon. + +The source selection code is therefore small: + +```python +requested = os.getenv("REACHY_GLADIATOR_CAMERA", "opencv").strip().lower() + +if requested == "opencv": + return OpenCVCameraFrameSource(_camera_index()) + +if requested == "reachy": + return ReachyMediaFrameSource(reachy_mini) +``` + +After a frame is captured, the latest frame is copied into shared state. Both +the detector and dashboard stream read from that newest frame. This lets the +same app use either a Pi USB webcam or the physical Reachy camera without +changing the gesture-recognition code. + +## Recognize thumbs with MediaPipe - gesture.py + +`reachy_gladiator_lp/gesture.py` runs the MediaPipe Gesture Recognizer. This is +the edge AI part of the app: the Raspberry Pi classifies camera frames locally +and only sends robot commands over the network. + +The recognizer loads this model bundle: + +```text +reachy_gladiator_lp/assets/gesture_recognizer.task +``` + +The `.task` file is a MediaPipe Tasks model bundle. It includes the trained +gesture-recognition model and the metadata MediaPipe needs to run it through +the Tasks API. + +Under the hood, MediaPipe Tasks commonly runs the model through TensorFlow Lite +for efficient on-device inference. On CPU builds, TensorFlow Lite can use the +XNNPACK delegate to speed up neural-network operations on Arm CPUs. + +The detector runs in a worker process instead of the main app thread. This +keeps MediaPipe inference separate from the robot-control loop, dashboard +responses, and camera capture. Other processes therefore don't pause the MediaPipe worker while it is classifying frames. + +The app uses the CPU delegate: + +```python +base_options = python.BaseOptions( + model_asset_path=model_path, + delegate=python.BaseOptions.Delegate.CPU, +) +``` + +The recognizer is configured for one hand in image mode: + +```python +options = vision.GestureRecognizerOptions( + base_options=base_options, + running_mode=vision.RunningMode.IMAGE, + num_hands=1, + min_hand_detection_confidence=0.5, + min_hand_presence_confidence=0.5, + min_tracking_confidence=0.5, +) +``` + +Each OpenCV frame is converted from BGR to RGB before MediaPipe sees it. BGR +and RGB contain the same red, green, and blue color channels, but in a different +order. OpenCV returns webcam frames as blue-green-red, while MediaPipe expects +red-green-blue: + +```python +rgb_frame = bgr_frame[:, :, ::-1].copy() +mp_image = mediapipe.Image(image_format=mediapipe.ImageFormat.SRGB, data=rgb_frame) +result = recognizer.recognize(mp_image) +``` + +The app then maps MediaPipe labels to its own labels: + +```python +if raw_label == "Thumb_Up": + label = "thumbs_up" +elif raw_label == "Thumb_Down": + label = "thumbs_down" +else: + label = None +``` + +The returned `GestureResult` also includes the thumb tip pixel position. The worker queues have `maxsize=1`, which keeps detection biased toward the newest camera frame. + +The app also warms the detector in the background while Reachy is preparing. +This means the first verdict window does not have to pay all of the detector +startup cost: + +```python +def warmup_detector() -> None: + warmed_detector = ThumbGestureDetector() + while not stop_event.is_set() and not capture_stop.is_set(): + frame = self._read_frame() + if frame is not None: + warmed_detector.detect(frame) + break + time.sleep(LOOP_SLEEP_S) +``` + +When the verdict window starts, the app reuses the warmed detector if it is +ready: + +```python +if detector is None: + detector_warmup_done.wait(timeout=GESTURE_WARMUP_WAIT_S) + with detector_warmup_lock: + detector = detector_warmup["detector"] + detector_warmup["detector"] = None +``` + +The debounce logic accepts a verdict only after two matching confident frames: + +```python +if ( + result.label in ("thumbs_up", "thumbs_down") + and result.confidence >= VERDICT_MIN_CONFIDENCE +): + if result.label == last_label: + consecutive += 1 + else: + last_label = result.label + consecutive = 1 + if consecutive >= 2: + return result.label, detector +``` + +## Send robot motion commands - moves.py + +`reachy_gladiator_lp/moves.py` contains small robot motion primitives: +`Salute`, `Sword Swing`, `Shield Up`, `Battle Cry`, victory, defeat, and +neutral. Each move accepts a `ReachyMini` instance and sends one or more +`goto_target()` commands. + +For example, the `salute()` move combines a head target with antenna targets: + +```python +reachy_mini.goto_target( + head=create_head_pose(pitch=-16, yaw=0, roll=0, degrees=True), + antennas=np.deg2rad([58, 58]), + duration=0.42, + method="minjerk", +) +``` + +`create_head_pose(...)` builds the head target in degrees. The antenna values +are converted to radians with `np.deg2rad(...)`. + +Some moves try to use body yaw: + +```python +reachy_mini.goto_target(body_yaw=yaw_rad, duration=duration, method=method) +``` + +The helper `_safe_body_yaw()` catches SDK or hardware cases where body yaw is unavailable and falls back to a smaller head motion. + +The move catalogue is deliberately simple: + +```python +MOVE_CATALOGUE: dict[str, MoveFn] = { + "Salute": salute, + "Sword Swing": sword_swing, + "Shield Up": shield_up, + "Battle Cry": battle_cry, +} +``` + +Adding a move means writing a new function and registering it in +`MOVE_CATALOGUE`. That makes the app loop eligible to select and run the move. +To display it cleanly in the dashboard, also add a matching entry to +`MOVE_DESCRIPTIONS`. + +The shuffled bag logic lives in `main.py`, not in the move functions: + +```python +names = list(gmoves.MOVE_CATALOGUE.keys()) +if not move_queue: + move_queue.extend(rng.sample(names, len(names))) + if last_move is not None and len(move_queue) > 1 and move_queue[0] == last_move: + move_queue.append(move_queue.pop(0)) + +return [move_queue.pop(0)] +``` + +The victory and defeat reactions are also just move functions. The state machine decides when to call them: + +```python +if verdict == "thumbs_up": + gmoves.victory(reachy_mini) +elif verdict == "thumbs_down": + gmoves.defeat(reachy_mini) +``` + +`main.py` decides what should happen, and `moves.py` describes how Reachy should move. + +## Render the dashboard + +Because `custom_app_url` is set, the Reachy app base class starts a FastAPI +server for this app. That server mounts `reachy_gladiator_lp/static/` at +`/static` and serves `static/index.html` at `/`. The files in `static/` are +therefore the dashboard you view in the browser: + +- `index.html` defines the dashboard layout. +- `main.js` polls `/status` and updates the DOM. +- `style.css` controls the arena theme, verdict colors, and responsive layout. +- `reachy_gladiator.png` is the gladiator image shown in the dashboard. + +The JavaScript does not control the robot. It only renders state produced by +the Python app. + +The dashboard camera feed is an MJPEG stream from `/video`. Python resizes each frame and encodes it as JPEG: + +```python +preview_frame = self._resize_preview_frame(frame) +ok, encoded = cv2.imencode( + ".jpg", + preview_frame, + [cv2.IMWRITE_JPEG_QUALITY, DASHBOARD_PREVIEW_JPEG_QUALITY], +) +``` + +`static/index.html` displays that stream in an image element, while +`static/main.js` polls `/status` for the round state, gesture confidence, and +verdict highlighting. This split keeps video transport simple. + +The command-line entry point wires the script environment into the Reachy SDK connection: + +```python +if __name__ == "__main__": + daemon_host = os.getenv("REACHY_GLADIATOR_DAEMON_HOST", "localhost") + daemon_port = int(os.getenv("REACHY_GLADIATOR_DAEMON_PORT", "8000")) + daemon_timeout = float(os.getenv("REACHY_GLADIATOR_DAEMON_TIMEOUT", "8.0")) + os.environ.setdefault("REACHY_GLADIATOR_CAMERA", "opencv") + os.environ.setdefault("REACHY_GLADIATOR_MEDIA_BACKEND", "no_media") + app = ReachyGladiatorLp() + app.wrapped_run(host=daemon_host, port=daemon_port, timeout=daemon_timeout) +``` + +Take some time to look through the files and try to understand the different parts. Feel free to ask a coding agent or LLM to help explain any areas you are unsure about. + +## What you learned + +You inspected how the app captures camera frames, runs MediaPipe gesture recognition on the Pi, debounces thumbs-up and thumbs-down verdicts, sends Reachy SDK motion commands, and renders a browser dashboard. You are now ready to experiment with adapting this project, or building your own app. diff --git a/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/use-the-app.md b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/use-the-app.md new file mode 100644 index 0000000000..355e4a741d --- /dev/null +++ b/content/learning-paths/cross-platform/build-a-reachy-robot-app-on-pi/use-the-app.md @@ -0,0 +1,44 @@ +--- +title: Look at the dashboard and use the application +weight: 5 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Open the dashboard + +If you are connected to the Raspberry Pi with VS Code Remote SSH, VS Code can forward the dashboard port to your laptop. Open the dashboard from your laptop browser: + +```text +http://localhost:8042 +``` + +If you are not using Remote SSH port forwarding, use the Pi IP address directly. + +Find the Raspberry Pi IP address: + +```bash +hostname -I +``` + +Then open the dashboard from any browser on the same network: + +```text +http://:8042 +``` + +This works even when the Pi is headless. The browser only needs network access +to the Pi and port `8042`. + +The app will countdown from 10, and then Reachy will start performing moves. Reachy will perform the same move three times, and then await your verdict. After a verdict is given, Reachy will perform the **Victory** or **Defeat** moves. Reachy will then randomly select another move the cycle will repeat. + +![Thumbs Up Victory#center](thumbs-up.gif "Thumbs Up Victory") + +![Thumbs Down Defeat#center](thumbs-down.gif "Thumbs Down Defeat") + +When you are finished, you can stop the Pi app with `Ctrl+C` in the terminal you ran the application from. + +## What you learned and what is next + +You opened the dashboard, watched the live camera and app state, gave thumbs-up and thumbs-down verdicts, and saw how edge AI gesture recognition changes Reachy's next action. This is a simple application premise using vision to control expressive robot outputs. Next you will understand how the app works and how you could take this further or make your own.