RL / ML learning Journey

I have been eyeballing the Microduck since I saw a couple of months back. I have been talking with the AI and put together a learning path for me to see if my interest is strong enough to spend $500 on a robot of my own.

Below is the learning path overview. I additionally have a the first 5 weeks detailed out further. I am going to start this week. I would like to see how much progress i make by Christmas. Then if i get enough Christmas monies I might purchase one.

High level learning plan

Weeks 1–2: Foundations (no robot needed)

Brush up on Python, NumPy, and PyTorch basics. The goal is to read a training loop comfortably, not master deep learning theory.
Play with the Microduck simulator on HF Spaces to get a feel for the robot's joints and gaits.

Weeks 3–5: RL fundamentals

Work through the free Hugging Face Deep RL Course. Focus on the units covering Q-learning, policy gradients, and PPO, since PPO is the workhorse for legged robots.
Train small agents in Gymnasium with Stable-Baselines3, like CartPole and then a MuJoCo walker. You'll build intuition for reward shaping, which is most of the real work.

Weeks 6–7: The Microduck stack

Clone pollen-robotics/microduck, run it locally, and reproduce one shipped policy end to end. Sit & stand is a good first one.
Read the reward function and observation space for that policy. Change one reward term and watch how the behavior shifts.

Weeks 8–9: Sim-to-real

Deploy the stock policy to the real duck, then deploy your retrained one.
Log IMU data in sim and on hardware and compare them. This teaches domain randomization and why policies fail on real floors. It's the same lesson F1TENTH teaches, in a new form.

Week 10: MLOps (your DevOps angle)

Move training to HF Jobs, add experiment tracking, and version each policy on the Hub with a proper model card.
Wrap it in a CI pipeline that retrains and evaluates in sim on every config change. For your FAANG and DevOps goals, this is arguably the most hireable part.

Weeks 11–12: Capstone, "something cool"
A few options, roughly easy to hard:

Laser chase: use the camera to track the accessory-pack laser pointer and steer the walk policy toward it.
Voice-commanded duck: a small speech or vision-language model from the Hub turns "go kick the ball" into a sequence of walk, kick, and grab calls.
A brand-new learned trick, like a dance or a spin, trained from scratch with your own reward function, published as a policy others can load.

Microduck ML Plan: Weeks 1–5

How to use this plan

By the end of week 5 you can train a PPO agent in MuJoCo, explain why it walks, and read the Microduck training code without getting lost. No robot is needed for any of it, so all five weeks can happen while your pre-order ships.

Time budget: about 8–10 hours a week. Roughly 40% concepts (reading, videos), 60% micro projects.
Rule of thumb: a concept isn't learned until a micro project uses it. If a project feels trivial, move on; if it breaks, that's where the learning is.
Machine: any modern laptop handles weeks 1–4 on CPU. Week 5 MuJoCo training runs on CPU too, just slowly; a free Google Colab GPU or Hugging Face Jobs covers the heavier runs.
Repo habit: one GitHub repo, one folder per week, a short README per micro project with what you tried and what you saw. This doubles as portfolio evidence later.

Setup (do this on day 1):

Python 3.11+ with uv or conda, and a fresh environment named duck
Install numpy, matplotlib, torch, gymnasium[mujoco], stable-baselines3, tensorboard
Create a Hugging Face account and a write token
Create the GitHub repo microduck-learning
Join the Pollen Robotics Discord

Week 1: The toolkit (NumPy, PyTorch, gradients)

Goal: be fluent enough in tensors and autograd that RL code later reads like plain Python.

Concepts

NumPy arrays, shapes, broadcasting and vectorized ops (replacing for-loops)
PyTorch tensors: shapes, dtypes, moving between CPU and GPU
Gradients and autograd: what loss.backward() actually computes
Gradient descent: learning rate, steps, why too big diverges and too small stalls
The training loop: forward pass → loss → backward → optimizer step → zero grads

Micro projects

Robot-arm math in NumPy. Compute the (x, y) position of a 2-joint planar arm from two angles, vectorized over 10,000 random angle pairs, then scatter-plot the reachable workspace. This is forward kinematics, the same math behind Microduck's legs.
Gradient descent by hand. Fit a line y = mx + b to noisy data with NumPy only, computing gradients yourself. Plot the loss curve for three learning rates on one chart.
Same thing in PyTorch. Redo the line fit using torch.autograd and torch.optim.SGD. Confirm your hand gradients match autograd's to 4 decimal places.
Inverse kinematics by optimization. Given a target (x, y), use PyTorch to optimize the two arm angles until the fingertip reaches it. You've just used gradient descent to control a robot.

Resources: the official PyTorch "Learn the Basics" tutorial; Andrej Karpathy's micrograd video if you want autograd from scratch (optional, ~2.5 hours, very worth it).

Week 2: Neural networks and the Microduck sim

Goal: understand what a "policy" is physically (a small neural net mapping sensor readings to motor targets) and see one running.

Concepts

Multi-layer perceptrons (MLPs): layers, activations, why nonlinearity matters
Supervised learning: inputs, labels, loss functions (MSE for regression, cross-entropy for classification)
Overfitting vs generalization: train/validation splits and why they matter
Observations and actions for a robot: joint angles, joint velocities, IMU orientation in; motor position targets out
Control frequency: Microduck runs its policy at 50 Hz, so the net has 20 ms per decision

Micro projects

Learn the kinematics. Train a small MLP to predict the Week 1 arm's fingertip position from its angles. Then train one for the reverse (position → angles) and see where it fails when two solutions exist.
Overfit on purpose. Train on only 50 samples with a huge network, plot train vs validation loss, then fix it with more data. Screenshot both curves for your README.
Play the Microduck simulator on its Hugging Face Space for 30 minutes. Write down every behavior you see and guess what its observations and rewards might be. You'll check these guesses in Week 5.
Time budget test. Build a 3-layer MLP with ~15 outputs (one per Microduck motor), time 1,000 forward passes on your CPU, and compute whether it fits in 20 ms. (It will, by a wide margin; knowing why is the point.)

Resources: 3Blue1Brown's neural networks series (chapters 1–4); PyTorch "Build the Neural Network" tutorial.

Week 3: Reinforcement learning fundamentals

Goal: understand how an agent learns from reward alone, with no labels, using tabular methods you can fully inspect.

Concepts

The RL loop: agent, environment, state, action, reward, episode
Markov Decision Processes (MDPs) and the Markov property
Return and discount factor (gamma): why future reward is worth less
Value functions V(s) and Q(s, a); the Bellman equation
Exploration vs exploitation: epsilon-greedy
Q-learning: the update rule and why it converges on small problems

Micro projects

Random agent baseline. Run a random policy on Gymnasium's CartPole-v1 and FrozenLake-v1, and record average episode reward over 100 episodes. Every later result gets compared to this number.
Q-learning from scratch. Implement tabular Q-learning for FrozenLake-v1 in about 50 lines of NumPy. Plot success rate over training episodes.
Knob-turning experiments. Rerun with gamma at 0.5, 0.9 and 0.99, and epsilon fixed vs decaying. Put the learning curves on one plot and write two sentences on what changed.
Design your own grid world. A 6×6 grid where a "duck" must reach a pond while avoiding a cat. Write the reward function yourself, then deliberately break it (e.g. +1 for every step) and watch the agent exploit it. This is your first taste of reward hacking, the #1 problem in robot RL.

Resources: Hugging Face Deep RL Course, Units 1–2 (intro, Q-learning); Sutton & Barto Reinforcement Learning: An Introduction, chapters 3 and 6 (free PDF online) for depth.

Week 4: Deep RL and PPO

Goal: replace the Q-table with a neural network and learn PPO, the algorithm behind most legged-robot policies, including sim-to-real work like Microduck's.

Concepts

Why tables break: continuous states (joint angles) can't be enumerated, so a neural net approximates the policy or value
Deep Q-Networks (DQN), briefly: replay buffer and target network
Policy gradients: directly nudging the policy toward actions that earned more reward
Actor-critic: a policy network (actor) plus a value network (critic) that reduces noise
PPO: clipped updates so the policy never changes too much in one step; key hyperparameters (learning rate, n_steps, batch size, clip range, entropy coefficient)
Reading training curves in TensorBoard: episode reward, episode length, entropy, value loss

Micro projects

PPO on CartPole with Stable-Baselines3. Train in under 20 lines, log to TensorBoard, and hit the 500-step max. Compare wall-clock time and final score against your Week 3 random baseline.
Continuous actions. Train PPO on Pendulum-v1, where the action is a torque value rather than left/right. This is the jump robot motors require.
Hyperparameter sweep. Run 4 variations of PPO on LunarLander-v3 (learning rate × 2 values, entropy coefficient × 2 values) with 3 random seeds each. Plot mean and spread per setting. Seeds matter more than beginners expect.
Push to the Hub. Upload your best LunarLander agent to Hugging Face with a model card and a replay video. This is exactly the workflow you'll use to publish Microduck policies later.

Resources: Hugging Face Deep RL Course, Units 3, 4 and 8 (DQN, policy gradients, PPO); OpenAI Spinning Up's PPO page for the clean math; Stable-Baselines3 docs "RL Tips and Tricks" page.

Week 5: MuJoCo, locomotion and the Microduck repo

Goal: train a legged walker in the same physics engine Microduck uses, then map what you learned onto the real Microduck training code.

Concepts

MuJoCo basics: bodies, joints, actuators, the model XML (MJCF), timestep vs control step
Locomotion observations: joint positions and velocities, body orientation from the IMU, commanded velocity, previous action
Reward shaping for walking: track the commanded velocity, stay upright, penalize energy, jerky actions and falling
Reward hacking in locomotion: hopping, sliding, vibrating in place
The sim-to-real gap: motor delay, friction, mass errors; domain randomization as the fix
Parallel environments: why robot RL runs hundreds of sims at once

Micro projects

Poke at a MuJoCo model. Load Hopper-v5 or Walker2d-v5, print its observation and action spaces, and open the viewer. Manually apply a sine wave to one joint and watch it move.
Train a walker. PPO on Walker2d-v5 for about 1M steps (Colab or HF Jobs if your CPU is slow). Record a video at 100k, 500k and 1M steps and compare the gaits.
Reward surgery. Copy the walker environment and change one reward term: drop the energy penalty, or double the forward-speed reward. Retrain and describe the new, probably weirder, gait.
Domain randomization lite. Randomize floor friction and body mass by ±20% each episode. Test the original policy and the randomized-trained policy on a "surprise" environment with high friction and see which one falls less.
Read the Microduck repo. Clone pollen-robotics/microduck, find the MJCF model, the observation definition, and the reward function for one shipped behavior (walk or sit & stand). Check them against your Week 2 simulator guesses and write a one-page "how Microduck learns to walk" note in your own words.

Resources: MuJoCo documentation "Overview" and "Modeling" pages; Gymnasium MuJoCo environment docs; the Microduck GitHub README and Discord.

Week 5 checkpoint

You're ready for Week 6 (reproducing a shipped Microduck policy) when you can answer these without looking anything up:

Explain in two sentences why PPO clips its updates.
Sketch Microduck's observation → policy → action loop and say what runs at 50 Hz.
Name three reward terms in a walking policy and what goes wrong if each is removed.
Explain domain randomization to a non-ML engineer.
Show a public Hugging Face model page for an agent you trained.

If two or more are shaky, spend an extra week re-running the Week 4 and 5 projects with a different environment (e.g. Ant-v5) rather than pushing ahead.