MAEPose: learning human pose from raw mmWave video without labels

Cameras are very good at telling us how a person is moving. They are also a poor fit for many of the places where that information would be most useful, such as a living room, a changing area or a rehabilitation space. They record faces and surroundings, they fail in the dark, and they cannot see through a curtain or a sheet.

Millimetre-wave (mmWave) radar avoids these problems. A small radar chip emits chirps at around 77 GHz and measures how they reflect off the body. The result carries information about distance and motion, but not a recognisable image of the person, and it works regardless of lighting or light, non-metallic occlusion. The hard part is turning that signal into something as precise as a skeleton.

In MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video, researchers at University College London show that radar signals can be treated as video. A model can then learn from hours of radar footage without any pose labels before it is taught to estimate pose. Across three datasets and a strict leave-one-person-out protocol, MAEPose reduces pose error by up to 22.1% relative to the strongest adapted baseline ( $p < 0.05$ ). When an unseen bystander walks through the scene, its error rises by only 6.5%.

Radar is already a video

Most prior mmWave pose systems first compress the raw signal into a simpler intermediate form. One common choice is a sparse point cloud obtained by peak detection. Another is a Micro-Doppler map that aggregates motion over a time window. Both choices add hand-tuned processing and throw away much of the spatiotemporal structure in the original signal.

The observation behind MAEPose is that a radar operating continuously already produces a sequence of 2D spectral images. Each frame is a Range–Doppler (RD) map, showing how much energy returns from each distance at each radial velocity. Stack these frames over time and you have a video. That puts recent progress in video representation learning within reach, including masked autoencoders (MAE), which learn by hiding most of the input and asking the model to fill it in.

A second observation concerns labels. Radar can record all day without anyone annotating it. Supervised pipelines leave that unlabelled footage unused, while self-supervised pre-training turns it into useful signal.

Photograph of the capture rig: an Intel RealSense RGB camera mounted above a Texas Instruments AWR1843BOOST radar board with a DCA1000EVM capture card, and a microcontroller synchroniser on top. A cascaded radar board is also mounted but labelled as not used. — Figure 1. The data collection platform. A Texas Instruments AWR1843BOOST radar with a DCA1000EVM capture card records mmWave signals, and an Intel RealSense D455 records RGB used only to derive ground-truth skeletons. A microcontroller trigger synchronises all sensors at 10 Hz.

How MAEPose works

MAEPose is trained in two stages, shown in the figure below.

Two-stage diagram. Stage 1: a 20-frame Range-Doppler video is split into 3D patches, 90% are randomly masked, a 12-block ViT encoder processes the visible patches, and a 4-block decoder reconstructs the masked ones. Stage 2: the pre-trained encoder processes the full unmasked video, and a spatiotemporal heatmap decoder outputs five frames of 56 by 56 joint heatmaps, which argmax converts to skeletons. — Figure 2. MAEPose architecture. Stage 1 learns a spatiotemporal representation by reconstructing masked radar video patches. Stage 2 transfers the encoder and trains a lightweight multi-frame heatmap decoder for pose estimation.

Stage 1: masked pre-training on radar video. A 20-frame RD clip is cut into spacetime patches of $2 \times 16 \times 16$ , giving $10 \times 14 \times 14 = 1{,}960$ tokens. MAEPose then hides 90% of them at random, so the encoder sees only about 196 tokens. The encoder is a standard Vision Transformer (12 blocks, 384 dimensions, 6 heads). A lightweight 4-block decoder receives the encoded visible tokens plus learnable mask tokens and reconstructs the missing patches. The loss is the mean squared error over the masked patches $\mathcal{M}$ only:

\mathcal{L}_{\text{MAE}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\lVert \hat{\mathbf{x}}_p - \mathbf{x}_p \right\rVert_2^2

The high masking ratio suits radar. Consecutive frames share nearly the same background, and only a small fraction of pixels in each frame carries body motion. Hiding most of the input stops the model from coasting on that redundancy and pushes it to model the motion itself.

Three rows of five Range-Doppler frames. Top: original frames with a bright vertical stripe and motion features. Middle: the same frames with 90% of patches blacked out. Bottom: reconstructed frames that recover the stripe and motion features while smoothing the background noise. — Figure 3. Masked reconstruction on an unseen RD clip. Top: original frames. Middle: the 90%-masked input the model actually receives. Bottom: the reconstruction. The Doppler structure is recovered, while the noisy background is largely smoothed away.

Stage 2: decoding pose as heatmaps. For fine-tuning, the reconstruction decoder is discarded and the pre-trained encoder processes the full, unmasked clip. Instead of pooling features into a vector and regressing joint coordinates, MAEPose attaches a small decoder. It consists of a 3D convolution over time followed by spatial upsampling from $14 \times 14$ to $56 \times 56$ . The output is a heatmap for each of 13 body joints in each of 5 output frames, trained against 2D Gaussians centred on the ground-truth joints.

Joint positions are then read off with an argmax over each heatmap $\mathbf{H}_k$ :

(\hat{u}_k, \hat{v}_k) = \arg\max_{(i,j)} \mathbf{H}_k(i, j), \qquad \hat{x}_k = \frac{\hat{v}_k}{W'}, \quad \hat{y}_k = \frac{\hat{u}_k}{H'}

This keeps the input and the output in the same visual form. The spatial layout the encoder has learned is preserved all the way to the prediction rather than collapsed early. Predicting several frames jointly also lets each frame draw on evidence from its neighbours.

Data and evaluation

The authors evaluated MAEPose on three datasets collected within UCL's research framework. Together they cover two rooms, 17 participants aged 20 to 50, and 21 distinct actions, totalling over 108,200 frames (about 180 minutes) of synchronised radar and skeleton data.

mmTryOn: 8 participants performing 11 clothes try-on gestures in a lab room of about 12 m².
mmMove: 9 participants performing 10 full-body movements in a studio of about 30 m².
mmYoga: the same 9 participants performing 9 yoga-inspired whole-body movements, which are harder to repeat consistently.

Ground truth consists of 13 2D keypoints extracted from the synchronised RGB frames with ViTPose. Every experiment uses leave-one-person-out cross-validation: the test person is excluded from both pre-training and fine-tuning. Differences are tested with a Friedman omnibus test followed by Bonferroni-corrected pairwise tests.

The baselines are CubeLearn (CNN+LSTM), PoseFormer (Transformer) and HuPR (mmWave spectrogram-based). They were originally designed for single frames or aggregated images. For a fair comparison, each baseline's input stage was replaced with the same 3D convolutional backbone so that all models consume identical radar video.

Results

On Range–Doppler input, MAEPose achieves the lowest error on all three datasets. The table reports mean per-joint position error (MPJPE, lower is better) and the percentage of keypoints within 5 cm (PCK@5cm, higher is better).

Dataset	Best baseline MPJPE (m)	MAEPose MPJPE (m)	Error reduction	MAEPose PCK@5cm
mmTryOn	0.0344	0.0268	22.1%	91.1%
mmMove	0.0385	0.0307	20.3%	85.3%
mmYoga	0.0447	0.0419	6.3%	75.4%
mmMove with bystander (zero-shot)	0.0404	0.0328	18.8%	84.2%

mmYoga is the hardest dataset for every model. Its movements are slower, and participants found them difficult to repeat consistently. The margin over baselines narrows there, but pre-training matters most: MPJPE falls from 0.0712 m with random initialisation to 0.0419 m.

The per-action breakdown shows a consistent pattern. MAEPose leads by the widest margins on dynamic movements, such as Stepping (28% lower error) and Chest expand (26% lower) on mmMove. For slow, subtle actions it still leads, but by less, because small motions produce fewer Doppler variations.

Radar chart of per-action MPJPE on mmMove for ten actions. The MAEPose polygon (blue) lies inside the CubeLearn, PoseFormer and HuPR polygons on every action. — Figure 4. Per-action MPJPE on mmMove (lower is better, closer to the centre). MAEPose, in blue, has the lowest error on every action, with the largest margins on dynamic movements.

The qualitative example below shows why heatmaps help. In this arm-raise sequence, MAEPose's skeletons stay closest to the grey ground truth, including the raised arms. The regression baselines follow the arm positions less closely.

Grid of five time steps. Rows show the Range-Doppler input, a 3D body mesh reference, MAEPose heatmaps, and skeletons from MAEPose, CNN+LSTM, PoseFormer and HuPR overlaid on a grey ground-truth skeleton. MAEPose skeletons follow the raised arms most closely. — Figure 5. Qualitative comparison on an arm-raise sequence from mmTryOn. A 3D body mesh derived from RGB is shown for reference in place of camera images, to protect participant privacy. Grey skeletons are ground truth.

Robustness to bystanders. Real rooms are rarely empty. To test this, the authors recorded an additional 6,040 frames in which a second person walked freely around the participant, sometimes between them and the radar, with the furniture rearranged. No model was retrained on this data. MAEPose's MPJPE rose from 0.0307 m to 0.0328 m, which the paper reports as a 6.5% increase, and it remained 18.8% below the best baseline. The authors attribute this to the spatiotemporal prior learned in pre-training. When a few frames are corrupted by stray reflections, the model can rely on context from the frames around them.

Which radar view to use. The same radar data can be rendered as Range–Doppler (motion) or Range–Azimuth (angle). The study compared RD, RA and a dual-stream fusion of both, and found RD ≈ Dual ≫ RA on every dataset. For MAEPose, RA increased error by 24% to 29% relative to RD, while fusion matched RD at roughly double the input cost. For pose, motion information turns out to matter more than angular position.

All models, MAEPose included, ran at over 110 frames per second on a single RTX 4090 GPU. That is well above the radar's 10 Hz frame rate.

Limitations

These results should be read in context.

Controlled indoor settings. All data comes from two rooms, and each fold holds out a person, not an environment. Generalisation to new rooms is not yet demonstrated, and the authors name it as the next step.
2D skeletons from a camera teacher. Ground truth is 13 2D keypoints produced by ViTPose from RGB frames, so any errors in those labels carry into both training and evaluation.
Modest scale. The study covers 17 participants and 21 actions. That is enough for rigorous cross-validation, but it is small compared with vision benchmarks.
Difficult motions remain difficult. On slow, low-motion actions and on mmYoga, the margin over baselines is smaller, reflecting the weaker Doppler signal those movements produce.

What's next

The paper's stated next step is cross-environment adaptation: fine-tuning MAEPose on a small amount of labelled data from a new room, to lower the cost of deploying in a new space. More broadly, the work suggests that treating radar as video, and pre-training on unlabelled footage, is a practical foundation for contactless human sensing. It applies to settings the paper studies, such as exercise and movement analysis, where cameras are unwelcome or unreliable.

This research direction informs our work on contactless sensing at Cymatics.

Citation

BibTeX

@misc{wei2026maepose,
  title  = {{MAEPose}: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on {mmWave} Video},
  author = {Wei, Xijia and Fang, Yuan and Chetty, Kevin and Cho, Youngjun and Bianchi-Berthouze, Nadia},
  year   = {2026},
  note   = {Preprint},
  institution = {University College London}
}