SKIA-rPPG
Spatial-Knowledge Informed Attention for Motion-Robust Remote Photoplethysmography
A motion-robust contactless vital-sign (BVP & heart rate) estimation framework combining a parameter-free static lower-face anatomical prior with a 3-stream 3D-CNN (SKIA-Net). Achieves 0.7522 Pearson correlation and 3.73 BPM HR MAE on UBFC-rPPG, accepted at ICMV 2026.
The Technical Bottleneck
Camera-based remote photoplethysmography (rPPG) measures subtle skin-color variations caused by cardiac blood volume pulse (BVP). However, non-rigid lower-face movements (talking, smiling, jaw motion) introduce severe intensity artifacts that overwhelm the physiological signal.
Why Existing Baselines Fail
Existing motion-robust rPPG methods rely on implicit spatial attention or optical-flow masks. We discovered that frame-varying optical-flow masks actually degrade pulse reconstruction because changing spatial weights across frames alters the sampled skin-color distribution. We therefore tested whether a fixed anatomical prior suppressing the lower 30% of the face provides a superior inductive bias.
Core Formulation & Insight
We introduce a static spatial-knowledge mask (α = 0.7 retaining the forehead and upper cheeks) and validate it on both classical CHROM and our compact 3-stream spatiotemporal network, SKIA-Net. SKIA-Net jointly encodes Raw frames (X), Static Masked frames (X ⊙ M), and Temporal-Difference frames (ΔX_t) using a 3D-CNN encoder and 1D temporal convolutional head trained via Negative Pearson Correlation Loss.
Architectural Design
Multi-stream spatiotemporal 3D-CNN fusing full facial context, anatomically stabilized upper-face skin regions, and frame-to-frame motion residuals.
SKIA multi-stream spatiotemporal architecture and evaluation pipeline (ICMV 2026).
1. Facial ROI Cropping & Static Anatomical Masking (α = 0.7)
Extracts facial clips resized to 96×96 / 128×128 and applies a temporally invariant mask M(i,j) = 1 for i < 0.7H to suppress mouth and jaw articulation without landmark jitter.
2. Three-Stream Representation Construction
Constructs aligned tensors ∈ ℝ^{T×H×W×3} for Raw appearance X, Masked appearance X ⊙ M, and Temporal differences ΔX_t = X_t − X_{t-1}.
3. 3D-CNN Spatiotemporal Encoder & Temporal Head
Concatenates the three streams channel-wise into a 3D convolutional encoder, spatial average pooling layer, and 1D temporal convolutional head producing per-frame BVP estimates.
4. Negative Pearson Loss & FFT Heart-Rate Estimation
Optimizes temporal waveform morphology directly via L = −corr(ŷ, y), followed by 0.7–2.5 Hz (42–150 BPM) bandpass filtering and FFT peak heart-rate extraction.
Technical Stack & Systems Engineering
Implementation Highlights
- →Built end-to-end video preprocessing, chrominance projection (CHROM), dense Farnebäck optical-flow dynamic masking, and 3D-CNN training pipelines.
- →Implemented statistical validation across all 42 UBFC-rPPG subjects using paired Wilcoxon signed-rank tests and masking-ratio sweeps (α ∈ {0.5, 0.6, 0.7, 0.8}).
Datasets & Protocol
Datasets & Corpora
- UBFC-rPPG Dataset (42 subjects for CHROM & masking ablations; 32 train / 10 test subject-disjoint split for SKIA-Net)
Evaluation Protocol
Compared CHROM vs. Static Masked CHROM vs. Dynamic Optical-Flow Soft Mask across 42 subjects, and evaluated PhysNet-lite, SKIA Raw, SKIA Masked, SKIA Raw+Masked, and SKIA Full on the 10-subject test set.
Tracked Evaluation Metrics
SKIA Full achieved 0.7522 Pearson correlation and 3.73 BPM HR MAE, while static lower-face masking improved CHROM on 88.1% of subjects (p = 2.0 × 10⁻⁷).
Improves over PhysNet-lite (0.2350) and SKIA Raw (0.6320)
Reduces error from 8.99 BPM (Raw) and 4.97 BPM (Raw+Masked)
Pearson 0.3507 → 0.4166 (88.1% winrate, p = 2.0 × 10⁻⁷)
Dynamic optical flow degrades winrate to 4.8%
Empirical Analysis & Interpretation
Combining raw, masked, and temporal-difference streams yields strong synergy: the masked stream stabilizes heart-rate estimation while the raw and temporal-difference streams preserve complementary skin-color and short-term pulse dynamics.
Static lower-face suppression (α = 0.7) isolating forehead and upper-cheek physiological regions.
What This System Contributes
- 01First author: formulated the static spatial-knowledge hypothesis, designed SKIA-Net, implemented the signal-processing and PyTorch pipelines, and authored the ICMV 2026 paper.
Boundary Conditions & Future Work
- ·Evaluated on UBFC-rPPG with a heuristic ratio mask (α = 0.7); landmark-adaptive masking on dedicated speech corpora is planned for future work.
SKIA: Spatial-Knowledge Informed Attention for Robust Remote Photoplethysmography under Lower-Face Motion Artifacts
Van-Truong Le, Viet-Tham Huynh, Minh-Triet Tran