Per clip, averaged over frames.
↑ higher better (PSNR·SSIM·RC-S·m/bg-PSNR) ·
↓ lower better (LPIPS·RC-T·MSE).
Ground truth: predictions are VAE-decoded (unavoidable), but for the object-removal sets the GT is the
pristine source RGB from the eval mirror —
not the VAE round-tripped mp4s — re-preprocessed (first-N frames + stretch resize) to align to the prediction. Which streams are scored depends on the source: monitor64/val40 (synthetic) have decoded fg+bg GT; RORD50/PROVE-M have a pristine clean plate + mask; PROVE-H has neither (only the reference-free RC metrics).
- psnr / lpips / ssim · _fg, _bg — full-frame, prediction vs the GT layer. fg and bg for synthetic; bg only for object-removal (vs the pristine clean plate). PSNR/SSIM
data_range=1; LPIPS = AlexNet normalize=True.
- psnr_fg_masked / lpips_fg_masked — fg fidelity for object-removal sets with no fg GT: the pristine composite is the GT (object present), scored only over the object mask.
exact definition
Mask = pristine source mask thresholded >0.5. PSNR: pixel MSE of fg_pred vs composite over the masked pixels only (unmasked excluded), then 10·log₁₀(1/MSE). LPIPS: both zeroed outside the mask, cropped to the mask's bbox (expanded to ≥32px), then LPIPS on that object crop.
- mpsnr ↑ (m-PSNR) / bgpsnr ↑ (bg-PSNR) — the object-removal PSNR split (PROVE paper): the removal result (
bg_pred) vs the pristine clean plate, decomposed by region.
region split
m-PSNR = PSNR over the object mask (the removed-object region) — did we fill the hole with the correct background? bg-PSNR = PSNR outside the mask — did we leave the rest untouched (preservation; usually high)? Each = MSE over that region's pixels only → 10·log₁₀(1/MSE). Together they decompose the full-frame psnr_bg.
- rc_s ↑ (RC-S) / rc_t ↓ (RC-T) — PROVE Removal Coherence (arXiv:2605.14534), DINOv2-giant patch features. Reference-free — need only the removal result + mask, so they also score PROVE-H (no clean plate). Computed on
bg_pred + the clip's own object mask.
how RC-S / RC-T work (DINOv2 MMD)
RC-S (spatial, ↑): does the removed region blend with its surroundings within a frame? Over each mask component's bounding square (expanded ~⅓), a sliding window (16-patch kernel, stride 8) computes RBF-MMD between DINOv2 features inside vs outside the mask; low MMD = the fill matches the background, reported as g_mmd = exp(−MMD/τ) (τ=3) so higher≈1 is better.
RC-T (temporal, ↓): is the removed region coherent across frames? For each consecutive pair, MMD between frame t and t+1 DINO features in the shared mask region — lower = more temporally stable.
Why not PSNR/SSIM for removal? Full-reference metrics reward copy-paste over genuine erasure and are dominated by the unchanged background; RC localizes to the edited region and needs no GT.