AnydoorMed: Reference-Guided Anomaly Inpainting for Medical Counterfactuals

AnydoorMed: Reference-Guided Anomaly Inpainting for Medical Counterfactuals

The University of Manchester
Completed in fulfilment of "Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation"

Abstract

High-fidelity data is essential for developing reliable computer-aided diagnostics in medical imaging, yet clinical datasets are difficult to obtain and often suffer from severe class imbalance, particularly for rare pathologies such as malignant breast lesions. Synthetic data offers a promising avenue to mitigate these limitations by augmenting existing datasets with diverse and realistic counterfactual examples. To be clinically useful, generated data must respect anatomical constraints, preserve fine-grained tissue structures, and allow controlled insertion of abnormalities. While recent diffusion-based methods enable anomaly synthesis via text or segmentation masks, such conditioning often fails to capture subtle structural variations. In contrast, diffusion inpainting has shown promising results on natural images but remains underexplored in medical imaging. This works proposes AnydoorMed, a reference-guided inpainting method for mammography that transfers lesion characteristics from a source scan to a target location, blending them seamlessly with surrounding tissue, supporting the generation of realistic counterfactual examples.

Architecture

This work adapts AnyDoor, a reference-based image inpainting method, to the medical domain. Additional input and output adaption layers were added to the variational autoencoder of StableDiffusion and finetuned on mammography scans from VinDr-Mammo. Gated cross-attention layers added within the pre-trained diffusion inpainting model were trained in a separate stage, with the medical autoencoder frozen. The anomaly's High Frequency map (HF map) was coloured purple for visualisation purposes only.

Results

Anomaly insertion results. AnydoorMed inserts the reference anomaly (second column), guided by the context and high-frequency map context (first column), into the healthy mammography scan (fourth column), producing the composited result (third column). The inpainted anomalies preserve some of the features present in the reference image, such as calcifications (first row). For all examples, the anomaly was composited semantically in the destination scan, within the breast tissue.

Anomaly replacement results. AnydoorMed replaces the anomaly from the original scan (fourth column), with the reference anomaly (second column), guided by the context and high-frequency map context (first column), producing the composited result (third column). This is done by removing the anomaly from the scan and using a similarly-sized reference as a condition. The inpainted anomalies preserve some of the features present in the reference image, such as the “excavation” from the first row. All generated scans look highly realistic, with anomalies being semantically blended within the breast tissue.

AnydoorMed results on the test set of VinDr-Mammo dataset according to image realism metrics.

Discussion

These examples illustrate some failure cases of AnydoorMed. From top to bottom: the inserted anomaly does not closely replicate the microcalcifications from the reference image (which may be undesirable in certain scenarios); the inpainting produces an anatomically implausible result due to the bounding box being placed primarily outside the breast tissue; and finally, the last example exhibits visible copy-and-paste artefacts.

@article{buburuzan2025reference, title={Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation}, author={Buburuzan, Alexandru}, journal={arXiv preprint arXiv:2507.23058}, year={2025} }