This repo contains the skript, notebooks and models resulting from the master project "Inpainting Detection" at the Ruhr University Bochum in the summer term 2023.
The objective of the second project of the seminar was to develop an effective image inpainting detector, with a specific focus on understanding and addressing the noise patterns found in inpainted images. Furthermore the project sought to answer the question if the noise patterns characteristic of modern inpainting pipelines could be effectively learned and detected without the use of specialized model architectures.
We trained our models on two datasets.
- The first dataset was provided to us. The dataset can be found at: https://ruhr-uni-bochum.sciebo.de/f/1777053235 (only square masks). Files ending in *realfake.webp are inpainted using the entire model. Files ending in fakefake.webp are re-encoded using the latent autoencoder and then blended using the mask (.mask.webp).
- The second dataset is an composition of the first one and additional generated data. The additional data can be found on huggingface. In this dataset the originial image is included (.original.webp) as well as the inpainted (.inpainted.webp) image that was generated by the difussion model.
Listing of all relevant files for this project.
- report.pdf: The report of the project
- baseline: contains the training scripts and model checkpoints for the baseline model.
- Nix: contains the training scripts, different model architectures and theri model checkpoints for the NIX model.
- data_generation: scripts that were used to generate the additional data
- inpainting.py: generation of the first data set
- qmd_masks.py: generation of images with the QuickDraw dataset
- seg_masks.py: generation of images with the MaskFormer classifier
- yolo_masks.py: generation of images with the YOLO classifier
- generate_SD2.1.py: generation of inpainted images with stable diffusion 2.1
