GithubHelp home page GithubHelp logo

tianzhengg / dct-net Goto Github PK

View Code? Open in Web Editor NEW

This project forked from menyifang/dct-net

0.0 0.0 0.0 126.39 MB

Official implementation of "DCT-Net: Domain-Calibrated Translation for Portrait Stylization", SIGGRAPH 2022 (TOG); Multi-style cartoonization

License: Apache License 2.0

Python 7.16% Jupyter Notebook 92.84%

dct-net's Introduction

DCT-Net: Domain-Calibrated Translation for Portrait Stylization

Official implementation of DCT-Net for Full-body Portrait Stylization.

DCT-Net: Domain-Calibrated Translation for Portrait Stylization,
Yifang Men1, Yuan Yao1, Miaomiao Cui1, Zhouhui Lian2, Xuansong Xie1,
1DAMO Academy, Alibaba Group, Beijing, China
2Wangxuan Institute of Computer Technology, Peking University, China
In: SIGGRAPH 2022 (TOG) arXiv preprint

google colab logo Hugging Face Spaces

Demo

demo_vid

News

(2023-02-20) Two new style pre-trained models (design, illustration) trained with combined DCT-Net and Stable-Diffusion are provided. The training guidance will be released soon.

(2022-10-09) The multi-style pre-trained models (3d, handdrawn, sketch, artstyle) and usage are available now.

(2022-08-08) The pertained model and infer code of 'anime' style is available now. More styles coming soon.

(2022-08-08) cartoon function can be directly call from pythonSDK.

(2022-07-07) The paper is available now at arxiv(https://arxiv.org/abs/2207.02426).

Web Demo

Requirements

  • python 3
  • tensorflow (>=1.14)
  • easydict
  • imageio[ffmpeg]
  • numpy
  • both CPU/GPU are supported

Quick Start

google colab logo

git clone https://github.com/menyifang/DCT-Net.git
cd DCT-Net

Installation

conda create -n dctnet python=3.7
conda activate dctnet
pip install --upgrade tensorflow-gpu==1.15 # GPU support, use tensorflow for CPU only
pip install "modelscope[cv]==1.3.0" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
pip install "modelscope[multi-modal]==1.3.0" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

Downloads

anime 3d handdrawn sketch artstyle
design illustration

Pre-trained models in different styles can be downloaded by

python download.py

Inference

  • from python SDK
python run_sdk.py
  • from source code
python run.py

Video cartoonization

demo_vid

video can be directly processed as image sequences, style choice [option: anime, 3d, handdrawn, sketch, artstyle, sd-design, sd-illustration]

python run_vid.py --video_path input.mp4 --save_path res.mp4 --style anime

Training

Data preparation

face_photo: face dataset such as [FFHQ](https://github.com/NVlabs/ffhq-dataset) or other collected real faces.
face_cartoon: 100-300 cartoon face images in a specific style, which can be self-collected or synthsized with generative models.

Due to the copyrighe issues, we can not provide collected cartoon exemplar for training. You can produce cartoon exemplars with the style-finetuned Stable-Diffusion (SD) models, which can be downloaded from modelscope or huggingface hubs.

The effects of some style-finetune SD models are as follows:

design watercolor illustration clipart flat
  • Generate stylized data, style choice [option: clipart, design, illustration, watercolor, flat]
python generate_data.py --style clipart

Train content calibration network

To-be-added

Geometry calibration

To-be-added

Train texture translator

To-be-added

Acknowledgments

Face detector and aligner are adapted from Peppa_Pig_Face_Engine and InsightFace.

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{men2022dct,
  title={DCT-Net: Domain-Calibrated Translation for Portrait Stylization},
  author={Men, Yifang and Yao, Yuan and Cui, Miaomiao and Lian, Zhouhui and Xie, Xuansong},
  journal={ACM Transactions on Graphics (TOG)},
  volume={41},
  number={4},
  pages={1--9},
  year={2022},
  publisher={ACM New York, NY, USA}
}

dct-net's People

Contributors

menyifang avatar ak391 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.