markmohr / awesome-sketch-synthesis

:books: A collection of papers about Sketch Synthesis (Generation).

Awesome-Sketch-Synthesis

A collection of papers about Sketch Synthesis (Generation). Mainly focus on stroke-level vector sketch synthesis.

Feel free to create a PR or an issue.

Outlines

• 0. Survey
• 1. Datasets
• 2. Sketch-Synthesis Approaches
  • 1) Semantic Concept-to-sketch
  • 2) Photo-to-sketch
  • 3) Text/Attribute-to-sketch
  • 4) 3D shape-to-sketch
  • 5) Art-to-sketch
• 3. Vector Graphics Generation

---

0. Survey

Paper	Source	Code/Project Link
Deep Learning for Free-Hand Sketch: A Survey	TPAMI 2022	[code]

1. Datasets

Here Vector strokes means having svg data. With photos means having the photo-sketch paired data.

Level	Dataset	Source	Vector strokes	With photos	Notes
Characters	Omniglot		✔️	❌	Alphabets characters
	KanjiVG		✔️	❌	Chinese characters
	Kuzushiji		❌	❌	Japanese characters
Icon	FIGR-8-SVG		✔️	❌	Icons with text descriptions
Systematic Symbol	SketchIME	ACM MM 2023	✔️	❌	Systematic sketches with semantic annotations
Instance-level	TU-Berlin	SIGGRAPH 2012	✔️	❌	Multi-category hand sketches
	Sketchy	SIGGRAPH 2016	✔️	✔️	Multi-category photo-sketch paired
	QuickDraw	ICLR 2018	✔️	❌	Multi-category hand sketches
	QMUL-Shoe-Chair-V2	CVPR 2016	✔️	✔️	Only two categories
	Sketch Perceptual Grouping (SPG)	ECCV 2018	✔️	❌	With part-level semantic segmentation information
	FaceX	AAAI 2019	✔️	❌	Labeled facial sketches
	Creative Sketch	ICLR 2021	✔️	❌	With annotated part segmentation
	ImageNet-Sketch	NeurIPS 2019	❌	❌	50 images for each of the 1000 ImageNet classes
	SEVA	NeurIPS 2023	✔️	✔️	90K human-generated sketches that vary in detail
Scene-level	SketchyScene	ECCV 2018	❌	✔️	With semantic/instance segmentation information
	CMPlaces	TPAMI 2018	❌	✔️	Cross-modal scene dataset
	Context-Skecth	Expressive 2018	❌	✔️	Context-based scene sketches for co-classification
	SketchyCOCO	CVPR 2020	❌	✔️	Scene sketch, segmentation and normal images
	FS-COCO	ECCV 2022	✔️	✔️	Scene sketches with text description
	SFSD	VC 2022	✔️	✔️	Completely hand-drawn scene sketches with label annotation
Drawing from photos	Photo-Sketching	WACV 2019	✔️	✔️	ScenePhoto-sketch paired
	Tracing-vs-Freehand	SIGGRAPH 2021	✔️	✔️	Tracings and freehand drawings of images
Drawing from 3D models	DifferSketching	SIGGRAPH Asia 2022	✔️	❌	3D model-sketch paired, with novice and professional ones
Portrait	CUFS	TPAMI 2009	❌	✔️	Face-sketch pairs
	APDrawing	CVPR 2019	❌	✔️	Portrait-sketch paired
	SKSF-A	EG 2024	❌	✔️	Face-sketch pairs of seven styles
Anime portrait	Danbooru-Parsing	TOG 2023	❌	✔️	For anime portrait parsing and anime translation
Children's Drawing	Amateur Drawings	TOG 2023	❌	✔️	With character bounding boxes, segmentation masks, and joint location annotations
Rough sketch	Da Vinci	CGI 2018	❌	✔️	Line drawing restoration dataset
	Rough Sketch Benchmark	SIGGRAPH Asia 2020	✔️	✔️	Rough and clean sketch pairs (only for evaluation)
CAD	ld3d	SIGGRAPH 2008	❌	❌	Line Drawings of 3D Shapes
	OpenSketch	SIGGRAPH Asia 2019	✔️	❌	Product Design Sketches
	SketchGraphs	ICML 2020 Workshop	✔️	❌	Sketches extracted from real-world CAD models
	Fusion 360 Gallery	SIGGRAPH 2021	✔️	❌	For 'sketch and extrude' designs
	FloorPlanCAD	ICCV 2021	✔️	❌	With instance and semantic annotations

2. Sketch-Synthesis Approaches

1) Semantic Concept-to-sketch

Level	Paper	Source	Code/Project Link
Instance-level	A Neural Representation of Sketch Drawings (sketch-rnn)	ICLR 2018	[Code] [Project] [Demo]
	Sketch-pix2seq: a Model to Generate Sketches of Multiple Categories		[Code]
	AI-Sketcher : A Deep Generative Model for Producing High-Quality Sketches	AAAI 2019	[Project]
	Stroke-based sketched symbol reconstruction and segmentation (stroke-rnn)	CGA 2019
	BézierSketch: A generative model for scalable vector sketches	ECCV 2020	[Code]
	Pixelor: A Competitive Sketching AI Agent. So you think you can beat me?	SIGGRAPH Asia 2020	[Project] [Code]
	Creative Sketch Generation	ICLR 2021	[Project] [Code]
	Computer-Aided Design as Language	arxiv 2105
	DoodleFormer: Creative Sketch Drawing with Transformers	ECCV 2022	[Project] [Code]
	SketchKnitter: Vectorized Sketch Generation with Diffusion Models	ICLR 2023	[Code]
	Self-Organizing a Latent Hierarchy of Sketch Patterns for Controllable Sketch Synthesis	TNNLS 2023	[Code]

---

2) Photo-to-sketch

• vector image generation

Data type	Paper	Source	Code/Project Link
Facial	Style and abstraction in portrait sketching	TOG 2013
Instance-level	Free-Hand Sketch Synthesis with Deformable Stroke Models	IJCV 2017	[Project] [code]
	Learning to Sketch with Shortcut Cycle Consistency	CVPR 2018	[Code1] [Code2]
	Learning Deep Sketch Abstraction	CVPR 2018
	CLIPasso: Semantically-Aware Object Sketching	SIGGRAPH 2022	[Project] [Code]
Scene-level	Sketch Generation with Drawing Process Guided by Vector Flow and Grayscale	AAAI 2021	[Code]
	CLIPascene: Scene Sketching with Different Types and Levels of Abstraction	ICCV 2023	[Project] [Code]
Technical Drawings	Deep Vectorization of Technical Drawings	ECCV 2020	[Project] [code]

• raster image generation

Type	Paper	Source	Code/Project Link
Facial	ArtLine	Online demo	[Code]
	APDrawingGAN: Generating Artistic Portrait Drawings from Face Photos with Hierarchical GANs	CVPR 2019	[Code] [Demo]
	Unpaired Portrait Drawing Generation via Asymmetric Cycle Mapping	CVPR 2020	[Code]
	Line Drawings for Face Portraits From Photos Using Global and Local Structure Based GANs	TPAMI 2020	[Code]
	Quality Metric Guided Portrait Line Drawing Generation from Unpaired Training Data	TPAMI 2022	[Code]
	Human-Inspired Facial Sketch Synthesis with Dynamic Adaptation	ICCV 2023	[Code]
	Stylized Face Sketch Extraction via Generative Prior with Limited Data	EG 2024	[Code] [Project]
Instance-level	Deep Factorised Inverse-Sketching	ECCV 2018
	Synthesizing human-like sketches from natural images using a conditional convolutional decoder	WACV 2020	[Code]
Anime	sketchKeras	online demo	[Code]
	LineDistiller	online demo	[Code]
	Anime2Sketch	online demo	[Code]
	Reference Based Sketch Extraction via Attention Mechanism	SIGGRAPH Asia 2022	[Code]
Scene-level	Photo-Sketching: Inferring Contour Drawings from Images	WACV 2019	[Code] [Project]
	Learning to generate line drawings that convey geometry and semantics	CVPR 2022	[Code] [Project]
Arbitrary	Semi-supervised reference-based sketch extraction using a contrastive learning	SIGGRAPH 2023	[Code] [Project]

---

3) Text/Attribute-to-sketch

• raster image generation

Type	Paper	Source	Code/Project Link
Facial	Text2Sketch: Learning Face Sketch from Facial Attribute Text	ICIP 2018
Scene-level	Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications	UIST 2019
Scene-level	Scones: Towards Conversational Authoring of Sketches	IUI 2020

• vector image generation

Type	Paper	Source	Code/Project Link
Arbitrary	Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts	arxiv 21.09	[code] [project]
Arbitrary	CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders	NeurIPS 2022	[code]
Arbitrary	StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation	IJCAI 2022	[code]
SVG	VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models	CVPR 2023	[project]
Arbitrary	SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation	BMVC 2023	[code]
Arbitrary	DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models	NeurIPS 2023	[project] [code]
Icon	IconShop: Text-Based Vector Icon Synthesis with Autoregressive Transformers	SIGGRAPH Asia 2023	[project]
SVG	Text-Guided Vector Graphics Customization	SIGGRAPH Asia 2023	[project]
SVG	SVGDreamer: Text Guided SVG Generation with Diffusion Model	CVPR 2024	[project] [code]
Arbitrary	SVGCraft: Beyond Single Object Text-to-SVG Synthesis with Comprehensive Canvas Layout	arxiv 24.04	[code]

---

4) 3D shape-to-sketch

Paper	Source	Code/Project Link
DeepShapeSketch : Generating hand drawing sketches from 3D objects	IJCNN 2019
Neural Contours: Learning to Draw Lines from 3D Shapes	CVPR 2020	[project] [code]
Cloud2Curve: Generation and Vectorization of Parametric Sketches	CVPR 2021	[project]
Neural Strokes: Stylized Line Drawing of 3D Shapes	ICCV 2021	[code]
Learning a Style Space for Interactive Line Drawing Synthesis from Animated 3D Models	PG 2022
CAD2Sketch: Generating Concept Sketches from CAD Sequences	SIGGRAPH Asia 2022	[project]

---

5) Art-to-sketch

Here we list sketch synthesis based on other image types, like Manga, line art, rough sketch, etc.

a) Line art

• Vector-to-Vector

Paper	Source	Code/Project Link	*-based
Closure-aware Sketch Simplification	SIGGRAPH Asia 2015	[Project]	Optimization
StrokeAggregator: Consolidating Raw Sketches into Artist-Intended Curve Drawings	SIGGRAPH 2018	[Project]	Optimization
StrokeStrip: Joint Parameterization and Fitting of Stroke Clusters	SIGGRAPH 2021	[Project] [code]	Optimization
StripMaker: Perception-driven Learned Vector Sketch Consolidation	SIGGRAPH 2023		Optimization
Region-Aware Simplification and Stylization of 3D Line Drawings	EG 2024		Optimization

• Raster-to-Vector (a.k.a. Vectorization)

Paper	Source	Code/Project Link	*-based
Topology-Driven Vectorization of Clean Line Drawings	TOG 2013		Optimization
Fidelity vs. Simplicity: a Global Approach to Line Drawing Vectorization	SIGGRAPH 2016	[Project]	Optimization
A Delaunay triangulation based approach for cleaning rough sketches	C&G 2018	[Code]	Optimization
Semantic Segmentation for Line Drawing Vectorization Using Neural Networks	EG 2018	[project] [code]	Learning
Deep Line Drawing Vectorization via Line Subdivision and Topology Reconstruction	PG 2019		Learning
Inertia-based Fast Vectorization of Line Drawings	PG 2019		Optimization
Vectorization of Line Drawings via Polyvector Fields	TOG 2019	[Code]	Optimization
Integer-Grid Sketch Simplification and Vectorization	SGP 2020	[Project] [Code]	Optimization
Deep Vectorization of Technical Drawings	ECCV 2020	[project] [code]	Learning
General Virtual Sketching Framework for Vector Line Art	SIGGRAPH 2021	[project] [code]	Learning
Keypoint-Driven Line Drawing Vectorization via PolyVector Flow	SIGGRAPH Asia 2021	[project]	Learning + Optimization
End-to-end Line Drawing Vectorization	AAAI 2022		Learning
Vectorizing Line Drawings of Arbitrary Thickness via Boundary-based Topology Reconstruction	CGF 2022		Heuristic
Singularity-Free Frame Fields for Line Drawing Vectorization	SGP 2023	[code]	Optimization

b) Rough sketch simplification / cleanup

• Datasets and benchmark

Paper	Source	Code/Project Link
A Benchmark for Rough Sketch Cleanup	SIGGRAPH Asia 2020	[Project] [Code]

• Methods

Paper	Source	Code/Project Link
Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup	SIGGRAPH 2016	[Code] [Project]
Mastering Sketching: Adversarial Augmentation for Structured Prediction	SIGGRAPH 2018	[Code] [Project]
Real-Time Data-Driven Interactive Rough Sketch Inking	SIGGRAPH 2018	[Code] [Project]
Perceptual-aware Sketch Simplification Based on Integrated VGG Layers	TVCG 2019

c) Manga (Comics)

Paper	Source	Code/Project Link
Deep extraction of manga structural lines	SIGGRAPH 2017	[Code]
Manga Filling Style Conversion with Screentone Variational Autoencoder	SIGGRAPH Asia 2020	[Project] [Code]
Generating Manga from Illustrations via Mimicking Manga Workflow	CVPR 2021	[Project] [Code]
MangaGAN: Unpaired Photo-to-Manga Translation Based on The Methodology of Manga Drawing	AAAI 2021
MARVEL: Raster Gray-level Manga Vectorization via Primitive-wise Deep Reinforcement Learning	TCSVT 2023

3. Vector Graphics Generation

Here we focus on learning-based vector graphics generation without depending on vector training data, and traditional vectorization algorithms.

• Learning with external black-box (non-differentiable) rendering simulator

Paper	Source	Code/Project Link
Synthesizing Programs for Images using Reinforced Adversarial Learning	ICML 2018	[Code]
Unsupervised Doodling and Painting with Improved SPIRAL	arxiv 1910	[Project]

• Learning with built-in differentiable rendering module

Paper	Source	Code/Project Link
Optimize and Reduce: A Top-Down Approach for Image Vectorization	AAAI 2024	[code]
Segmentation-Based Parametric Painting	arxiv 23.11	[code] [project]
Editable Image Geometric Abstraction via Neural Primitive Assembly	ICCV 2023
Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region	ACM MM 2023	[code]
Intelli-Paint: Towards Developing More Human-Intelligible Painting Agents	ECCV 2022	[project]
Towards Layer-wise Image Vectorization	CVPR 2022	[code] [project]
Paint Transformer: Feed Forward Neural Painting with Stroke Prediction	ICCV 2021	[code]
Combining Semantic Guidance and Deep Reinforcement Learning For Generating Human Level Paintings	CVPR 2021	[code]
Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes	CVPR 2021	[code]
Im2Vec: Synthesizing Vector Graphics without Vector Supervision	CVPR 2021	[Project] [code]
Stylized Neural Painting	CVPR 2021	[Code] [project]
Learning to Paint With Model-based Deep Reinforcement Learning	ICCV 2019	[code]
Strokenet: A neural painting environment	ICLR 2019	[Code]
Neural Painters: A learned differentiable constraint for generating brushstroke paintings	arxiv 1904	[Code]
Learning to Sketch with Deep Q Networks and Demonstrated Strokes	arxiv 1810
Unsupervised Image to Sequence Translation with Canvas-Drawer Networks	arxiv 1809

• Vectorization

Paper	Source	Code/Project Link
Depixelizing pixel art	SIGGRAPH 2011
Perception-Driven Semi-Structured Boundary Vectorization	SIGGRAPH 2018	[Webpage]
PolyFit: Perception-aligned Vectorization of Raster Clip-art via Intermediate Polygonal Fitting	SIGGRAPH 2020	[Webpage] [Code]
ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis	TVCG 2021
TCB-Spline-Based Image Vectorization	TOG 2022
Image vectorization and editing via linear gradient layer decomposition	SIGGRAPH 2023

awesome-sketch-synthesis's Issues

Additional paper

I am not sure if you have seen KidPen: a stroke-based method for kid-style sketches synthesis from photos, but I think it takes a good approach to the sketch generation problem with impressive results in terms of shape abstraction. The one caveat is that it relies on human interaction for the segmentation portion of the task, but it is still a good read.

Dataset

for portrait dataset, please add
CUFS http://mmlab.ie.cuhk.edu.hk/archive/facesketch.html
SKSF-A https://github.com/kwanyun/SKSF-A

code link for Reference based sketch extraction via attention mechanism

Hi, thanks for the awesome repository

I have link for the code that you missed
https://github.com/ref2sketch/ref2sketch

thank you

New paper

Hi thanks for the awesome page

I have the new paper which is follow-up study of "Reference based sketch extraction via attention mechanism"

new paper's title is "Semi-supervised reference-based sketch extraction using a contrastive learning framework"

it is accepted to upcoming siggraph 2023 journal track

project page : https://chanuku.github.io/Semi_ref2sketch/
code : https://github.com/Chanuku/semi_ref2sketch_code
dataset (4skst) : https://github.com/Chanuku/4skst

Thank you

dataset download

Thanks for your work.
But the download address of QMUL-Shoe-Chair-V2 can not open.
Would you mind provide the datasets ?
Thanks for your help