zlzeng / deepfloorplan

Deep Floor Plan Recognition using a Multi-task Network with Room-boundary-Guided Attention

By Zhiliang ZENG, Xianzhi LI, Ying Kin Yu, and Chi-Wing Fu

[2021/07/26: updated download link]

[2019/08/28: updated train/test/score code & dataset]

[2019/07/29: updated demo code & pretrained model]

Introduction

This repository contains the code & annotation data for our ICCV 2019 paper: 'Deep Floor Plan Recognition Using a Multi-Task Network with Room-Boundary-Guided Attention'. In this paper, we present a new method for recognizing floor plan elements by exploring the spatial relationship between floor plan elements, model a hierarchy of floor plan elements, and design a multi-task network to learn to recognize room-boundary and room-type elements in floor plans.

Requirements

• Please install OpenCV
• Please install Python 2.7
• Please install tensorflow-gpu

Our code has been tested by using tensorflow-gpu==1.10.1 & OpenCV==3.1.0. We used Nvidia Titan Xp GPU with CUDA 9.0 installed.

Python packages

• [numpy]
• [scipy]
• [Pillow]
• [matplotlib]

Data

We share all our annotations and train-test split file here. Or download the annotation using the link in file "dataset/download_links.txt". The additional round plan is included in the annotations.

Our annotations are saved as png format. The name with suffixes "_wall.png", "_close.png" and "_room.png" are denoted "wall", "door & window" and "room types" label, respectively. We used these labels to train our multi-task network.

The name with suffixes "_close_wall.png" is the combination of "wall", "door & window" label. We don't use this label in our paper, but maybe useful for other tasks.

The name with suffixes "_multi.png" is the combination of all the labels. We used this kind of label to retrain the general segmentation network.

We also provide our training data on R3D dataset in "tfrecord" format, which can improve the loading speed during training.

To create the "tfrecord" training set, please refer to the example code in "utils/create_tfrecord.py"

All the raw floor plan image please refer to the following two links:

• R2V: https://github.com/art-programmer/FloorplanTransformation.git
• R3D: http://www.cs.toronto.edu/~fidler/projects/rent3D.html

Usage

To use our demo code, please first download the pretrained model, find the link in "pretrained/download_links.txt" file, unzip and put it into "pretrained" folder, then run

python demo.py --im_path=./demo/45719584.jpg 

To train the network, simply run

python main.py --pharse=Train

Run the following command to generate network outputs, all results are saved as png format.

python main.py --pharse=Test

To compute the evaluation metrics, please first inference the results, then simply run

python scores.py --dataset=R3D

To use our post-processing method, please first inference the results, then simply run

python postprocess.py

or

python postprocess.py --result_dir=./[result_folder_path]

Citation

If you find our work useful in your research, please consider citing:

---

@InProceedings{zlzeng2019deepfloor,
	author = {Zhiliang ZENG, Xianzhi LI, Ying Kin Yu, and Chi-Wing Fu},
	title = {Deep Floor Plan Recognition using a Multi-task Network with Room-boundary-Guided Attention},
	booktitle = {IEEE International Conference on Computer Vision (ICCV)},
	year = {2019}
}

---

deepfloorplan's Issues

scipy.misc is depreacated

I can't run the demo nor train the model because everywhere you use scipy for imread, imresize I'm running error

Do you mind providing an update ?

Thanks !

where the tensor 'inputs:0' define?

In the demo.py to get the inputs in the graph when infer using the 'graph.get_tensor_by_name('inputs:0')'. But I cannot find where define this in the main.py when build the graph.

How I solve the problem about r2v dataset and roomtype error

First of all, thanks to @zlzeng sent me txt of origin name by correct order, which give me feasibility to modify these files into correct correspondence. If you need the raw txt, check the file below.
test.txt
train.txt
val.txt

If you don't want to know what happens, just find the summary in the end.

To train model with r2v dataset, we need to correct the correspondence in txt file and change a little bit code. Let's see how to get through this. You can download r2v origin data from here.

Modify txt file
First I merged zlzeng's txt and project's txt, the result looks like:

../dataset/floorplan_image/04/94/1b4b5b3a8dcbe2135310d067eb6e/0001.jpg ../dataset/jp/test/1_wall.png ../dataset/jp/test/1_close.png ../dataset/jp/test/1_rooms.png ../dataset/jp/test/1_close_wall.png
../dataset/floorplan_image/08/fe/cb1ded96be76713683c50e731f3f/0001.jpg ../dataset/jp/test/2_wall.png ../dataset/jp/test/2_close.png ../dataset/jp/test/2_rooms.png ../dataset/jp/test/2_close_wall.png
../dataset/floorplan_image/04/8c/723a3389ae4aaaa11d3d97123e2d/0001.jpg ../dataset/jp/test/3_wall.png ../dataset/jp/test/3_close.png ../dataset/jp/test/3_rooms.png ../dataset/jp/test/3_close_wall.png
../dataset/floorplan_image/03/c0/fd5d2bb20c86db0c8f24a83b8a3c/0003.jpg ../dataset/jp/test/4_wall.png ../dataset/jp/test/4_close.png ../dataset/jp/test/4_rooms.png ../dataset/jp/test/4_close_wall.png
../dataset/floorplan_image/00/23/500a167ab926df601817631d6733/0002.jpg ../dataset/jp/test/5_wall.png ../dataset/jp/test/5_close.png ../dataset/jp/test/5_rooms.png ../dataset/jp/test/5_close_wall.png
../dataset/floorplan_image/01/e5/df6969c71c4cb3b786cde09b4efc/0002.jpg ../dataset/jp/test/6_wall.png ../dataset/jp/test/6_close.png ../dataset/jp/test/6_rooms.png ../dataset/jp/test/6_close_wall.png
../dataset/floorplan_image/08/3c/4144463ce4d677723e9032358408/0002.jpg ../dataset/jp/test/7_wall.png ../dataset/jp/test/7_close.png ../dataset/jp/test/7_rooms.png ../dataset/jp/test/7_close_wall.png
../dataset/floorplan_image/09/22/b9bd969fff7820dbdbc660b6df26/0001.jpg ../dataset/jp/test/8_wall.png ../dataset/jp/test/8_close.png ../dataset/jp/test/8_rooms.png ../dataset/jp/test/8_close_wall.png
../dataset/floorplan_image/02/d4/6ab7b8396d8f24c31ea2e8a0e2f2/0003.jpg ../dataset/jp/test/9_wall.png ../dataset/jp/test/9_close.png ../dataset/jp/test/9_rooms.png ../dataset/jp/test/9_close_wall.png
../dataset/floorplan_image/00/99/28fbde7b272adad6da35f0bc16c3/0001.jpg ../dataset/jp/test/10_wall.png ../dataset/jp/test/10_close.png ../dataset/jp/test/10_rooms.png ../dataset/jp/test/10_close_wall.png

Then I try to run main.py --phase test, it works well but when I check ./out directory, I found that results were overwritten by itself because of the name of origin file is almost the same like 0001.jpg. So we have to change name of images and path in txt.
I write a simple script to do this task:

Before running the script below, you should download those files and put it into ./dataset
r2v_train.txt
r2v_test.txt

# DeepFloorplan/utils/txt_modify.py
import os


TRAIN_TXT = "r2v_train.txt"
TEST_TXT = "r2v_test.txt"


if __name__ == "__main__":
    for txt_name in [TRAIN_TXT, TEST_TXT]:
        root_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "../../dataset"))
        print(root_path)
        txt_path = os.path.join(root_path, txt_name)
        print(txt_path)

        with open(txt_path) as file:
            lines = file.read().splitlines()

        path = [[line.split("\t")[0], line.split("\t")[1]] for line in lines]
        
        target_path = []
        for origin, target in path:
            root, origin_name = os.path.split(origin)
            target_name = os.path.basename(target).split("_")[0] + ".jpg"
            
            try:
                os.rename(origin, os.path.join(root, target_name))
            except OSError:
                print("No such file or directory {}".format(origin))
            
            target_path.append(os.path.join(root, target_name))
        
        path = [line.split("\t") for line in lines]
        for line, target_path in zip(path, target_path):
            line[0] = target_path
            
        with open(txt_name, "w+") as file:
            for line in path:
                file.write("\t".join(line))
                file.write("\r\n")

When I run this script I got an IO Error so I add try block to figure out what happened:

No such file or directory ../dataset/floorplan_image/04/94/1b4b5b3a8dcbe2135310d067eb6e/1.jpg
No such file or directory ../dataset/floorplan_image/08/fe/cb1ded96be76713683c50e731f3f/2.jpg
No such file or directory ../dataset/floorplan_image/04/8c/723a3389ae4aaaa11d3d97123e2d/3.jpg
No such file or directory ../dataset/floorplan_image/03/c0/fd5d2bb20c86db0c8f24a83b8a3c/4.jpg
No such file or directory ../dataset/floorplan_image/00/23/500a167ab926df601817631d6733/5.jpg
No such file or directory ../dataset/floorplan_image/01/e5/df6969c71c4cb3b786cde09b4efc/6.jpg
No such file or directory ../dataset/floorplan_image/08/3c/4144463ce4d677723e9032358408/7.jpg
No such file or directory ../dataset/floorplan_image/09/22/b9bd969fff7820dbdbc660b6df26/8.jpg
No such file or directory ../dataset/floorplan_image/02/d4/6ab7b8396d8f24c31ea2e8a0e2f2/9.jpg
No such file or directory ../dataset/floorplan_image/00/99/28fbde7b272adad6da35f0bc16c3/10.jpg

Seem like we loss 10 images. I checked directory and find out that there are ten duplication usages in txt file. These ten images are used in both training and test task.

To solve this problem, we have three choices (I choose solution 2)

1. Training a model use first merged txt file (without change filename)
2. copy these images and rename it by yourself
3. find all code of read image and use try block to catch the error

Finally, we have correct txt file and ready for training. Before training in large dataset, you should check the output of model whether room type can be predicted.

Modify code
If the room type can't be predicted, replaceself.loss_type = 'balanced' in main.py with self.loss_type = ''. More information please check this issue.

Simple version
To summarize, you should

1. download r2v origin data from here
2. download the txt file and put it into ./dataset
   r2v_train.txt
   r2v_test.txt
3. run the script and solve the duplication problem
4. follow "the steps of training and validation in this project"

Those are the steps of training and validation in this project

1. prepare data and txt
2. run ./utils/create_tfrecord.py to create tfrecord of data (r2v or r3d)
3. run ./main.py --phase train (sr2v or r3d)
4. run ./main.py --phase test (r2v or r3d)
5. check result from ./out
6. run ./scores.py for calculate ACC and IoU

Hope helpful.
Please reply if there are any mistake or suggestion.

Jurassic Python

Dear authors,

This is a great piece of code and I thank you for it. My question is: why Python 2.7? The timestamp on your paper shows 2019. Python 3.0 dates back to December 2008, and Python 2 was put on death row 5 years ago, seriously looking like 2020 will be the year of the final execution.

Is this due to some library being tied to Python 2.7?

Are you planning on porting this to 3? If not, could I please try and do it myself? My plan would be to:

1. humbly try to port it;
2. submit it to you for checking;
3. keep mention to your authorship at all times.

Again, many thanks.
Ricardo

I can not find any train-input-img in r2v

obtain the raw image

Hi,

For both these two link below:

R2V: https://github.com/art-programmer/FloorplanTransformation.git
R3D: http://www.cs.toronto.edu/~fidler/projects/rent3D.html

I did not get the raw image successfully.

For the first link, the authors said: "We don't have the permission to share the rasterized images, which are from the LIFULL dataset. Here we only share our vector-graphics annotations which might be helpful for other tasks.".

For the second link, the dataset link does not work.

Anyway, Could you please instruct me to obtain your training and testing data? Thank you so much!

How can i get coords of bounding box of closings?

☝

How to get the LIFULL HOME’S dataset?

Hi,

When I browsed the official website of the LIFULL HOME’S dataset, I found that there is only a Japanese application form, so you also filled out this application form to get this dataset?

Thanks,

Zhengda Lu

The blanced entropy loss can't work

I run the training code in r3d dataset, and can't get any room used the trained model.
The EVAL_pretrained file shows as follows:

Model at epoch 68: overall accuracy = 0.6127, mean_acc = 0.2182
epoch 68: 0th label: accuracy = 0.9814
epoch 68: 1th label: accuracy = 0.0
epoch 68: 2th label: accuracy = 0.0
epoch 68: 3th label: accuracy = 0.0
epoch 68: 4th label: accuracy = 0.0
epoch 68: 5th label: accuracy = 0.0
epoch 68: 6th label: accuracy = 0.0
epoch 68: 9th label: accuracy = 0.0
epoch 68: 10th label: accuracy = 0.9828

can you share any idea with the problem??

pretrained network is gone

The download link: https://mycuhk-my.sharepoint.com/:f:/g/personal/1155052510_link_cuhk_edu_hk/EgyJhisy04hNnxKncWl5zksBf9zDKDpMJ7c0V-q53_pxuA?e=P0BjZd and gives an error 404, perhaps someone could reupload it or post a link here?

I also hope to get the pretrained model

I also hope to get the pretrained model

Originally posted by @deisler134 in #18 (comment)

Original image proportions

Hello! If I understand correctly, the model is intended to work on 512x512 images, but what about arbitrary image sizes? If we scale images to 512x512, we will almost always lose the original floor plans proportions, do we not? How is this issue tackled? Even if we resize to 512x512 for inference and then scale to original, the results will probably not be very good. Thank you in advance for all clarifications.

I see in the paper that windows and doors have the same label, are you able to differentiate between the two?

Upgrading python and tensorflow

Dear authors,

it would be nice if you can upgrade python and tensorflow. Thank you!

only r_b type can be classified

I try to use tensorboard and find that room type image is completely black,only room boundary can be classified.I didn't modify the t code of train part.So anyone come to similar results like me ?I check the code and don't know what the problem is.Thank you .

A bug when retraining

Hi, when running python main.py --phase=train
I get the following error:

start training
max_step = 40000, max_ep = 223, num_batch = 179
Start Training!
Traceback (most recent call last):
  File "main.py", line 310, in <module>
    main(FLAGS)
  File "main.py", line 300, in main
    model.train(loader_dict, num_batch)
  File "main.py", line 144, in train
    [loss_value, update_value] = sess.run([loss, optim])	
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 877, in run
    run_metadata_ptr)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1100, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
    run_metadata)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
	 [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_UINT8], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]
	 [[Node: shuffle_batch/_31 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_173_shuffle_batch", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op u'shuffle_batch', defined at:
  File "main.py", line 310, in <module>
    main(FLAGS)
  File "main.py", line 296, in main
    loader_dict, num_batch = data_loader_bd_rm_from_tfrecord(batch_size=1)
  File "/home/jupyter/cloned/DeepFloorplan/net.py", line 25, in data_loader_bd_rm_from_tfrecord
    loader_dict = read_bd_rm_record('../dataset/r3d.tfrecords', batch_size=batch_size, size=512)
  File "./utils/tf_record.py", line 354, in read_bd_rm_record
    batch_size=batch_size, capacity=batch_size*256, num_threads=1, min_after_dequeue=batch_size*32)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 1300, in shuffle_batch
    name=name)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 846, in _shuffle_batch
    dequeued = queue.dequeue_many(batch_size, name=name)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 476, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3480, in queue_dequeue_many_v2
    component_types=component_types, timeout_ms=timeout_ms, name=name)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
    return func(*args, **kwargs)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
    op_def=op_def)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
    self._traceback = tf_stack.extract_stack()

OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
	 [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_UINT8], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]
	 [[Node: shuffle_batch/_31 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_173_shuffle_batch", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

My installments are as defined in to README.md. What can be the problem?

init_with_pretrain_vgg=False

Dear authors,

This is a great piece of code and I thank you for it. My question is:
why init_with_pretrain_vgg=False, if i want to use the pretrain model, how can i get the ckpt?

3Q

Input images missing from the dataset

Hi!

It seems that the input images such as "'../dataset/jp/train/100_input.jpg'" are missing from the "jp" dataset. Is there any way to acquire them? @zlzeng

reproruce acc of R3D in the paper

hi

Great work, I rerun your checkpoint but can not reproduce your results of your paper.

My results of rerunning your checkpoint is

Overall_acc 0.899

background acc 0.984 | closet 0.659 | Bathroom 0.817 |LivingRoom&Kitchen&DiningRoom 0.819 |BedRoom 0.760 |Hall 0.641 | Balcony 0.716 | Wall 0.957 | Door 0.774

Is it a reasonable one.

obtain the R2V raw image

Thank you very much for publishing such an excellent work! I would like to obtain the original R2V JPG format image, thank you!

OutOfRangeError (see above for traceback): RandomShuffleQueue

OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
[[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_UINT8], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]
[[Node: shuffle_batch/_31 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_173_shuffle_batch", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

I only use one dataset R3D,I don't know does it matter?

Dataset link is dead