Skip to content

Commit 6e05ff4

Browse files
amlankaramlankar
committed
Create README.md
0 parents  commit 6e05ff4

File tree

4 files changed

+174
-0
lines changed

4 files changed

+174
-0
lines changed

‎Docs/model.png‎

250 KB
Loading

‎Docs/polydemo.gif‎

600 KB
Loading

‎Docs/tool.png‎

187 KB
Loading

‎README.md‎

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
# PolygonRNN++
2+
3+
This is the official PyTorch reimplementation of Polygon-RNN++ (CVPR 2018). This repository allows you to train new Polygon-RNN++ models, and run our demo tool on local machines. For technical details, please refer to:
4+
5+
**Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++**
6+
[David Acuna](http://www.cs.toronto.edu/~davidj/)\*, [Huan Ling](http:///www.cs.toronto.edu/~linghuan/)\*, [Amlan Kar](http://www.cs.toronto.edu/~amlan/)\*, [Sanja Fidler](http://www.cs.toronto.edu/~fidler/) (\* denotes equal contribution)
7+
CVPR 2018
8+
**[[Paper](https://arxiv.org/abs/1803.09693)] [[Video](https://www.youtube.com/watch?v=evGqMnL4P3E)] [[Project Page](http://www.cs.toronto.edu/polyrnn/)] [[Demo](https://goo.gl/forms/aWEHgFTyWfeA5oUt1)]**
9+
<img src = "Docs/model.png" width="56%"/>
10+
<img src = "Docs/polydemo.gif" width="42%"/>
11+
12+
# Where is the code?
13+
To get the code, please [signup](http://www.cs.toronto.edu/polyrnn/code_signup/) here. We will be using GitHub to keep track of issues with the code and to update on availability of newer versions (also available on website and through e-mail to signed up users).
14+
15+
If you use this code, please cite:
16+
17+
@inproceedings{AcunaCVPR18,
18+
title={Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++},
19+
author={David Acuna and Huan Ling and Amlan Kar and Sanja Fidler},
20+
booktitle={CVPR},
21+
year={2018}
22+
}
23+
24+
@inproceedings{CastrejonCVPR17,
25+
title = {Annotating Object Instances with a Polygon-RNN},
26+
author = {Lluis Castrejon and Kaustav Kundu and Raquel Urtasun and Sanja Fidler},
27+
booktitle = {CVPR},
28+
year = {2017}
29+
}
30+
31+
# Contents
32+
1. [Reproduction Results](#results)
33+
2. [Environment Setup](#environment-setup)
34+
3. [Tool](#tool)
35+
1. [Backend](#backend)
36+
2. [Frontend](#frontend)
37+
4. [Testing Models](#testing-models)
38+
5. [Training Models](#training-models)
39+
1. [Data](#data)
40+
2. [Training MLE Model](#training-mle-model)
41+
3. [Training RL Model](#training-rl-model)
42+
4. [Training Evaluator](#training-evaluator)
43+
5. [Training GGNN](#training-ggnn)
44+
45+
# Results
46+
These are the reproduction results from this repository as compared to the paper
47+
48+
| Training Type | Num first points | LSTM Beam Size | Before | Now |
49+
|:-------------:|:----------------:|:--------------:|:------:|:-----:|
50+
| MLE + Att | 1 | 1 | 65.43 | 66.35 |
51+
| MLE + Att + RL | 1 | 1 | 67.17 | 67.45 |
52+
| MLE + Att + Evaluator | 5 | 1 | 69.72 | 71.05 |
53+
| MLE + Att + Evaluator | 5 | 8 | 70.21 | 70.91 |
54+
| MLE + Att + Evaluator + GGNN | 5 | 8 | 71.38 | 72.05 |
55+
| MLE + Att + Evaluator + GGNN | 5 | 1 | - | 72.08 |
56+
| MLE + Att + Evaluator + GGNN (Shared Encoder) | 5 | 8 | - | 72.22 |
57+
| MLE + Att + Evaluator + GGNN (Shared Encoder) | 5 | 1 | - | **72.33** |
58+
59+
**Note:** Benchmarked forward pass speed for the tool (with 5 first points and 1 beam size) is 0.3 seconds per interaction on a TitanXp
60+
61+
**Note:** Shared Encoder refers to sharing the Resnet between the graph network and the convLSTM network. In the original paper, the two networks were kept separate.
62+
63+
# Environment Setup
64+
All the code has been run and tested on Ubuntu 16.04, Python 2.7.12, Pytorch 0.4.0, CUDA 9.0, TITAN X/Xp and GTX 1080Ti GPUs
65+
66+
- Get code after [signing up](http://www.cs.toronto.edu/polyrnn/code_signup/)
67+
- Go into the downloaded code directory
68+
```
69+
cd <path_to_downloaded_directory>
70+
```
71+
- Setup python environment
72+
```
73+
virtualenv env
74+
source env/bin/activate
75+
pip install -r requirements.txt
76+
```
77+
- Add the project to PYTHONPATH
78+
```
79+
export PYTHONPATH=$PWD
80+
```
81+
82+
# Tool
83+
- [Setup](#environment-setup) your environment
84+
- Download the MLE+RL+Evaluator+GGNN model after getting access by [signing up](http://www.cs.toronto.edu/polyrnn/code_signup/)
85+
86+
## Backend
87+
- Launch backend (flask server) with,
88+
```
89+
python Tool/tool.py --exp Experiments/tool.json --reload <path_to_model> --port <port> --image_dir Tool/frontend/static/img/
90+
```
91+
92+
## Frontend
93+
- Edit Tool/frontend/static/js/polygon.js and change globalFolder to the appropriate
94+
directory based on where you cloned the repository.
95+
- With python2.7, run
96+
```
97+
cd Tool/frontend/
98+
python -m SimpleHTTPServer
99+
```
100+
- On your browser, navigate to localhost:8000. You should see a page like
101+
<img src = "Docs/tool.png" width="100%"/>
102+
103+
**Note:** Replace SimpleHTTPServer with http.server if you are using python3 for the server
104+
105+
**Note:** You can setup your own image directory by editing Tool/frontend/static/js/polygon.js and passing that path to Tool/tool.py
106+
from the command line. This image directory MUST contain the pre-defined images that are defined in Tool/frontend/index.html
107+
108+
# Testing Models
109+
- [Setup](#environment-setup) your environment
110+
- Download pretrained models after getting access by [signing up](http://www.cs.toronto.edu/polyrnn/code_signup/)
111+
112+
```
113+
python Scripts/prediction/generate_annotation.py --exp <path_to_corresponding_experiment> --reload <path_to_checkpoint> --output_dir <path_to_store_predictions>
114+
```
115+
- You can check predicted/GT masks for every instance in the output_dir
116+
- To get scores, run
117+
```
118+
python Scripts/get_scores.py --pred <path_to_preds> --output <path_to_file_to_save_results>
119+
```
120+
121+
# Training Models
122+
123+
## Data
124+
125+
### Cityscapes
126+
- Download the Cityscapes dataset (leftImg8bit\_trainvaltest.zip) from the official [website](https://www.cityscapes-dataset.com/downloads/) [11 GB]
127+
- Download our processed annotation files from [here](http://www.cs.toronto.edu/~amlan/data/polygon/cityscapes.tar.gz) [68 MB]
128+
- From the root directory, run the following command with appropriate paths to get the annotation files ready for your machine
129+
```
130+
python Scripts/data/change_paths.py --city_dir <path_to_downloaded_leftImg8bit_folder> --json_dir <path_to_downloaded_annotation_file> --output_dir <output_dir>
131+
```
132+
133+
### Custom Dataset
134+
To train on your custom datasets, you have one of two options:
135+
- Prepare annotation data similar to our annotation files and use our default DataProvider
136+
- Implement your own DataProvider following the cityscapes implementation for your own data
137+
138+
## Training
139+
- [Setup](#environment-setup) your environment
140+
- Download the pre-trained Pytorch Resnet-50 from [here](https://download.pytorch.org/models/resnet50-19c8e357.pth)
141+
- **Note** - While resuming training, always resume from end of epoch checkpoints to produce reproducible results!
142+
143+
### Training MLE model
144+
- Edit the experiment file at [Experiments/mle.json](Experiments/mle.json) and change paths for your machine
145+
- From the root directory, run
146+
```
147+
python Scripts/train/train_ce.py --exp Experiments/mle.json --reload <optional_if_resuming_training>
148+
```
149+
- You can view progress on Tensorboard (logs are at <experiment\_dir>/logs/)
150+
151+
### Training RL model
152+
- Edit the experiment file at [Experiments/rl.json](Experiments/rl.json) and change paths for your machine
153+
- In the experiment file, set xe\_initializer to the best MLE model
154+
- From the root directory, run
155+
```
156+
python Scripts/train/train_rl.py --exp Experiments/mle.json --reload <optional_if_resuming_training>
157+
```
158+
- **Note** - You might have to play with hyperparameters a bit to achieve stable training, especially temperature, lr and lr\_decay
159+
160+
### Training Evaluator
161+
- Edit the experiment file at [Experiments/evaluator.json](Experiments/evaluator.json) and change paths for your machine
162+
- In the experiment file, set xe\_initializer to the best RL model
163+
- From the root directory, run
164+
```
165+
python Scripts/train/train_evaluator.py --exp Experiments/evaluator.json --reload <optional_if_resuming_training>
166+
```
167+
168+
### Training GGNN
169+
- Edit the experiment file at [Experiments/ggnn.json](Experiments/ggnn.json) and change paths for your machine
170+
- In the experiment file, set xe\_initializer to the best Evaluator model
171+
- From the root directory, run
172+
```
173+
python Scripts/train/train_ggnn.py --exp Experiments/ggnn.json --reload <optional_if_resuming_training>
174+
```

0 commit comments

Comments
 (0)