charlie880624/STDC
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
STDC/docs_kneron/stdc_step_by_step.md
chingning.chen 8f8f145abb doc: stdc_step_by_step
2022-03-31 17:35:11 +08:00

14 KiB
Raw Blame History

Step 0: Environment

Prerequisites

Note: You need to run pip uninstall mmcv first if you have mmcv installed. If mmcv and mmcv-full are both installed, there will be ModuleNotFoundError.

Install MMSegmentationKN

  1. Install PyTorch. You can follow Official PyTorch Installation Instruction to install PyTorch using Conda:

    conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch -y

  2. Install mmcv-full. We recommend you installing mmcv-full using pip:

    pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11.0/index.html

Please replace cu113 and torch1.11.0 in the url to your desired one. For example, to install the mmcv-full with CUDA 11.1 and PyTorch 1.9.0, use the following command:

```shell
pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.9.0/index.html
```

If you see error messages while installing mmcv-full, please check if your installation instruction matches your installed version of PyTorch and Cuda, and see MMCV pip Installation Instruction for different versions of MMCV compatible to different PyTorch and CUDA versions.

  1. Clone the MMSegmentationKN repository.

    git clone https://github.com/kneron/MMSegmentationKN.git
cd MMSegmentationKN

  2. Install required python packages for building and installing MMSegmentationKN.

    pip install -r requirements/optional.txt  # only if you want to train / evaluate on CityScapes dataset
pip install -r requirements/runtime.txt
pip install -v -e .  # or "python setup.py develop"

Step 1: Training Models on Standard Datasets

MMSegmentationKN provides many existing and existing semantic segmentation models in Model Zoo, and supports several standard datasets like CityScapes, Pascal Context, Coco Stuff, ADE20K, etc. Here we demonstrate how to train STDC-Seg, a semantic segmentation algorithm, on CityScapes, a well-known semantic segmentation dataset.

  1. Download CityScapes Dataset

Go to CityScapes Official Website and click Download link on the top of the page. If you're not logged in, it will navigate you to login page.

If it is the first time visiting CityScapes website, to download CityScapes dataset, you have to register an account.

Click the Register link and it will navigate you to the registeration page. Fill in all the required fields, accept the terms and conditions, and click the Register button. If everything goes well, you will see Registration Successful on the page and recieve a registration confirmation mail in your email inbox. Click on the link provided in the confirmation mail, login with your newly registered account and password, and you should be able to download the CityScapes dataset.

Download leftImg8bit_trainvaltest.zip (images) and gtFine_trainvaltest.zip (labels) and place them onto your server.

  1. Dataset Preparation

We suggest that you extract the zipped files to somewhere outside the project directory and symlink (ln) the dataset root to MMSegmentationKN/data so you can use the dataset outside this project, as shown below:

# Replace all "path/to/your" below with where you want to put the dataset!

# Extracting Cityscapes
mkdir -p path/to/your/cityscapes
unzip leftImg8bit_trainvaltest.zip -d path/to/your/cityscapes
unzip gtFine_trainvaltest.zip -d path/to/your/cityscapes

# symlink dataset to MMSegmentationKN/data  # where "MMSegmentationKN" is the repository you cloned in step 0-4
mkdir -p MMSegmentationKN/data
ln -s $(realpath path/to/your/cityscapes) MMSegmentationKN/data

# Replace all "path/to/your" above with where you want to put the dataset!

Then, we need cityscapesScripts to preprocess the CityScapes dataset. If you followed our Step 0, you should have python package cityscapesScripts installed (if no, execute pip install cityscapesScripts command).

# Replace "path/to/your" with where you want to put the dataset!
export CITYSCAPES_DATASET=$(realpath path/to/your/cityscapes)
csCreateTrainIdLabelImgs

Wait several minutes and you'll see something like this:

Processing 5000 annotation files
Progress: 100.0 %

The files inside the dataset folder should be something like:

MMSegmentationKN/data/cityscapes
├── gtFine
│   ├── test
│   │   ├── ...
│   ├── train
│   │   ├── ...
│   ├── val
│   │   ├── frankfurt
│   │   │   ├── frankfurt_000000_000294_gtFine_color.png
│   │   │   ├── frankfurt_000000_000294_gtFine_instanceIds.png
│   │   │   ├── frankfurt_000000_000294_gtFine_labelIds.png
│   │   │   ├── frankfurt_000000_000294_gtFine_labelTrainIds.png
│   │   │   ├── frankfurt_000000_000294_gtFine_polygons.png
│   │   │   ├── ...
│   │   ├── ...
├── leftImg8bit
│   ├── test
│   │   ├── ...
│   ├── train
│   │   ├── ...
│   ├── val
│   │   ├── frankfurt
│   │   │   ├── frankfurt_000000_000294_leftImg8bit.png
│   │   ├── ...
...

It's recommended that you symlink the dataset folder to mmdetection folder. However, if you place your dataset folder at different place and do not want to symlink, you have to change the corresponding paths in the config file.

Now the dataset should be ready for training.

  1. Train STDC-Seg on CityScapes

Short-Term Dense Concatenate Network (STDC network) is a light-weight network structure for convolutional neural network. If we apply this network structure to semantic segmentation task, it's called STDC-Seg. It's first introduced in Rethinking BiSeNet For Real-time Semantic Segmentation .

We only need a configuration file to train a deep learning model in either the original MMSegmentation or MMSegmentationKN. STDC-Seg is provided in the original MMSegmentation repository, but the original configuration file needs some modification due to our hardware limitation so we can apply the trained model to our Kneron dongle.

To make a configuration file compatible with our device, we have to:

Change the mean and std value in image normalization to mean=[128., 128., 128.] and std=[256., 256., 256.].

Shrink the input size during inference phase. The original CityScapes image size is too large (2048(w)x1024(h)) for our device; 1024(w)x512(h) might be good for our device.

To achieve this, you can modify the img_scale in test_pipeline and img_norm_cfg in the configuration file configs/_base_/datasets/cityscapes.py. However, here in MMSegmentationKN, we provide a modified STDC-Seg configuration file (configs/stdc/kn_stdc1_in1k-pre_512x1024_80k_cityscapes.py) so we can easily apply the trained model to our device.

To train STDC-Seg compatible with our device, just execute:

cd MMSegmentationKN
python tools/train.py configs/stdc/kn_stdc1_in1k-pre_512x1024_80k_cityscapes.py

And MMSegmentationKN will generate work_dirs/kn_stdc1_in1k-pre_512x1024_80k_cityscapes folder and save the configuration file and all checkpoints there.

Step 2: Test Trained Model

tools/test.py is a script that generates inference results from test set with our pytorch model and evaluates the results to see if our pytorch model is well trained (if --eval argument is given). Note that it's always good to evluate our pytorch model before deploying it.

python tools/test.py \
    work_dirs/kn_stdc1_in1k-pre_512x1024_80k_cityscapes/kn_stdc1_in1k-pre_512x1024_80k_cityscapes.py \
    work_dirs/kn_stdc1_in1k-pre_512x1024_80k_cityscapes/latest.pth \
    --eval mIoU
  • kn_stdc1_in1k-pre_512x1024_80k_cityscapes/kn_stdc1_in1k-pre_512x1024_80k_cityscapes.py can be your training config.
  • kn_stdc1_in1k-pre_512x1024_80k_cityscapes/latest.pth can be your model checkpoint.

The expected result of the command above should be something similar to the following text (the numbers may slightly differ):

...
+---------------+-------+-------+
|     Class     |  IoU  |  Acc  |
+---------------+-------+-------+
|      road     | 97.49 | 98.59 |
|    sidewalk   | 80.17 | 88.71 |
|    building   | 89.52 | 95.25 |
|      wall     | 57.92 | 66.99 |
|     fence     |  55.5 | 70.15 |
|      pole     | 38.93 | 47.51 |
| traffic light | 49.95 | 59.97 |
|  traffic sign |  62.1 | 70.05 |
|   vegetation  | 89.02 | 95.27 |
|    terrain    | 60.18 | 72.26 |
|      sky      | 91.84 | 96.34 |
|     person    | 68.98 | 84.35 |
|     rider     | 47.79 | 60.98 |
|      car      | 91.63 | 96.48 |
|     truck     | 74.31 | 83.52 |
|      bus      | 80.24 | 86.83 |
|     train     | 66.45 | 76.78 |
|   motorcycle  | 48.69 | 58.18 |
|    bicycle    | 65.81 | 81.68 |
+---------------+-------+-------+
Summary:

+------+-------+-------+
| aAcc |  mIoU |  mAcc |
+------+-------+-------+
| 94.3 | 69.29 | 78.42 |
+------+-------+-------+

Step 3: Export ONNX

tools/deployment/pytorch2onnx.py is a script provided by MMSegmentationKN to help users to convert our trained pytorch model to ONNX:

python tools/pytorch2onnx_kneron.py \
    work_dirs/kn_stdc1_in1k-pre_512x1024_80k_cityscapes/kn_stdc1_in1k-pre_512x1024_80k_cityscapes.py \
    --checkpoint work_dirs/kn_stdc1_in1k-pre_512x1024_80k_cityscapes/latest.pth \
    --output-file work_dirs/kn_stdc1_in1k-pre_512x1024_80k_cityscapes/latest.onnx
  • kn_stdc1_in1k-pre_512x1024_80k_cityscapes/kn_stdc1_in1k-pre_512x1024_80k_cityscapes.py can be your training config.
  • kn_stdc1_in1k-pre_512x1024_80k_cityscapes/latest.pth can be your model checkpoint.
  • kn_stdc1_in1k-pre_512x1024_80k_cityscapes/latest.onnx can be any other path. Here for convenience, the ONNX file is placed in the same folder of our pytorch checkpoint.

Step 4: Convert ONNX File to NEF Model for Kneron Platform

Step 4-1: Install Kneron toolchain docker:

Step 4-2: Mout Kneron toolchain docker

  • Mount a folder (e.g. '/mnt/hgfs/Competition') to toolchain docker container as /data1. The converted ONNX in Step 3 should be put here. All the toolchain operation should happen in this folder.
sudo docker run --rm -it -v /mnt/hgfs/Competition:/data1 kneron/toolchain:latest

Step 4-3: Import KTC and required lib in python shell

  • Here we demonstrate how to go through all Kneron Toolchain (KTC) flow through Python API:
import ktc
import numpy as np
import os
import onnx
from PIL import Image

Step 4-4: Optimize the onnx model

onnx_path = '/data1/latest.onnx'
m = onnx.load(onnx_path)
m = ktc.onnx_optimizer.onnx2onnx_flow(m)
onnx.save(m,'latest.opt.onnx')

Step 4-5: Configure and load data necessary for ktc, and check if onnx is ok for toolchain

# npu (only) performance simulation
km = ktc.ModelConfig((&)model_id_on_public_field, "0001", "720", onnx_model=m)
eval_result = km.evaluate()
print("\nNpu performance evaluation result:\n" + str(eval_result))

Step 4-6: quantize the onnx model

We sampled 3 images from Cityscapes dataset (3 images) as quantization data. To test our quantized model:

  1. Download the zip file
  2. Extract the zip file as a folder named cityscapes_minitest
  3. Put the cityscapes_minitest into docker mounted folder (the path in docker container should be /data1/cityscapes_minitest)

The following script will do some preprocess(should be the same as training code) on our quantization data, and put it in a list:

import os
from os import walk

img_list = []
for (dirpath, dirnames, filenames) in walk("/data1/cityscapes_minitest"):
    for f in filenames:
        fullpath = os.path.join(dirpath, f)
        
        image = Image.open(fullpath)
        image = image.convert("RGB")
        image = Image.fromarray(np.array(image)[...,::-1])
        img_data = np.array(image.resize((640, 640), Image.BILINEAR)) / 256 - 0.5
        print(fullpath)
        img_list.append(img_data)

Then perform quantization. The BIE model will be generated at /data1/output.bie.

# fixed-point analysis
bie_model_path = km.analysis({"input": img_list})
print("\nFixed-point analysis done. Save bie model to '" + str(bie_model_path) + "'")

Step 4-7: Compile

The final step is compile the BIE model into an NEF model.

# compile
nef_model_path = ktc.compile([km])
print("\nCompile done. Save Nef file to '" + str(nef_model_path) + "'")

You can find the NEF file at /data1/batch_compile/models_720.nef. models_720.nef is the final compiled model.

Step 5: Run NEF model on KL720