autoware_tensorrt_rtmdet#

Purpose#

RTMDet is a real-time instance segmentation model which can be used for detecting objects like cars, pedestrians, bicycles, etc. in a scene. This package provides a ROS2 interface for RTMDet using TensorRT.

Inner-workings / Algorithms#

Inputs / Outputs#

Input#

Name	Type	Description
`in/image`	`sensor_msgs/Image`	The input image

Output#

Name	Type	Description
`out/objects`	`tier4_perception_msgs/DetectedObjectsWithFeature`	The detected objects with 2D bounding boxes and scores
`out/mask`	`autoware_internal_msgs/SegmentationMask`	The instance segmentation mask
`out/color_mask`	`sensor_msgs/Image`	The colorized image of instance segmentation mask for visualization
`out/debug_image`	`sensor_msgs/Image`	The image with 2D bounding boxes for visualization

Parameters#

Name	Type	Description	Default	Range
is_publish_debug_image	boolean	If true, debug image which contain bounding boxes is published at runtime.	False	N/A
is_publish_color_mask	boolean	If true, debug image which contain colorful masks is published at runtime.	False	N/A
model_path	string	Path to ONNX file.	$(var data_path)/tensorrt_rtmdet/$(var model_name).onnx	N/A
color_map_path	string	Path to color map file.	$(var data_path)/tensorrt_rtmdet/color_map.csv	N/A
plugin_paths	array	Path to plugin files.	['./build/tensorrt_rtmdet/libtensorrt_rtmdet_plugin.so']	N/A
input_image_mean	array	Mean values for the model.	[103.53, 116.28, 123.675]	N/A
input_image_std_dev	array	Standard deviation values for model.	[57.375, 57.12, 58.395]	N/A
score_threshold	float	A threshold value of existence probability score, all of objects with score less than this threshold are ignored.	0.3	≥0.0 ≤1.0
nms_threshold	float	A threshold value of NMS.	0.0	≥0.0 ≤1.0
mask_threshold	float	A threshold value of pixel values in the output mask.	0.7	≥0.0 ≤1.0
precision	string	Operation precision to be used on inference. Valid value is one of: [fp32, fp16, int8].	fp16	N/A
dla_core_id	float	If positive ID value is specified, the node assign inference task to the DLA core.	-1	N/A
quantize_first_layer	boolean	If true, set the operating precision for the first (input) layer to be fp16. This option is valid only when precision==int8.	False	N/A
quantize_last_layer	boolean	If true, set the operating precision for the last (output) layer to be fp16. This option is valid only when precision==int8.	False	N/A
profile_per_layer	boolean	If true, profiler function will be enabled. Since the profile function may affect execution speed, it is recommended to set this flag true only for development purpose.	False	N/A
clip_value	float	If positive value is specified, the value of each layer output will be clipped between [0.0, clip_value]. This option is valid only when precision==int8 and used to manually specify the dynamic range instead of using any calibration.	0.0	N/A
preprocess_on_gpu	boolean	If true, pre-processing is performed on GPU.	True	N/A
apply_erosion	boolean	If true, apply erosion to the output mask.	False	N/A
calibration_image_list_path	string	Path to a file which contains path to images. Those images will be used for int8 quantization.		N/A

Assumptions / Known limits#

Onnx model#

A sample model is provided in autoware_data folder by ansible script on env preparation stage. If you cannot find the model, you follow instructions from the link to download the model.

Package acceptable model generation#

Users can generate their own model using official RTMDet repository. Please refer to the official repository

Label file#

A sample label file (named label.txt) and instance segmentation color map file (name color_map.csv) are also downloaded automatically during env preparation process.

These files are used to map the class index to class name and color respectively.

Reference repositories#

https://github.com/open-mmlab/mmdetection/tree/3.x/configs/rtmdet#rtmdet-an-empirical-study-of-designing-real-time-object-detectors

Citation#

@misc{lyu2022rtmdetempiricalstudydesigning,
title={RTMDet: An Empirical Study of Designing Real-Time Object Detectors},
author={Chengqi Lyu and Wenwei Zhang and Haian Huang and Yue Zhou and Yudong Wang and Yanyi Liu and Shilong Zhang and Kai Chen},
year={2022},
eprint={2212.07784},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2212.07784},
}