Skip to content

TransFusion

TransFusion is a LiDAR-based 3D object detection model integrated under the detection3d task namespace. It uses a sparse-voxel frontend (hard voxelization + sparse 3D convolution encoder) with a SECOND backbone, SECONDFPN neck, and native TransFusion detection head.

Summary

Property Value
Task 3D object detection
Modality LiDAR
Input Point cloud
Output 3D bounding boxes and class scores
Architecture Sparse voxel encoder + SECOND + SECONDFPN + TransFusion head
Datasets T4Dataset, nuScenes

Available Configurations

The LiDAR-only BEVFusion experiments run the same sparse-voxel encoder, SECOND backbone, SECONDFPN neck, and TransFusion head stack. The TransFusion experiments below train that stack without the BEVFusion wrapper.

Experiment configs live under autoware_ml/configs/tasks/detection3d/transfusion/ and follow the CenterPoint layout: a base.yaml with the model and export modules, a <dataset>/base_<range>.yaml with the range-dependent settings and transforms, a voxel-level base, and one leaf per database. The bundled leaf is detection3d/transfusion/nuscenes/voxel0075_second_secfpn_b16_30e_nuscenes_54m.

For T4Dataset the bundled configs stop at the bases t4dataset/base_120m.yaml and t4dataset/base_voxel0170_120m.yaml. They declare the classes the model trains and require the taxonomy that folds the labels of the database onto them, so a T4Dataset experiment lives in an external config root, see Configuration.

Training

An experiment is trained through the standard entrypoint, with --weights supplying a pretrained checkpoint when the config declares weights: ???:

autoware-ml train --config-name detection3d/transfusion/nuscenes/voxel0075_second_secfpn_b16_30e_nuscenes_54m

Checkpoints are written to mlruns/<experiment_group_name>/<experiment_name>/<run_id>/artifacts/checkpoints/ as best.ckpt and last.ckpt.

Evaluation

autoware-ml test \
    --config-name detection3d/transfusion/nuscenes/voxel0075_second_secfpn_b16_30e_nuscenes_54m \
    --weights mlruns/transfusion/NuScenesDataset-base-v1.0.0/nuscenes/voxel0075_second_secfpn_b16_30e_nuscenes_54m/<run_id>/artifacts/checkpoints/best.ckpt

Evaluation runs on a single device by default so metrics are deterministic. Pass --use-config-devices to keep the trainer.devices from the config.

Deployment

autoware-ml deploy \
    --config-name detection3d/transfusion/nuscenes/voxel0075_second_secfpn_b16_30e_nuscenes_54m \
    --weights mlruns/transfusion/NuScenesDataset-base-v1.0.0/nuscenes/voxel0075_second_secfpn_b16_30e_nuscenes_54m/<run_id>/artifacts/checkpoints/best.ckpt \
    --release v1.0.0

The model exports one ONNX main body wrapping the voxel encoder, sparse encoder, backbone, neck, and head.

Tensor Direction Description
voxels Input Voxelized LiDAR features
num_points Input Point count per voxel
coors Input Voxel coordinates with the batch index
cls_score0 Output Per-proposal class scores, the sigmoid of the class heatmap times the query heatmap score
bbox_pred0 Output Per-proposal raw regression channels (center, height, dim, vel)
dir_cls_pred0 Output Per-proposal yaw regression as (sin, cos), decoded with atan2

The height channel of bbox_pred0 is the z of the box gravity center, which is how autoware_lidar_transfusion reads it.

Exports land in mlruns/<experiment_group_name>/<experiment_name>/<run_id>/artifacts/exports/ of a dedicated deploy run linked to the training run. Every module is stamped with its provenance and the --release version. Omit --release only for throwaway exports, which are stamped unversioned. The current verification scope covers ONNX export. TensorRT engine generation has not been validated yet.

Implementation

Path Description
autoware_ml/models/detection3d/main_modules/transfusion.py TransFusion model wrapper
autoware_ml/models/detection3d/encoders/voxel.py Hard voxelization feature encoder
autoware_ml/models/detection3d/encoders/sparse/sparse_encoder.py Sparse 3D convolution encoder
autoware_ml/models/detection3d/backbones/second.py SECOND backbone
autoware_ml/models/detection3d/necks/second_fpn.py SECONDFPN neck
autoware_ml/models/detection3d/heads/transfusions/transfusion_head.py TransFusion detection head
autoware_ml/models/detection3d/task_modules/ Shared assigners, costs, coders
autoware_ml/datamodule/t4dataset/dataset.py T4Dataset records as a dataset
autoware_ml/datamodule/nuscenes/dataset.py nuScenes dataset
autoware_ml/datamodule/common/detection3d.py 3D boxes of a record
autoware_ml/configs/datamodule/ Datamodule, dataset and transform configurations
autoware_ml/preprocessing/detection3d/point_pillar_preprocessor.py Pillar preprocessing
autoware_ml/configs/tasks/detection3d/transfusion/ Experiment configurations

Acknowledgment

The Autoware-ML TransFusion implementation was ported from the official mmdetection3d project by OpenMMLab.