BEVFusion¶

BEVFusion is a camera-LiDAR 3D object detection model integrated under the detection3d task namespace. It combines a sparse-voxel LiDAR branch (hard voxelization + sparse 3D convolution encoder), a multiview image branch, a LiDAR-depth-guided DepthLSSTransform view transform, and a convolutional BEV fusion layer before the TransFusion detection head. The bundled experiments are LiDAR-only: they leave camera_network unset and skip the image branch entirely.

Summary¶

Available Configurations¶

Experiment configs live under autoware_ml/configs/tasks/detection3d/bevfusion/<dataset>/ and are passed to the CLI as the path relative to tasks/. Every bundled leaf is LiDAR-only and sits under a bevfusion_lidar/ directory. Camera-LiDAR experiments will add a sibling directory that configures the camera branch and fuser.

The leaf name spells out its variant: the voxel size, the backbone and neck, the batch size, the epoch count and the detection range, so the values live in one place, the config. The configs are layered so that each file owns one concern:

For T4Dataset the bundled configs stop at the bases t4dataset/base_120m.yaml and t4dataset/bevfusion_lidar/base_lidar_voxel0170_120m.yaml. They declare the classes the model trains and require the taxonomy that folds the labels of the database onto them, so a T4Dataset experiment lives in an external config root, see Configuration.

Training¶

The nuScenes config trains from scratch on the nuscenes/nuscenes_base database:

autoware-ml train --config-name detection3d/bevfusion/nuscenes/bevfusion_lidar/lidar_voxel0075_second_secfpn_b16_30e_nuscenes_54m

For a pipeline validation run:

autoware-ml train \
    --config-name detection3d/bevfusion/nuscenes/bevfusion_lidar/lidar_voxel0075_second_secfpn_b16_30e_nuscenes_54m \
    +trainer.fast_dev_run=true

Checkpoints are written to mlruns/<experiment_group_name>/<experiment_name>/<run_id>/artifacts/checkpoints/ as best.ckpt and last.ckpt. best.ckpt is the validation with the highest val/det3d/mAP, and early stopping watches the same metric. Resume an interrupted run with --resume-checkpoint pointing at last.ckpt.

Evaluation¶

autoware-ml test \
    --config-name detection3d/bevfusion/nuscenes/bevfusion_lidar/lidar_voxel0075_second_secfpn_b16_30e_nuscenes_54m \
    --weights mlruns/bevfusion/NuScenesDataset-base-v1.0.0/nuscenes/lidar_voxel0075_second_secfpn_b16_30e_nuscenes_54m/<run_id>/artifacts/checkpoints/best.ckpt

Evaluation runs on a single device by default so metrics are deterministic. Pass --use-config-devices to keep the trainer.devices from the config.

Deployment¶

autoware-ml deploy \
    --config-name detection3d/bevfusion/nuscenes/bevfusion_lidar/lidar_voxel0075_second_secfpn_b16_30e_nuscenes_54m \
    --weights mlruns/bevfusion/NuScenesDataset-base-v1.0.0/nuscenes/lidar_voxel0075_second_secfpn_b16_30e_nuscenes_54m/<run_id>/artifacts/checkpoints/best.ckpt \
    --release v1.0.0

The export produces the ONNX modules for autoware_universe/perception/autoware_bevfusion. LiDAR-only configurations export a single bevfusion_lidar.onnx main body restricted to the first three inputs below, and the runtime node consumes it unchanged. Camera-LiDAR models export two modules: bevfusion_image_backbone.onnx encodes raw uint8 multiview images (the training-time 1 / 255 normalization is baked into the graph) into image_feats, and bevfusion_camera_lidar.onnx is the main body consuming those features together with the LiDAR depth maps and precomputed bev_pool metadata. The camera-LiDAR main body is not consumed by the current runtime node: its inputs changed and the node has not been updated yet, see Runtime compatibility.

Every main-body module returns the runtime detection interface: bbox_pred with the raw regression channels (center, height, dim, rot, vel) per proposal, score with per-proposal confidences, and label_pred with per-proposal class labels. Metric-space decoding happens in the runtime node. The height channel is the z of the box gravity center, which is how autoware_bevfusion reads it.

Exports land in mlruns/<experiment_group_name>/<experiment_name>/<run_id>/artifacts/exports/ of a dedicated deploy run linked to the training run. Every module is stamped with its provenance and the --release version. Omit --release only for throwaway exports, which are stamped unversioned. TensorRT engine generation is disabled (deploy.tensorrt.enabled=false) because the runtime builds engines itself using the custom sparse-convolution and bev_pool plugins.

Runtime compatibility¶

Breaking change: camera-LiDAR main body inputs¶

The bevfusion_camera_lidar.onnx main body no longer takes points, lidar2image and img_aug_matrix. It takes depth_maps instead. Previously the graph projected the point cloud onto every camera itself. Now the sparse depth maps are built outside the graph by LiDARDepthSparseTransform (autoware_ml/transforms/camera_lidar/depth.py), both in the training pipeline and in the export sample. The kept input is now sized by num_frustum_points rather than num_kept.

autoware_bevfusion (lib/bevfusion_trt.cpp) still binds points, img_aug_matrix and lidar2image in fusion mode and does not build depth maps, so a camera-LiDAR model exported from this version does not load in the current node, and the camera-LiDAR ONNX published for the current node does not match this export. LiDAR-only exports are unaffected. No shipped experiment runs the camera branch yet, so nothing deployed is broken today, but the runtime must change before the first camera-LiDAR model from this repository ships.

A second behaviour change, outside the export ABI: a camera-only model (lidar_network: null) now feeds the image BEV straight into the detection head. The BEV backbone and neck (pts_backbone / pts_neck) live on the LiDAR branch and do not run without it, unlike upstream BEVFusion where every BEV map goes through them. A camera-only head therefore has to take the view-transform channels as in_channels. There is no camera-only export, so the runtime is not affected.

Planned runtime change (autoware_bevfusion)¶

The node has to build the depth maps that the graph used to build, with the same geometry it already uses for lidar2image and img_aug_matrix.

1. Depth map kernel. Add a CUDA kernel in lib/preprocess/preprocess_kernel.cu that projects the sweep points (points_d_, the same buffer fed to voxelization) onto every camera and writes a (num_cameras, 1, roi_height, roi_width) float32 buffer, reproducing LiDARDepthSparseTransform.build_depth_maps.
2. Bindings. In addCameraNetworkIO, addCameraProfileDims, setTensorAddress and configureTensorRTInputs, replace points, img_aug_matrix and lidar2image with depth_maps (Dims{4, {num_cameras, 1, roi_height, roi_width}}, min = opt = max). Drop points_d_ from the engine inputs (it stays as the voxelization buffer) and img_aug_matrix_d_ from device memory, and keep the host-side img_aug_matrices_ and the lidar2image values for the kernel. Bindings are by name, so the new input order does not matter.
3. Pipeline. Allocate depth_maps_d_ (num_cameras * roi_height * roi_width floats) in fusion mode, clear it in clearDeviceMemory, and launch the kernel in preProcess after generateSweepPoints and before configureTensorRTInputs. No new ROS parameters are needed: num_cameras, roi_height and roi_width already exist in ml_package_bevfusion_camera_lidar.param.yaml.
4. ABI transition. Either gate the fusion bindings on the input names of the loaded network (points present means the old graph, depth_maps the new one) for one release, or bump the camera-LiDAR model package version and switch outright. The node README, the parameter schema and CHANGELOG.rst must document the new input set.
5. Validation. Export a camera-LiDAR model from this repository, dump the depth_maps the export sample used and compare them with the kernel output for the same frame and calibration (equal everywhere except duplicate-hit pixels), then compare bbox_pred / score / label_pred between the PyTorch model and the TensorRT engine on a few frames.

Implementation¶

Acknowledgment¶

The Autoware-ML BEVFusion implementation was ported from the official mmdetection3d project by OpenMMLab.

• Repository: https://github.com/open-mmlab/mmdetection3d
• License: Apache License 2.0
• Paper: Liu, Zhijian, et al. "BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation" ICRA, 2023.