PointTransformerV3¶

PointTransformerV3 is a LiDAR encoder for LiDAR 3D semantic segmentation and 3D object detection. The shared component is the hierarchical PTv3 encoder (PointTransformerV3Encoder); each task owns its head:

• Segmentation: PTv3Seg3DDecoderHead unpools the deepest encoder stage back to full resolution through the encoder skip chain and classifies each point.
• Detection: PTv3Det3DBEVNeck fuses the two coarsest encoder stages (PTv3Det3DFeatureFusion), scatters them onto a dense BEV canvas at the detection-head resolution, and feeds a TransFusionHead.

The models read single lidar frames, every point as (x, y, z, intensity). The voxel feature encoder turns each voxel into the mean x, y, z and intensity of its points, 4 features per voxel, and the detection heads predict no velocity. The 6 cm teacher reads the current frame and one past sweep, every point with a time lag. Its voxel feature encoder places each voxel at its current frame points and describes the past and future points relative to that position, 17 features per voxel, and its detection head predicts velocity. Segmentation is scored on the current frame points only.

Because the detection branch taps the encoder only, finetuning the segmentation head with a frozen encoder cannot change detection outputs - the segmentation and detection checkpoints of a joint model stay independently finetunable and stackable at deploy time.

The training path uses flash-attn for serialized attention. The export path automatically disables flash attention and falls back to the standard attention implementation so ONNX export remains supported.

Summary¶

Available Configurations¶

The T4Dataset rows are bases. They declare the classes the model trains and require the taxonomy that folds the labels of the database onto them, so a T4Dataset experiment that binds one lives in an external config root, see Configuration.

Training¶

autoware-ml train --config-name segmentation3d/ptv3/nuscenes/voxel005_b16_50e_nuscenes_51m
autoware-ml train --config-name detection3d/ptv3/nuscenes/voxel005_b16_30e_nuscenes_51m
autoware-ml train --config-name multi/ptv3/nuscenes/voxel005_b16_100e_nuscenes_51m

For a pipeline validation run:

autoware-ml train \
    --config-name segmentation3d/ptv3/nuscenes/voxel005_b16_50e_nuscenes_51m \
    +trainer.fast_dev_run=true

Mixed precision¶

spconv has no autocast support, so under trainer.precision=16-mixed its convolutions run in fp32. If t4sparse is installed (see Installation), sparse convolutions run 16-bit t4sparse kernels under autocast instead. Checkpoints are interchangeable. The two libraries' indice_key caches are incompatible, so disable autocast around whole stages only.

Evaluation¶

autoware-ml test \
    --config-name segmentation3d/ptv3/nuscenes/voxel005_b16_50e_nuscenes_51m \
    --weights mlruns/ptv3_seg3d/NuScenesDataset-base-v1.0.0/nuscenes/voxel005_b16_50e_nuscenes_51m/<run_id>/artifacts/checkpoints/best.ckpt

autoware-ml test \
    --config-name detection3d/ptv3/nuscenes/voxel005_b16_30e_nuscenes_51m \
    --weights mlruns/ptv3/NuScenesDataset-base-v1.0.0/nuscenes/voxel005_b16_30e_nuscenes_51m/<run_id>/artifacts/checkpoints/best.ckpt

Deployment¶

PointTransformerV3 ONNX export is available. The generic TensorRT stage remains disabled in Autoware-ML because PTv3 requires a runtime with matching sparse convolution plugins.

autoware-ml deploy \
    --config-name segmentation3d/ptv3/nuscenes/voxel005_b16_50e_nuscenes_51m \
    --weights mlruns/ptv3_seg3d/NuScenesDataset-base-v1.0.0/nuscenes/voxel005_b16_50e_nuscenes_51m/<run_id>/artifacts/checkpoints/best.ckpt \
    deploy.tensorrt.enabled=false

autoware-ml deploy \
    --config-name detection3d/ptv3/nuscenes/voxel005_b16_30e_nuscenes_51m \
    --weights mlruns/ptv3/NuScenesDataset-base-v1.0.0/nuscenes/voxel005_b16_30e_nuscenes_51m/<run_id>/artifacts/checkpoints/best.ckpt \
    deploy.tensorrt.enabled=false

The deployment command switches PTv3 attention blocks into non-flash export mode automatically.

Every shipped detection config, including the LitePT ones that inherit from multi/ptv3, exports the detection head with TensorRT-fusable attention. Bf16 is required for numerically stable operator fusion, so the configs pair the head flag with an fp16 precision for the detection-head module:

model:
  bbox_head:
    use_bf16_cross_attention: true
deploy:
  onnx:
    modules:
      ptv3_det3d_head:
        precision: fp16

Both settings are required to enable the fused path, and the resulting engine requires an SM80 or newer GPU. Export refuses to run with only one of them set. To fall back to the stable but slower non-fused attention path, disable both:

autoware-ml deploy \
    --config-name detection3d/ptv3/nuscenes/voxel005_b16_30e_nuscenes_51m \
    --weights <checkpoint> \
    model.bbox_head.use_bf16_cross_attention=false \
    deploy.onnx.modules.ptv3_det3d_head.precision=fp32

The exported ONNX model returns both pred_labels and pred_probs. The probability output is produced by a final softmax layer, while training and evaluation continue to use logits inside the Lightning model.

Deployment uses an explicit PTv3 export wrapper and a copied encoder, so the training model is not mutated when export-only sparse-convolution and serialization settings are applied.

The height channel of the detection head's bbox_pred output is the z of the box gravity center. Earlier releases regressed the top face (z + 0.5 * h), and the autoware_ptv3 node still undoes that shift with z = height - box_h * 0.5 in detection3d_postprocess.cu. A model trained with this release therefore needs the node updated to z = height (no half-height subtraction) before it is deployed. Models trained before this release keep working with the current node.

ONNX Preprocessing Contract¶

The exported PTv3 encoder ONNX expects all pooling-shape metadata to be generated by preprocessing outside the engine. N_0 is the input voxel count, P the number of points a voxel keeps (max_num_points of the data preprocessing), and O is the number of serialization orders.

Npad_k = ceil(N_k / patch_size_k) * patch_size_k is the padded length of level k. The fill is described in Attention window fill at export; the runtime must build patch_order with the per-level patch_sizes stamped into the encoder's ONNX metadata.

For every encoder pooling stage i, with input voxel count N_i and pooled output voxel count M_i, preprocessing also provides:

Because preprocessing resolves every pooling shape ahead of time, the exported graph contains no data-dependent pooling shape discovery. Pooled feature reduction is implemented with native ONNX Gather and the autoware::SegmentCSR plugin.

Attention window fill at export¶

Serialized attention runs on fixed-size windows of K = patch_size tokens, so a sample of n voxels needs its last window filled unless n is a multiple of K:

• n % K == 0 - the windows are exact, nothing to fill.
• otherwise, n > K - the last window borrows tokens from the preceding one to reach K, which is what training does.
• otherwise, n < K - there is nothing to borrow from, so the sample's own tokens are cycled round-robin to equalize their weight in the softmax as far as possible.

Dynamic window sizes and masking are both awkward to express in TensorRT without a performance penalty, hence this fill. Preprocessing computes the fill into patch_order with build_patch_order (encoders/ptv3.py), which the deployed runtime reimplements.

Split-export module contract¶

The split export produces one graph per deploy.onnx.modules entry:

• encoder - the encoder; outputs per-stage features point_feat_0 … point_feat_{S-1} (finest to deepest, S encoder stages). It consumes the per-stage pooling metadata except serialized_pooling_i_cluster, which only drives head-side unpooling and enters the head graphs as pooling_cluster_i instead.
• seg3d_head - consumes all per-stage features plus the per-pooling pooling_cluster_i tensors (the serialized_pooling_i_cluster metadata) and outputs pred_labels/pred_probs. For every decoder stage i with attention blocks (dec_depths[i] > 0) the graph additionally consumes that stage's serialization metadata, under the same names as the encoder inputs: serialized_pooling_{i-1}_patch_order, serialized_pooling_{i-1}_serialized_inverse and serialized_pooling_{i-1}_grid_coord (stage 0 instead takes the base patch_order, serialized_inverse and grid_coord). The rule is implemented once in seg3d_head_export_input_names and must be mirrored by deployment consumers from the artifact's dec_depths.
• det3d_head - consumes point_feat_{S-2}, point_feat_{S-1}, pooling_cluster_{S-2}, and point_grid_coord_{S-2} and outputs the detection head tensors.

Implementation¶

Acknowledgment¶

The Autoware-ML PointTransformerV3 implementation was ported from the official PointTransformerV3 project by Pointcept.

• Repository: https://github.com/Pointcept/PointTransformerV3
• License: MIT
• Paper: Wu, Xiaoyang, et al. "Point Transformer V3: Simpler, Faster, Stronger." CVPR, 2024.