Metrics¶

A metric accumulates data over an epoch and produces a scalar report at the end. Metrics are attached to a model from config and run during validation and test. Losses are handled by the model itself, not by metrics.

The design separates two roles:

• A suite (MetricSuite, a torchmetrics.Metric) is a task state engine. It owns the accumulated state, its cross-GPU reduction, and the dispatch across the range and filter axes. It does not decide which metrics run.
• A metric (Metric) is a small, self-contained, injectable object. It computes its own numbers from the state the suite builds, and declares which stages it runs in and which filter it reads.

Which metrics run, in which stages, and over which slices is pure configuration. The suite is just the engine that feeds them.

What runs in each split¶

Each metric declares its stages. The convention is that cheap headline metrics run in both val and test, while the heavier reporting runs only in test, so validation epochs stay fast. A suite is cloned only for the stages where at least one of its components reports, so a heavy test-only suite does not exist at validation time at all.

Lifecycle¶

A suite runs the standard torchmetrics contract across an epoch.

• update(eval_out) runs once per batch on each GPU. It folds the batch into the suite's state and never talks to other GPUs. Everything that needs per-frame context that is not a tensor, such as the ego pose or the scene's lanelet map, happens here.
• sync runs once at epoch end, inside compute. torchmetrics combines every GPU's state using the reduction declared for each state. This is the only cross-GPU step.
• compute() runs after sync. Components are grouped by their filter, the suite builds one state per filter and range window, and every stage-applicable metric evaluates the state its filter selects.

result(stage) sets the reporting stage and calls compute. The mixin clones a suite per stage it reports at and resets it at epoch start, so each instance reports for exactly one stage. A suite declares the eval-output keys it needs through required_keys(), which folds in the keys its active components and their filters require, and the first batch fails loud when one is missing.

sequenceDiagram
    participant L as Lightning
    participant M as Model
    participant S as Suite
    participant Me as Metric
    loop each val batch
        L->>M: on_validation_batch_end(outputs, batch)
        M->>M: build_eval_output(batch, outputs)
        M->>S: update(eval_out)
    end
    L->>M: on_validation_epoch_end()
    M->>S: result(stage)
    S->>S: compute() syncs state across GPUs
    S->>S: state_for(range, filter) builds the state
    S->>Me: evaluate(state, stage) for each metric in this stage
    Me-->>S: MetricReport with bare names
    S-->>M: MetricReport placed in filter slices and range windows
    M->>L: log under val/prefix/key

Built-in suites¶

Each piece of state is registered with add_state(name, default, dist_reduce_fx), the reduce function torchmetrics uses to combine that state across GPUs.

A confusion matrix is a bounded sufficient statistic, so its counts sum across ranks. Detection matching is score ordered inside each frame and the point-level metrics read raw points, so those states stay per-frame list elements gathered with no reduction. Evaluation filters need the ego pose and the scene's lanelet map, which are not tensors, so their keep-masks are computed per frame at update and stored as boolean states, and DDP gathers only tensors.

seg3d_frames is a list with one entry per frame carrying that frame's point coordinates, predicted and target labels, per-class scores, and the per-frame metadata configured filters need (ego2global, lanelet_map_path). required_keys() extends the base tuple with what the configuration demands: a detection suite with a collision provider also requires ego2global and lanelet_map_path, and a ground-truth point filter requires gt_num_points.

A metric records its values under bare names such as mAP_car in a MetricReport. Every value travels with a typed MetricKey that names the suite prefix, the taxonomy view, the filter_name, the metric name and the range_suffix, and renders them as {prefix}/{view}/{filter_name}/{name}_{range_suffix} with the empty levels left out. The suite places each metric's keys in the filter slice and range window it evaluated, the model places each suite's keys under its prefix, and the key is logged as {split}/{rendered key}, for example test/det3d/region_road/mAP_0m_50m. A report rejects a duplicate key at every level and the log payload rejects two keys that render to one name, so two metrics, two filters or two suites can never silently overwrite each other. A value is read back by its key, report.metric_value(MetricKey(...)), and report.metric_names() lists the keys. Two suites of the same class run side by side under distinct configured prefixes, and the mixin rejects duplicate prefixes.

Class structure¶

classDiagram
    class TorchMetric["torchmetrics.Metric"] {
        +add_state(name, default, dist_reduce_fx)
        +update(eval_out)
        +compute()
        +reset()
    }
    class MetricSuite {
        +prefix : str
        +components : list[Metric]
        +required_keys()
        +update(eval_out)*
        +state_for(range, filter)*
        +compute()
        +result(stage)
    }
    class Metric {
        +stages : frozenset[EvalStage]
        +filter : MetricFilter
        +evaluate(state, stage)*
    }
    class TaskSuite["Detection3DMetricSuite / Segmentation3D...Suite"]
    class TaskMetric["MeanAP / IoU / ..."]
    TorchMetric <|-- MetricSuite
    MetricSuite <|-- TaskSuite
    Metric <|-- TaskMetric
    MetricSuite o-- "0..*" Metric : runs injected

Method marks:

A suite implements update and state_for and declares its prefix and required keys. A metric implements evaluate and declares stages and an optional filter. The suite holds a list of metrics it was given and runs each one against the state its filter selects. Adding a metric means adding a Metric subclass and listing it in config, never editing the suite.

Data requirements¶

The baseline metrics need nothing beyond the model's predictions and the ground truth. Each optional capability adds a concrete requirement, checked loud at the first batch.

The map-based features never read a map path directly. They pass the frame's scene_token to a map provider, and an injected resolver turns that identifier into the scene's lanelet2_map.osm. Any data source that can name a map file per scene and attach a per-frame ego pose in the map frame can use them.

A missing eval-output key fails loud at the first batch, naming the suite and the key. A missing map is the one tolerated absence: the scene is excluded from the map-based slices and the collision metrics, kept in the whole-scene and corridor metrics, and the per-filter coverage is logged every epoch (see Coverage below). Boxes without velocity components score the worst-case velocity error instead of failing.

Evaluation axes¶

Every metric can be reported over orthogonal, purely configurational axes.

Ranges. Radial distance windows. Every key a metric emits is also emitted per range with a distance suffix. Detection clips boxes per range, segmentation buckets points.

Region filter. RegionFilter keeps only the elements whose position, transformed to the map frame by the frame's ego pose, falls inside a chosen set of lanelet regions. Detection tests the whole box footprint, segmentation tests each point. Any metric combined with a region filter becomes its on-road or on-walkway variant. The bundled configs define the road slice as the union of the lanelet primitives road, road_shoulder, crosswalk, drivable_area, intersection_area and crosswalk_polygon, and the walkway slice as the walkway primitive.

An optional margin erodes the outer border of the union of the mapped regions, so noisy points at the edge of the mapped surface stop counting in the filtered slice.

Corridor filter. CorridorFilter keeps only the elements inside a straight corridor ahead of ego: a fixed-width forward strip in the ego frame. The strip carries no length bound of its own, because every key is range-binned anyway, so the range axis already slices the corridor by distance. It needs no map and no pose, so the slice covers every scene. A detection box counts when any part of its footprint overlaps the strip.

Collision filter. CollisionFilter keeps only the elements inside the ego collision area, everything ego could collide with within the horizon under bounded steering, clipped to the drivable lanelets so it follows the road on bends. It is a reporting slice like the region filter, computed per frame from the same collision model the collision metrics use. A detection box counts when any part of its footprint meets the area, a segmentation point when it lies inside. Only ego is propagated here and every element is tested where it stands; the objects' own reachable sets enter through the collision metrics, not through the filter.

Class catalog. Every dataset family keeps a class catalog per task in configs/metrics/<dataset>/<task>/class_catalog/, one row per class name with its behaviour group, the distance it is evaluated up to (max_distance), its collision motion_model and max_speed, and whether the partial detection score checks it. A model reads the rows of the classes it trains, and a trained class the catalog does not list stops the run.

Behaviour groups. class_groups folds the trained classes onto behaviour-equivalent groups. Confusing a bus for a truck does not change how the vehicle drives, so inside a group it counts as a hit. Grouping always replaces the class axis, never extends it: the same suite is configured twice, once per class and once grouped under its own prefix, and both views share one accumulation contract.

Coverage. A scene without a lanelet map cannot be map-filtered, so region and collision slices exclude its frames while the whole-scene and corridor metrics keep them. The suites count per-filter frame coverage and log it every epoch, so a slice computed over fewer scenes is never silent.

The collision model¶

The collision model is not a filter but mechanics. It extends the collision filter: the same region construction (one shared ego agent, the same clipped area) with a time rule applied on top. Configured on the suite as the CollisionTTC provider, it computes a time to collision (TTC) per detection box once per frame at update time, from the ego pose, the class-specific collision sets, and the scene's lanelet map. The metrics that declare needs_ttc read the stored TTC at compute time.

The model answers one question: if ego never sees this object, what is the worst that can follow? Every agent moves at its class maximum speed under its tightest turn, forward or in reverse, so the collision set is a bound, not a prediction. A lead at matched speed is not exempt, it may brake. A wheeled body is swept as a rectangle placed at every time step along the arc, clipped to the drivable area and reduced to the part still connected to the driven path. The rectangle sits around the pose's reference point: a detection box is centred, ego is measured at its rear axle.

The time to collision (TTC) of an object is the first time step at which its collision set and ego's overlap. An object that cannot reach ego within the horizon has an infinite TTC.

The model runs on the following constants. Values marked config are set in the bundled dataset configs and tunable there, the rest are code defaults.

Default slices¶

The bundled dataset configs attach every metric to every filter slice (whole scene, road, walkway, corridor, collision area). Every key is also emitted per radial range bin and once more per behaviour group through the grouped twin suite. The partial-detection score is wired in the joint detection plus segmentation configs only (the detection ground truth rides in seg3d_frames) and is never grouped, its box to class mapping is bound to the trained label space.

Detection metrics¶

The detection state matches predictions to ground truth greedily in score order, center distance by default with a corner-distance cost selectable per suite. Matching is memoized per class and threshold, so every component reads the same match curves.

Mean AP¶

MeanAP is the center-distance average precision over its match thresholds, reported per class and as the class mean. It is the headline metric that also runs at validation. At test it adds per-class ground-truth counts, match counts, the maximum F1, and the optimal-confidence operating point of each curve.

Heading AP¶

HeadingAP weights every true positive by its heading score, so a detector that finds objects but points them the wrong way scores lower than plain AP.

NDS¶

Nds folds the mean AP and the true-positive errors, taken at a single configured operating threshold, into one composite detection score.

TP errors¶

TpErrors reports the translation, orientation, scale, and velocity errors of true positives at configured recall operating points, plus the optimal-confidence operating point.

Corner displacement error¶

CornerError couples position, size, and yaw into one distance in meters, measured where planning collides with things: the box outline. It also exposes footprint inflation that center distance is blind to.

Heading-flip rate¶

HeadingFlipRate counts true positives whose heading is reversed by about 180 degrees. Corner displacement forgives a flip because the outline barely moves, yet a flipped heading inverts the object's velocity direction and breaks tracking, so it is penalized on its own.

Signed nearest-surface error¶

NearestSurfaceError measures the error of the object face nearest to ego, which is what stopping distance is computed against. The sign is the safety signal: positive means the predicted near face sits farther than the truth and ego brakes late, negative means over-caution.

Critical FP / FN¶

CriticalFPFN reports two numbers that are never averaged: false positives in ego's path cause phantom braking (usability), false negatives in ego's path mean driving toward something unseen (safety). An object is critical when its collision TTC is within the horizon. Requires the suite's collision provider.

Both are reported as curves over the confidence threshold, so the operating point trade-off is visible.

Collision-risk-weighted mAP¶

CollisionWeightedMeanAP weights every object's contribution to precision and recall by an exponentially decaying function of its TTC, so mistakes on objects about to matter dominate the score. Requires the suite's collision provider.

Calibration error¶

CalibrationError bins predictions by score and measures the gap between the mean score and the empirical precision in each bin. Reported pooled and macro-averaged per class, so a dominant class cannot hide a badly calibrated rare one.

Confident-error rate¶

ConfidentErrorRate reports, over the false positives, the fraction the model was confident about. A high-score phantom is what triggers a hard brake, so this is the usability release blocker. The missed-object axis has no score to read and belongs to the critical FN curve.

Confusion matrix¶

ConfusionMatrix pairs predictions and ground truth with class-agnostic matching and counts the label of each matched pair, so a car predicted where a truck stands lands off the diagonal. It is the label-confusion view among detections that did match, not a recall metric.

Frame and box counts¶

FrameCount reports how many frames the other metrics scored, as num_frames. A region-filtered component counts only the frames its filter covered, so a slice reported over fewer scenes than the whole-scene metrics shows it next to its scores. Range clipping removes boxes, never frames, so the component declares itself range_invariant and the suite reports it once per slice instead of once per range window. The segmentation suites ship the same component under autoware_ml.metrics.segmentation3d.frame_count, reading the frame count every suite keeps in its base class: one whole-scene counter, and the coverage counter for each filter slice.

BoxCounts reports how many ground-truth and predicted boxes the other metrics scored, in total and per class. The ground truth is already filtered by the minimum point count and the class range of the suite, and both sides are clipped to the evaluated range, so a change of the eval filters shows up in the counts next to the scores.

Occlusion-aware recall split¶

Not a component but a suite pattern: with per-box point counts (gt_num_points) in the eval output, a box with zero points is occluded. Two extra suites under distinct prefixes, one keeping visible boxes and one keeping everything, differ exactly by the occlusion effect on recall.

Segmentation metrics¶

IoU, accuracy, precision / recall / F1¶

IoU, Accuracy, and PrecisionRecallF1 read the accumulated confusion matrix and report per class and macro-averaged values. IoU is the headline metric that also runs at validation.

Confusion matrix¶

ConfusionMatrix emits the accumulated point confusion counts, rows true class, columns predicted class. In a grouped suite the matrix is already folded onto the behaviour groups.

Error clusters¶

ErrorClusters reports the misclassification rate together with the connected error clusters: how many contiguous wrong regions exist and how large they are. A hundred scattered wrong points are noise, the same hundred points in one blob are a phantom obstacle.

Neighbourhood-tolerant error rate¶

NeighbourhoodTolerantErrorRate counts a misclassified point as an error only when the model predicted its true class on no point within a small radius. Point-wise labels are not perfect, and for driving a handful of flipped points do not matter as long as the right class is present right there.

Small-object partial-detection score¶

PartialDetection3DScore groups segmentation points inside each small-object ground-truth box and rewards partial hits with a saturating credit: for a pedestrian or a cone, classifying even a few points correctly is far better than none, which point-averaged mIoU cannot see. A diagnostic metric of the joint detection plus segmentation suite (datasets/<dataset>/det3d_seg3d), whose seg3d_frames carry the detection ground-truth boxes. The detection class catalog marks the classes it reports with partial_detection.

Calibration error¶

CalibrationError is the expected calibration error over the per-point confidence: a model that says 90 percent road should be right about 90 percent of the time. Reported overall and macro-averaged per predicted class.

Uncertainty usefulness¶

UncertaintyUsefulness measures how well the per-point predictive entropy separates correct from misclassified points, as the AUROC of the binary "is this point wrong" task. 0.5 means the uncertainty is meaningless, higher means it can flag the model's own mistakes.

Confident-error rate¶

ConfidentErrorRate reports the fraction of misclassified points predicted with high confidence, measured as low predictive entropy. An error the model is unsure about is recoverable, a confident error is what hides an obstacle.

Attaching metrics¶

model.metrics is a list of suites, so a joint segmentation and detection model lists several. Each suite is given its components, and each component its stages and optional filter. The suites live in the metrics config groups of each database and read the tunable pieces (class names, ranges, filters, groups) from interpolation variables, because Hydra replaces a list wholesale rather than merging it, so overriding a suite directly would mean restating every field.

model:
  metrics:
    - _target_: autoware_ml.metrics.detection3d.suite.Detection3DMetricSuite
      class_names: ${taxonomy.detection3d.class_names}
      max_distances: ${metric_params.detection3d.max_distances}
      ranges: ${metric_ranges}
      components:
        - { _target_: autoware_ml.metrics.detection3d.mean_ap.MeanAP, stages: [val, test] }
        - { _target_: autoware_ml.metrics.detection3d.nds.Nds, stages: [test] }
        - { _target_: autoware_ml.metrics.detection3d.mean_ap.MeanAP, stages: [test],
            filter: '${metric_params.detection3d.filters.region_filters.road}' }

The same component listed once more with a filter is one more reported slice.

What a model provides¶

One method. It maps the typed forward outputs and model inputs to the flat dict the suites read. Model-specific work like box decoding happens here, and per-frame metadata the dataset supplies (ego pose, scene token) is passed through.

class ModelA(ModuleBaseModel):
    def build_eval_output(self, batch, outputs):
        return detection3d_eval_output(self.decode_outputs(batch, outputs), batch)

The mixin feeds this dict into every attached suite. The model never calls update, compute, or result.

Writing a custom metric¶

A metric is the unit of extension. Subclass Metric, declare the stages it runs in (or accept the default), and read the suite's state. The example adds a per-class accuracy view to segmentation without touching the suite.

class PerClassAccuracy(Metric):
    def evaluate(self, state, stage):
        report = MetricReport()
        for i in range(state.num_classes):
            if bool(state.has_support[i]):
                report.record(f"acc_class_{i}", float(state.recall[i].item()))
        return report

Add it to the suite's components list in config and its keys appear under the suite prefix. A new metric family that needs new state is a new suite, which implements update and state_for.

Distributed runs¶

Losses are means, so a mean across GPUs is correct. Metrics are not always linear, so each state declares how it combines and torchmetrics applies it before computing. After sync the state is identical on every rank, so the result is logged without sync_dist.