Why a heatmap is not a model explanation

A saliency map estimates which image regions influenced an output. It is a useful visual check and a common first step in reviewing a vision model.

It can help an engineer decide where to look first in a particular image. That is a legitimate inspection question. A visualization may expose a reason to examine a case more closely, or help organize a review with people who know the application. Its usefulness does not depend on answering every question about the model.

The limitation is one of scope, not a claim that every heatmap is misleading. An estimate of region influence is different from an explanation of the learned structure producing the output. A visually plausible highlighted area is not, on its own, an assessment of what the model learned.

Engineers usually need more than where. They need to know what learned structure produced the output, why the model failed on a case, and whether the input has moved outside what the model learned. A heatmap alone does not answer those questions.

Detection errors illustrate the gap. A TIDE-style error breakdown separates classification errors, localization errors, duplicate detections, background detections and missed detections. Together, these categories help engineers distinguish error types during review, and a heatmap alone does not provide this diagnosis.

A heatmap also does not decide what to change in a model. Even when it helps locate an unexpected emphasis, the engineering team still needs to investigate the model and the data before choosing a response.

Structural explanation addresses the model's learned structure. With Xpdeep, prediction and structural explanation share the same forward pass. The explanation is not another image laid over an output. The full distinction from post-hoc analysis is in Post-hoc explanation vs structural model control.

This gives the review a different object: the structure associated with the prediction. Engineers can investigate model behaviour rather than infer everything from a highlighted image. Structural explanation describes what the model learned; it is not, by itself, a causal account of the physical scene.

Attributed domain state adds another question, the input's relation to what was learned. It is reported separately from prediction confidence, and a confident output with a plausible saliency map does not make that question disappear.

Xpdeep connects to the customer's or supplier's perception models; it does not supply the perception model. Relevant architectures include YOLO, D-FINE and DETR detectors, 3D camera detection, BEVFusion-type camera and LiDAR fusion, and ViT video models.

The structural design loop has already informed ADAS model design.

Start with the observed behaviour and the application's assessment criteria. Use the heatmap as a visual check if it helps, then examine the structural explanation and attributed domain state as separate evidence. Compare the case with the data and conditions used to assess the model. Keep the observed error distinct from any proposed explanation of its cause.

If a change is needed, Native enables targeted structural editing and controlled correction; Retrofit inspects a frozen PyTorch model. To choose between them, read Native vs Retrofit.

The customer decides whether an investigation supports a model change and how that change is validated. The practical aim is a review that can state what was observed, what evidence supports the interpretation and what remains unresolved.

On image and video, Xpdeep provides structural explanation, attributed domain state and design-loop capabilities. Domain state is explained in Knowing when the model has left what it learned; the modality scope is on Technology. Customer rules determine ACT, HOLD or ESCALATE, with people handling exceptions as required.