CRC Press · Taylor & Francis · Handbook of Deep Learning Methods for Digital Image Processing

Explainable Transformer-Based Multimodal Image Fusion for Edge Vision Systems

Vatsalya Bhadaurya Website ORCID

Code & Data Download Model View Results
0FPS, real-time at 256²
0million parameters
0× smaller student
0fusion benchmarks

01 Abstract

Edge vision systems must fuse complementary imaging modalities under tight compute budgets while staying interpretable. We present TCMF-Net, a Transformer-guided Cross-Modal Fusion Network that pairs a shared-weight pyramid-Transformer encoder, a deformable cross-modal attention (DCAF) module for implicit cross-modal registration, and a modality-aware gate for adaptive weighting. The entire study is fully reproducible on a single 6 GB laptop GPU: it outperforms classical fusion baselines on TNO, RoadScene and MFNet, generalises zero-shot, extends to segmentation, distils to a 0.048 M-parameter student, and produces explanations that are quantitatively faithful.

02 Highlights

Deformable cross-attention

Learns 2-D sampling offsets to register modalities implicitly. In the ablation it is the single biggest driver of mutual-information gain.

Explainable by design

Occlusion-based modality contribution, a learned gate, and attribution faithfulness that measurably beats a random baseline.

Real-time on a laptop

5.25 M parameters, 73 FPS at 256² on an RTX 4050; distilled to a 0.048 M student above 1000 FPS.

Reproducible

Released code, splits and seeds. Every number regenerates with one command, with no fabricated results.

03 Architecture

Two source images flow through one shared pyramid-Transformer encoder. Deformable cross-modal attention and a per-pixel gate fuse them, and a U-Net decoder produces the fused image together with an optional segmentation map.

TCMF-Net architecture
Figure 1. TCMF-Net: shared PVTv2 encoder, DCAF with a modality gate, U-Net decoder, and weight-map / segmentation heads.
1Shared encoder

PVTv2 pyramid Transformer, four scales, one set of weights for both modalities.

2DCAF

Deformable cross-modal attention at 1/8 and 1/16. Query is visible, key and value are infrared.

3Modality gate

A per-pixel, per-channel gate decides how much of each stream to trust.

4Fusion head

A full-resolution weight-map head preserves input-level sharpness.

04 Fusion results

Under one identical protocol, TCMF-Net outperforms classical baselines on the information- and gradient-oriented metrics across all three benchmarks (MFNet is evaluated zero-shot). Select a metric and a dataset:

Higher is better. A lower SSIM for TCMF-Net is expected, because selective fusion deliberately departs from each single source in order to combine both.

Ablation: what each component contributes (TNO)

Removing DCAF drops mutual information the most. The gate is near-neutral on these aggregate metrics, which we report honestly.

Show the full results table (every metric, every dataset)

Best per metric and dataset in bold. TCMF-Net, Average and Laplacian pyramid are all run under one identical protocol with a single shared metric implementation.

05 See it fuse

Real outputs from the released model. Drag the slider to reveal the fused result over the visible input.

visible fused
‹ ›
Visible
Fused
infrared, visible, fused
Figure 2. Infrared, Visible, and Fused. Thermal signatures and visible texture are combined.

06 Explainability

Interpretability is built in, not bolted on. Three complementary lenses reveal which sensor drove a decision and whether the explanation can be trusted.

Modality contribution

Occlusion attribution: replace one modality with its mean and measure how much the fused output changes.

Learned gate

The per-pixel gate g, read straight from the network, shows its modality preference at every location.

Faithfulness & stability

Deletion and insertion curves against a random baseline, plus robustness of attributions to input noise.

Which modality drives the fusion?

TNO · balanced scene

RoadScene · thermal-rich

Infrared contributes 71% on thermal-rich driving scenes but only 48% on TNO. The fusion adapts to the scene rather than fixing a ratio, matching the intuition that thermal cues matter most at night.

Are the explanations trustworthy?

Faithfulness: attribution vs random baseline

Stability under input noise
0.032
0.023

Mean change in the gate under Gaussian noise (σ = 0.02) on a [0,1] scale. Lower is more stable: the attributions barely move, so they are not artefacts of input noise.

modality gate
Figure 3. The learned modality gate g (warmer indicates higher infrared weight) over a night driving scene. Warm objects and their edges receive more infrared weight.

07 Efficiency and reach

73FPS at 256² on an RTX 4050
5.25Mparameters, 9.3 GFLOPs
0.048Mdistilled student, 8.5× faster
55%smaller via INT8 (ONNX)
36.2%MFNet RGB-thermal segmentation mIoU
18.8%NYU RGB-depth segmentation mIoU

Distillation retention: EN 97%, SSIM 102%, MI 117%, QAB/F 88%. Most of the quality is kept at 109× fewer parameters. Pretrained weights (teacher, distilled student, ONNX and INT8) are in the weights folder.

RGB-thermal segmentation example
Figure 4. The fusion backbone extends to segmentation. On an RGB-thermal night scene (MFNet) it localises the safety-critical classes, car 64.7% and person 56.3% IoU, for an overall 36.2% mIoU.

08 Key findings

  1. Deformable cross-attention is the engine. Removing it causes the largest drop in mutual information (4.63 to 4.27), identifying it as the main driver of cross-modal information transfer.
  2. The learned fusion generalises. Trained only on TNO and RoadScene, the model still beats classical baselines on MFNet with no fine-tuning, transferring to an unseen sensor domain.
  3. Explanations are faithful, not decorative. Attributions reliably beat a random baseline on deletion and insertion, and infrared accounts for 71% of the fused response on thermal-rich driving scenes.
  4. Quality survives heavy compression. A 0.048 M student, 109× smaller and 8.5× faster, retains 88 to 117% of the teacher across metrics.
  5. Honesty over hype. We report where the method leads (entropy, mutual information, QAB/F, efficiency) and where it does not (SSIM, dense segmentation), and release the code so every number can be checked.

09 Citation

@incollection{bhadaurya2026tcmfnet,
  title     = {Explainable Transformer-Based Multimodal Image Fusion for Edge Vision Systems},
  author    = {Bhadaurya, Vatsalya},
  booktitle = {Handbook of Deep Learning Methods for Digital Image Processing},
  publisher = {CRC Press, Taylor and Francis Group},
  year      = {2026},
  note      = {Project page: https://tcmf-net.vatsalyaaa.me/ , Code: https://github.com/VatsalyaBhadaurya/TCMF-Net}
}