Fusing thermal and event data for visible spectrum image reconstruction

Reconstructing visible spectrum images from unconventional sensors is a timely and relevant problem in computer vision. In settings where standard cameras fail or are not allowed, thermal and event-based cameras can offer complementary advantages—robustness to darkness, fog, motion, and high dynamic range conditions—while also being privacy-preserving and energy efficient. However, their raw data is hard to read, and most computer vision models are designed and pretrained on standard visible inputs, making direct integration of unconventional data challenging. In this work, we ask whether it is possible, given a paired system that simultaneously records thermal and event data, to recover the kind of information people associate with the visible spectrum. We propose a simple dual-encoder, gated-fusion network that synthesizes visible-like images from thermal frames and event streams. The thermal branch captures structure and coarse appearance; the event branch models spatio-temporal changes and adds more detailed information. Their outputs are combined together and finally decoded into a colored image. We train and test the proposed solution on a paired thermal–visible–event dataset. Results show that this approach can recover plausible visible images producing better results than single-modality baselines, both quantitatively and qualitatively