Our IEEE ICIP 2025 paper, with Mohammad Arif Hossain, Weiqi Liu, and Nirwan Ansari, proposes a unified framework for hyperspectral image classification that combines an energy-based generative model with morphological attention. This post explains the reasoning behind that combination, because each half answers a different weakness of the usual approach and the two turn out to need each other.
The problem: labels are scarce, pixels are not
A hyperspectral image is a cube with two spatial dimensions and a spectral dimension of a hundred or more narrow bands. The task is to assign a land-cover class to every pixel. Benchmark scenes contain hundreds of thousands of pixels, but only a small fraction are labelled, and some classes have only a handful of labelled examples.
Almost every model in this area is discriminative. It learns a decision boundary between classes from the labelled pixels and learns nothing about the classes themselves. That leaves most of the scene unused. The unlabelled pixels carry a great deal of information about how spectra are distributed, which materials cluster together, and where the gaps between them lie, and a discriminative loss has no way to use any of it.
What an energy-based model learns
An energy-based model learns a scalar function that assigns low energy to inputs that look like the data and high energy to everything else. Unlike a normalised density model, it never has to compute a partition function, which makes it flexible enough to sit on top of the same feature extractor a classifier uses.
Training follows a simple intuition. Push the energy down on real pixels, and push it up on inputs the model currently considers plausible but that do not come from the data. Over time the low-energy region contracts around the true data manifold. Crucially, this objective needs no labels, so every pixel in the scene contributes to it.
When the classifier and the energy function share their features, the generative objective becomes a regulariser for the classifier. The features have to describe where the data actually lies, not just which side of a boundary each labelled pixel falls on. A classifier built on those features is pushed towards decision boundaries that pass through low-density gaps in the data rather than through dense regions that happen to contain no labelled examples. On small training sets, that is exactly the failure a purely discriminative model is prone to.
Why the spectral view is not enough
A generative prior over spectra captures what materials look like, but not how they are arranged. Land-cover classes have characteristic shapes. Roads are thin and linear, fields are compact, buildings are blocky with sharp corners. Two pixels with similar spectra can belong to different classes, and the clearest evidence for which class a pixel belongs to is often the shape of the region around it.
Learned spatial attention can in principle discover this, but it has to learn shape from the labels, and there are not enough labels to learn it reliably. This is the same scarcity problem appearing again in the spatial dimension.
Morphological attention as a shape prior
Mathematical morphology is a classical toolkit built for exactly this kind of reasoning. Operators such as erosion, dilation, opening, and closing probe an image with a structuring element and respond to the size and shape of the structures they encounter. An opening removes bright structures smaller than the structuring element; a closing fills in dark ones. Applied at several scales, these operators describe the geometry around each pixel without any training at all.
Morphological attention uses those responses to drive the spatial attention weights. Instead of learning from scratch which neighbours matter, the attention module starts from a description of local shape that already separates thin structures from compact ones and interiors from boundaries. Neighbours that belong to a different structure are suppressed, which matters most at class boundaries, where a spatial patch straddles two materials and a naive model averages them together.
Why unify them
The two components address the same underlying problem, too few labels, from two directions. The energy-based objective supplies a training signal from unlabelled pixels; the morphological operators supply spatial structure without needing a training signal at all. Used together on a shared representation, the generative objective shapes features that already encode local geometry, and the classifier draws on a representation informed by both the spectral distribution and the spatial layout of the scene.
Either half alone leaves a gap. A generative prior without a shape prior learns what spectra look like but blurs boundaries. Morphological attention without a generative objective still depends entirely on the labelled pixels to learn how to use the shape information it is given.
Practical notes
Energy-based training has a reputation for instability, and it is earned. Two habits made it manageable here.
Treat the generative term as a regulariser, not the goal. The point is better classification, not convincing samples. Weighting the energy objective so that it shapes the features without dominating the classification loss kept training stable, and judging progress by validation accuracy rather than sample quality kept the attention on the metric that matters.
Let fixed operators carry what they can. Morphological operators have no parameters to overfit. Every piece of structure they provide is one less thing the network must learn from a few hundred labels, which leaves the learned components free to focus on what genuinely requires learning.
What carries over
The broader lesson is to match the inductive bias to the data. Hyperspectral scenes are dominated by unlabelled pixels and by objects with characteristic shapes, so the framework encodes both directly: a generative objective that uses the unlabelled data, and a morphological prior that encodes shape. The same discipline of reading the data before choosing the model applies well beyond remote sensing.
The full paper is available at doi.org/10.1109/ICIP55913.2025.11084715.