Hyperspectral image classification has been a recurring thread in my research since 2019, and looking back over the sequence of models, each one was a reaction to a specific failure of the one before. This post traces that sequence, because the reasoning behind each step is more transferable than any individual architecture.
The data shape is the whole problem
A hyperspectral image is a cube: two spatial dimensions and a spectral dimension with a hundred or more narrow bands. The task is usually to assign a land-cover class to every pixel. Two facts about this data dominate every design decision.
First, the spectral dimension carries most of the information. A single pixel’s spectrum is often enough to identify its material, which is why the earliest methods treated each pixel as an independent vector and ignored its neighbours entirely.
Second, labelled pixels are scarce. A benchmark scene may have a few hundred labelled samples per class, and some classes have far fewer. Any model with a large parameter count relative to this will memorise the training pixels and fail on their neighbours.
Pixel-wise multidimensional convolutions
The first model I built took the spectral-first view seriously. Instead of a 2D network over spatial patches, it applied convolutions along the spectral axis of each pixel, treating the spectrum as a signal with local structure, and combined this with a modest spatial neighbourhood. The multidimensional part refers to running convolutions across both axes with kernels shaped to match: long and thin along the spectrum, small in space.
This outperformed purely spectral classifiers because adjacent bands are highly correlated and a convolution exploits that. It also exposed the next problem. The spectral dimension is redundant, and a network that consumes all bands at full resolution spends most of its capacity on redundancy.
Regularised SVD as a learned front end
The standard response to spectral redundancy is dimensionality reduction before the network, typically principal component analysis. This works but throws away information the classifier might have wanted, because the reduction is chosen without reference to the labels.
The alternative I tried was to build a singular value decomposition into the model as a differentiable front end, with a regulariser that discourages the network from relying on components with small singular values. The reduction is then shaped by the classification loss rather than fixed in advance, and the regulariser gives the effect of a soft, data-dependent choice of how many components to keep. On small training sets this mattered, because it reduced the parameter count downstream without committing to a dimensionality that might be wrong for a given scene.
Residual attention in both domains
By 2021 the field had converged on spectral-spatial networks that process both dimensions jointly, and the question became how to weight them. Not every band is equally informative for every class, and not every neighbouring pixel is equally relevant, particularly at class boundaries where a spatial patch straddles two materials.
Attention answers both questions. A spectral attention module produces a weight per band conditioned on the input, suppressing bands that are noisy or uninformative for the current pixel. A spatial attention module produces a weight per neighbour, suppressing pixels that belong to a different class. Wrapping both in residual connections was essential: on small datasets, an attention module that can fail gracefully to the identity mapping trains far more reliably than one that must produce useful weights from the start. The resulting spectral-spatial residual attention network was the first model in this sequence that felt robust across scenes rather than tuned to one.
Dynamic kernels
Fixed convolutional kernels assume that the right receptive field is the same everywhere in the image. In a hyperspectral scene it is not. Homogeneous regions benefit from large spatial context; boundaries need small kernels that do not mix classes. The dynamic kernel network addressed this by predicting kernel parameters from the local input, so the effective receptive field adapts to the content. This is attention in a different guise, applied to the filter rather than to the features, and it produced the cleanest class boundaries of anything in the sequence.
Energy-based generative models with morphological attention
Every model above is discriminative: it learns a boundary between classes and nothing about the classes themselves. On tiny training sets this is wasteful. The unlabelled pixels, which vastly outnumber the labelled ones, carry information about the structure of the data that a discriminative model cannot use.
An energy-based model learns a scalar energy function that is low on the data manifold and high elsewhere. Trained on all pixels, labelled or not, it captures the structure of the scene. Combined with a classifier that shares its features, it acts as a powerful regulariser: the classifier is pushed to make decisions that are consistent with where the data actually lies, not just with the few labelled points.
Morphological attention is the spatial component of this framework. Land-cover classes have characteristic shapes, such as roads that are thin and linear and fields that are compact, and morphological operators from classical image processing capture exactly these properties. Using them to drive attention weights gives the model a shape prior that a learned spatial attention module struggles to recover from a few hundred samples. Unifying the energy-based objective with morphological attention gave a framework that handled both the scarce-label problem and the boundary problem at once.
What carried over
Three principles survived every iteration.
Match the inductive bias to the data shape. Every gain came from encoding something known about hyperspectral data, whether spectral locality, redundancy, class shape, or the abundance of unlabelled pixels, into the model rather than hoping the model would learn it.
Attention is a weighting problem. Spectral, spatial, kernel, or morphological, every attention mechanism here answers the same question: which parts of the input should influence this decision? Framing it that way makes it obvious where the next one belongs.
Small data punishes ambition. The models that worked were the ones with a graceful fallback, whether a residual identity path, a regularised reduction, or a generative prior, when the labelled data was insufficient to specify the answer.
Those principles are the reason this line of work connects to what I do now. Graph-structured power system data and hyperspectral cubes look nothing alike, but the discipline of reading the data shape before choosing the model is the same.