Radiometrically Accurate Spatial Resolution Enhancement of Spectral Imagery for Improved Exploitation

This research project seeks to address the challenging problem of sharpening remotely sensed hyperspectral imagery with higher resolution multispectral or panchromatic imagery, with a focus on maintaining the radiometric accuracy of the resulting sharpened imagery. Traditional spectral sharpening methods focus on producing results with visually accurate color reproduction, without regard for overall radiometric accuracy. This is done primarily to support exploitation through visual interpretation. However, hyperspectral imagery (HSI) is not exploited through purely visual means. Instead, HSI is processed through (semi-) automated workflows to produce various resulting products. Tasks of interest include material classification and identification, anomaly detection, and target detection. For many of these applications, individual pixel spectra are compared against either other in-scene spectral signatures, or library signatures. Consequently, maintaining radiometric accuracy is a key goal of spatial resolution enhancement of hyperspectral imagery. Here, our focus will be in characterizing the issues related to accurate sharpening, and in particular understanding how the scene spatial - spectral content and complexity impact the ability to sharpen a particular image, or portion of an image. 

This project has three main tasks.  The first task seeks to understand the current state of the art, as well as the impacts of various scene contents on the accuracy of different sharpening approaches.  “Traditional” panchromatic sharpening algorithms generally focus on improving the color improvement of multispectral images and are typically assessed as such with image-wide metrics of spectral similarity and error. However, significant errors in individual pixels may occur which will have a strong impact on applications such as target detection. Our goal here is to understand how traditional algorithms will perform when applied to hyperspectral imagery, and assessed against the application of target detection. 

Sharpening algorithms were tested against two hyperspectral images.  The data were collected with the SpecTIR Propec-TIR hyperspectral sensor and cover the spectral range from 400 - 2500 nm with approximately 5 nm spectral resolution.  The spatial resolution is 1m.  The agricultural scene contains a field with over 100 targets of known material and location, while the urban scene was a collection of opportunity with little ground truth.  However the two scenes provide an interesting challenge due to the dramatically differing scene content.  The scenes were downsampled by a factor of 4 relative to the original data. Per-pixel spectral comparisons show sometimes good agreement between the sharpened image and the original, however, there are also many instances of significant variation.  This is true in both scenes, and in general happens on edges or in relatively “dark” areas of the scene. None of the algorithms tested consistently produce high quality spectral signatures in the sharpened imagery. 

The second task considers the problem of developing a new approach to sharpening Vis-NIR hyperspectral imagery with high resolution multispectral imagery.   A novel learning framework was developed to approach the problem.  The philosophy is that we know the high spatial resolution, low spectral resolution answer, we know the low spatial resolution, high spectral resolution answer, and we need to learn the fusion of them while maintaining radiometric accuracy.  The key to this approach is in application of loss functions that consider both known answers.  However, we leverage the requirement on the algorithm that we can not require the method to have labeled data for the training process.  The framework can be seen in Figure {Messinger-NURI-figure1.png}.  ROC curve results, from application of the ACE target detector against felt targets in the scene, are shown in Figure {Messinger-NURI-figure2.png}.  These results demonstrate that the new algorithm consistently provides radiometrically accurate sharpening products and can be used successfully with a target detection algorithm.

Finally, a third task planned and executed an experiment campaign using the RIT UAV hyperspectral imaging system to collected data at multiple altitudes.  This alleviates the need to simulate data and will provide the opportunity for quantitative assessments of novel algorithms developed under this program.  A ground photo of the target layout is shown in Figure {Messinger-NURI-figure3.jpg}.  These images will be used in future years of this project.