Multispectral Imaging: Looking Beyond the Visible Light

Multispectral imaging is a technique that captures light across a broad range of spectral bands, extending beyond what the human eye can see, including infrared and ultraviolet light.

This approach significantly surpasses traditional color imaging by revealing details invisible to the naked eye. Using this method to gather rich information benefits various applications, including analyzing crop health and detecting skin diseases.

At the heart of multispectral imaging is the principle, that different materials possess distinctive spectral signatures(unique patterns of light absorption, reflection, or emission by different materials).

These signatures represent the unique ways in which a material interacts with light across various wavelengths. While some materials absorb certain wavelengths more readily, others might reflect or emit them with greater intensity.

By studying these interactions, it is possible to distinguish among different materials.

About us: Viso Suite is the computer vision infrastructure that completes the entire machine learning pipeline. By implementing Viso Suite, ML teams have successfully cut the time-to-value of their applications down to just three days. Learn more about Viso Suite by booking a demo with our team.

A Brief History

The origin of Multispectral imaging took place in the early 20th century when scientists began experimenting with photography techniques that could capture light beyond the visible spectrum.

• Early Days:
  • The early 1900s: Experiments with special cameras to capture pictures beyond human vision (infrared, ultraviolet).
• Military Use:
  • World War II: Soldiers used infrared cameras to see hidden enemy positions during night missions.
• Satellites Use:
  • One of the most significant milestones in the evolution of multispectral imaging.
  • 1972: A satellite named Landsat 1 launched with a special camera that could see Earth in different colors beyond what our eyes see.
• Size Improvements:
  •  Technology improved, making sensors smaller and more powerful. This allowed for handheld devices and cameras on drones.
• New uses:
  • Farmers started to use Multispectral imaging technology to check on crop health and manage water more efficiently.
  • Doctors started to use it for medical diagnosis.

How does Multispectral Imaging Work

We can see with our eyes due to light reflected by an object. The reflected light from the object reaches our eyes, and our brain processes it, making us see the object. This reflected light is electromagnetic radiation (a form of energy that propagates as both electrical and magnetic waves traveling in packets of energy called photons).

What is the Electromagnetic Spectrum?

The electromagnetic spectrum encompasses a continuous range of electromagnetic radiation, organized by wavelength. It is segmented into bands, each named for radiation within certain wavelength intervals. This spectrum spans from very short gamma rays to extremely long radio waves.

Each band serves specific purposes, catering to various scientific, technological, and practical applications.

Different Bands Wavelength

1. Ultraviolet (UV) imaging: This spans from 100 to 400 nanometers (nm), and serves forensic analysis, art restoration, and environmental monitoring.
2. Visible Light: 400 to 700 nm: This is the part of the spectrum that the human eye can see, comprising the colors from violet through red.
   • Blue (450 – 495 nm):  Used in atmospheric studies and water body analysis due to its strong absorption by water and sensitivity to atmospheric particles.
   • Green (495 – 570 nm): Used for analyzing plant health as plants reflect green light, giving them their characteristic color. It’s also used in studying deep water bodies as green light penetrates water deeper than other visible wavelengths.
   • Red (620 – 750 nm): Critical for assessing vegetation through the analysis of chlorophyll absorption.
3. Near-Infrared (NIR) 700 to 1,400 nm: This band is significant for agricultural and ecological research. Plants reflect NIR light strongly, which makes this band ideal for assessing plant biomass, health, and stress.
4. Short Wave Infrared (SWIR)  1,400 to 3,000 nm: This wavelength excels in penetrating atmospheric haze, enhancing its utility for geological mapping and mineral identification. It enables the differentiation between various minerals, and the assessment of moisture content in soil and vegetation, proving invaluable in fire detection and management.
5. Mid-Infrared (MIR) 3,000 to 8,000 nm: These bands are less commonly used in traditional MSI due to their relevance mostly in thermal imaging. They are important for measuring temperature differences, detecting thermal anomalies, and in applications requiring thermal data such as environmental monitoring, military surveillance, and volcanic activity monitoring.

Significance of Spectral Bands

Specific bands can identify materials based on their spectral signatures because different materials reflect and absorb light differently across the spectrum.

Analyzing the reflection of light in various spectral bands allows for discerning the condition of objects, vegetation, and surfaces. Healthy vegetation, for example, absorbs blue- and red-light energy to fuel photosynthesis and produce chlorophyll. Consequently, a plant with more chlorophyll reflects more near-infrared energy compared to an unhealthy plant.

Steps Involved in Multispectral Imaging

MSI systems capture images in specific spectral bands, selecting these bands based on the application’s requirements and the unique spectral signatures of the materials of interest.

Unlike traditional cameras that capture images in three primary colors (red, green, and blue) corresponding to the visible spectrum, MSI systems can capture light in both the visible and invisible parts of the spectrum, including UV, visible light, and near-infrared. The steps involved in Multispectral Imaging Camera:

• Sensor Technology: MSI employs sensors that are sensitive to different parts of the electromagnetic spectrum. These sensors detect the energy reflected or emitted by objects within their specific spectral bands.  MSI systems can use filter wheels, dichroic mirrors, or prism-based systems to separate the incoming light into the desired spectral bands before it reaches the sensor.
• Image Acquisition: The system captures multiple images of the same scene, each in a different spectral band. It accomplishes this either simultaneously, by employing a sensor array equipped with filters for different bands, or sequentially, by swapping the filters in front of a single sensor or by conducting multiple passes over the target area.
• Data Processing: The system then processes the raw images. This step involves correcting atmospheric interference and sensor noise.
• Analysis and Interpretation: After obtaining the images, the next step involves analyzing the spectral data to identify materials and assess conditions. The analysis occurs by comparing the observed spectral signatures against known signatures of specific materials. Additionally, it includes the use of statistical and machine learning algorithms to classify areas of the image based on their spectral properties.

Types of Multispectral Imaging Camera Setup

There are several different camera setups and configurations used in multispectral imaging, each tailored to specific needs and applications. Here are some of the most common setups:

Single Sensor with Filter Wheel

This setup involves a single-sensor camera that captures one wavelength at a time. A motorized filter wheel is placed in front of the sensor, with different optical filters. The camera takes multiple images in sequence, each with a different filter, to cover the desired spectral bands. This method is relatively slow since it captures images sequentially, but it’s simpler and more cost-effective.

Array of Monochromatic Cameras

This configuration uses multiple cameras, each equipped with a different spectral filter. All cameras capture their respective wavelength range simultaneously. This setup is faster than the filter wheel method since it captures all spectral bands at once but requires precise alignment of the cameras to ensure the images overlap perfectly.

Beam Splitter with Multiple Sensors

In this setup, a beam splitter divides a single optical input into multiple paths, each leading to a different sensor equipped with its spectral filter. This design enables the simultaneous capture of all spectral bands, akin to employing an array of cameras, but achieves better alignment through optical splitting and filtering.

Foveon-like Sensor

Some specialized sensors use a layered design, similar to the Foveon X3 sensor used in certain color cameras. Each layer is sensitive to a different range of wavelengths, allowing the sensor to capture multiple spectral bands simultaneously in a single shot. However, these sensors have limitations in the number of bands they can capture and are less common in multispectral imaging applications.

Filter-on-Chip Cameras

Some newer camera sensors have an array of microscopic filters directly on the sensor chip, allowing different pixels to capture different spectral bands. Whereas traditional methods require multiple shots with different filters and complex setups with a beam splitter and various sensors, the FoC approach simplifies the process. This setup can capture multispectral images in a single shot without needing external filters or beam splitters, though it sacrifices spatial resolution for spectral information. Dedicating each pixel to a specific spectral band decreases the effective spatial resolution of the image.

Integration with Various Technologies

Multispectral Cameras are integrated into various technologies such as drones and satellites. This integration enhances the utility of the camera, allowing it to perform various tasks.

Drones (Unmanned Aerial Vehicles, UAVs)

Drones equipped with multispectral cameras can capture high-resolution images of specific areas. For eg, this integration into drones allows for the usage of computer vision in agriculture, where the drone is used for detailed information about crop health and soil conditions, environmental monitoring, and land management.

Satellites (remote sensing)

Satellite platforms equipped with multispectral sensors provide comprehensive coverage of the Earth’s surface, making it possible to monitor changes in land use, vegetation cover, and environmental conditions on a global scale. Satellites like Landsat have been instrumental in long-term environmental monitoring, climate change research, and natural resource management.

Ground Vehicles

Multispectral imaging is widely used in Autonomous and Robotic Systems. In areas such as environmental assessment, and exploration. For eg, the Mars rover utilizes Pancams for multispectral imaging. These systems can operate in varied terrains and conditions.

Laboratory Instruments

High-precision multispectral imaging systems are used in research laboratories for analyzing materials, biological samples, or chemical compositions. These systems are capable of capturing l tissue health, blood flow, or other physiological features.

Multispectral Imaging vs Hyperspectral Imaging

Multispectral imaging (MSI), traditional imaging (such as RGB imaging), and hyperspectral imaging (HSI) are three distinct technologies used for capturing images, each with its unique capabilities and applications. Here’s a comparison highlighting their main differences:

Traditional Imaging (RGB)

• Spectral Bands: Captures images in three primary wavelength bands (red, green, blue) corresponding to the visible light spectrum. This is what the human eye can see.
• Resolution and Speed: Offers high spatial resolution and fast imaging speed, suitable for everyday photography and video.
• Applications: Widely used in smartphones and DSLR cameras.
• Limitations: Cannot capture information beyond what is visible to the naked eye.

Multispectral Imaging (MSI)

• Spectral Bands: Captures images in several (usually between 4 and 10) discrete spectral bands, which include visible, near-infrared (NIR), and shortwave infrared (SWIR) bands.
• Resolution and Speed: Offers a balanced spatial resolution and spectral information.
• Applications: Used in monitoring environmental, and land analysis.
• Limitations: Limited by the number of spectral bands it can capture, which doesn’t provide as detailed spectral information as hyperspectral imaging.

Hyperspectral Imaging (HSI)

• Spectral Bands: Captures images in a very large number (often hundreds) of narrow and contiguous spectral bands across a wide range of the electromagnetic spectrum.
• Resolution and Speed: Highest spectral resolution but lower spatial resolution compared to RGB and MSI. Slow in speed as it captures a lot of information.
• Applications: Used in mineralogy for identifying minerals, and in environmental monitoring for detailed vegetation analysis.
• Limitations: Produces a large volume of data, which requires significant processing power and storage.

Successful Applications of MSI in various fields

Precision Farming and Crop Monitoring: The use of MSI in precision agriculture has enabled farmers to monitor crop health, optimize water usage, and detect diseases or pests early. For instance, the Normalized Difference Vegetation Index (NDVI), derived from MSI data, has become a fundamental tool for assessing plant health and vigor. This technology has led to increased yields, reduced environmental impact, and improved efficiency in farming practices.

Revealing Hidden Masterpieces in Art: Multispectral imaging has unveiled hidden details in historical paintings and manuscripts that were previously unseen. A notable example is the analysis of Leonardo da Vinci’s notebook. The analysis successfully revealed a figure, possibly drawn and then erased by da Vinci himself. Experts believe it might belong to a series of “fugitive images” occasionally found in his work.

Skin Cancer Detection: In healthcare and, more specifically, dermatology, MSI has been employed to enhance the early detection of skin cancer, including melanoma. By analyzing spectral images, dermatologists can differentiate between malignant and benign lesions with greater accuracy. This non-invasive diagnostic tool has improved early detection rates, significantly impacting patient outcomes and treatment success.

Quality Control in Industrial Manufacturing: In the food industry, MSI has been successfully applied to ensure the quality and safety of products. For example, MSI systems are used to detect foreign objects in food items on production lines or to assess the ripeness and quality of fruits and vegetables without damaging them. This application has improved product quality, consumer safety, and operational efficiency in food processing and manufacturing.

Deep Learning and Multispectral Imaging

Deep learning with multispectral imaging is used for object detection and classification. Convolutional Neural Networks (CNNs), excel at automatically extracting and learning features from images. When applied to multispectral imaging, these models can identify patterns across different spectral bands that may not be visible or distinguishable to the human eye or through traditional image processing techniques.

The models can leverage this diverse spectral information to improve object detection and classification tasks. For instance, using models like YOLOv3 trained on multispectral datasets. The integration enhances the model’s ability to distinguish and classify objects under various conditions, such as low light or obscured visibility.

Moreover, the Mask R-CNN (Region-Based Convolutional Neural Network) method is used for instance segmentation.

Challenges and Limitations

• Complexity: Compared to standard RGB cameras, multispectral systems require more complex setups. They often involve multiple cameras or sensors capturing data in different spectral bands. This can make them bulkier, more expensive, and require specialists.
• Data Handling: The data collected by multispectral imaging is usually quite large. This poses challenges and limitations such as storage and processing power.
• Spectral Resolution: While multispectral imaging provides more spectral information than RGB imaging, it still captures data in a limited number of spectral bands. This can be a constraint compared to hyperspectral imaging, which captures a continuous spectrum and can provide more detailed information about the material properties of the imaged objects.
• Object Distance: Multispectral imaging can become less effective for objects at varying distances. Ensuring all objects in the image are captured within the same spatial footprint across all spectral bands can be challenging, especially for close-up applications.