Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
projects:earth_observation:sentinel_2_data_processing_handbook [2024/12/06 09:57] – created kymkiprojects:earth_observation:sentinel_2_data_processing_handbook [2024/12/06 16:15] (current) – kymki
Line 1: Line 1:
-==== The Sentinel 2 Data Processing Handbook ====+<html> 
 +<head> 
 +  <meta charset="UTF-8"> 
 +  <meta name="viewport" content="width=device-width, initial-scale=1.0"> 
 +  <title>Ramble Meter</title> 
 +  <style> 
 +    /* Container for the entire Ramble Meter */ 
 +    .ramble-meter-container { 
 +      display: flex; 
 +      align-items: center; 
 +      justify-content: center; 
 +      margin: 20px 0; 
 +    } 
 + 
 +    /* Ramble Meter */ 
 +    .ramble-meter { 
 +      position: relative; 
 +      width: 200px; /* Scaled down */ 
 +      height: 40px; /* Scaled down */ 
 +      background: linear-gradient(to right, green, yellow, orange, red); 
 +      border-radius: 20px; 
 +      overflow: hidden; 
 +      box-shadow: 0 2px 8px rgba(0, 0, 0, 0.3); 
 +    } 
 + 
 +    /* Needle */ 
 +    .needle { 
 +      position: absolute; 
 +      top: 5px; /* Adjust for centering */ 
 +      left: 50%; /* Default position */ 
 +      width: 4px; /* Visible width */ 
 +      height: 30px; 
 +      background: black; 
 +      border-radius: 2px; 
 +      z-index: 2; 
 +      box-shadow: 0 0 10px rgba(0, 0, 0, 0.5); /* Glow effect */ 
 +    } 
 + 
 +    /* Label in the middle of the meter */ 
 +    .meter-label { 
 +      position: absolute; 
 +      top: 50%; 
 +      left: 50%; 
 +      transform: translate(-50%, -50%); 
 +      font-family: Arial, sans-serif; 
 +      font-size: 12px; /* Adjusted for smaller size */ 
 +      font-weight: bold; 
 +      color: #fff; 
 +      text-shadow: 0 2px 4px rgba(0, 0, 0, 0.5); 
 +      z-index: 3; 
 +    } 
 + 
 +    /* Tooltip styling */ 
 +    .tooltip { 
 +      visibility: hidden; 
 +      width: 250px; 
 +      background-color: #222; 
 +      color: #fff; 
 +      text-align: center; 
 +      padding: 10px; 
 +      border-radius: 5px; 
 +      position: absolute; 
 +      top: 50%; /* Vertically aligned */ 
 +      left: 110%; /* Position to the right of the meter */ 
 +      transform: translateY(-50%); 
 +      font-size: 12px; 
 +      z-index: 10; 
 +      box-shadow: 0 2px 5px rgba(0, 0, 0, 0.3); 
 +    } 
 + 
 +    /* Show tooltip on hover */ 
 +    .ramble-meter:hover + .tooltip { 
 +      visibility: visible; 
 +    } 
 + 
 +    /* Wrapper for positioning the tooltip and meter */ 
 +    .ramble-wrapper { 
 +      position: relative; 
 +      display: flex; 
 +      align-items: center; 
 +    } 
 +  </style> 
 +  <script> 
 +    document.addEventListener("DOMContentLoaded", () => { 
 +      const needle = document.querySelector(".needle"); 
 +      const rambleMeter = document.querySelector(".ramble-meter"); 
 +      const level = rambleMeter.getAttribute("data-level"); // Read the level from the attribute (0-100) 
 +      needle.style.left = `${Math.min(Math.max(level, 0), 100)}%`; // Clamp between 0 and 100 
 +    }); 
 +  </script> 
 +</head> 
 +<body> 
 +  <!-- Ramble Meter --> 
 +  <div class="ramble-meter-container"> 
 +    <div class="ramble-wrapper"> 
 +      <div class="ramble-meter" data-level="20"> <!-- Adjust level here (0-100) --> 
 +        <div class="meter-label">Ramble Meter</div> 
 +        <div class="needle"></div> 
 +      </div> 
 +      <div class="tooltip">Warning: This post may still be inconsistent and may contain some underdeveloped chapters.</div> 
 +    </div> 
 +  </div> 
 +</body> 
 +</html>
  
-This post is work in progress. You have been warned!+====== The Sentinel 2 Data Processing Handbook ======
  
-=== Introduction ===+===== Introduction =====
 The aim of me writing this document is for anyone with little prior knowledge of sentinel-2 data and its processing to learn all they need to know in order to get started working with it, and in particular the generation of an L2A product using the sen2cor processor. What that means should not be obvious to the reader. If it is, perhaps you can still gain some knowledge in skimming through the text, as I will try to explain a number of terms and concepts that are sources of confusion and are typically dropped into the litterature without much further explanation. The aim of me writing this document is for anyone with little prior knowledge of sentinel-2 data and its processing to learn all they need to know in order to get started working with it, and in particular the generation of an L2A product using the sen2cor processor. What that means should not be obvious to the reader. If it is, perhaps you can still gain some knowledge in skimming through the text, as I will try to explain a number of terms and concepts that are sources of confusion and are typically dropped into the litterature without much further explanation.
  
Line 10: Line 113:
 To understand what we need to do in order to reach the level-2 A product starting from the raw satellite output, it is first neccessary to review what steps are involved in producing the level-1 input product from the "raw" level-0 data. We need to get some basics covered. What the sensor measures and how its ability to do those measurements is quantified is described briefly in the follwing two subsections. See these sections as building a common vocabulary for the later sections. To understand what we need to do in order to reach the level-2 A product starting from the raw satellite output, it is first neccessary to review what steps are involved in producing the level-1 input product from the "raw" level-0 data. We need to get some basics covered. What the sensor measures and how its ability to do those measurements is quantified is described briefly in the follwing two subsections. See these sections as building a common vocabulary for the later sections.
  
-Resolution metrics+==== Resolution metrics ====
 Firstly we will briefly talk about the metrics by which we measure resolution of data obtained and derived from the satellite sensors. Firstly we will briefly talk about the metrics by which we measure resolution of data obtained and derived from the satellite sensors.
  
Line 29: Line 132:
 Finally, there is [[https://sentinels.copernicus.eu/web/sentinel/user-guides/Sentinel-2-msi/resolutions/radiometric|radiometric resolution]], which measures the range of brightness values that can be recorded by the satellite and is bounded by the interger type - 12 bit in the case of Sentinel-2 - used to store the data which limits it to integers between 0 to 4095. This can be thought of as the "dynamic range" in photography terms. Finally, there is [[https://sentinels.copernicus.eu/web/sentinel/user-guides/Sentinel-2-msi/resolutions/radiometric|radiometric resolution]], which measures the range of brightness values that can be recorded by the satellite and is bounded by the interger type - 12 bit in the case of Sentinel-2 - used to store the data which limits it to integers between 0 to 4095. This can be thought of as the "dynamic range" in photography terms.
  
-== Units ==+==== Units ====
  
 As hinted at in the resolution section above, the radiance values (energy flux) recorded at the sensor are rescaled into a digital representation from a unit often measured in watt/(steradian/square meter). Steradian, or "square radian" here being the SI unit for volumetric angles compared to the radian being the unit for planar angles. As hinted at in the resolution section above, the radiance values (energy flux) recorded at the sensor are rescaled into a digital representation from a unit often measured in watt/(steradian/square meter). Steradian, or "square radian" here being the SI unit for volumetric angles compared to the radian being the unit for planar angles.
Line 41: Line 144:
 These two terms - reflectance and radiance - are sometimes used interchangebly and in others with minor variations depending on the physics of the system being observed. In both cases it is worth remembering that the sensor doesnt measure either of them but rather the energy flux incident on the sensor, which can then be converted (or re-scaled) to an unsigned interger that represents the physical variable. These two terms - reflectance and radiance - are sometimes used interchangebly and in others with minor variations depending on the physics of the system being observed. In both cases it is worth remembering that the sensor doesnt measure either of them but rather the energy flux incident on the sensor, which can then be converted (or re-scaled) to an unsigned interger that represents the physical variable.
  
-==== Sentinel 2 data processing ====+===== S2 Data Processing Concepts =====
  
 The sections below detail the different steps or processing levels that the raw data from the satellite is subject to after having been recieved at the ground station. The sections below detail the different steps or processing levels that the raw data from the satellite is subject to after having been recieved at the ground station.
Line 47: Line 150:
 There is always error in physical measurements, and we will start by considering the sources of error or image distortion that apply to satellite data. In general terms there are four categories of sources for satellite image distortion, and they are briefly described below. There is always error in physical measurements, and we will start by considering the sources of error or image distortion that apply to satellite data. In general terms there are four categories of sources for satellite image distortion, and they are briefly described below.
  
-=== The Sensor ===+==== The Sensor ====
  
 The digital output from the sensor deployed on the satellite is subject to degradation, as well as minute mechanical differences for a type of sensor in a production series. Therefore, it is not suitable to use the raw output from them as spectral measures. Instead they must be scaled into a number of a certain bit depth by using suitable calibration coefficients. This process converts digital sensor numbers to what is typically refered to in the litterature as at-sensor radiance. This is different from sensor calibration, which is the process of modelling sensor degradation and deriving calibrations coefficients as a function of sensor usage over time. The digital output from the sensor deployed on the satellite is subject to degradation, as well as minute mechanical differences for a type of sensor in a production series. Therefore, it is not suitable to use the raw output from them as spectral measures. Instead they must be scaled into a number of a certain bit depth by using suitable calibration coefficients. This process converts digital sensor numbers to what is typically refered to in the litterature as at-sensor radiance. This is different from sensor calibration, which is the process of modelling sensor degradation and deriving calibrations coefficients as a function of sensor usage over time.
  
-=== The Sun ===+==== The Sun ====
  
 Once the at-sensor radiance values are obtained, the next step would account for solar radiation interference and earth-sun geometry, which is dependent on latitude and datum, to obtain values of top-of-atmosphere (TOA) reflectance. The effect of this source of distortions depend on solar power (which varies over time), the solar elevation angle (determining the amount of light reflected) and the Earth-Sun distance (which also varies over time). Once the at-sensor radiance values are obtained, the next step would account for solar radiation interference and earth-sun geometry, which is dependent on latitude and datum, to obtain values of top-of-atmosphere (TOA) reflectance. The effect of this source of distortions depend on solar power (which varies over time), the solar elevation angle (determining the amount of light reflected) and the Earth-Sun distance (which also varies over time).
Line 57: Line 160:
 Thankfully, while solar flux can be a complicated process to model depending on the need for detail, the Earth and the Sun are reasonably well-behaved celestial bodies and the geometric relationships between them are easily calculated. It is worth noting that in some literature, correcting for this is sometimes grouped together with the sensor correction step. Thankfully, while solar flux can be a complicated process to model depending on the need for detail, the Earth and the Sun are reasonably well-behaved celestial bodies and the geometric relationships between them are easily calculated. It is worth noting that in some literature, correcting for this is sometimes grouped together with the sensor correction step.
  
-=== The atmosphere ===+==== The atmosphere ====
  
-== Top of Atmosphere Radiance ==+=== Top of Atmosphere Radiance ===
  
 Radiation, both from terrestrial and interstellar sources, has varying probability of scattering in parts of the volume segment from the surface of the earth to the sensor platform of the satellite. As such, the observed radiation is a superposition of all of those scattering processes distributed over space and wavelength. What that mix of scattered radiation consists of depends on a great variety of physical processes - Rayleigh scattering in the atmosphere, for instance, results in a large part of the blue region of the visible spectrum scattering in the atmosphere. Radiation, both from terrestrial and interstellar sources, has varying probability of scattering in parts of the volume segment from the surface of the earth to the sensor platform of the satellite. As such, the observed radiation is a superposition of all of those scattering processes distributed over space and wavelength. What that mix of scattered radiation consists of depends on a great variety of physical processes - Rayleigh scattering in the atmosphere, for instance, results in a large part of the blue region of the visible spectrum scattering in the atmosphere.
Line 71: Line 174:
 However, some atmospheric effects are highly variable over the Earth’s surface and can be difficult to correct in Landsat imagery. While it is not always necessary to atmospherically correct Landsat data to surface values, there are instances where this level of correction is needed. In general, absolute atmospheric corrections are needed when (1) an empirical model is being created for application beyond the data used to develop it, (2) there is a comparison being made to ground reflectance data such as a field-based spectroradi- ometer, or (3) as an alternative to relative correction when comparisons are being made across multiple images. All atmospheric correction methods have associated assump- tions about the target and the nature of the atmospheric particles or emissivity (for land surface temperature). There are numerous atmospheric correction methods available, ranging from simple approaches that use only within-image information such as dark object subtraction (Chavez 1988), to more complex and data-intensive approaches such as the method used for the Landsat Ecosystem Disturbance Adaptive Processing System (LEDAPS) products (Masek et al. 2006) However, some atmospheric effects are highly variable over the Earth’s surface and can be difficult to correct in Landsat imagery. While it is not always necessary to atmospherically correct Landsat data to surface values, there are instances where this level of correction is needed. In general, absolute atmospheric corrections are needed when (1) an empirical model is being created for application beyond the data used to develop it, (2) there is a comparison being made to ground reflectance data such as a field-based spectroradi- ometer, or (3) as an alternative to relative correction when comparisons are being made across multiple images. All atmospheric correction methods have associated assump- tions about the target and the nature of the atmospheric particles or emissivity (for land surface temperature). There are numerous atmospheric correction methods available, ranging from simple approaches that use only within-image information such as dark object subtraction (Chavez 1988), to more complex and data-intensive approaches such as the method used for the Landsat Ecosystem Disturbance Adaptive Processing System (LEDAPS) products (Masek et al. 2006)
  
-=== The Topography ===+==== The Topography ====
  
 The processes of georeferencing (alignment of imagery to its correct geographic location) and orthorectifying (correction for the effects of relief and view direction on pixel location) are components of geometric correction necessary to ensure the exact positioning of an image. Imagery can be positioned relative to the datum, topog- raphy, or other data types, including reference data and additional geospatial layers that might be used in the analyses. The processes of georeferencing (alignment of imagery to its correct geographic location) and orthorectifying (correction for the effects of relief and view direction on pixel location) are components of geometric correction necessary to ensure the exact positioning of an image. Imagery can be positioned relative to the datum, topog- raphy, or other data types, including reference data and additional geospatial layers that might be used in the analyses.
Line 85: Line 188:
 Correcting for these they may be computationally costly and the corrections themselves are imperfect. They may introduce artifacts in the data, for instance, or only partially capable of correcting for the effect they are designed for. The correction for these distortions and how they are addressed are often mentioned in bullet-point fashion in documentation, but without much reference, reasoning or justification. Correcting for these they may be computationally costly and the corrections themselves are imperfect. They may introduce artifacts in the data, for instance, or only partially capable of correcting for the effect they are designed for. The correction for these distortions and how they are addressed are often mentioned in bullet-point fashion in documentation, but without much reference, reasoning or justification.
  
-== Level 0 == +===== S2 Processing Levels ===== 
 +==== Level-0 ====
  
 The Sentinel-2 [[https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-2-msi/product-types/level-0|Level 0 product,]] unlike L1C, is not available for public access and is the first processing level performed by the Payload Data Ground Segment (PDGS). Its processing step takes the MSI raw data as input from th Copernicus Ground Segment and .. The Sentinel-2 [[https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-2-msi/product-types/level-0|Level 0 product,]] unlike L1C, is not available for public access and is the first processing level performed by the Payload Data Ground Segment (PDGS). Its processing step takes the MSI raw data as input from th Copernicus Ground Segment and ..
Line 94: Line 198:
     packages Instrument Source Packets obtained from the satellite ground station network into granules     packages Instrument Source Packets obtained from the satellite ground station network into granules
          
-== Level-0 Consolidated ==+=== Level-0 Consolidated ===
  
 This intermediary product contains L-0 and all the meta-data required for subsequent L1 processing. These packets of data are compressed and stored. Like the L0 product L0C is not available to the public. This intermediary product contains L-0 and all the meta-data required for subsequent L1 processing. These packets of data are compressed and stored. Like the L0 product L0C is not available to the public.
  
-== Level-1 A ==+==== Level-1 A ====
  
 This processing step refers to the decompression of the L0C product and applies no processing beyond that. This processing step refers to the decompression of the L0C product and applies no processing beyond that.
  
-== Level-1 B ==+==== Level-1 B ====
  
    1.  Level-1B radiometric processing, including:    1.  Level-1B radiometric processing, including:
Line 119: Line 223:
    6. Level-1B imagery compression utilises the JPEG2000 algorithm.    6. Level-1B imagery compression utilises the JPEG2000 algorithm.
        
-== Level 1 ==+==== Level 1 ====
  
 ‌‌Sentinel-2 MSI L1C data undergoes a number of pre-processing steps before it is ready for use. These steps include radiometric and geometric correction, atmospheric correction, and cloud and water masking.‌‌‌‌‌‌‌‌DEM https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi/definitions ‌‌Sentinel-2 MSI L1C data undergoes a number of pre-processing steps before it is ready for use. These steps include radiometric and geometric correction, atmospheric correction, and cloud and water masking.‌‌‌‌‌‌‌‌DEM https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi/definitions
Line 133: Line 237:
 Overall, the pre-processing steps applied to Sentinel-2 MSI L1C data are designed to correct for various factors that can affect the quality of the data and to make the data more consistent and usable for various applications. Overall, the pre-processing steps applied to Sentinel-2 MSI L1C data are designed to correct for various factors that can affect the quality of the data and to make the data more consistent and usable for various applications.
  
-== Outlook and advice == +===== Outlook and advice =====
  
 We recommend taking a parsimonious approach to preprocessing; correct the artifacts necessary for a particular application, but avoid unnecessary steps that may introduce additional artifacts without gaining additional value (Song et al. 2001, Riaño et al. 2003, Kennedy et al. 2009). We recommend taking a parsimonious approach to preprocessing; correct the artifacts necessary for a particular application, but avoid unnecessary steps that may introduce additional artifacts without gaining additional value (Song et al. 2001, Riaño et al. 2003, Kennedy et al. 2009).