VanderSat Data Flags

Introduction

VanderSat strives to provide the best and most reliable data. Therefore our data comes with a set of flags that supply additional information on our data. These flags indicate possible data issues as well as information on data usage and quality. This document lists the flags that come with our Soil Moisture, Vegetation Optical Depth and Land Surface Temperature products and how to interpret them.

How to retrieve the data flags

VanderSat product have an associated data flags file. The flags can be retrieved via de VanderSat API (VanderSat API User Guide). The API name for the flag file associated with a products can be derived as follows by prefixing the product name with QF-. So if the product name is SM-LN_V001_100 the associated flag file can be retrieved using the product name QF-SM-LN_V001_100.

There is an exception to this rule: the flag file for VOD and SM are shared per band. This means that there are different flag files for each band (C, L, X, the second part of the product name) but there is only a flag file created for the soil moisture product.

Example flag file product mappings
Product Flag file
SM-SMAP-LN-DESC_V001_100 QF-SM-SMAP-LN-DESC_V001_100
SM-AMSR2-XN-DESC_V001_100 QF-SM-AMSR2-XN-DESC_V001_100
TEFF-AMSR2-DESC_V001_100 QF-TEFF-AMSR2-DESC_V001_100
VOD-AMSRE-C1N-DESC_V001_100 QF-VOD-AMSRE-C1N-DESC_V001_100

For details on how to retrieve the actual data files see the API Guide (Requesting spatial maps).

Flag types

The VanderSat data comes with three possible types of flags:

static flags:These are layers that flag certain areas based on (mostly) static features such as terrain slope
critical flags:Critical flags indicate that data is not to be used. As such we replace the data with a missing value. However, when retrieving our data you can acces the original data by looking at the second band in the geotiff and netcdf files. All values in the flag file > 127 are critical
non critical flags:
 A non critical flag means that the data can be used with caution, taking into account the information given in the flag. The data is shown as normal in the viewer and stored similare to non-flagged data in the geotiff and netcdf files. All values in the flag file <= 127 are non-critical.

Note

Critical and non-critical flags are combined into one flag file for each product and given for each time a product is made.

Available flags

Note

The flags below are so-called static flags that are available as separate products in our system.

Complex (steep) terrain

The complex terrain flag indicates areas in which the complex terrain (steep topography) may have influenced our algorithm. In many cases the data is still fine but care must be taken when using the data.

Affected products:

  • Soil Moisture
  • VOD
  • Land Surface Temperature
  • Products derived from any of the above such as DRZSM

Build-up area

The build-up area flag indicates paved/build-up areas. Although this does not hamper our algorithm as such, the obtained values for soil moisture and VOD cannot be interpreted the same as for natural areas. Land Surface Temperature is not affected.

Affected products:

  • Soil Moisture
  • VOD
  • Products derived from any of the above such as DRZSM

Note

The following flags are band/product specific non critical flags

1. Dense Vegetation (high VOD)

This flag indicates high vegetation cover. For those areas soil moisture retrieval is less reliable. Depending on the frequency, the flags value for vegetation corrected soil moisture and vod are applied:

  • C/X band: medium VOD threshold = 0.45 / high VOD threshold = 0.6
  • L band: medium VOD threshold = 0.3 / high VOD threshold = 0.4

Affected products:

  • Soil Moisture
  • Products derived from any of the above such as DRZSM

2. Low Soil Moisture

The retrieved soil moisture is lower than the estimated wilting point.

Affected products:

  • Soil Moisture
  • Products derived from any of the above such as DRZSM

3. High Soil Moisture

The retrieved soil moisture is higher than the estimated porosity.

Affected products:

  • Soil Moisture
  • Products derived from any of the above such as DRZSM

4. Possibly influenced by Snow or severe rainfall

Part of the footprints touch an area flagged as severe rainfall or snow. See 9. Snow or severe rainfall .

Affected products:

  • Soil Moisture
  • VOD
  • Land Surface Temperature
  • Products derived from any of the above such as DRZSM

5. Possibly influenced by RFI

Part of the footprints touch an area flagged as RFI. See 12. RFI detected.

Affected products:

  • Soil Moisture
  • VOD
  • Products derived from any of the above such as DRZSM

7. Possible frozen Soil

This flag indicates areas where frozen soil may occur (Teff < 273.15 K).

  • Soil Moisture
  • VOD
  • Land Surface Temperature
  • Products derived from any of the above such as DRZSM

Note

The following flags are band specific critical flags (VOD, SM and Teff).

8. Frozen Soil

This flag indicates areas where frozen soil is detected. A conservative value for Teff if -10C is used to avoid masking valid data. Data between 0 and -10C is marked with a non-critical flag. For areas marked as frozen soil no Soil Moisture or VOD is given. The data is filtered out during the processing in LPRM.

Affected products:

  • Soil Moisture
  • VOD
  • Land Surface Temperature
  • Products derived from any of the above such as DRZSM

9. Snow or severe rainfall

This flag indicates areas where snow or severe precipitation is detected. For areas marked as snow no Soil Moisture, VOD or Land Surface Temperature is given they are filtered out during the processing of the raw satellite data. Flagging citerium apllied during downscaling is: \((36V - 18V) < -2.5\)

Affected products:

  • Soil Moisture
  • VOD
  • Land Surface Temperature
  • Products derived from any of the above such as DRZSM

10. High vegetation

This flag indicates if the mpdi of brightness temperatures at the given frequency is too low for the algorithm to converge. The formula is: \(\frac{BT_V-BT_H}{BT_V+BT_H} < 0.0001\). The data is filtered out during LPRM processing.

Affected products:

  • Soil Moisture
  • VOD
  • Products derived from any of the above such as DRZSM

11. No overpass

If one or more brightness temperature frequency is missing (due to a combination of different satellite swaths), the no coverage flag is produced during downscaling.

Affected products:

  • Soil Moisture
  • VOD
  • Products derived from any of the above such as DRZSM

12. RFI detected

If a footprint is contaminated more than 25% with RFI during downscaling it is removed from processing. Footprints will less contamination than 25% are retained but the data is marked as influenced by RFI.

Affected products:

  • Soil Moisture
  • VOD
  • Products derived from any of the above such as DRZSM

13. Instrumental Flaws (e.g. along the edges of the swath)

If the brightness temprerature at 36.5GhZ V produces values either over 400K or under 0.9 * the water temperature, the data is considered as unrealistic and is removed.

Affected products:

  • Soil Moisture
  • VOD
  • Land Surface Temperature
  • Products derived from any of the above such as DRZSM

14. Out of valid range

Soil moisture valued retrived are outside of the valid range. This affects both the soil moisture and the retrived VOD. In essence this means that no valid solution was found.

Affected products:

  • Soil Moisture
  • VOD
  • Products derived from any of the above such as DRZSM

15. Open water/Water body

This flag indicates areas that are deemed to be open water (rivers, lakes sea etc). For areas marked as open water no Soil Moisture, VOD or Land Surface Temperature is given they are filtered out during the processing of the raw satellite data.

Affected products:

  • Soil Moisture
  • VOD
  • Land Surface Temperature
  • Products derived from any of the above such as DRZSM

DRZSM quality indicator

The Derived Root Zone Soil Moisture (DRZSM) quality indicator is an indicator for the amount of days for which a valid soil moisture measurement was used. Like the DRZSM product, it is calculated recursively and depends on the amount of days with a valid measurement and the chosen T value. This indicator ranges from 0-100 % and describes for each pixel the level of confidence that the DRZSM has had sufficient data of acceptable quality within the characteristic time scale T. The quality indicator is calculated according to:

\[q(t) = c + q(t-1) * e^{(-dt/T)}\]

with

\[c = (1-e^{-(1/T)})\]

if a valid measurement is used and otherwise

\[c = 0\]

Note that for satellite products that do not have daily coverage, the quality maximum will converge to a value lower than 100 %.

Affected products:

  • Derived Root Zone Soil Moisture

Flags data file description for LST, VOD and SM

During the processing of the data for each product an associated flag file including all the corresponding flags is also created. The relation between product name and flag name is described in How to retrieve the data flags.

This file is a int16 map with each number associated with one or multiple flags. The bit values contain decimal numbers (1 and 0), the position of which indicates the flag. Below, the flags are split into two sections, the first section with the non-critical flags, the second section with the critical flags.

Position of VanderSat data flags for VOD and SM. A 1 indicates a flag position, a 0 indicates a reserved position
Section 2 Section 1
011111111 1011111
Position of VanderSat data flags for Teff (LST). A 1 indicates a flag position, a 0 indicates a reserved position
Section 2 Section 1
010101011 1001000

Some examples of how to read the flags are given below:

64 decimal:which is 0000000001000000 in binary indicates only the flag at position 7 is set. This indicates a non-critical flag for frozen soil
128 decimal:which is 0000000010000000 in binary indicates only the flag at position 8 is set. This indicates a critical flag for frozen soil

Decoding a flag file using python

The example code below shows how to decode the flags:

flagstr = [
    '1. Dense Vegetation (high VOD)',
    '2. Low Soil Moisture',
    '3. High Soil Moisture',
    '4. Possibly influenced by Snow or severe rainfall',
    '5. Possibly influenced by RFI',
    '6.',
    '7. Possible frozen soil',
    '8. Frozen Soil',
    '9. Snow or severe rainfall',
    '10. High vegetation',
    '11. No overpass',
    '12. RFI detected',
    '13. Instrument flaw',
    '14. Out of valid range',
    '15. Open water/Water body',
    '16.',
]



numbers = [12,64,128,141,32768]


for number in numbers:
    pos = 15
    flags= format(number,'016b')
    print("=================================================")
    print("Decimal number: " + str(number))
    print("Binary string: " + flags)
    print("--------------- Flags set -----------------------")
    for fl in flags:
        if fl == '1':
            print(flagstr[pos])
        pos = pos - 1
    print("=================================================\n")

Running the script gives the following output:

=================================================
Decimal number: 12
Binary string: 0000000000001100
--------------- Flags set -----------------------
4. Possibly influenced by Snow or severe rainfall
3. High Soil Moisture
=================================================

=================================================
Decimal number: 64
Binary string: 0000000001000000
--------------- Flags set -----------------------
7. Possible frozen soil
=================================================

=================================================
Decimal number: 128
Binary string: 0000000010000000
--------------- Flags set -----------------------
8. Frozen Soil
=================================================

=================================================
Decimal number: 141
Binary string: 0000000010001101
--------------- Flags set -----------------------
8. Frozen Soil
4. Possibly influenced by Snow or severe rainfall
3. High Soil Moisture
1. Dense Vegetation (high VOD)
=================================================

=================================================
Decimal number: 32768
Binary string: 1000000000000000
--------------- Flags set -----------------------
16.
=================================================

Flags data file description for DRZSM

The quality indicator is stored as a byte integer map without scaling and includes a value between 0 and 100 % indicating the estimated data quality.