Frequently Asked Questions
Table Of Contents
- Data access
- Data format
- Temporal availability
- Spatial availability
- Noise estimates and flags
- Data inconsistencies
- Data characteristics
- Reading and handling data files
- Data usage in models
... use the product as the "absolute" truth
Before merging the active and passive merged products into a combined active+passive product we first scale both datasets into the dynamic range of the GLDAS-Noah surface soil moisture fields. We perform this processing step to obtain a final product in absolute volumetric units [m3/m3].
Even though the original dynamics of the remote sensing observations are preserved, this step imposes the absolute values and dynamic range (min-max) of the GLDAS-Noah product on the combined product. As a consequence, the combined product CANNOT be considered an independent dataset representing absolute true soil moisture. Hence, the statistal comparison metrics like root-mean-square-difference and bias based on our combined dataset are scientifically not meaningful. However, the CCI SM prodcuts can be used as a reference for computing correlation statistics or the unbiased root-mean-square-difference.
I have registered but not received any data access information
Once you have registered to access the data sets you will receive an immediate e-mail, asking you to confirm your e-mail address. Once you have confirmed your e-mail address we will verify your registration details. If all is in order you will receive an e-mail with login credentials for an ftp server.
If you have not received the login details within 5 working days after your confirmation of e-mail address, please re-check your e-mail inbox. If you still do not find the login details please re-register.
Help, I cannot access your ftp-server! What to do?!
The ftp site is only accessible via SFTP (secure file transfer protocol). You will need to use ftp software that can handle this protocol. Examples of such software are WinSCP or Filezilla. You will need to set the communication port to 22. We regularly check that the ftp server is active. If you cannot access it please recheck the login credentials, and port settings, and re-try. If you cannot access to the ftp site after number of attempts over a couple of days, (perhaps the site is busy, or we are performing some site maintenance), please contact us via the e-mail addresses provided in e-mail you received with the login credentials.
How can the data be unzipped?
We suggest to use http://www.7-zip.org/
Could you send us the data in another format (e.g. Geotiff, ASCII)?
Unfortunately not. We distribute our data sets only in NetCDF, which is an internationally recognized climate data standard. Most software packages and scripting languages (e.g. IDL, Matlab, R) have built-in readers for this format.
What is the unit of the soil moisture product and how can I convert them?
Our data set is provided as volumetric soil moisture [m3/m3]. Based on basic soil properties (e.g. porosity) these units can be converted into other units. See or for the most common conversion equations.
Are data also available for the period after 2014?
At the moment the data record spans the period 1978-2013.
In the foreseen phase 2 of the project a regular yearly update is scheduled, with the product, up to the end of the proceeding year, being available towards the end of the current year.
Will the ECV product be available in near real time?
No, the ECV product will not be available in near real time. However, a product update, including more recent observations is planned for the second year of the CCI-SM project.
Why are data so irregularly available during the first years of the observation period (1978-1987)?
During this period, the ECV product is only based on the SMMR radiometer. SMMR had a 24 hr on-off cycle to save power, but this was sometimes changed. For example in 1986 there is a period with daily observations (they switched the 24 hr on-off cycle off). So, the observation density changes over time. In addition, SMMR observes the Earth surface at 12:00 and 24:00 local solar time, which sometimes leads to a shift of one day for the night-time observations.
Is the SM-ECV also available as monthly values?
Unfortunately not, but you can calculate them by averaging the daily observations for each month.
How can the daily values be transformed on monthly data?
Just sum the valid soil moisture observations for a particular month and divide this sum by the number of days with valid soil moisture observations for that month (and not by the total number of days of that month).The swath doesn't need to be taken into account because we only have a maximum of one observation per day. In the future we'll produce aggregated monthly products according to the standards of Obs4Mips: https://www.earthsystemcog.org/projects/obs4mips/
Why are for some areas no data available?
For areas with dense vegetation (tropical, boreal forests), strong topography (mountains), ice cover (Greenland, Antarctica, Himalayas), a large fractional coverage of water, or extreme desert areas we are not able to make meaningful soil moisture retrievals. Hence, we mask them. You can find these flags in the distributed NetCDF files.
Why do the provided soil moisture images appear in data stripes?
Especially images of the first years from 1978 onwards show clearly these data stripes. This is a typical characteristic in the oberservation through satellite microwave instruments. Microwave images from the earth's surface are taken while the satellite is orbitting the earth in fixed paths. These paths represent the data stripes on the images. If we move forward in time, the spatial data availability is getting higher and higher, and the data stripes are getting closer and closer. This is due to the fact that not only the number of available input data sources (satellites) is growing, but also the technology of satellites intruments is getting better and better.
Why do I get no soil moisture data at all for some days?
Some image files do not provide any soil moisture data at all. All values are NaN. We call these images "blank" or "empty" days. Because of many reasons, e.g. technical failures, there is no data available for that day. Especially the SMMR and the AMI-WS (ERS1/2) instruments are known for their data outages causing these blank days. Other instruments also have short time periods with no data availability. In most cases these empty periods are replaced or filled with data from the remaining microwave sensor(s). So blank days are most likely experienced on days where only one sensor is used as input source, which then fails to deliver data for that time.
Why are there no observations during the winter months in the Northern Hemisphere?
When the soil is frozen or covered with soil, we are not able to make a meaningful soil moisture retrieval. Such observations are masked and indicated with flag number 1 in the NetCDF file.
Why do sometimes data gaps appear in the time series?
Based on the sensitivity to vegetation density, we decided for each pixel whether to use either the scatterometer or the radiometer retrievals, or to use both in a synergistic way (See for details). This merging scheme may lead to data gaps in the following situations:
One of the sensors fails. This is for example the case between 2001 and 2006 in Western Europe, parts of Siberia, parts of North and South America, due to failure of the onboard storage capacity of ERS-2.
Changes in observation wavelength (frequency) may lead to increased sensitivity to vegetation . Hence, larger areas need to be masked. This is for example visible for the period after 1987 where based on the SSM/I Ku-band observations, the extent of masked areas increases with respect to the preceding SMMR period (C-Band).
We are currently working on a method to fill up many of the data gaps with lower quality observations. We hope to provide you with this update with the next product version.
Is the data available at 12.5 km or even higher spatial resolution?
On the following site you can find more info on our products. :
The 12.5 km product is only available on a non-regular swath grid and as near-real-time product, which has a lower quality than e.g. the ASCAT Warp 5.5 25 km products. If you don't need the data for NRT applications, I'd recommend to use one of the latter or the ESA CCI SM active product (available for download at this website).
There is currently no operational soil moisture product available at a higher spatial resolution, e.g. 1 km.
Noise estimates and flags
How should I interpret the soil moisture noise layer?
The noise layer gives you information about the random uncertainty of the retrieval and, hence, about the quality of it. The noise (or error) values were calculated using error propagation methods (See the Algorithm Theoretical Base Document - active for the scatterometer error propagation method and Agorithm Theoretical Base Document - Passive for the radiometer error propagation method. These methods only take into account uncertainties stemming from input data and the retrieval methods. Notice, that our noise products is still in a preliminary stage and error estimates of radiometer and scatterometer input products do not give us exactly the same information. Thus, the hamonisation of noise/errors is still immature. Besides, the radiometer methodology was initially developed for night-time observations of the AMSR-E sensor and, with minor adaptions, applied to the other radiometers. Care should be taken into inter-comparing noise levels from different radiometers as the error estimates depend on calibration parameters which are individually tuned for the different radiometers.
Why do not all soil moisture retrievals have a noise estimate?
The error propagation method for radiometer products can be applied to night-time observations only. Hence, the day-time observations have no uncertainty estimates. Our intention is to fill this gap for the next product release.
Why does the merged passive product does not have uncertainty estimates after 2 July 2012?
From July 2012 onwards the passive CCI product is based on LPRM retrievals from the AMSR2 sensor. The AMSR2 product version that we used for the generation of CCI ECV_SM v02.0 was still very preliminary at that time, and did not yet contain uncertainty estimates.
A more mature AMSR2 product, including error estimates, can now be found on ftp://hydro1.sci.gsfc.nasa.gov/data/s4pa/WAOB/LPRM_AMSR2_A_SOILM3.001/.
Why is data flag number 3 not any longer used after September 2010?
Due to a change in the processing chain retrievals that prior to September 2010 received flag 3 (“invalid data due to other reason and/or no convergence in the retrieval”) are now being entirely removed. Hence the flag number 3 is replaced with a missing data value (127b). We plan to restore this inconsistency for the next product update.
Why do I see a jump in soil moisture values between different swaths?
For AMI-WS and ASCAT soil moisture values may show
jumps where ascending and descending swaths overlap with each other, e.g. in the higher northern latitudes. This is a natural phenomenon related to the differences in overpass time (up to 24h). Potentially different soil moisture values may result from precipitation or evaporation taking place between the two observation time steps. We therefore recommend to use the original observation time (t0) and not the nominal overpass time if you want to make a direct comparison e.g. with in-situ observations.
What are the specific observation periods of each sensors/satellites used for blending the ECV SM 02.0?
The sensors used for each period are best described by the graphic below:
Is the "active product" of ECV SM 02.0 the simple average of the ASCAT soil moisture product at both ascending (21:30) and descending (09:30) overpasses?
This is partly true: CCI SM active product for the period 2007-2012 is the average of asc and desc if both are available at 0:00 +/- 12h. If not, then either the asc or desc observation is used.
Where can I get the passive microwave soil moisture products that are used as input for ESA CCI SM?
Either you can use the passive data provided via CCI Soil Moisture or you get access to the LPRM data: http://www.falw.vu/~jeur/lprm/
Where can I get the active microwave soil moisture products that are used as input for ESA CCI SM?
If you are interested in the ASCAT data only (i.e. not including the period prior to 2007) we would recommend to directly use the ASCAT product as provided at the following URL: http://rs.geo.tuwien.ac.at/products/461b7121-9c07-5022-8c40-23aba622c74c/320899/
Even though the differences are not large, this product is always the latest version available and is as close as possible to the original observations.
Reading and handling data files
Product version 0.1 was published as NetCDF3-classic, which has no data file compression capabilities. Since version 02.0 the soil moisture data are stored in NetCDF4-classic file format featuring data compression. Furthermore, the data files are Climate Forecast (CF) Convention compliant. The NetCDF originator (UCAR) lists a various number of software programs for manipulating or displaying NetCDF data:
Also, there are examples demonstrating how to read (and write) NetCDF files in different programming languages:
For quickly reading, viewing, and plotting the CCI Soil Moisture data files we recommend this set of freely available software:
What is it all about the scaling factor found in the NetCDF files?
For the data variables "sm" and "sm_uncertainty" there is a scaling factor, which is 1E-4 for the Passive and the Combined, and 1E-2 for the Active product, applied to the data values. As already mentioned in the Product Specification Document, under normal circumstances there is no need to take care for this scaling factor, as most NetCDF readers are automatically deploying it. However, there are also readers ignoring it. In this case the appropriate factor must be applied to the read data. Compare the Python and the IDL example below. The scaling factor itself is stored as NetCDF attribute to the corresponding data variable and can be retrieved through any NetCDF reader and viewer (e.g. Panoply). Here is an example code in Python for reading and printing the scaling factor for the NetCDF variable "sm":
How do I read the soil moisture data in Matlab?
The following example shows how to read the soil moisture data variable, and print its data values. The version of the data product to be read is v02.1 or higher. In command window (version R2014a) type in:
How do I read the soil moisture data in IDL v8.x?
How do I read the soil moisture data in Python 2.7.x?
The source code shown here has been proven to work in a Linux as well as in a Cygwin v1.7.35 environment. The netCDF4 module is needed to read the NetCDF4-classic data files, and the matplotlib module is used only to quickly plot the read image. If matplotlib is not available, the plot commands can be omitted in this demonstration. The version of the data product to be read is v02.1 or higher.
How can I open the ECV SM data files in R?