The round robin for algorithm comparison and product validation will be an important mechanism to inform and engage the international scientific community in the CCI activities. In the meteorological, land surface, and climate-modelling community round robins, usually referred to as intercomparisons or assessment exercises, belong to the regular scientific undertakings. Such intercomparisons are very useful to objectively assess similarities and differences between models.
In remote sensing intercomparison activities are currently not as common as in the modelling community. This simply relates to the fact that several well validated, but nevertheless ‘competing’ data sets, are required to make sense of such intercomparisons. Intercomparison exercises can be seen as a sign of maturity of a particular remote sensing community. With regards soil moisture several intercomparison studies were carried out but none of these studies attempted to broadly involve the international community. This will be one of the goals of the round robins as carried within the Soil Moisture CCI project.
Round robin timeline
Release of Round Robin Data Package and protocol: June 2012
Invitation of participants: June 2012
Delivery of Level 2 retrievals by participants: 30th November 2012
Evaluation of Level 2 retrievals by CCI team: January 2013
Feedback of results to participants: February 2013
PVASR Product Validation and Algorithm Selection Report: April 2013
Round Robin exercise for soil moisture retrieval
The Round Robin exercise aims at identifying the most appropriate algorithms for soil moisture retrieval. Various external research groups thereby apply their methodologies and demonstrate their performance. The Round Robin exercise allows comparing different algorithms and provides insight into their capacity to meet the targeted criteria defined in the Product Specification Document. The entire effort is organised as transparent as possible. All participants work within a standardised environment: they have the same information about context and objective and they work with the same dataset. The scope of the exercise, the input dataset, the expected output, the validation dataset and the evaluation methodology are defined in advance and are described in the Round Robin Protocol document.
Two separate Round Robin exercises will be carried out. One is for algorithms to be applied to the scatterometer measurements and the other one is for passive algorithms. The two Round Robins will be based on ASCAT (scatterometer) and AMSR-E (passive) Level 1 data sets from the 5-years period 2007 to 2011 respectively. This period was chosen because of the increasing availability of high-quality in-situ soil moisture networks data in the most recent years. Any conclusions that can be drawn from these two round robins can also safely be assumed to apply to the predecessor instruments, i.e. for ERS-1/2 SCAT in case of ASCAT and SMMR, TMI, and Windsat in case of AMSR-E.
The assessment of the different algorithms participating in the round robin exercises for ASCAT and AMSR-E will follow the rules as defined in the Product Validation Plan (PVP). Amongst others statistical measures, the skills in terms of the bias-free root mean square error and correlation will be estimated using in situ soil moisture data. Also, new statistical approaches such as the triple collocation method will be applied. Based on these statistics, one might expect that it is straightforward to select the most appropriate algorithms for each instrument. However, previous inter-comparisons showed that there is usually no best performing algorithm. Thus, the result of the round robins for ASCAT and AMSR-E will, with all likelihood, not present a clear winner. Only for some algorithms, or parts thereof, the failure to represent particular physical processes correctly will become apparent. These instances of a clear identification of a model failure are expected to be the most important outcomes of the round robin exercises as they will allow improving the affected algorithms in the long run, provided that new algorithms can be found that can solve these problems.
- No upcoming events available