Digital Opportunities for Better Agricultural Policies

Chapter 14.  Case Study 9: Connecting the dots to create a data infrastructure: The US National Soil Moisture Network

Abstract

The objective of this case study is to provide the example of the National Soil Moisture Network (NSMN) initiative in the United-States, which intends to address the fragmentation and heterogeneity of data coverage for the tracking of soil moisture. Its intent is to combine data from satellites with the data captured from state level in situ networks, to build a national inventory that can inform policy management and decisions.

Context: Disconnect between different layers of data about soil moisture in the United States prevents their reuse for comprehensive water policies

Soil moisture matters to inform policy makers

Soil moisture is an important element for the assessment of water use and water demand. Soil moisture data are critical for assessing:

• Drought conditions and operational drought monitoring

• Flood forecasting

• Land surface modelling (Component in weather and climate models, to simulate the exchange of surface water and energy fluxes at the soil–atmosphere interface.)

• Crop yield estimation

• Water supply forecasting

• Operational hydrologic models

• Weather forecasting

Mesoscale in situ meteorological observations provide the data used for weather and climate forecasting and decision-making.1 The data they create are essential to a large range of stakeholder communities. These include state environmental and emergency management agencies, water managers, farmers, energy producers and distributors, the transportation sector, the commercial sector, media, and the general public (Mahmood et al., 2017[1]).

The history of the development of soil monitoring in the United States

Two types of technologies are used for the monitoring of soils water content in the United States. The first system relies on direct in situ instruments; the second relies on remote sensing. Each approach has strengths and weaknesses, and each was designed and has evolved to meet specific purposes and goals.

The in-situ monitoring of surface soil moisture in the United States is not new. The first surface weather observations began on the East Coast in the late seventeenth century (Fiebrich, 2009[2]). Those sparse initiatives were then organised into a network of volunteer weather observers by the Smithsonian Institution, following the Surgeon General, army, and General Land Office request for regular observations at widespread locations.2 From the 1970s, technological improvement allowed the process of automation of weather data collection with the development of sensors and multiple-function data processors at remote sites. Cellular and satellite communication systems also allowed for a more rapid transfer of information. Developments ultimately lead to the creation of automated weather networks, continuously providing data, such as soil moisture and temperature enabling near-real-time decisions. Those systems rely on sensors which provide instantaneous estimates of soil moisture at discrete depths, but also can provide event detection, event ID, location sensing, and local control of actuators (devise responding to a digital signal, for instance a valve in an irrigation system). As technology continues to progress, the concept of micro-sensing and wireless connection of these nodes promise many new application areas, including the monitoring of environmental conditions and precision agriculture.

Most data used by researchers are still mostly at the 30 km scale. Those mesoscale networks, also called mesonets, were principally resulting from initiatives at the State level. Their coverage across the United States is different in intensity, but also in nature and were developed for specific diverse purposes. In addition, some of those networks operate on a fee basis to fund themselves in part or in whole.

Table 14.1. Statewide mesonet

State	Network	Total number of real-time stations
Alabama	North Alabama Climate Network	22
Alabama	University of South Alabama Mesonet (CHILI)	25
Arizona	Arizona Meteorological Network	21
Arkansas	Arkansas State Plant Board Weather Network	50
California	California Irrigation Management Information System	152
Colorado	Colorado Agricultural Meteorological Network	75
Delaware	Delaware Environmental Observing System	57
Florida	Florida Automated Weather Network	42
Georgia	Georgia Automated Weather Network	82
Illinois	Illinois Climate Network	19
Iowa	Iowa Environmental Mesonet	17
Kansas	Kansas Mesonet	51
Kentucky	Kentucky Mesonet	66
Louisiana	Lousiana Agroclimatic Information System	9
Michigan	Enviroweather	82
Minnesota	Minnesota Mesonet	8
Missouri	Missouri Mesonet	24
Nebraska	Nebraska Mesonet	68
New Jersey	New Jersey Weather and Climate Network	61
New Mexico	New Mexico Climate Network	6
New York	New York Mesonet	101
North Carolina	North Carolina ECONet	40
North Dakota	North Dakota Agricultural Weather Network	90
Oklahoma	Oklahoma Mesonet	120
South Dakota	South Dakota Mesonet	25
Texas	West Texas Mesonet	98
Utah	Utah Agricultural Weather Network	32
Washington	Washington AgWeatherNet	176
Total	1 619

Source: Mahood et al (2016).

While those networks are useful for some of applications, the understanding of a range of natural phenomenon requires a broader coverage, national if not beyond borders. In addition, the understanding of the dynamics of soil moisture requires more information than soil moisture data point estimates. It requires knowledge about soil characteristics, not only surface composition, but also composition across multiple soil depths, weather patterns, and land use information. These data layers are available but in disparate data networks and from different sources.

More recently, capacities have increased, in particular in the context of the development of new data storage, processing and aggregation capacities (big data). Today, sensors networks are more than networks creating data. Their use relies on algorithms and communication protocols. Data layers captured by sensors networks are compiled with other data to make predictions and support decision making of a range of stakeholders, from farmers to policy makers. For example, a range of environmental conditions, including information about soil moisture and nutrients, can be combined by an algorithm to estimate crop health and production quality over time. The more precise networks, such as wireless sensors networks (WSN), providing data at the farm level, can be integrated into decision systems for precision agriculture or water management. For example, a sensor can measure levels of humidity and be linked to an actuator responsible for opening an irrigation valve.

The other technology available is remote sensing, which presents the advantage of allowing contiguous data coverage across the United States.

The use of digital technologies to improve the coordination and use of soil moisture data

The problem

While a large amount of data exists and could support researchers and policy makers, it is not used to its full potential.

Sensors networks developed at the farm level are the only one providing information at a level of granularity fine enough for farmers to use for decision making, analysis and monitoring of soil moisture on farm. However, those are often private and systems are proprietary. In addition, the data is considered3 as belonging to either the farmers or the company providing the service and therefore not easily accessible by other stakeholders, including researchers and the government.4

At the level of mesonets (States), many in-situ sensors networks have been developed in the United States, but they are distributed unevenly, with some geographic areas being more densely covered than others. In addition, they are not always publically accessible and protected by paywalls. Finally, remote sensing data is gaining in popularity now that multiple high quality platforms are providing near daily products. However, data provided is still at a coarse resolution. Models can also be used to provide better spatial coverage, but they are limited by input data layers (primarily precipitation) and have water budget closure issues. Indeed, many factors influence how soil moisture varies, including soil properties, topography, vegetation, land cover or land use and climate.

The production of an accurate representation of soil moisture at an informative scale enabling policy management and monitoring and with regular coverage and comparative data on conterminous United States the 48 adjoining States plus Washington, DC on the continent of North America (CONUS) has been a challenge. This was the result of a lack of technical capacity (data processing and management) but also of the existence of parallel independent and non-coordinated developments of networks across the United States. Consequently, soil moisture observations have been poorly integrated into assessments of vulnerability, for instance early warning systems for droughts and floods.

The central issue is the traditional investment behaviour conflicts in infrastructure in networks industries (fixed networks to deliver services based on lumpy initial investments and presenting elements of natural monopoly), which calls for role of the public sector as a provider of public services. In the case of the United States, this is also a problem linked to devolution and differences in investment priorities among States. The realisation of this issue and of the opportunity cost of the lack of coordination for policy management triggered an effort to promote the better integration of soil moisture data across the United States, spawning the National Soil Moisture Network (NSMN) initiative.

Overview of the National Soil Moisture Network (NSMN) initiative5

In 2013, the National Integrated Drought Information System (NIDIS) initiated discussion for the development of a Coordinated National Soil Moisture Network. This network is intended to be based on Federal and State in situ monitoring networks, satellite remote sensing missions, and numerical modelling and draw on the experiences of the soil moisture community. Such a platform would aim to improve knowledge of soil moisture status across multiple spatial and temporal scales and over multiple soil depths.

The initiative brought together experts from the USDA Natural Resources Conservation Service’s Soil Climate Analysis Network (SCAN) and Snow Telemetry (SNOTEL) in situ instrument networks (Schaefer, Cosh and Jackson, 2007[3]); the NOAA Climate Reference Network (Diamond et al., 2013[4]); state in-situ networks; remote sensing and modelling experts from NASA, NOAA, and USDA; and soil moisture database managers from academia and federal and state governments.

The first workshop held in 2013 highlighted a range of issues in relation to soil moisture data. At the conclusion of the workshop launching the initiative (McNutt, Verdin and Darby, 2013[5]), the need was identified for improving metadata and calibration and validation of soil moisture data as well as data integration, leading to the creation of the initiative for the development of the National Soil Moisture Network (NSMN). The original objective was to develop a high-resolution gridded soil moisture resource, accessible to the public through a web portal. Therefore, the project brought together in situ measurements of soil moisture from the federal networks, in combination with a range of other databases, including the NRCS SSURGO, which provides a unique gridded database of soil properties and satellite (PRISM) data.

As a result of the first workshop, NIDIS funded a pilot project operated by the North American Soil Moisture Database at Texas A&M University, focusing on the southern plains networks of the U.S. to demonstrate the feasibility of the process. Challenges included data transfer protocols, storage, and data gaps from intermittent connectivity to stations.

Additional NSMN workshops were held in 2016, 2017 and 2018, with the last meeting held in Lincoln Nebraska at the National Drought Mitigation Center. As a result of this workshop, an Executive Committee was formed and a charter was drafted to provide more structure to the effort.

Rationale and lessons learned from the development of a National Soil Moisture Network6

According to the type of public services required, to the institutional environment and initial conditions of the network, enabling the development of a data infrastructure might require different types of actions and role for the government, whether as a central planner, as a regulator setting interoperability standards or to directly develop the data infrastructure and create markets for usage rights. The role of the government is likely to change according to how advanced those networks are, whether the service provided can be marketable and whether the costs of providing the services are higher than the benefits.7

Lesson 1. There is path dependency from previous policy-making and infrastructure development on the data infrastructure and governance

In the case of the NSMN, issues result from path dependency from the devolution of investments decisions to sub-national scales, investments that were taken rationally according to priorities in each particular State. While there was a logic in this devolution, today, this lack of coordination and alignment of objectives creates inefficiencies in terms of data creation, management, potentially spiralling down into poor water policies and managements in a context of information gap.

A lot of data is already being produced which could be better exploited by cutting across administrative borders. A maintenance updating of the current mesonet infrastructure and the creation of an enabling environment for data sharing might be enough for the creation of an efficient data infrastructure. Both the public and private sectors – Farmers, policy makers and the community – would benefit from better preparation and resilience to drought.

The NSMN initiative acted as a catalyser, bringing together institutions and creating awareness about the specificity of soil moisture monitoring, highly variable in space, depth and time.

Lesson 2. There could be a role for the government to support the development of infrastructure

A traditional question for network infrastructure is where the role of government stops and where the role of private sector starts. There might be cases requiring broader government support to the financing of network infrastructure, including in less economically important areas.

In addition, questions about the sharing of data according to the status, definition and value (economic and social) provided to the different users of data produced by private systems is still to be discussed. Discussions at the Global Forum for Agriculture held at the OECD in May 20188 highlighted a range of diverging views in relation to data ownership, and the issue of privacy. In particular, asking the question of which types of information and derived conclusions can be left with the private sector and which need to be managed (governed) by public authorities.

The NSMN is in the process of using existing networks, but most current networks were created with spatially restricted objectives (states, watersheds, etc.) and the concluding report of the workshop, which launched the creation of the NSMN in 2013 (McNutt, Verdin and Darby, 2013[5]) mentioned that an increase in the number of monitoring sites would be the most important improvement in the overall depiction of soil moisture. Indeed, drought risk and water flows do not end at regional borders nor are they only an issue within one land cover group, and more specifically highly productive areas: water management, policies and drought risk require a comprehensive understanding of soil water dynamics at a high resolution across all landscapes.

To be considered too is that new sources of data will likely become available in the future with the digital transformation of agriculture. Although the sharing of data between stakeholders is not a given, the private sector might have incentives to develop infrastructure when it holds promises of profitability. For instance, WSN can be developed to support decision systems, more likely to be sold to farmers. Consequently, WSN would produce a lot of information, in specific areas, such as high density farming areas, to which services can be provided to make investment profitable. However, there might still be constraints to the sharing of such data. In addition, the quality and veracity of data obtained via private application of new digital technologies to support policy-making would need to be ensured.

Finally, the creation of a network of sensors and of information needed to monitor soil moisture in ways that allow the provision of public services such as drought early warning systems, and to inform water policy and management, requires coverage of all geographic areas, whether cultivated or not. Such cases identify a clear role for the government as it might not be economically viable for the private sector to develop infrastructure in some areas, which are nevertheless important for the understanding of ecosystems dynamics and forecasting. In addition, myopic views of hydrologic concerns have major impacts downstream.

Lesson 3. There might be a role for some regulatory oversight or central planners when there is a collective gain to coordination but no returns or private incentives

This leads to the second issue highlighted by this case study, which is interoperability standards. Any network or platform, whether public or private is developed to answer specific questions, achieve certain purposes, explaining their difference of approach to data creation, management and codification. As a consequence, the data produced might not be “fit for purpose” and create biases if used in modelling and analytics that depart from the initial goals. While there can be collective gains to coordination, there might not necessarily be private or return incentives. In such cases, networks lend themselves to some form of regulatory oversight or a central planner.9

With the creation of the NSMN, an important need was for the data produced to be usable for a diversity of objectives and by a diversity of end users. The first step of the NSMN was the coordination of existing networks, bringing together current entities in a common format. As such, the NSMN also acted as a standards marker: effectively leveraging the full variety of existing networks and modelling efforts first relied on consistent calibration and validation practices and metadata characterisation.

References

[4] Diamond, H. et al. (2013), “U.S. Climate Reference Network after One Decade of Operations: Status and Assessment”, Bulletin of the American Meteorological Society, Vol. 94/4, pp. 485-498, http://dx.doi.org/10.1175/bams-d-12-00170.1.

[2] Fiebrich, C. (2009), “The History of Surface Weather Observations in the United States”, Earth-Science Reviews, Vol. 93/April, pp. 77-84, http://dx.doi.org/10.1016/j.earscirev.2009.01.001.

[1] Mahmood, R. et al. (2017), “Mesonets: Mesoscale Weather and Climate Observations for the United States”, Bulletin of the American Meteorological Society, Vol. 98/7, pp. 1349-1361, http://dx.doi.org/10.1175/bams-d-15-00258.1.

[5] McNutt, C., J. Verdin and L. Darby (2013), Developing a Coordinated National Soil Moisture Network, https://www.drought.gov/drought/sites/drought.gov.drought/files/media/reports/rpt-2013-Developing-Coordinated-National-Soil-Moisture-Network.pdf.

[3] Schaefer, G., M. Cosh and T. Jackson (2007), “The USDA Natural Resources Conservation Service Soil Climate Analysis Network (SCAN)”, Journal of Atmospheric and Oceanic Technology, Vol. 24/12, pp. 2073-2077, http://dx.doi.org/10.1175/2007jtecha930.1.

Notes

← 1.  Roughly spanning a 30 km radius or grid box around a given location

← 2.  The foundation for today’s National Weather Service Cooperative Observing Program (COOP).

← 3.  See section discussing data ownership and the example of the Privacy and Security Principles for Farm Data in the United States in main report.

← 4.  See main report for a discussion about data ownership. In the United-States, farmers are reluctant to share data about their production system.

← 5.  Information about the development of the NSMN was taken from sources available on the website of the National Integrated Drought Information System, https://www.drought.gov/drought/sites/drought.gov.drought/.

← 6.  Or worldwide but this would go further than the premises of this document which is focusing on domestic policies. Initiatives at the G20 are aiming to develop such knowledge sharing international networks. See G20 Meeting of Agriculture Ministers declaration, 27-28 July 2018, Buenos Aires, Argentina: “We highlight the importance of enhancing the quantity and quality of soil data and information and support the sharing of knowledge and technology to measure, restore, rejuvenate and maintain soil health.”

← 7.  Although cost-benefit analysis are often difficult to implement for issues in relation to the environment and disaster risk reduction.

← 8.  Global Forum on Agriculture (GFA), 14-15 May 2018, Digital technologies in food and agriculture: Reaping the benefits. http://www.oecd.org/agriculture/events/oecd-global-forum-on-agriculture/.

← 9.  WMO is in the process of developing standards for soil moisture network development.