COSMOS-UK API - Example Python snippets for accessing data

This notebook aims to provide a basic introduction in how you might interact with the COSMOS-UK API programatically, for example to automate the process of bringing COSMOS-UK data into your workflows.

More information about COSMOS-UK can be found on our website: https://cosmos.ceh.ac.uk/.

A detailed general user guide for COSMOS-UK, including information about sensors, quality control checks and other information about how our data is collected, is available here: https://cosmos.ceh.ac.uk/resources/user_guide.

Imports and helper functions

Here, we are just importing the required Python modules and defining some static helper functions that will be used throughout the rest of the code.

[1]:

from datetime import datetime
import io
import json
import requests
import zipfile

import pandas as pd

[2]:

BASE_URL = 'https://cosmos-api.ceh.ac.uk'

[3]:

def get_api_response(url, csv=False):
    """ Helper function to send request to API and get the response

    :param str url: The URL of the API request
    :param bool csv: Whether this is a CSV request. Default False.
    :return: API response
    """
    # Send request and read response
    print(url)
    response = requests.get(url)

    if csv:
        return response
    else:
        # Decode from JSON to Python dictionary
        return json.loads(response.content)

Data collections

The COSMOS-UK API can provide data for sites at two time steps, known as ‘collections’:

  • 30 minutes (30M)

  • Daily (1D)

Each collection has their own set of variables (known as ‘parameters’) that data is available for. You can find these parameters by interrogating the collection metadata with a call to the root collection URL. The parameter listing contains various information about each parameter that may be useful.

[4]:

def get_collection_parameter_info(params):
    """ A function for wrangling the collection information into a more visually appealing format!
    """
    df = pd.DataFrame.from_dict(params)
    df = df.T[['label', 'description', 'unit', 'sensorInfo']]

    df['unit_symbol'] = df['unit'].apply(lambda x: x['symbol']['value'])
    df['unit_label'] = df['unit'].apply(lambda x: x['label'])
    df['sensor_depth'] = df['sensorInfo'].apply(lambda x: None if pd.isna(x) else x['sensor_depth']['value'])

    df = df.drop(['sensorInfo', 'unit'], axis=1)

    return df

30 minute (30M) parameters

[5]:

collection_30M_url = f'{BASE_URL}/collections/30M'
collection_30M_meta = get_api_response(collection_30M_url)

# Get the information about the parameter names from the metadata dictionary
collection_30M_params = collection_30M_meta['parameter_names']

collection_30M_params_df = get_collection_parameter_info(collection_30M_params)
display(collection_30M_params_df)

https://cosmos-api.ceh.ac.uk/collections/30M
label description unit_symbol unit_label sensor_depth
g1 Soil Heat Flux 1 30 minute soil heat flux from heat flux plate 1 W m-2 watts per square metre None
g2 Soil Heat Flux 2 30 minute soil heat flux from heat flux plate 2 W m-2 watts per square metre None
lwin Longwave Radiation - Incoming 30 minute incoming longwave radiation W m-2 watts per square metre None
lwout Longwave Radiation - Outgoing 30 minute outgoing longwave radiation W m-2 watts per square metre None
pa Atmospheric Pressure 30 minute atmospheric pressure hPa hectopascal None
precip Precipitation (Pluvio) 30 minute precipitation from pluvio gauge mm millimetre None
precip_raine Precipitation (rainE) 30 minute precipitation from rainE gauge mm millimetre None
precip_tipping Precipitation (Tipping Bucket) 30 minute precipitation from tipping bucket gauge mm millimetre None
q Absolute Humidity 30 minute absolute humidity g m-3 gram per cubic metre None
rh Relative Humidity 30 minute relative humidity % percent None
rn Net Radiation 30 minute net radiation W m-2 watts per square metre None
snow_depth Snow Depth 30 minute snow depth (Only given on snow days) mm millimetre None
stp_tsoil10 STP 03 Soil Temperature at 10cm 30 minute soil temperature at 10cm. STP sensor °C degrees Celsius 10
stp_tsoil2 STP 01 Soil Temperature at 2cm 30 minute soil temperature at 2cm. STP sensor °C degrees Celsius 2
stp_tsoil20 STP 04 Soil Temperature at 20cm 30 minute soil temperature at 20cm. STP sensor °C degrees Celsius 20
stp_tsoil5 STP 02 Soil Temperature at 5cm 30 minute soil temperature at 5cm. STP sensor °C degrees Celsius 5
stp_tsoil50 STP 05 Soil Temperature at 50cm 30 minute soil temperature at 50cm. STP sensor °C degrees Celsius 50
swin Shortwave Radiation - Incoming 30 minute incoming shortwave radiation W m-2 watts per square metre None
swout Shortwave Radiation - Outgoing 30 minute outgoing shortwave radiation W m-2 watts per square metre None
ta Air Temperature 30 minute air temperature °C degrees Celsius None
tdt10_tsoil TDT 10 - Soil Temperature at 50cm 30 minute soil temperature at 50cm*. TDT senso... °C degrees Celsius 50
tdt10_vwc TDT 10 - Volumetric Water Content at 50cm 30 minute volumetric water content at 50cm*. T... % percent 50
tdt1_tsoil TDT 01 - Soil Temperature at 10cm 30 minute soil temperature at 10cm. TDT sensor 1 °C degrees Celsius 10
tdt1_vwc TDT 01 - Volumetric Water Content at 10cm 30 minute volumetric water content at 10cm. TD... % percent 10
tdt2_tsoil TDT 02 - Soil Temperature at 10cm 30 minute soil temperature at 10cm. TDT sensor 2 °C degrees Celsius 10
tdt2_vwc TDT 02 - Volumetric Water Content at 10cm 30 minute volumetric water content at 10cm. TD... % percent 10
tdt3_tsoil TDT 03 - Soil Temperature at 5cm 30 minute soil temperature at 5cm. TDT sensor 3 °C degrees Celsius 5
tdt3_vwc TDT 03 - Volumetric Water Content at 5cm 30 minute volumetric water content at 5cm. TDT... % percent 5
tdt4_tsoil TDT 04 - Soil Temperature at 5cm 30 minute soil temperature at 5cm. TDT sensor 4 °C degrees Celsius 5
tdt4_vwc TDT 04 - Volumetric Water Content at 5cm 30 minute volumetric water content at 5cm. TDT... % percent 5
tdt5_tsoil TDT 05 - Soil Temperature at 15cm 30 minute soil temperature at 15cm. TDT sensor 5 °C degrees Celsius 15
tdt5_vwc TDT 05 - Volumetric Water Content at 15cm 30 minute volumetric water content at 15cm. TD... % percent 15
tdt6_tsoil TDT 06 - Soil Temperature at 15cm 30 minute soil temperature at 15cm. TDT sensor 6 °C degrees Celsius 15
tdt6_vwc TDT 06 - Volumetric Water Content at 15cm 30 minute volumetric water content at 15cm. TD... % percent 15
tdt7_tsoil TDT 07 - Soil Temperature at 25cm 30 minute soil temperature at 25cm. TDT sensor 7 °C degrees Celsius 25
tdt7_vwc TDT 07 - Volumetric Water Content at 25cm 30 minute volumetric water content at 25cm. TD... % percent 25
tdt8_tsoil TDT 08 - Soil Temperature at 25cm 30 minute soil temperature at 25cm. TDT sensor 8 °C degrees Celsius 25
tdt8_vwc TDT 08 - Volumetric Water Content at 25cm 30 minute volumetric water content at 25cm. TD... % percent 25
tdt9_tsoil TDT 09 - Soil Temperature at 50cm 30 minute soil temperature at 50cm*. TDT senso... °C degrees Celsius 50
tdt9_vwc TDT 09 - Volumetric Water Content at 50cm 30 minute volumetric water content at 50cm*. T... % percent 50
ux Wind Speed - X Component 30 minute x component of wind speed m/s metres per second None
uy Wind Speed - Y Component 30 minute y component of wind speed m/s metres per second None
uz Wind Speed - Z Component 30 minute z component of wind speed m/s metres per second None
wd Wind Direction 30 minute wind direction ° degree (angle) None
ws Wind Speed 30 minute wind speed m/s metres per second None

Daily (1D) parameters

[6]:

collection_1D_url = f'{BASE_URL}/collections/1D'
collection_1D_meta = get_api_response(collection_1D_url)

# Get the information about the parameter names from the metadata dictionary
collection_1D_params = collection_1D_meta['parameter_names']

collection_1D_params_df = get_collection_parameter_info(collection_1D_params)
display(collection_1D_params_df)

https://cosmos-api.ceh.ac.uk/collections/1D
label description unit_symbol unit_label sensor_depth
albedo Albedo Mean albedo value between 10:00 and 14:00 1 dimensionless None
cosmos_vwc COSMOS Volumetric Water Content Daily volumetric water content (soil moisture)... % percent None
cts_mod_corr COSMOS Neutron Counts (corrected) Daily corrected counts from CRNS tube count count None
d86_75m D86 75M Daily effective depth of COSMOS VWC measuremen... cm centimetre None
g1 Soil Heat Flux 1 Daily soil heat flux from heat flux plate 1 W m-2 watts per square metre None
g2 Soil Heat Flux 2 Daily soil heat flux from heat flux plate 2 W m-2 watts per square metre None
lwin Longwave Radiation - Incoming Daily incoming longwave radiation MJ m-2 day-1 megajoule per square metre per day None
lwout Longwave Radiation - Outgoing Daily outgoing longwave radiation MJ m-2 day-1 megajoule per square metre per day None
pa Atmospheric Pressure Daily atmospheric pressure hPa hectopascal None
pe Potential Evaporation Daily potential evaporation mm millimetre None
precip Precipitation (Pluvio) Daily precipitation from pluvio gauge mm millimetre None
precip_raine Precipitation (rainE) Daily precipitation from rainE gauge mm millimetre None
precip_tipping Precipitation (Tipping Bucket) Daily precipitation from tipping bucket gauge mm millimetre None
q Absolute Humidity Daily absolute humidity g m-3 gram per cubic metre None
rh Relative Humidity Daily relative humidity % percent None
rn Net Radiation Daily net radiation MJ m-2 day-1 megajoule per square metre per day None
snow Snow Whether or not snow occurred (Daily) 1 dimensionless None
snow_depth Snow Depth Daily snow depth (Only given on snow days) mm millimetre None
stp_tsoil10 STP 03 Soil Temperature at 10cm Daily soil temperature at 10cm. STP sensor °C degrees Celsius 10
stp_tsoil2 STP 01 Soil Temperature at 2cm Daily soil temperature at 2cm. STP sensor °C degrees Celsius 2
stp_tsoil20 STP 04 Soil Temperature at 20cm Daily soil temperature at 20cm. STP sensor °C degrees Celsius 20
stp_tsoil5 STP 02 Soil Temperature at 5cm Daily soil temperature at 5cm. STP sensor °C degrees Celsius 5
stp_tsoil50 STP 05 Soil Temperature at 50cm Daily soil temperature at 50cm. STP sensor °C degrees Celsius 50
swe_crns Snow Water Equivalent Daily snow water equivalent (only given on sno... mm millimetre None
swin Shortwave Radiation - Incoming Daily incoming shortwave radiation MJ m-2 day-1 megajoule per square metre per day None
swout Shortwave Radiation - Outgoing Daily outgoing shortwave radiation MJ m-2 day-1 megajoule per square metre per day None
ta Air Temperature Daily air temperature °C degrees Celsius None
ta_max Air Temperature - Maximum Daily maximum air temperature °C degrees Celsius None
ta_min Air Temperature - Minimum Daily minimum air temperature °C degrees Celsius None
tdt10_tsoil TDT 10 - Soil Temperature at 50cm Daily soil temperature at 50cm*. TDT sensor 10... °C degrees Celsius 50
tdt10_vwc TDT 10 - Volumetric Water Content at 50cm Daily volumetric water content at 50cm*. TDT s... % percent 50
tdt1_tsoil TDT 01 - Soil Temperature at 10cm Daily soil temperature at 10cm. TDT sensor 1 °C degrees Celsius 10
tdt1_vwc TDT 01 - Volumetric Water Content at 10cm Daily volumetric water content at 10cm. TDT se... % percent 10
tdt2_tsoil TDT 02 - Soil Temperature at 10cm Daily soil temperature at 10cm. TDT sensor 2 °C degrees Celsius 10
tdt2_vwc TDT 02 - Volumetric Water Content at 10cm Daily volumetric water content at 10cm. TDT se... % percent 10
tdt3_tsoil TDT 03 - Soil Temperature at 5cm Daily soil temperature at 5cm. TDT sensor 3 °C degrees Celsius 5
tdt3_vwc TDT 03 - Volumetric Water Content at 5cm Daily volumetric water content at 5cm. TDT sen... % percent 5
tdt4_tsoil TDT 04 - Soil Temperature at 5cm Daily soil temperature at 5cm. TDT sensor 4 °C degrees Celsius 5
tdt4_vwc TDT 04 - Volumetric Water Content at 5cm Daily volumetric water content at 5cm. TDT sen... % percent 5
tdt5_tsoil TDT 05 - Soil Temperature at 15cm Daily soil temperature at 15cm. TDT sensor 5 °C degrees Celsius 15
tdt5_vwc TDT 05 - Volumetric Water Content at 15cm Daily volumetric water content at 15cm. TDT se... % percent 15
tdt6_tsoil TDT 06 - Soil Temperature at 15cm Daily soil temperature at 15cm. TDT sensor 6 °C degrees Celsius 15
tdt6_vwc TDT 06 - Volumetric Water Content at 15cm Daily volumetric water content at 15cm. TDT se... % percent 15
tdt7_tsoil TDT 07 - Soil Temperature at 25cm Daily soil temperature at 25cm. TDT sensor 7 °C degrees Celsius 25
tdt7_vwc TDT 07 - Volumetric Water Content at 25cm Daily volumetric water content at 25cm. TDT se... % percent 25
tdt8_tsoil TDT 08 - Soil Temperature at 25cm Daily soil temperature at 25cm. TDT sensor 8 °C degrees Celsius 25
tdt8_vwc TDT 08 - Volumetric Water Content at 25cm Daily volumetric water content at 25cm. TDT se... % percent 25
tdt9_tsoil TDT 09 - Soil Temperature at 50cm Daily soil temperature at 50cm*. TDT sensor 9 ... °C degrees Celsius 50
tdt9_vwc TDT 09 - Volumetric Water Content at 50cm Daily volumetric water content at 50cm*. TDT s... % percent 50
wd Wind Direction Daily wind direction ° degree (angle) None
ws Wind Speed Daily wind speed m/s metres per second None

Data flags parameters

Alongside each data parameter (as listed above), there is a corresponding ‘flag’ parameter. The flag parameters mirror the structure of their data parameter data part, and hold information about the data value itself. There are currently three values that the flag parameters can be:

M : Missing - Where data has been lost and not infilled.

I : Infilled - Where missing data has been filled in directly using an infill method. The only infilling method currently in use is interpolation, used for gaps smaller than 10 values. More information is available in the supporting documentation of the EIDC data holding.

E : Estimated - Where the value contains some degree of uncertainty. This can be for one of two reasons:

  • The value was aggregated with less than a full set of data. For example, a daily mean temperature where some of the sub daily values were missing.

  • The value was aggregated or derived with some degree of infilled data. For example, if net radiation was calculated using an infilled short wave radiation value.

A null value in the flag parameter indicates a ‘normal’ data value.

Note: The default is not to return the flag parameters alongside the data parameters in order to reduce the size of the request. You can use the query parameter flags=true to include the flag parameters. See below for examples.

Site list

Both the 30M and 1D data collections contain data for the same set of COSMOS-UK sites. You can get information about the sites from calling to the /locations endpoint of each collection (shown below for the 1D collection).

Note: ‘End date’ in the table below is the last available date there is data available for each site. For currently open sites, this will be the date this notebook was created. For closed sites, this will be the date the site was closed.

[7]:

site_info_url = f'{BASE_URL}/collections/1D/locations'
site_info_response = get_api_response(site_info_url)

site_info = {}
for site in site_info_response['features']:
    site_id = site['id']
    site_name = site['properties']['label']
    coordinates = site['geometry']['coordinates']
    date_range = site['properties']['datetime']
    start_date, end_date = date_range.split('/')

    other_info = site['properties']['siteInfo']
    other_info = {key: d['value'] for key, d in other_info.items()}

    site_info[site_id] = {'site_name': site_name,
                          'coordinates': coordinates,
                          'start_date': start_date,
                          'end_date': end_date} | other_info

site_info_df = pd.DataFrame.from_dict(site_info).T
display(site_info_df)

https://cosmos-api.ceh.ac.uk/collections/1D/locations
site_name coordinates start_date end_date altitude bulk_density bulk_density_sd land_cover lattice_water lattice_water_sd soil_organic_carbon soil_organic_carbon_sd soil_type
ALIC1 Alice Holt [51.153551, -0.858232] 2015-03-06T13:30:00Z 2023-08-07T00:00:00Z 80.0 0.84 None Broadleaf woodland 0.025 None 0.042 None Mineral soil
BALRD Balruddery [56.482297, -3.1114881] 2014-05-15T17:30:00Z 2023-08-07T00:00:00Z 130.0 1.35 0.14 Arable and horticulture 0.018 0.00066 0.023 0.00076 Mineral soil
BICKL Bickley Hall [53.02635, -2.7005297] 2015-01-28T17:00:00Z 2023-08-07T00:00:00Z 78.0 1.25 0.23 Grassland 0.01 0.00042 0.02 0.0036 Mineral soil
BUNNY Bunny Park [52.86073, -1.12685] 2015-01-27T00:30:00Z 2023-08-07T00:00:00Z 39.0 1.53 0.11 Arable and horticulture 0.008 0.0012 0.016 0.0015 Mineral soil
CARDT Cardington [52.105601, -0.424644] 2015-06-24T10:00:00Z 2023-08-07T00:00:00Z 29.0 1.1 0.17 Improved grassland 0.016 0.0018 0.04 0.0052 Mineral soil
CGARW Cwm Garw [51.951295, -4.746634] 2016-06-29T11:00:00Z 2023-08-07T00:00:00Z 299.0 0.79 0.26 Improved grassland 0.022 0.00086 0.048 0.0015 Mineral soil
CHIMN Chimney Meadows [51.708021, -1.4787658] 2013-10-02T13:30:00Z 2023-08-07T00:00:00Z 65.0 1.32 0.095 Improved grassland 0.011 0.0018 0.027 0.0016 Calcareous mineral soil
CHOBH Chobham Common [51.367821, -0.597484] 2015-02-24T14:00:00Z 2023-08-07T00:00:00Z 47.0 0.73 0.37 Heather grassland 0.003 0.0011 0.031 0.0084 Organic soil over mineral soil
COCHN Cochno [55.941421, -4.4035431] 2017-08-23T00:00:00Z 2020-11-16T00:00:00Z 168.0 0.76 0.21 Improved grassland 0.019 0.0018 0.068 0.014 Mineral soil
COCLP Cockle Park [55.216013, -1.6943736] 2014-11-21T14:30:00Z 2023-08-07T00:00:00Z 87.0 1.18 0.084 Arable and horticulture 0.02 0.0015 0.033 0.0018 Mineral soil
CRICH Crichton [55.043057, -3.583386] 2014-12-02T15:00:00Z 2023-08-07T00:00:00Z 42.0 1.1 0.15 Arable and horticulture 0.011 0.0012 0.045 0.0068 Mineral soil
EASTB Easter Bush [55.867392, -3.207115] 2014-08-13T11:30:00Z 2023-08-07T00:00:00Z 208.0 1.07 0.19 Improved grassland 0.019 0.0015 0.033 0.0035 Mineral soil
ELMST Elmsett [52.094647, 0.9930645] 2016-08-11T15:30:00Z 2023-08-07T00:00:00Z 76.0 1.2 0.18 Arable and horticulture 0.015 0.00096 0.022 0.0041 Calcareous mineral soil
EUSTN Euston [52.383178, 0.7847112] 2016-03-31T16:00:00Z 2023-08-07T00:00:00Z 18.0 1.18 0.2 Improved grassland 0.003 0.00026 0.029 0.0023 Mineral soil
FINCH Fincham [52.617773, 0.5107284] 2017-06-07T11:30:00Z 2023-08-07T00:00:00Z 15.0 1.26 0.13 Arable and horticulture 0.007 0.00055 0.02 0.0011 Calcareous mineral soil
FIVET Fivemiletown [54.298468, -7.291954] 2018-06-26T15:30:00Z 2023-08-07T00:00:00Z 174.0 0.86 0.24 Arable and horticulture 0.014 0.0019 0.039 0.0095 Mineral soil
GISBN Gisburn Forest [54.023711, -2.384689] 2014-08-15T12:00:00Z 2023-08-07T00:00:00Z 246.0 0.71 0.29 Coniferous woodland 0.021 0.0037 0.061 0.0035 Mineral soil
GLENS Glensaugh [56.914403, -2.5621547] 2014-05-14T11:30:00Z 2023-08-07T00:00:00Z 399.0 0.36 0.33 Heather 0.014 0.0034 0.203 0.059 Organic soil
GLENW Glenwherry [54.838087, -6.0045994] 2016-06-15T15:00:00Z 2023-08-07T00:00:00Z 274.0 0.58 0.24 Improved grassland 0.024 0.0019 0.153 0.011 Organic soil
HADLW Hadlow [51.228582, 0.320276] 2016-10-27T14:30:00Z 2023-08-07T00:00:00Z 33.0 1.1 0.15 Improved grassland 0.028 0.0019 0.031 0.0014 Mineral soil
HARTW Hartwood Home [55.810254, -3.828995] 2014-05-20T13:30:00Z 2023-08-07T00:00:00Z 225.0 1.02 0.15 Improved grassland 0.033 0.0066 0.043 0.0037 Mineral soil
HARWD Harwood Forest [55.216677, -2.025488] 2015-05-22T12:00:00Z 2022-06-28T09:00:00Z 300.0 0.35 0.23 Coniferous forest 0.009 0.0034 0.304 0.1 Organic soil
HENFS Henfaes Farm [53.225198, -4.01255] 2015-12-17T13:00:00Z 2023-08-07T00:00:00Z 287.0 0.86 0.26 Acid grassland 0.022 0.001 0.077 0.0041 Mineral soil
HILLB Hillsborough [54.446959, -6.068526] 2016-06-14T17:00:00Z 2023-08-07T00:00:00Z 146.0 1.11 0.18 Improved grassland 0.021 0.0042 0.042 0.0052 Mineral soil
HLACY Holme Lacy [52.02088, -2.6621054] 2018-04-11T00:00:00Z 2023-08-07T00:00:00Z 76.0 1.27 0.16 Arable and horticulture 0.017 0.00084 0.022 0.0013 Mineral soil
HOLLN Hollin Hill [54.110665, -0.959477] 2014-03-25T12:30:00Z 2023-08-07T00:00:00Z 82.0 0.96 0.18 Improved grassland 0.025 0.0034 0.032 0.012 Mineral soil
HYBRY Heytesbury [51.20277, -2.079616] 2017-08-16T12:30:00Z 2023-08-07T00:00:00Z 166.0 0.78 0.27 Calcareous grassland 0.006 0.0021 0.066 0.011 Calcareous mineral soil
LIZRD The Lizard [50.03266, -5.19999] 2014-10-17T12:00:00Z 2023-08-07T00:00:00Z 85.0 0.78 0.3 Grassland 0.014 0.0023 0.058 0.027 Mineral soil
LODTN Loddington [52.610159, -0.8264188] 2016-04-26T15:00:00Z 2023-08-07T00:00:00Z 186.0 1.07 0.1 Arable and horticulture 0.041 0.0023 0.036 0.0023 Mineral soil
LULLN Lullington Heath [50.79372, 0.18887] 2014-12-16T15:30:00Z 2023-08-07T00:00:00Z 119.0 0.8 0.22 Calcareous grassland 0.006 0.0017 0.043 0.011 Calcareous mineral soil
MOORH Moor House [54.659417, -2.4678] 2014-12-04T11:30:00Z 2023-08-07T00:00:00Z 565.0 0.61 0.38 Acid grassland 0.014 0.0025 0.076 0.029 Mineral soil
MOREM Moreton Morrell [52.199407, -1.563081] 2018-11-15T00:00:00Z 2023-08-07T00:00:00Z 53.0 1.09 0.23 Improved grassland 0.026 0.0038 0.035 0.0053 Mineral soil
MORLY Morley [52.548146, 1.0342313] 2014-05-14T05:00:00Z 2023-08-07T00:00:00Z 55.0 1.48 0.2 Arable and horticulture 0.016 0.002 0.017 0.0038 Mineral soil
NWYKE North Wyke [50.773479, -3.905963] 2014-10-16T09:00:00Z 2023-08-07T00:00:00Z 181.0 1.16 0.15 Arable and horticulture 0.02 0.0014 0.037 0.0021 Mineral soil
PLYNL Plynlimon [52.453337, -3.762572] 2014-11-05T12:30:00Z 2023-08-07T00:00:00Z 542.0 0.61 0.33 Acid grassland 0.02 0.0015 0.098 0.0097 Organic soil
PORTN Porton Down [51.120071, -1.681482] 2014-12-18T10:30:00Z 2023-08-07T00:00:00Z 146.0 0.91 0.14 Improved grassland 0.004 0.0011 0.049 0.0059 Calcareous mineral soil
RDMER Redmere [52.44577, 0.42104] 2015-02-10T10:30:00Z 2018-09-20T00:00:00Z 3.0 0.59 0.097 Arable and horticulture 0.056 0.0059 0.238 0.053 Organic soil
REDHL Redhill [51.26287, 0.4291348] 2016-02-18T16:00:00Z 2023-08-07T00:00:00Z 91.0 1.23 0.15 Orchard 0.011 0.00076 0.024 0.0026 Calcareous mineral soil
RISEH Riseholme [53.261647, -0.5259074] 2016-05-04T16:00:00Z 2023-08-07T00:00:00Z 53.0 1.15 0.18 Improved grassland 0.022 0.0051 0.032 0.0043 Calcareous mineral soil
ROTHD Rothamsted [51.813787, -0.378304] 2014-07-25T08:00:00Z 2023-08-07T00:00:00Z 131.0 1.32 0.15 Arable and horticulture 0.018 0.0011 0.021 0.0036 Mineral soil
SHEEP Sheepdrove [51.530244, -1.481903] 2013-10-24T14:30:00Z 2023-08-07T00:00:00Z 170.0 1.03 0.14 Arable and horticulture 0.027 0.006 0.059 0.0086 Mineral soil
SOURH Sourhope [55.479876, -2.229989] 2014-11-19T11:00:00Z 2023-08-07T00:00:00Z 487.0 0.55 0.29 Acid grassland 0.021 0.0034 0.086 0.041 Mineral soil
SPENF Spen Farm [53.86886, -1.31886] 2016-11-23T10:00:00Z 2023-08-07T00:00:00Z 57.0 1.49 0.19 Arable and horticulture 0.011 0.0013 0.019 0.002 Calcareous mineral soil
STGHT Stoughton [52.601667, -1.047046] 2015-08-19T06:30:00Z 2023-08-07T00:00:00Z 130.0 1.35 0.14 Arable and horticulture 0.018 0.0028 0.027 0.0071 Mineral soil
STIPS Stiperstones [52.581248, -2.9446865] 2014-11-06T15:00:00Z 2023-08-07T00:00:00Z 432.0 0.52 0.25 Improved grassland 0.016 0.004 0.104 0.036 Organic soil
SYDLG Sydling [50.828373, -2.527891] 2018-11-27T00:00:00Z 2023-08-07T00:00:00Z 249.0 1.05 0.19 Improved grassland 0.02 0.0015 0.035 0.0029 Mineral soil
TADHM Tadham Moor [51.207417, -2.82881] 2014-10-14T12:00:00Z 2023-08-07T00:00:00Z 7.0 0.31 0.093 Improved grassland 0.029 0.0033 0.314 0.059 Organic soil
WADDN Waddesdon [51.839436, -0.948405] 2013-11-04T18:30:00Z 2023-08-07T00:00:00Z 98.0 1.01 0.16 Improved grassland 0.021 0.0021 0.034 0.0023 Mineral soil
WIMPL Wimpole [52.132078, -0.044411] 2019-09-10T00:00:00Z 2023-08-07T00:00:00Z 30.0 1.18 0.16 Arable and horticulture 0.015 0.0039 0.035 0.0021 Mineral soil
WRTTL Writtle [51.733896, 0.417923] 2017-07-04T10:30:00Z 2023-08-07T00:00:00Z 44.0 1.15 0.23 Improved grassland 0.019 0.0013 0.035 0.0033 Mineral soil
WYTH1 Wytham Woods [51.77728, -1.33846] 2013-11-26T00:30:00Z 2016-10-01T00:00:00Z 109.0 0.96 0.19 Broadleaf woodland 0.017 0.0047 0.028 0.0044 Mineral soil

Data queries

To get hold of the data for COSMOS-UK sites, there are a number of different ‘queries’ that you can use against each data collection. They are:

  • Location / site ID

  • Position

  • Cube

  • Radius

JSON response

Data responses can either be in JSON or CSV format. First we will look at the JSON response - this is the default. The JSON response is essentially loaded as a nested Python dictionary. We can access the parts of the dictionary we need to build a dataframe of the COSMOS-UK data. The following is a function that we can re-use to wrangle the dictionary into a Pandas dataframe:

[8]:

def read_json_collection_data(json_response):
    """ Wrangle the response JSON from a COSMOS-API data collection request into a more usable format - in this case a Pandas Dataframe

    :param dict json_response: The JSON response dictionary returned from a COSMOS-API data collection request
    :return: Dataframe of data
    :rtype: pd.DataFrame
    """
    # The response is a list of dictionaries, one for each requested site

    # You can choose how you want to build your dataframes.  Here, I'm just loading all stations into one big dataframe.
    # But you could modify this for your own use cases.  For example you might want to build a dictionary of {site_id: dataframe}
    # to keep site data separate, etc.
    master_df = pd.DataFrame()

    for site_data in resp['coverages']:
        # Read the site ID
        site_id = site_data['dct:identifier']

        # Read the time stamps of each data point
        time_values = pd.DatetimeIndex(site_data['domain']['axes']['t']['values'])

        # Now read the values for each requested parameter at each of the time stamps
        param_values = {param_name: param_data['values'] for param_name, param_data in site_data['ranges'].items()}

        # And put everything into a dataframe
        site_df = pd.DataFrame.from_dict(param_values)
        site_df['datetime'] = time_values
        site_df['site_id'] = site_id

        site_df = site_df.set_index(['datetime', 'site_id'])
        master_df = pd.concat([master_df, site_df])

    return master_df

Location (site ID) query

/collections/<collection_id>/locations/<site_id>

The location query fetches data for a given site ID. You can constrain the query by specifing a given date or date range and one or more parameter names (see section below)

The default without additional query parameters is to return data for all parameters, for the latest single timestep available.

Single site ID

[9]:

site_id = 'CHIMN'
query_url = f'{BASE_URL}/collections/1D/locations/{site_id}'
resp = get_api_response(query_url)

df = read_json_collection_data(resp)
display(df)

https://cosmos-api.ceh.ac.uk/collections/1D/locations/CHIMN
albedo cosmos_vwc cts_mod_corr d86_75m g1 g2 lwin lwout pa pe ... tdt6_tsoil tdt6_vwc tdt7_tsoil tdt7_vwc tdt8_tsoil tdt8_vwc tdt9_tsoil tdt9_vwc wd ws
datetime site_id
2023-08-07 00:00:00+00:00 CHIMN 0.21 40.0 885.12869 14.53812 -1.2 -0.8 28.2 33.7 1011.6 3.5 ... 16.1 28.0 16.0 32.3 16.2 23.6 16.1 8.4 252.9 2.0

1 rows × 51 columns

Multiple site IDs

You can also provide a semi-colon separated list of site IDs. This returns data for all sites listed. For example:

[10]:

site_ids = ['CHIMN', 'ROTHD', 'WRTTL']
query_url = f'{BASE_URL}/collections/1D/locations/{";".join(site_ids)}'
resp = get_api_response(query_url)

df = read_json_collection_data(resp)
display(df)

https://cosmos-api.ceh.ac.uk/collections/1D/locations/CHIMN;ROTHD;WRTTL
albedo cosmos_vwc cts_mod_corr d86_75m g1 g2 lwin lwout pa pe ... tdt6_tsoil tdt6_vwc tdt7_tsoil tdt7_vwc tdt8_tsoil tdt8_vwc tdt9_tsoil tdt9_vwc wd ws
datetime site_id
2023-08-07 00:00:00+00:00 CHIMN 0.210 40.0 885.12869 14.53812 -1.2 -0.8 28.2 33.7 1011.6 3.5 ... 16.1 28.0 16.0 32.3 16.2 23.6 16.1 8.4 252.9 2.0
ROTHD 0.171 32.8 1661.41515 15.84727 3.2 1.8 28.0 34.2 1003.4 3.6 ... NaN NaN NaN NaN NaN NaN NaN NaN 281.0 2.4
WRTTL 0.178 27.4 1730.81202 19.37998 -0.4 -0.5 27.7 34.2 1014.0 3.8 ... 15.4 29.9 15.5 30.1 15.4 29.3 15.3 11.1 277.2 2.9

3 rows × 51 columns

Constraining the request with query parameters

Date constraints

Lets add some query parameters to select a date range. There are 4 ways to specify a date query parameter:

  • A single date in the format: datetime=<date>

  • A date range in the format: datetime=<start_date>/<end_date>

  • A date range with an open end (i.e. fetch all from start date): datetime=<start_date>..

  • A date range with an open beginning (i.e. fetch all up to end date): datetime=..<end_date>

Note that dates always need to be provided in the format: “YYYY-mm-ddTHH:MM:SSZ”, for example: “2023-03-31T10:30:00Z”

[11]:

def format_datetime(dt):
    return dt.strftime("%Y-%m-%dT%H:%M:%SZ")

[12]:

# An example of using specific start and end dates
start_date = format_datetime(datetime(2023, 3, 1))
end_date = format_datetime(datetime(2023, 3, 6))
query_date_range = f'{start_date}/{end_date}'

query_url = f'{BASE_URL}/collections/1D/locations/{site_id}?datetime={query_date_range}'
resp = get_api_response(query_url)

df = read_json_collection_data(resp)
display(df)

https://cosmos-api.ceh.ac.uk/collections/1D/locations/CHIMN?datetime=2023-03-01T00:00:00Z/2023-03-06T00:00:00Z
albedo cosmos_vwc cts_mod_corr d86_75m g1 g2 lwin lwout pa pe ... tdt6_tsoil tdt6_vwc tdt7_tsoil tdt7_vwc tdt8_tsoil tdt8_vwc tdt9_tsoil tdt9_vwc wd ws
datetime site_id
2023-03-01 00:00:00+00:00 CHIMN 0.182 43.3 870.38390 14.04427 0.3 0.6 28.3 29.4 1021.2 0.8 ... 5.2 31.6 5.3 34.8 5.3 35.5 6.0 11.1 33.2 3.4
2023-03-02 00:00:00+00:00 CHIMN 0.209 42.6 873.54724 14.14392 -1.9 -2.4 24.3 29.2 1018.3 1.3 ... 5.4 31.5 5.5 34.8 5.6 35.4 6.1 11.2 45.5 3.4
2023-03-03 00:00:00+00:00 CHIMN 0.195 43.0 871.49535 14.08666 -2.3 -3.4 27.9 29.2 1021.4 0.8 ... 5.3 31.5 5.5 34.8 5.6 35.4 6.1 11.1 34.8 2.6
2023-03-04 00:00:00+00:00 CHIMN 0.193 41.6 877.72452 14.29089 -2.7 -4.2 27.6 29.1 1021.7 0.9 ... 5.3 31.4 5.4 34.7 5.5 35.3 6.1 11.0 9.6 2.4
2023-03-05 00:00:00+00:00 CHIMN 0.194 44.3 865.99066 13.90633 -3.8 -6.4 26.4 28.6 1010.8 0.7 ... 5.1 31.4 5.3 34.7 5.4 35.3 6.0 11.1 277.0 1.7
2023-03-06 00:00:00+00:00 CHIMN 0.199 42.6 873.31374 14.14392 -1.0 -1.7 27.9 29.3 997.4 0.9 ... 5.0 31.3 5.2 34.7 5.3 35.4 5.9 11.1 253.4 2.5

6 rows × 51 columns

Parameter constraints

We can constrain the data response further by providing a list of parameter names:

[13]:

# Specify a subset of paramater names
param_names = ['albedo', 'cosmos_vwc', 'pe']

query_url = f'{BASE_URL}/collections/1D/locations/{site_id}?datetime={query_date_range}&parameter-name={",".join(param_names)}'
resp = get_api_response(query_url)

df = read_json_collection_data(resp)
display(df)

https://cosmos-api.ceh.ac.uk/collections/1D/locations/CHIMN?datetime=2023-03-01T00:00:00Z/2023-03-06T00:00:00Z&parameter-name=albedo,cosmos_vwc,pe
albedo cosmos_vwc pe
datetime site_id
2023-03-01 00:00:00+00:00 CHIMN 0.182 43.3 0.8
2023-03-02 00:00:00+00:00 CHIMN 0.209 42.6 1.3
2023-03-03 00:00:00+00:00 CHIMN 0.195 43.0 0.8
2023-03-04 00:00:00+00:00 CHIMN 0.193 41.6 0.9
2023-03-05 00:00:00+00:00 CHIMN 0.194 44.3 0.7
2023-03-06 00:00:00+00:00 CHIMN 0.199 42.6 0.9

By specifying ‘flags=true’, we can return the flag parameters alongside the data parameters:

[14]:

# Add flags=false to the query:
query_url = f'{BASE_URL}/collections/1D/locations/{site_id}?datetime={query_date_range}&parameter-name={",".join(param_names)}&flags=true'
resp = get_api_response(query_url)

df = read_json_collection_data(resp)
display(df)

https://cosmos-api.ceh.ac.uk/collections/1D/locations/CHIMN?datetime=2023-03-01T00:00:00Z/2023-03-06T00:00:00Z&parameter-name=albedo,cosmos_vwc,pe&flags=true
albedo albedo_flag cosmos_vwc cosmos_vwc_flag pe pe_flag
datetime site_id
2023-03-01 00:00:00+00:00 CHIMN 0.182 None 43.3 None 0.8 E
2023-03-02 00:00:00+00:00 CHIMN 0.209 None 42.6 None 1.3 E
2023-03-03 00:00:00+00:00 CHIMN 0.195 None 43.0 None 0.8 E
2023-03-04 00:00:00+00:00 CHIMN 0.193 None 41.6 None 0.9 E
2023-03-05 00:00:00+00:00 CHIMN 0.194 None 44.3 None 0.7 E
2023-03-06 00:00:00+00:00 CHIMN 0.199 None 42.6 None 0.9 E

Position, cube and radius queries

Other methods are available for requesting multiple sites, however these are potentially less useful than the location query shown above. They have been included in the API for completeness and conformance with the OGC Environmental Data Retrieval API standards.

As with the location query, you can constrain the query further by specifing a given date or date range and one or more parameter names.

Position

/collections/<collection_id>/position?<coord(s)>

The position query fetches data for site(s) nearest to the given coordinate(s). You can specifiy coordinates in 2 ways:

  • A single POINT coordinate to return just one site

    • for example: /collections/1D/position?coords=POINT(-1.12685 52.86073)

  • A set of MULTIPOINT coordinates to return multiple sites

    • for example: /collections/1D/position?coords=MULTIPOINT((-1.4787658 51.708021),(-2.4678 54.659417),(-0.044411 52.132078))

The default coordinate reference system is WGS84 (i.e. specifying latitude and longitude coordinates). Alternatively, you can specify crs=OSGB36 as an additional query parameter to use British National Grid coordinates.

Cube

/collections/<collection_id>/cube?<bbox>

The cube query fetches data for sites within a given bounding box. The bounding box is given in the format: bbox=<min x>,<min y>,<max x>,<max y>

For example: /collections/1D/cube?bbox=-0.8,50,0,53

Radius

/collections/<collection_id>/radius?<coord(s)>

The radius query fetches data for sites within a circle of a given radius, centred at a given point (or points). The radius query parameter can be omitted, in which case this performs as described in the Position query above.

For example: /collections/30M/radius?coords=POINT(-1, 52)&within=10&within-units=km

This fetches data for all sites within a 10km radius of the given point.

CSV response

All of the data collection queries described above can return CSV file(s) instead of JSON if required. Simply add f=csv as an additional query parameter to the query URL.

A CSV file per site is created and returned within a zip archive.

[20]:

# Create a CSV query URL for multiple sites

site_ids = ['CHIMN', 'MOORH', 'WIMPL']

start_date = format_datetime(datetime(2023, 3, 1))
end_date = format_datetime(datetime(2023, 3, 3))
query_date_range = f'{start_date}/{end_date}'

query_url = f'{BASE_URL}/collections/1D/locations/{";".join(site_ids)}?datetime={query_date_range}&f=csv'

resp = get_api_response(query_url, csv=True)

zip_file = zipfile.ZipFile(io.BytesIO(resp.content))

https://cosmos-api.ceh.ac.uk/collections/1D/locations/CHIMN;MOORH;WIMPL?datetime=2023-03-01T00:00:00Z/2023-03-03T00:00:00Z&f=csv

[21]:

# List files within the returned zip archive

for csv_name in zip_file.namelist():
    print(csv_name)

COSMOS_UK_CHIMN_1D_202303010000_202303030000.csv
COSMOS_UK_MOORH_1D_202303010000_202303030000.csv
COSMOS_UK_WIMPL_1D_202303010000_202303030000.csv

[22]:

# The CSV files are formatted with some metadata at the top.  You can read the CSV files into a Pandas dataframe with code such as:

csv_file = zip_file.open(csv_name)

df = pd.read_csv(csv_file, index_col=0, skiprows=[0, 1, 2, 3, 4, 6, 7])
display(df)
albedo cosmos_vwc cts_mod_corr d86_75m g1 g2 lwin lwout pa pe ... tdt3_vwc tdt4_vwc tdt5_vwc tdt6_vwc tdt7_vwc tdt8_vwc tdt9_vwc tdt10_vwc wd ws
parameter-id
2023-03-01T00:00:00Z 0.181 37.4 1570.37808 16.57967 2.2 2.2 27.6 29.5 1026.2 0.7 ... 38.5 39.2 38.9 39.0 29.4 39.1 23.6 19.1 27.2 2.2
2023-03-02T00:00:00Z 0.196 40.9 1541.40998 15.95570 -0.3 -1.6 24.7 29.3 1023.5 1.3 ... 38.5 39.0 39.0 39.0 29.4 39.1 23.6 19.1 38.5 2.6
2023-03-03T00:00:00Z 0.179 37.9 1565.63872 16.48513 -1.7 -1.2 27.8 29.5 1026.2 0.9 ... 38.5 38.9 38.8 39.0 29.4 39.0 23.6 19.1 20.0 2.2

3 rows × 51 columns

Request limits

Given the large number of stations, variables, and timesteps, there is potential for data requests to the API to be very large, particularly for the 30 minute time step collection. Therefore, it is necessary for a limit to be set on the size of the data request so that the data can be returned in a timely manner without the API server or your local connection being overwhelmed.

The COSMOS-UK API has a ‘credit’ system to help you manage your data requests in a transparent way. The current credit limit is set to 15,000,000. Credits are calculated by:

(Number of sites) x (number of parameters) x (number of timesteps)

Note: Remember that the default is to always return the flag parameters alongside the data parameters. This will effectively double the number of parameters in the request - so bear that in mind when working about request credit totals.

Example representative credit totals:

  • 1 site, all parameters (including flags), 1 year of 30 minute data = 1 * 50 * 17,520 = 876,000 credits

  • 1 site, all parameters (including flags), 10 years of 30 minute data = 1 * 50 * 175,200 = 8,760,000 credits

  • All sites, all parameters (including flags), 10 years of daily data = 51 * 68 * 3,650 = 12,658,200 credits

A request that exceeds the credit limit will be returned a 413 error code with a description of how many credits you used in the failed request.

[23]:

# Request 30 minute data for all sites, for all parameters, for all timesteps
query_url = f'{BASE_URL}/collections/30M/cube?bbox=-9,49.75,2,61&datetime=..2023-03-01T00:00:00Z'
resp = get_api_response(query_url)
print(resp)

https://cosmos-api.ceh.ac.uk/collections/30M/cube?bbox=-9,49.75,2,61&datetime=..2023-03-01T00:00:00Z
{'code': 413, 'name': 'Request Entity Too Large', 'description': 'Too much data has been requested for a single request. Maximum request size is 15,000,000 credits. Your request would use 288,564,525 credits. Credits are calculated by (sites x parameters x time steps).'}