Ctrl+K

disdrodb.api package

disdrodb.api package#

Submodules#

disdrodb.api.checks module#

DISDRODB Checks Functions.

disdrodb.api.checks.check_base_dir(base_dir: str)[source]#

Raise an error if the path does not end with DISDRODB.

disdrodb.api.checks.check_campaign_name(campaign_name)[source]#

Check the campaign name is upper case !.

disdrodb.api.checks.check_data_source(data_source)[source]#

Check the data_source name is upper case !.

disdrodb.api.checks.check_directories_inside(dir_path)[source]#

Check there are directories inside the specified dir_path.

disdrodb.api.checks.check_is_within_processed_directory(path)[source]#

Check the path is within the DISDRODB Processed directory.

disdrodb.api.checks.check_is_within_raw_directory(path)[source]#

Check the path is within the DISDRODB Raw directory.

disdrodb.api.checks.check_issue_dir(data_source, campaign_name, base_dir=None)[source]#

Check existence of the issue directory. If does not exists, raise an error.

disdrodb.api.checks.check_issue_file(data_source, campaign_name, station_name, base_dir=None)[source]#

Check existence of a valid issue YAML file. If does not exists, raise an error.

disdrodb.api.checks.check_metadata_dir(product, data_source, campaign_name, base_dir=None)[source]#

Check existence of the metadata directory. If does not exists, raise an error.

disdrodb.api.checks.check_metadata_file(product, data_source, campaign_name, station_name, base_dir=None, check_validity=True)[source]#

Check existence of a valid metadata YAML file. If does not exists, raise an error.

disdrodb.api.checks.check_path(path: str) → None[source]#

Check if a path exists.

Parameters

path (str) – Path to check.

Raises

FileNotFoundError – If the path does not exist.

disdrodb.api.checks.check_path_is_a_directory(dir_path, path_name='')[source]#

Check that the path exists and is directory.

disdrodb.api.checks.check_processed_dir(processed_dir)[source]#

Check input, format and validity of the processed_dir directory path.

Parameters

processed_dir (str) – Path to the campaign directory in the DISDRODB/Processed directory tree

Returns

Path of the processed campaign directory

Return type

str

disdrodb.api.checks.check_product(product)[source]#

Check DISDRODB product.

disdrodb.api.checks.check_raw_dir(raw_dir: str, station_name: str) → None[source]#

Check validity of raw_dir content.

Steps:

  1. Check that raw_dir is a valid directory path

  2. Check that raw_dir follows the expect directory structure

  3. Check that each station_name directory contains data

  4. Check that for each station_name the mandatory metadata.yml is specified.

  5. Check that for each station_name the mandatory issue.yml is specified.

Parameters

  • raw_dir (str) – Input raw campaign directory.

  • station_name (str) – Station name.

  • verbose (bool, optional) – Whether to verbose the processing. The default is False.

disdrodb.api.checks.check_sensor_name(sensor_name: str, product: str = 'L0A') → None[source]#

Check sensor name.

Parameters

  • sensor_name (str) – Name of the sensor.

  • product (str) – DISDRODB product.

Raises

  • TypeError – Error if sensor_name is not a string.

  • ValueError – Error if the input sensor name has not been found in the list of available sensors.

disdrodb.api.checks.check_station_dir(product, data_source, campaign_name, station_name, base_dir=None)[source]#

Check existence of the station data directory. If does not exists, raise an error.

disdrodb.api.checks.check_station_has_data(product, data_source, campaign_name, station_name, base_dir=None)[source]#

Check the station data directory has data inside. If not, raise an error.

disdrodb.api.checks.check_url(url: str) → bool[source]#

Check url.

Parameters

url (str) – URL to check.

Returns

True if url well formatted, False if not well formatted.

Return type

bool

disdrodb.api.checks.check_valid_campaign_dir(campaign_dir)[source]#

Check the validity of a campaign directory path.

Used to check validity of raw_dir and processed_dir.

The path must be */DISDRODB/<Raw/Processed>/<DATA_SOURCE>/<CAMPAIGN_NAME>

disdrodb.api.checks.has_available_station_files(product, data_source, campaign_name, station_name, base_dir=None)[source]#

Return True if data are available for the given product and station.

disdrodb.api.configs module#

Retrieve sensor configuration files.

disdrodb.api.configs.available_sensor_names(product: str = 'L0A') → sorted[source]#

Get available names of sensors.

Returns

  • sensor_names (list) – Sorted list of the available sensors

  • product (str) – DISDRODB product. By default, it returns the sensors available for DISDRODB L0A products.

disdrodb.api.configs.get_sensor_configs_dir(sensor_name: str, product: str) → str[source]#

Retrieve configs directory.

Parameters

  • sensor_name (str) – Name of the sensor.

  • product (str) – DISDRODB product.

Returns

config_sensor_dir – Config directory.

Return type

str

Raises

ValueError – Error if the config directory does not exist.

disdrodb.api.configs.read_config_file(sensor_name: str, product: str, filename: str) → dict[source]#

Read a config yaml file and return the dictionary.

Parameters

  • sensor_name (str) – Name of the sensor.

  • filename (str) – Name of the file.

Returns

Content of the config file.

Return type

dict

Raises

ValueError – Error if file does not exist.

disdrodb.api.create_directories module#

Tools to create Raw, L0A and L0B DISDRODB directories.

disdrodb.api.create_directories.create_directory_structure(processed_dir, product, station_name, force)[source]#

Create directory structure for L0B and higher DISDRODB products.

disdrodb.api.create_directories.create_initial_station_structure(data_source, campaign_name, station_name, base_dir=None)[source]#

Create the DISDRODB Data Archive structure for a single station.

disdrodb.api.create_directories.create_issue_directory(base_dir, data_source, campaign_name)[source]#

Create issue directory.

disdrodb.api.create_directories.create_l0_directory_structure(raw_dir, processed_dir, station_name, force, product)[source]#

Create directory structure for the first L0 DISDRODB product.

If the input data are raw text files, use product = "L0A" If the input data are raw netCDF files, use product = "L0B"

product = "L0A" will call run_l0a. product = "L0B" will call run_l0b_nc.

disdrodb.api.create_directories.create_metadata_directory(base_dir, product, data_source, campaign_name)[source]#

Create metadata directory.

disdrodb.api.create_directories.create_station_directory(base_dir, product, data_source, campaign_name, station_name)[source]#

Create station data directory.

disdrodb.api.create_directories.create_test_archive(test_base_dir, data_source, campaign_name, station_name, base_dir=None, force=False)[source]#

Create test DISDRODB Archive for a single existing station.

This function is used to make a copy of metadata and issue files of a stations. This enable to then test data download and DISDRODB processing.

disdrodb.api.info module#

Retrieve file information from DISDRODB products file names and filepaths.

disdrodb.api.info.get_campaign_name_from_filepaths(filepaths)[source]#

Return the DISDROB campaign name of the specified files.

disdrodb.api.info.get_end_time_from_filepaths(filepaths)[source]#

Return the end time of the specified files.

disdrodb.api.info.get_info_from_filepath(filepath)[source]#

Retrieve file information dictionary from filepath.

disdrodb.api.info.get_key_from_filepath(filepath, key)[source]#

Extract specific key information from a list of filepaths.

disdrodb.api.info.get_key_from_filepaths(filepaths, key)[source]#

Extract specific key information from a list of filepaths.

disdrodb.api.info.get_product_from_filepaths(filepaths)[source]#

Return the DISDROB product name of the specified files.

disdrodb.api.info.get_start_end_time_from_filepaths(filepaths)[source]#

Return the start and end time of the specified files.

disdrodb.api.info.get_start_time_from_filepaths(filepaths)[source]#

Return the start time of the specified files.

disdrodb.api.info.get_station_name_from_filepaths(filepaths)[source]#

Return the DISDROB station name of the specified files.

disdrodb.api.info.get_version_from_filepaths(filepaths)[source]#

Return the DISDROB product version of the specified files.

disdrodb.api.info.infer_base_dir_from_path(path: str) → str[source]#

Return the disdrodb base directory from a file or directory path.

Assumption: no data_source, campaign_name, station_name or file contain the word DISDRODB!

Parameters

path (str) – path can be a campaign_dir (raw_dir or processed_dir), or a DISDRODB file path.

Returns

Path of the DISDRODB directory.

Return type

str

disdrodb.api.info.infer_campaign_name_from_path(path: str) → str[source]#

Return the campaign name from a file or directory path.

Assumption: no data_source, campaign_name, station_name or file contain the word DISDRODB!

Parameters

path (str) – path can be a campaign_dir (raw_dir or processed_dir), or a DISDRODB file path.

Returns

Name of the campaign.

Return type

str

disdrodb.api.info.infer_data_source_from_path(path: str) → str[source]#

Return the data_source from a file or directory path.

Assumption: no data_source, campaign_name, station_name or file contain the word DISDRODB!

Parameters

path (str) – path can be a campaign_dir (raw_dir or processed_dir), or a DISDRODB file path.

Returns

Name of the data source.

Return type

str

disdrodb.api.info.infer_disdrodb_tree_path(path: str) → str[source]#

Return the directory tree path from the base_dir directory.

Current assumption: no data_source, campaign_name, station_name or file contain the word DISDRODB!

Parameters

path (str) – path can be a campaign_dir (raw_dir or processed_dir), or a DISDRODB file path.

Returns

Path inside the DISDRODB archive. Format: DISDRODB/<Raw or Processed>/<DATA_SOURCE>/<CAMPAIGN_NAME>/...

Return type

str

disdrodb.api.info.infer_disdrodb_tree_path_components(path: str) → list[source]#

Return a list with the component of the disdrodb_path.

Parameters

path (str) – path can be a campaign_dir (raw_dir or processed_dir), or a DISDRODB file path.

Returns

Path element of the DISDRODB archive. Format: [base_dir, <Raw or Processed>, data_source`, ``campaign_name, …]

Return type

list

disdrodb.api.info.infer_path_info_dict(path: str) → dict[source]#

Return a dictionary with the base_dir, data_source and campaign_name of the disdrodb_path.

Parameters

path (str) – path can be a campaign_dir (raw_dir or processed_dir), or a DISDRODB file path.

Returns

Dictionary with the path element of the DISDRODB archive. Valid keys: "base_dir", "data_source", "campaign_name"

Return type

list

disdrodb.api.io module#

Routines tot extract information from the DISDRODB infrastructure.

disdrodb.api.io.available_campaigns(product, data_sources=None, return_tuple=True, base_dir=None)[source]#

Return campaigns for which stations data are available.

disdrodb.api.io.available_data_sources(product, base_dir=None)[source]#

Return data sources for which stations data are available.

disdrodb.api.io.available_stations(product, data_sources=None, campaign_names=None, station_names=None, return_tuple=True, base_dir=None)[source]#

Return stations for which data are available on disk.

disdrodb.api.path module#

Define paths within the DISDRODB infrastructure.

disdrodb.api.path.define_campaign_dir(product, data_source, campaign_name, base_dir=None, check_exists=False)[source]#

Return the campaign directory in the DISDRODB infrastructure.

Parameters

  • product (str) – The DISDRODB product. It can be "RAW", "L0A", or "L0B".

  • data_source (str) – The data source. Must be specified if campaign_name is specified.

  • campaign_name (str) – The campaign name.

  • base_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

  • check_exists (bool, optional) – Whether to check if the directory exists. By default False.

Returns

station_dir – Station data directory path

Return type

str

disdrodb.api.path.define_config_dir(product)[source]#

Define the config directory path of a given DISDRODB product.

disdrodb.api.path.define_issue_dir(data_source, campaign_name, base_dir=None, check_exists=False)[source]#

Return the issue directory in the DISDRODB infrastructure.

Parameters

  • data_source (str) – The data source.

  • campaign_name (str) – The campaign name.

  • base_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

  • check_exists (bool, optional) – Whether to check if the directory exists. By default False.

Returns

issue_dir – Station data directory path

Return type

str

disdrodb.api.path.define_issue_filepath(data_source, campaign_name, station_name, base_dir=None, check_exists=False)[source]#

Return the station issue filepath in the DISDRODB infrastructure.

Parameters

  • data_source (str) – The data source.

  • campaign_name (str) – The campaign name.

  • station_name (str) – The station name.

  • base_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

  • check_exists (bool, optional) – Whether to check if the directory exists. By default False.

Returns

issue_dir – Station data directory path

Return type

str

disdrodb.api.path.define_l0a_filename(df, processed_dir, station_name: str) → str[source]#

Define L0A file name.

Parameters

  • df (pd.DataFrame) – L0A DataFrame

  • processed_dir (str) – Path of the processed directory

  • station_name (str) – Name of the station

Returns

L0A file name.

Return type

str

disdrodb.api.path.define_l0a_filepath(df: DataFrame, processed_dir: str, station_name: str) → str[source]#

Define L0A file path.

Parameters

  • df (pd.DataFrame) – L0A DataFrame.

  • processed_dir (str) – Path of the processed directory.

  • station_name (str) – Name of the station.

Returns

L0A file path.

Return type

str

disdrodb.api.path.define_l0a_station_dir(processed_dir: str, station_name: str) → str[source]#

Define L0A directory.

Parameters

  • processed_dir (str) – Path of the processed directory

  • station_name (str) – Name of the station

Returns

L0A directory path.

Return type

str

disdrodb.api.path.define_l0b_filename(ds, processed_dir, station_name: str) → str[source]#

Define L0B file name.

Parameters

  • ds (xr.Dataset) – L0B xarray Dataset

  • processed_dir (str) – Path of the processed directory

  • station_name (str) – Name of the station

Returns

L0B file name.

Return type

str

disdrodb.api.path.define_l0b_filepath(ds: Dataset, processed_dir: str, station_name: str, l0b_concat=False) → str[source]#

Define L0B file path.

Parameters

  • ds (xr.Dataset) – L0B xarray Dataset.

  • processed_dir (str) – Path of the processed directory.

  • station_name (str) – ID of the station

  • l0b_concat (bool) – If False, the file is specified inside the station directory. If True, the file is specified outside the station directory.

Returns

L0B file path.

Return type

str

disdrodb.api.path.define_l0b_station_dir(processed_dir: str, station_name: str) → str[source]#

Define L0B directory.

Parameters

  • processed_dir (str) – Path of the processed directory

  • station_name (int) – Name of the station

Returns

Path of the L0B directory

Return type

str

disdrodb.api.path.define_metadata_dir(product, data_source, campaign_name, base_dir=None, check_exists=False)[source]#

Return the metadata directory in the DISDRODB infrastructure.

Parameters

  • product (str) – The DISDRODB product. It can be "RAW", "L0A", or "L0B".

  • data_source (str) – The data source.

  • campaign_name (str) – The campaign name.

  • base_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

  • check_exists (bool, optional) – Whether to check if the directory exists. By default False.

Returns

metadata_dir – Station data directory path

Return type

str

disdrodb.api.path.define_metadata_filepath(product, data_source, campaign_name, station_name, base_dir=None, check_exists=False)[source]#

Return the station metadata filepath in the DISDRODB infrastructure.

Parameters

  • product (str) – The DISDRODB product. It can be “RAW”, “L0A”, or “L0B”.

  • data_source (str) – The data source.

  • campaign_name (str) – The campaign name.

  • station_name (str) – The station name.

  • base_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

  • check_exists (bool, optional) – Whether to check if the directory exists. By default False.

Returns

metadata_dir – Station data directory path

Return type

str

disdrodb.api.path.define_station_dir(product, data_source, campaign_name, station_name, base_dir=None, check_exists=False)[source]#

Return the station data directory in the DISDRODB infrastructure.

Parameters

  • product (str) – The DISDRODB product. It can be "RAW", "L0A", or "L0B".

  • data_source (str) – The data source.

  • campaign_name (str) – The campaign name.

  • station_name (str) – The station name.

  • base_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

  • check_exists (bool, optional) – Whether to check if the directory exists. By default False.

Returns

station_dir – Station data directory path

Return type

str

disdrodb.api.path.get_disdrodb_path(base_dir, product, data_source='', campaign_name='', check_exists=True)[source]#

Return the directory in the DISDRODB infrastructure.

If data_source and campaign_name are not specified it return the product directory.

If data_source is specified, it returns the data_source directory.

If campaign_source is specified, it returns the campaign_name directory.

Parameters

  • base_dir (str) – The disdrodb base directory

  • product (str) – The DISDRODB product. It can be "RAW", "L0A", or "L0B".

  • data_source (str, optional) – The data source. Must be specified if campaign_name is specified.

  • campaign_name (str, optional) – The campaign name.

  • check_exists (bool, optional) – Whether to check if the directory exists. By default True.

Returns

dir_path – Directory path

Return type

str

Module contents#

Contents