disdrodb.routines package

disdrodb.routines package#

Submodules#

disdrodb.routines.l0 module#

Implement DISDRODB L0 processing.

disdrodb.routines.l0.run_l0a_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0A processing of a specific DISDRODB station when invoked from the terminal.

This function is intended to be called through the disdrodb_run_l0a_station command-line interface.

Parameters:

data_source (str) – The name of the institution (for campaigns spanning multiple countries) or the name of the country (for campaigns or sensor networks within a single country). Must be provided in UPPER CASE.
campaign_name (str) – The name of the campaign. Must be provided in UPPER CASE.
station_name (str) – The name of the station.
force (bool, optional) – If True, existing data in the destination directories will be overwritten. If False (default), an error will be raised if data already exists in the destination directories.
verbose (bool, optional) – If True (default), detailed processing information will be printed to the terminal. If False, less information will be displayed.
parallel (bool, optional) – If True, files will be processed in multiple processes simultaneously with each process using a single thread. If False (default), files will be processed sequentially in a single process, and multi-threading will be automatically exploited to speed up I/O tasks.
debugging_mode (bool, optional) – If True, the amount of data processed will be reduced. Only the first 3 raw data files will be processed. The default value is False.
data_archive_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

disdrodb.routines.l0.run_l0b_station(data_source, campaign_name, station_name, remove_l0a: bool = False, force: bool = False, verbose: bool = True, parallel: bool = True, debugging_mode: bool = False, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0B processing of a specific DISDRODB station when invoked from the terminal.

This function is intended to be called through the disdrodb_run_l0b_station command-line interface.

Parameters:

data_source (str) – The name of the institution (for campaigns spanning multiple countries) or the name of the country (for campaigns or sensor networks within a single country). Must be provided in UPPER CASE.
campaign_name (str) – The name of the campaign. Must be provided in UPPER CASE.
station_name (str) – The name of the station.
force (bool, optional) – If True, existing data in the destination directories will be overwritten. If False (default), an error will be raised if data already exists in the destination directories.
verbose (bool, optional) – If True (default), detailed processing information will be printed to the terminal. If False, less information will be displayed.
parallel (bool, optional) – If True, files will be processed in multiple processes simultaneously, with each process using a single thread to avoid issues with the HDF/netCDF library. If False (default), files will be processed sequentially in a single process, and multi-threading will be automatically exploited to speed up I/O tasks.
debugging_mode (bool, optional) – If True, the amount of data processed will be reduced. Only 100 rows sampled from 3 L0A files will be processed. The default value is False.
remove_l0a (bool, optional) – Whether to remove the processed L0A files. The default value is False.
data_archive_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

disdrodb.routines.l0.run_l0c_station(data_source, campaign_name, station_name, remove_l0b: bool = False, force: bool = False, verbose: bool = True, parallel: bool = True, debugging_mode: bool = False, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0C processing of a specific DISDRODB station when invoked from the terminal.

The DISDRODB L0A and L0B routines just convert source raw data into netCDF format. The DISDRODB L0C routine ingests L0B files and performs data homogenization. The DISDRODB L0C routine takes care of:

removing duplicated timesteps across files,
merging/splitting files into daily files,
regularizing timesteps for potentially trailing seconds,
ensuring L0C files with unique sample intervals.

Duplicated timesteps are automatically dropped if their variable values coincides, otherwise an error is raised.

This function is intended to be called through the disdrodb_run_l0c_station command-line interface.

Parameters:

data_source (str) – The name of the institution (for campaigns spanning multiple countries) or the name of the country (for campaigns or sensor networks within a single country). Must be provided in UPPER CASE.
campaign_name (str) – The name of the campaign. Must be provided in UPPER CASE.
station_name (str) – The name of the station.
force (bool, optional) – If True, existing data in the destination directories will be overwritten. If False (default), an error will be raised if data already exists in the destination directories.
verbose (bool, optional) – If True (default), detailed processing information will be printed to the terminal. If False, less information will be displayed.
parallel (bool, optional) – If True, files will be processed in multiple processes simultaneously, with each process using a single thread to avoid issues with the HDF/netCDF library. If False (default), files will be processed sequentially in a single process, and multi-threading will be automatically exploited to speed up I/O tasks.
debugging_mode (bool, optional) – If True, the amount of data processed will be reduced. Only the first 3 files will be processed. The default value is False.
remove_l0b (bool, optional) – Whether to remove the processed L0B files. The default value is False.
data_archive_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

disdrodb.routines.l1 module#

Implement DISDRODB L1 processing.

disdrodb.routines.l1.define_l1_logs_filename(campaign_name, station_name, start_time, end_time, temporal_resolution)[source][source]#: Define L1 logs filename.

disdrodb.routines.l1.run_l1_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = True, parallel: bool = True, debugging_mode: bool = False, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L1 processing of a specific DISDRODB station when invoked from the terminal.

The L1 routines just filter the raw drop spectrum and compute basic statistics. The L1 routine expects as input L0C files where each file has a unique sample interval.

This function is intended to be called through the disdrodb_run_l1_station command-line interface.

Parameters:

data_source (str) – The name of the institution (for campaigns spanning multiple countries) or the name of the country (for campaigns or sensor networks within a single country). Must be provided in UPPER CASE.
campaign_name (str) – The name of the campaign. Must be provided in UPPER CASE.
station_name (str) – The name of the station.
force (bool, optional) – If True, existing data in the destination directories will be overwritten. If False (default), an error will be raised if data already exists in the destination directories.
verbose (bool, optional) – If True (default), detailed processing information will be printed to the terminal. If False, less information will be displayed.
parallel (bool, optional) – If True, files will be processed in multiple processes simultaneously, with each process using a single thread to avoid issues with the HDF/netCDF library. If False (default), files will be processed sequentially in a single process, and multi-threading will be automatically exploited to speed up I/O tasks.
debugging_mode (bool, optional) – If True, the amount of data processed will be reduced. Only the first 3 files will be processed. The default value is False.
data_archive_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

disdrodb.routines.l2 module#

Implements routines for DISDRODB L2 processing.

disdrodb.routines.l2.define_l2e_logs_filename(campaign_name, station_name, start_time, end_time, temporal_resolution)[source][source]#: Define L2E logs filename.

disdrodb.routines.l2.define_l2m_logs_filename(campaign_name, station_name, start_time, end_time, model_name, temporal_resolution)[source][source]#: Define L2M logs filename.

disdrodb.routines.l2.run_l2e_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = True, parallel: bool = True, debugging_mode: bool = False, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Generate the L2E product of a specific DISDRODB station when invoked from the terminal.

This function is intended to be called through the disdrodb_run_l2e_station command-line interface.

This routine generates L2E files. Files are defined based on the DISDRODB archive settings options. The DISDRODB archive settings allows to produce L2E files either per custom block of time (i.e day/month/year) or per blocks of (rainy) events.

For stations with varying measurement intervals, DISDRODB defines a separate list of partitions for each measurement interval option. In other words, DISDRODB does not mix files with data acquired at different sample intervals when resampling the data.

L0C product generation ensure creation of files with unique sample intervals.

Parameters:

data_source (str) – The name of the institution (for campaigns spanning multiple countries) or the name of the country (for campaigns or sensor networks within a single country). Must be provided in UPPER CASE.
campaign_name (str) – The name of the campaign. Must be provided in UPPER CASE.
station_name (str) – The name of the station.
force (bool, optional) – If True, existing data in the destination directories will be overwritten. If False (default), an error will be raised if data already exists in the destination directories.
verbose (bool, optional) – If True (default), detailed processing information will be printed to the terminal. If False, less information will be displayed.
parallel (bool, optional) – If True, files will be processed in multiple processes simultaneously, with each process using a single thread to avoid issues with the HDF/netCDF library. If False (default), files will be processed sequentially in a single process, and multi-threading will be automatically exploited to speed up I/O tasks.
debugging_mode (bool, optional) – If True, the amount of data processed will be reduced. Only the first 3 files will be processed. The default value is False.
data_archive_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

disdrodb.routines.l2.run_l2m_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = True, parallel: bool = True, debugging_mode: bool = False, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L2M processing of a specific DISDRODB station when invoked from the terminal.

This function is intended to be called through the disdrodb_run_l2m_station command-line interface.

Parameters:

data_source (str) – The name of the institution (for campaigns spanning multiple countries) or the name of the country (for campaigns or sensor networks within a single country). Must be provided in UPPER CASE.
campaign_name (str) – The name of the campaign. Must be provided in UPPER CASE.
station_name (str) – The name of the station.
force (bool, optional) – If True, existing data in the destination directories will be overwritten. If False (default), an error will be raised if data already exists in the destination directories.
verbose (bool, optional) – If True (default), detailed processing information will be printed to the terminal. If False, less information will be displayed.
parallel (bool, optional) – If True, files will be processed in multiple processes simultaneously, with each process using a single thread to avoid issues with the HDF/netCDF library. If False (default), files will be processed sequentially in a single process, and multi-threading will be automatically exploited to speed up I/O tasks.
debugging_mode (bool, optional) – If True, the amount of data processed will be reduced. Only the first 3 files will be processed. The default value is False.
data_archive_dir (str, optional) – The base directory of DISDRODB, expected in the format <...>/DISDRODB. If not specified, the path specified in the DISDRODB active configuration will be used.

disdrodb.routines.options module#

Implements ProcessingOption class for DISDRODB routines.

class disdrodb.routines.options.L0CProcessingOptions(sensor_name)[source][source]#

Bases: object

Define L0C product processing options.

Define DISDRODB L0C product processing options.

class disdrodb.routines.options.L1ProcessingOptions(filepaths, parallel, sensor_name, temporal_resolutions=None)[source][source]#

Bases: object

Define L1 product processing options.

Define DISDRODB L1 product processing options.

get_folder_partitioning(temporal_resolution)[source][source]#: Return the folder partitioning for a specific L1 product.

get_product_options(temporal_resolution)[source][source]#: Return product options dictionary for a specific L1 product.

group_files_by_temporal_partitions(temporal_resolution)[source][source]#: Return files partitions dictionary for a specific L1 product.

class disdrodb.routines.options.L2ProcessingOptions(product, filepaths, parallel, temporal_resolution, sensor_name)[source][source]#

Bases: object

Define L2 products processing options.

Define DISDRODB L2 products processing options.

disdrodb.routines.options.check_availability_radar_simulations(options)[source][source]#: Check radar simulations are possible for L2E and L2M products.

disdrodb.routines.options.get_l2m_model_settings_directory(products_configs_dir)[source][source]#: Retrieve path to the product options directory.

disdrodb.routines.options.get_l2m_model_settings_files(products_configs_dir)[source][source]#: Retrieve path to the product options directory.

disdrodb.routines.options.get_model_options(model_name, products_configs_dir=None)[source][source]#: Return DISDRODB L2M product model options.

disdrodb.routines.options.get_product_custom_options_path(products_configs_dir, product, temporal_resolution, sensor_name=None)[source][source]#: Retrieve path to the product temporal resolution custom options.

disdrodb.routines.options.get_product_global_options_path(products_configs_dir, product, sensor_name=None)[source][source]#: Retrieve path to the product global options.

disdrodb.routines.options.get_product_options(product, temporal_resolution=None, sensor_name=None, products_configs_dir=None)[source][source]#

Return DISDRODB product options.

If temporal resolution is not provided, it returns the global product option. If temporal resolution is provided, it provides the custom product options, if specified. If product=”L1” and sensor_name is specified, it customize product options also by sensor.

disdrodb.routines.options.get_product_options_directory(products_configs_dir, product, sensor_name=None)[source][source]#: Retrieve path to the product options directory.

disdrodb.routines.options.get_product_temporal_resolutions(product, sensor_name=None)[source][source]#: Return DISDRODB products temporal resolutions.

disdrodb.routines.options_validation module#

class disdrodb.routines.options_validation.ArchiveOptions(*, strategy: Literal['time_block', 'event'], strategy_options: TimeBlockStrategyOptions | EventStrategyOptions, folder_partitioning: str)[source][source]#

Bases: CustomBaseModel

Archive options configuration.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

folder_partitioning: str#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

strategy: Literal['time_block', 'event']#

strategy_options: TimeBlockStrategyOptions | EventStrategyOptions#

classmethod validate_folder_partitioning(v)[source][source]#: Validate folder partitioning.

validate_strategy_options()[source][source]#: Ensure strategy_options match the selected strategy.

class disdrodb.routines.options_validation.ArchiveOptionsTimeBlock(*, strategy: Literal['time_block', 'event'], strategy_options: TimeBlockStrategyOptions | EventStrategyOptions, folder_partitioning: str)[source][source]#

Bases: ArchiveOptions

Archive options configuration for L0C and L1 products (time_block only).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod validate_strategy_early(v)[source][source]#: Validate that strategy is ‘time_block’ and fail early if not.

class disdrodb.routines.options_validation.EventStrategyOptions(*, variable: str, detection_threshold: float, neighbor_min_size: int, neighbor_time_interval: str, event_max_time_gap: str, event_min_duration: str, event_min_size: int)[source][source]#

Bases: CustomBaseModel

Strategy options for event strategy.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

detection_threshold: float#

event_max_time_gap: str#

event_min_duration: str#

event_min_size: int#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

neighbor_min_size: int#

neighbor_time_interval: str#

classmethod validate_time_intervals(v)[source][source]#: Validate time interval strings using check_temporal_resolution.

variable: str#

class disdrodb.routines.options_validation.L0CProductGlobalConfig(*, archive_options: ArchiveOptionsTimeBlock)[source][source]#

Bases: CustomBaseModel

L0C product configuration model.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

archive_options: ArchiveOptionsTimeBlock#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class disdrodb.routines.options_validation.L1ProductConfig(*, archive_options: ArchiveOptionsTimeBlock)[source][source]#

Bases: CustomBaseModel

L0C product configuration model.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

archive_options: ArchiveOptionsTimeBlock#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class disdrodb.routines.options_validation.L1ProductGlobalConfig(*, archive_options: ArchiveOptionsTimeBlock, temporal_resolutions: list[str])[source][source]#

Bases: L1ProductConfig, TemporalResolutionsValidationMixin

L1 product configuration model.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

temporal_resolutions: list[str]#

class disdrodb.routines.options_validation.L2EProductConfig(*, archive_options: ArchiveOptions, product_options: L2EProductOptions, radar_enabled: bool, radar_options: RadarOptions)[source][source]#

Bases: CustomBaseModel

L2E product configuration model.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

archive_options: ArchiveOptions#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

product_options: L2EProductOptions#

radar_enabled: bool#

radar_options: RadarOptions#

class disdrodb.routines.options_validation.L2EProductGlobalConfig(*, archive_options: ArchiveOptions, product_options: L2EProductOptions, radar_enabled: bool, radar_options: RadarOptions, temporal_resolutions: list[str])[source][source]#

Bases: L2EProductConfig, TemporalResolutionsValidationMixin

L1 product configuration model.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

temporal_resolutions: list[str]#

class disdrodb.routines.options_validation.L2EProductOptions(*, compute_spectra: bool, compute_percentage_contribution: bool, compute_partial_number_concentrations: bool, partial_number_concentration_bins: list[float] | None, minimum_ndrops: int, minimum_nbins: int, minimum_rain_rate: float, fall_velocity_model: str, minimum_diameter: float, maximum_diameter: float, minimum_velocity: float, maximum_velocity: float, keep_mixed_precipitation: bool, above_velocity_fraction: float | None, above_velocity_tolerance: float, below_velocity_fraction: float | None, below_velocity_tolerance: float, maintain_drops_smaller_than: float, maintain_drops_slower_than: float, maintain_smallest_drops: bool, remove_splashing_drops: bool)[source][source]#

Bases: CustomBaseModel

L2E product-specific options.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

above_velocity_fraction: float | None#

above_velocity_tolerance: float#

below_velocity_fraction: float | None#

below_velocity_tolerance: float#

compute_partial_number_concentrations: bool#

compute_percentage_contribution: bool#

compute_spectra: bool#

fall_velocity_model: str#

keep_mixed_precipitation: bool#

maintain_drops_slower_than: float#

maintain_drops_smaller_than: float#

maintain_smallest_drops: bool#

maximum_diameter: float#

maximum_velocity: float#

minimum_diameter: float#

minimum_nbins: int#

minimum_ndrops: int#

minimum_rain_rate: float#

minimum_velocity: float#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

partial_number_concentration_bins: list[float] | None#

remove_splashing_drops: bool#

validate_diameter_range()[source][source]#: Validate that maximum_diameter > minimum_diameter.

classmethod validate_fall_velocity_model(fall_velocity_model)[source][source]#: Validate fall velocity model.

classmethod validate_partial_number_concentration_bins(bins)[source][source]#: Validate bins list for partial number concentration computation.

validate_velocity_range()[source][source]#: Validate that maximum_velocity > minimum_velocity.

class disdrodb.routines.options_validation.L2MModelConfig(*, psd_model: str, optimization: str, optimization_settings: dict[str, Any])[source][source]#

Bases: CustomBaseModel

L2M model configuration validation.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

optimization: str#

optimization_settings: dict[str, Any]#

psd_model: str#

classmethod validate_optimization(optimization)[source][source]#: Validate optimization method.

validate_optimization_settings()[source][source]#: Validate that optimization_settings matches the optimization method.

classmethod validate_psd_model(psd_model)[source][source]#: Validate psd_model.

class disdrodb.routines.options_validation.L2MProductConfig(*, models: list[str], archive_options: ArchiveOptions, product_options: L2MProductOptions, radar_enabled: bool, radar_options: RadarOptions)[source][source]#

Bases: CustomBaseModel

L2M product configuration model.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

archive_options: ArchiveOptions#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

models: list[str]#

product_options: L2MProductOptions#

radar_enabled: bool#

radar_options: RadarOptions#

classmethod validate_models(models)[source][source]#: Validate L2M models list.

class disdrodb.routines.options_validation.L2MProductGlobalConfig(*, models: list[str], archive_options: ArchiveOptions, product_options: L2MProductOptions, radar_enabled: bool, radar_options: RadarOptions, temporal_resolutions: list[str])[source][source]#

Bases: L2MProductConfig, TemporalResolutionsValidationMixin

L2M product configuration model.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

temporal_resolutions: list[str]#

class disdrodb.routines.options_validation.L2MProductOptions(*, fall_velocity_model: str, diameter_min: float, diameter_max: float, diameter_spacing: float, gof_metrics: bool, minimum_ndrops: int, minimum_nbins: int, minimum_rain_rate: float)[source][source]#

Bases: CustomBaseModel

L2M product-specific options.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

diameter_max: float#

diameter_min: float#

diameter_spacing: float#

fall_velocity_model: str#

gof_metrics: bool#

minimum_nbins: int#

minimum_ndrops: int#

minimum_rain_rate: float#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

validate_diameter_grid()[source][source]#: Validate diameter grid configuration.

validate_diameter_range()[source][source]#: Validate that diameter_max > diameter_min.

classmethod validate_fall_velocity_model(fall_velocity_model)[source][source]#: Validate fall velocity model.

Bases: CustomBaseModel

Radar simulation options.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

axis_ratio_model: str | list[str]#

canting_angle_std: int | float | list[int | float]#

diameter_max: int | float | list[int | float]#

diameter_min: int | float | list[int | float]#

elevation_angle: int | float | list[int | float]#

classmethod ensure_list(v)[source][source]#: Normalize single values to lists.

frequency: str | int | float | list[str | int | float]#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

num_points: int | float | list[int | float]#

permittivity_model: str | list[str]#

classmethod validate_axis_ratio_model(axis_ratio_models)[source][source]#: Validate axis ratio models.

classmethod validate_frequency_bands(frequencies)[source][source]#: Validate radar frequency bands.

classmethod validate_permittivity_model(permittivity_models)[source][source]#: Validate permittivity models.

water_temperature: int | float | list[int | float]#

class disdrodb.routines.options_validation.TemporalResolutionsValidationMixin[source][source]#

Bases: object

Mixin for temporal resolutions validation.

classmethod validate_temporal_resolutions(temporal_resolutions: list[str])[source][source]#: Validate temporal resolutions list.

class disdrodb.routines.options_validation.TimeBlockStrategyOptions(*, freq: str)[source][source]#

Bases: CustomBaseModel

Strategy options for time_block strategy.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

freq: str#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'hide_error_urls': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod validate_freq(v)[source][source]#: Validate frequency using check_freq function.

disdrodb.routines.options_validation.validate_all_product_yaml_files(products_configs_dir)[source][source]#

Validate all DISDRODB product YAML configuration files.

Raises:: pydantic.ValidationError – If any YAML file validation fails with detailed information.

disdrodb.routines.options_validation.validate_l2m_model_configs(products_configs_dir: str)[source][source]#

Validate all L2M model configuration files.

Parameters:: products_configs_dir (str) – Path to products configuration directory.
Raises:: pydantic.ValidationError – If any L2M model configuration is invalid.

disdrodb.routines.options_validation.validate_product_configuration_structure(products_configs_dir)[source][source]#

Validate the DISDRODB products configuration directory structure.

Parameters:

products_configs_dir (str) – Path to the products configuration directory.

Raises:

FileNotFoundError – If required directories or files are missing.
ValueError – If directory structure is invalid.

disdrodb.routines.options_validation.validate_products_configurations(products_configs_dir=None)[source][source]#: Validate the DISDRODB products configuration files.

disdrodb.routines.options_validation.validate_temporal_resolution_consistency(products_configs_dir)[source][source]#

Validate temporal resolution consistency across products for each sensor.

Raises warnings if:

L1 temporal_resolutions doesn’t include all L2E and L2M temporal resolutions
L2E temporal_resolutions doesn’t include all L2M temporal resolutions

disdrodb.routines.wrappers module#

DISDRODB CLI routine wrappers.

disdrodb.routines.wrappers.create_summary(data_sources=None, campaign_names=None, station_names=None, parallel=False, temporal_resolution='1MIN', data_archive_dir=None, metadata_archive_dir=None)[source][source]#

Create summary figures and tables for a set of DISDRODB station.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.
temporal_resolution (str) – Temporal resolution of the summary. The default value is 1MIN.

disdrodb.routines.wrappers.create_summary_station(data_source, campaign_name, station_name, parallel=False, temporal_resolution='1MIN', data_archive_dir=None)[source][source]#: Create summary figures and tables for a DISDRODB station.

disdrodb.routines.wrappers.run(data_sources=None, campaign_names=None, station_names=None, l0a_processing: bool = True, l0b_processing: bool = True, l0c_processing: bool = True, remove_l0a: bool = False, remove_l0b: bool = False, l1_processing: bool = True, l2e_processing: bool = True, l2m_processing: bool = True, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the complete processing chain of DISDRODB stations.

This function allows to launch the complete processing of many DISDRODB stations with a single command. From the list of all available DISDRODB stations, it runs the processing of the stations matching the provided data_sources, campaign_names and station_names.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
l0a_processing (bool) – Whether to launch processing to generate L0A Apache Parquet file(s) from raw data. The default value is True.
l0b_processing (bool) – Whether to launch processing to generate L0B netCDF4 file(s) from L0A data. The default value is True.
l0c_processing (bool) – Whether to launch processing to generate L0C netCDF4 file(s) from L0B data. The default value is True.
remove_l0a (bool) – Whether to keep the L0A files after having generated the L0B netCDF products. The default value is False.
remove_l0b (bool) – Whether to remove the L0B files after having produced L0C netCDF files. The default value is False.
l1_processing (bool) – Whether to launch processing to generate L1 netCDF4 file(s) from L0C data. The default value is True.
l2e_processing (bool) – Whether to launch processing to generate L2E netCDF4 file(s) from L1 data. The default value is True.
l2m_processing (bool) – Whether to launch processing to generate L2M netCDF4 file(s) from L1 data. The default value is True.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L0A, it processes just the first 3 raw data files. For L0B, it processes 100 rows sampled from 3 L0A files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir (str (optional)) – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l0(data_sources=None, campaign_names=None, station_names=None, l0a_processing: bool = True, l0b_processing: bool = True, l0c_processing: bool = True, remove_l0a: bool = False, remove_l0b: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0 processing of DISDRODB stations.

This function allows to launch the processing of many DISDRODB stations with a single command. From the list of all available DISDRODB stations, it runs the processing of the stations matching the provided data_sources, campaign_names and station_names.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
l0a_processing (bool) – Whether to launch processing to generate L0A Apache Parquet file(s) from raw data. The default value is True.
l0b_processing (bool) – Whether to launch processing to generate L0B netCDF4 file(s) from L0A data. The default value is True.
l0c_processing (bool) – Whether to launch processing to generate L0C netCDF4 file(s) from L0B data. The default value is True.
remove_l0a (bool) – Whether to keep the L0A files after having generated the L0B netCDF products. The default value is False.
remove_l0b (bool) – Whether to remove the L0B files after having produced all L0C netCDF files. The default value is False.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L0A, it processes just the first 3 raw data files. For L0B, it processes 100 rows sampled from 3 L0A files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l0_station(data_source, campaign_name, station_name, l0a_processing: bool = True, l0b_processing: bool = True, l0c_processing: bool = True, remove_l0a: bool = False, remove_l0b: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0 processing of a specific DISDRODB station from the terminal.

Parameters:

data_source (str) – Institution name (when campaign data spans more than 1 country), or country (when all campaigns (or sensor networks) are inside a given country). Must be UPPER CASE.
campaign_name (str) – Campaign name. Must be UPPER CASE.
station_name (str) – Station name
l0a_processing (bool) – Whether to launch processing to generate L0A Apache Parquet file(s) from raw data. The default value is True.
l0b_processing (bool) – Whether to launch processing to generate L0B netCDF4 file(s) from L0A data. The default value is True.
l0b_processing – Whether to launch processing to generate L0C netCDF4 file(s) from L0B data. The default value is True.
l0c_processing (bool) – Whether to launch processing to generate L0C netCDF4 file(s) from L0C data. The default is True.
remove_l0a (bool) – Whether to keep the L0A files after having generated the L0B netCDF products. The default value is False.
remove_l0b (bool) – Whether to remove the L0B files after having produced L0C netCDF files. The default is False.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is True.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L0A, it processes just the first 3 raw data files for each station. For L0B, it processes 100 rows sampled from 3 L0A files for each station. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l0a(data_sources=None, campaign_names=None, station_names=None, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0A processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is True.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process.
debugging_mode (bool) – If True, it processes just the first 3 raw data files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l0a_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0A processing of a station by invoking the disdrodb_run_l0a_station command in the terminal.

Parameters:

data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data to process for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.
data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l0b(data_sources=None, campaign_names=None, station_names=None, remove_l0a: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0B processing of DISDRODB stations.

This function allows to launch the processing of many DISDRODB stations with a single command. From the list of all available DISDRODB L0A stations, it runs the processing of the stations matching the provided data_sources, campaign_names and station_names.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
remove_l0a (bool) – Whether to keep the L0A files after having generated the L0B netCDF products. The default value is False.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is True.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L0B, it processes 100 rows sampled from 3 L0A files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l0b_station(data_source, campaign_name, station_name, remove_l0a: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0B processing of a station by invoking the disdrodb_run_l0b_station command in the terminal.

Parameters:

data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.
data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
remove_l0a (bool, optional) – Whether to keep the L0A files after generating L0B netCDF files. The default value is False.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data processed for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.

disdrodb.routines.wrappers.run_l0c(data_sources=None, campaign_names=None, station_names=None, remove_l0b: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0C processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
remove_l0b (bool) – Whether to remove the L0B files after having produced L0C netCDF files. The default value is False.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L1B, it processes just 3 L0B files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l0c_station(data_source, campaign_name, station_name, remove_l0b: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0C processing of a station by invoking the disdrodb_run_l0c_station command in the terminal.

Parameters:

data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
remove_l0b (bool, optional) – Whether to remove the L0B files after generating L0C netCDF files. The default value is False.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data processed for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.
data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l1(data_sources=None, campaign_names=None, station_names=None, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L1 processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L1B, it processes just 3 L0B files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l1_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L1 processing of a station by invoking the disdrodb_run_l1_station command in the terminal.

Parameters:

data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data processed for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.
data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l2e(data_sources=None, campaign_names=None, station_names=None, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L2E processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L2E, it processes just 3 L1 files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l2e_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L2E processing of a station by invoking the disdrodb_run_l2e_station command in the terminal.

Parameters:

data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data processed for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.
data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l2m(data_sources=None, campaign_names=None, station_names=None, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L2M processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L2MB, it processes just 3 L0B files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_l2m_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L2M processing of a station by invoking the disdrodb_run_l2m_station command in the terminal.

Parameters:

data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data processed for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.
data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.run_station(data_source, campaign_name, station_name, l0a_processing: bool = True, l0b_processing: bool = True, l0c_processing: bool = True, remove_l0a: bool = False, remove_l0b: bool = False, l1_processing: bool = True, l2e_processing: bool = True, l2m_processing: bool = True, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the complete processing chain of a specific DISDRODB station from the terminal.

Parameters:

data_source (str) – Institution name (when campaign data spans more than 1 country), or country (when all campaigns (or sensor networks) are inside a given country). Must be UPPER CASE.
campaign_name (str) – Campaign name. Must be UPPER CASE.
station_name (str) – Station name
l0a_processing (bool) – Whether to launch processing to generate L0A Apache Parquet file(s) from raw data. The default value is True.
l0b_processing (bool) – Whether to launch processing to generate L0B netCDF4 file(s) from L0A data. The default value is True.
l0c_processing (bool) – Whether to launch processing to generate L0C netCDF4 file(s) from L0B data. The default value is True.
remove_l0a (bool) – Whether to keep the L0A files after having generated the L0B netCDF products. The default value is False.
remove_l0b (bool) – Whether to remove the L0B files after having produced L0C netCDF files. The default is False.
l1_processing (bool) – Whether to launch processing to generate L1 netCDF4 file(s) from L0C data. The default value is True.
l2e_processing (bool) – Whether to launch processing to generate L2E netCDF4 file(s) from L1 data. The default value is True.
l2m_processing (bool) – Whether to launch processing to generate L2M netCDF4 file(s) from L1 data. The default value is True.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is True.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L0A, it processes just the first 3 raw data files for each station. For L0B, it processes 100 rows sampled from 3 L0A files for each station. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir (str (optional)) – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.wrappers.validate_processing_flags_and_get_required_product(**processing_flags)[source][source]#

Validate processing flags and determine the required input product.

Validates that the processing chain is logically consistent and returns the product needed as input for the earliest requested processing step.

The processing chain must be contiguous - you cannot skip intermediate levels. For example: l1_processing=True, l2e_processing=False, l2m_processing=True is invalid.

Raises:: ValueError – If no processing levels are enabled or if the processing chain has gaps.
Returns:: The required input product for the processing chain.
Return type:: str

Module contents#

DISDRODB L0 software.

disdrodb.routines.create_summary(data_sources=None, campaign_names=None, station_names=None, parallel=False, temporal_resolution='1MIN', data_archive_dir=None, metadata_archive_dir=None)[source][source]#

Create summary figures and tables for a set of DISDRODB station.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.
temporal_resolution (str) – Temporal resolution of the summary. The default value is 1MIN.

disdrodb.routines.create_summary_station(data_source, campaign_name, station_name, parallel=False, temporal_resolution='1MIN', data_archive_dir=None)[source][source]#: Create summary figures and tables for a DISDRODB station.

disdrodb.routines.get_model_options(model_name, products_configs_dir=None)[source][source]#: Return DISDRODB L2M product model options.

disdrodb.routines.get_product_options(product, temporal_resolution=None, sensor_name=None, products_configs_dir=None)[source][source]#

Return DISDRODB product options.

disdrodb.routines.run(data_sources=None, campaign_names=None, station_names=None, l0a_processing: bool = True, l0b_processing: bool = True, l0c_processing: bool = True, remove_l0a: bool = False, remove_l0b: bool = False, l1_processing: bool = True, l2e_processing: bool = True, l2m_processing: bool = True, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the complete processing chain of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
l0a_processing (bool) – Whether to launch processing to generate L0A Apache Parquet file(s) from raw data. The default value is True.
l0b_processing (bool) – Whether to launch processing to generate L0B netCDF4 file(s) from L0A data. The default value is True.
l0c_processing (bool) – Whether to launch processing to generate L0C netCDF4 file(s) from L0B data. The default value is True.
remove_l0a (bool) – Whether to keep the L0A files after having generated the L0B netCDF products. The default value is False.
remove_l0b (bool) – Whether to remove the L0B files after having produced L0C netCDF files. The default value is False.
l1_processing (bool) – Whether to launch processing to generate L1 netCDF4 file(s) from L0C data. The default value is True.
l2e_processing (bool) – Whether to launch processing to generate L2E netCDF4 file(s) from L1 data. The default value is True.
l2m_processing (bool) – Whether to launch processing to generate L2M netCDF4 file(s) from L1 data. The default value is True.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L0A, it processes just the first 3 raw data files. For L0B, it processes 100 rows sampled from 3 L0A files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir (str (optional)) – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l0(data_sources=None, campaign_names=None, station_names=None, l0a_processing: bool = True, l0b_processing: bool = True, l0c_processing: bool = True, remove_l0a: bool = False, remove_l0b: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0 processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
l0a_processing (bool) – Whether to launch processing to generate L0A Apache Parquet file(s) from raw data. The default value is True.
l0b_processing (bool) – Whether to launch processing to generate L0B netCDF4 file(s) from L0A data. The default value is True.
l0c_processing (bool) – Whether to launch processing to generate L0C netCDF4 file(s) from L0B data. The default value is True.
remove_l0a (bool) – Whether to keep the L0A files after having generated the L0B netCDF products. The default value is False.
remove_l0b (bool) – Whether to remove the L0B files after having produced all L0C netCDF files. The default value is False.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L0A, it processes just the first 3 raw data files. For L0B, it processes 100 rows sampled from 3 L0A files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l0_station(data_source, campaign_name, station_name, l0a_processing: bool = True, l0b_processing: bool = True, l0c_processing: bool = True, remove_l0a: bool = False, remove_l0b: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0 processing of a specific DISDRODB station from the terminal.

Parameters:

data_source (str) – Institution name (when campaign data spans more than 1 country), or country (when all campaigns (or sensor networks) are inside a given country). Must be UPPER CASE.
campaign_name (str) – Campaign name. Must be UPPER CASE.
station_name (str) – Station name
l0a_processing (bool) – Whether to launch processing to generate L0A Apache Parquet file(s) from raw data. The default value is True.
l0b_processing (bool) – Whether to launch processing to generate L0B netCDF4 file(s) from L0A data. The default value is True.
l0b_processing – Whether to launch processing to generate L0C netCDF4 file(s) from L0B data. The default value is True.
l0c_processing (bool) – Whether to launch processing to generate L0C netCDF4 file(s) from L0C data. The default is True.
remove_l0a (bool) – Whether to keep the L0A files after having generated the L0B netCDF products. The default value is False.
remove_l0b (bool) – Whether to remove the L0B files after having produced L0C netCDF files. The default is False.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is True.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L0A, it processes just the first 3 raw data files for each station. For L0B, it processes 100 rows sampled from 3 L0A files for each station. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l0a(data_sources=None, campaign_names=None, station_names=None, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0A processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is True.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process.
debugging_mode (bool) – If True, it processes just the first 3 raw data files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l0a_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0A processing of a station by invoking the disdrodb_run_l0a_station command in the terminal.

Parameters:

data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data to process for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.
data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l0b(data_sources=None, campaign_names=None, station_names=None, remove_l0a: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0B processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
remove_l0a (bool) – Whether to keep the L0A files after having generated the L0B netCDF products. The default value is False.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is True.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L0B, it processes 100 rows sampled from 3 L0A files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l0b_station(data_source, campaign_name, station_name, remove_l0a: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0B processing of a station by invoking the disdrodb_run_l0b_station command in the terminal.

Parameters:

data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.
data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
remove_l0a (bool, optional) – Whether to keep the L0A files after generating L0B netCDF files. The default value is False.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data processed for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.

disdrodb.routines.run_l0c(data_sources=None, campaign_names=None, station_names=None, remove_l0b: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0C processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
remove_l0b (bool) – Whether to remove the L0B files after having produced L0C netCDF files. The default value is False.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L1B, it processes just 3 L0B files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l0c_station(data_source, campaign_name, station_name, remove_l0b: bool = False, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L0C processing of a station by invoking the disdrodb_run_l0c_station command in the terminal.

Parameters:

data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
remove_l0b (bool, optional) – Whether to remove the L0B files after generating L0C netCDF files. The default value is False.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data processed for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.
data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l1(data_sources=None, campaign_names=None, station_names=None, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L1 processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L1B, it processes just 3 L0B files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l1_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L1 processing of a station by invoking the disdrodb_run_l1_station command in the terminal.

Parameters:

data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data processed for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.
data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l2e(data_sources=None, campaign_names=None, station_names=None, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L2E processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L2E, it processes just 3 L1 files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l2e_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L2E processing of a station by invoking the disdrodb_run_l2e_station command in the terminal.

Parameters:

data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data processed for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.
data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l2m(data_sources=None, campaign_names=None, station_names=None, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L2M processing of DISDRODB stations.

Parameters:

data_sources (list) – Name of data source(s) to process. The name(s) must be UPPER CASE. If campaign_names and station are not specified, process all stations. The default value is None.
campaign_names (list) – Name of the campaign(s) to process. The name(s) must be UPPER CASE. The default value is None.
station_names (list) – Station names to process. The default value is None.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is False.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L2MB, it processes just 3 L0B files. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_l2m_station(data_source, campaign_name, station_name, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the L2M processing of a station by invoking the disdrodb_run_l2m_station command in the terminal.

Parameters:

data_source (str) – The name of the data source.
campaign_name (str) – The name of the campaign.
station_name (str) – The name of the station.
force (bool, optional) – If True, overwrite existing data in destination directories. The default value is False.
verbose (bool, optional) – If True, print detailed processing information to the terminal. The default value is False.
debugging_mode (bool, optional) – If True, reduce the amount of data processed for debugging. The default value is False.
parallel (bool, optional) – If True, process files in multiple processes simultaneously. The default value is True.
data_archive_dir – The directory path where the local DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines.run_station(data_source, campaign_name, station_name, l0a_processing: bool = True, l0b_processing: bool = True, l0c_processing: bool = True, remove_l0a: bool = False, remove_l0b: bool = False, l1_processing: bool = True, l2e_processing: bool = True, l2m_processing: bool = True, force: bool = False, verbose: bool = False, debugging_mode: bool = False, parallel: bool = True, data_archive_dir: str | None = None, metadata_archive_dir: str | None = None)[source][source]#

Run the complete processing chain of a specific DISDRODB station from the terminal.

Parameters:

data_source (str) – Institution name (when campaign data spans more than 1 country), or country (when all campaigns (or sensor networks) are inside a given country). Must be UPPER CASE.
campaign_name (str) – Campaign name. Must be UPPER CASE.
station_name (str) – Station name
l0a_processing (bool) – Whether to launch processing to generate L0A Apache Parquet file(s) from raw data. The default value is True.
l0b_processing (bool) – Whether to launch processing to generate L0B netCDF4 file(s) from L0A data. The default value is True.
l0c_processing (bool) – Whether to launch processing to generate L0C netCDF4 file(s) from L0B data. The default value is True.
remove_l0a (bool) – Whether to keep the L0A files after having generated the L0B netCDF products. The default value is False.
remove_l0b (bool) – Whether to remove the L0B files after having produced L0C netCDF files. The default is False.
l1_processing (bool) – Whether to launch processing to generate L1 netCDF4 file(s) from L0C data. The default value is True.
l2e_processing (bool) – Whether to launch processing to generate L2E netCDF4 file(s) from L1 data. The default value is True.
l2m_processing (bool) – Whether to launch processing to generate L2M netCDF4 file(s) from L1 data. The default value is True.
force (bool) – If True, overwrite existing data into destination directories. If False, raise an error if there are already data into destination directories. The default value is False.
verbose (bool) – Whether to print detailed processing information into terminal. The default value is True.
parallel (bool) – If True, the files are processed simultaneously in multiple processes. Each process will use a single thread to avoid issues with the HDF/netCDF library. By default, the number of process is defined with os.cpu_count(). If False, the files are processed sequentially in a single process. If False, multi-threading is automatically exploited to speed up I/0 tasks.
debugging_mode (bool) – If True, it reduces the amount of data to process. For L0A, it processes just the first 3 raw data files for each station. For L0B, it processes 100 rows sampled from 3 L0A files for each station. The default value is False.
data_archive_dir (str (optional)) – The directory path where the DISDRODB Data Archive is located. The directory path must end with <...>/DISDRODB. If None, it uses the data_archive_dir path specified in the DISDRODB active configuration.
metadata_archive_dir (str (optional)) – The directory path where the DISDRODB Metadata Archive is located. The directory path must end with <...>/DISDRODB-METADATA/DISDRODB. If None, it uses the metadata_archive_dir path specified in the DISDRODB active configuration.

disdrodb.routines package

Contents

disdrodb.routines package#

Submodules#

disdrodb.routines.l0 module#

disdrodb.routines.l1 module#

disdrodb.routines.l2 module#

disdrodb.routines.options module#

disdrodb.routines.options_validation module#

disdrodb.routines.wrappers module#

Module contents#