poets.io package¶
Submodules¶
poets.io.download module¶
Provides download functions for FTP/SFTP, HTTP and local data sources.
- poets.io.download.download_ftp(download_path, host, directory, port, username, password, filedate, dirstruct, begin, end=None)[source]¶
Download data via SFTP
Parameters: download_path : str, optional
Path where to save the downloaded files.
host : str
Link to host.
directory : str
Path to data on host.
port : int
Port to host.
username : str
Username for source.
password : str
Passwor for source.
filedate : dict
Dict which points to the date fields in the filename
dirstruct : list of str
Folder structure on host, each list element represents a subdirectory
begin : datetime.datetime
Set either to first date of remote repository or date of last file in local repository
end : datetime.datetime, optional
Entered in [years]. End year is not downloaded anymore, defaults to datetime.date.today()
Returns: bool
true if data is available, false if not
- poets.io.download.download_http(download_path, host, directory, filename, filedate, dirstruct, begin, end=None)[source]¶
Download data via HTTP
Parameters: download_path : str, optional
Path where to save the downloaded files.
host : str
Link to host.
directory : str
Path to data on host.
filename : str
Structure/convention of the file name.
filedate : dict
Dict which points to the date fields in the filename.
dirstruct : list of str
Folder structure on host, each list element represents a subdirectory.
begin : datetime.date
Set either to first date of remote repository or date of last file in local repository.
end : datetime.date, optional
Set to today if none given.
Returns: bool
true if data is available, false if not
- poets.io.download.download_sftp(download_path, host, directory, port, username, password, filedate, dirstruct, begin, end=None)[source]¶
Download data via SFTP
Parameters: download_path : str, optional
Path where to save the downloaded files.
host : str
Link to host.
directory : str
Path to data on host.
port : int
Port to host.
username : str
Username for source.
password : str
Passwor for source.
filedate : dict
Dict which points to the date fields in the filename.
dirstruct : list of str
Folder structure on host, each list element represents a subdirectory.
begin : datetime.datetime
Set either to first date of remote repository or date of last file in local repository.
end : datetime.datetime, optional
Entered in [years]. End year is not downloaded anymore, defaults to datetime.datetime.now()
Returns: bool
True if data is available, false if not.
poets.io.source_base module¶
- class poets.io.source_base.BasicSource(name, filename, filedate, temp_res, rootpath, host, protocol, username=None, password=None, port=22, directory=None, dirstruct=None, begin_date=datetime.datetime(2000, 1, 1, 0, 0), variables=None, nan_value=None, dest_nan_value=-99, dest_regions=None, dest_sp_res=0.25, dest_temp_res='dekad', dest_start_date=datetime.datetime(2000, 1, 1, 0, 0))[source]¶
Bases: object
Base Class for data sources.
Parameters: name : str
Name of the data source.
filename : str
Structure/convention of the file name.
filedate : dict
Position of date fields in filename, given as tuple.
temp_res : str
Temporal resolution of the source.
rootpath : str
Root path where all data will be stored.
host : str
Link to data host.
protocol : str
Protocol for data transfer.
username : str, optional
Username for data access.
password : str, optional
Password for data access.
port : int, optional
Port to data host, defaults to 22.
directory : str, optional
Path to data on host.
dirstruct : list of strings
Structure of source directory, each list item represents a subdirectory.
begin_date : datetime, optional
Date from which on data is available, defaults to 2000-01-01.
variables : string or list of strings, optional
Variables used from data source, defaults to [‘dataset’].
nan_value : int, float, optional
Nan value of the original data as given by the data provider.
dest_nan_value : int, float, optional
NaN value in the final NetCDF file.
dest_regions : list of str, optional
Regions of interest where data should be resampled to.
dest_sp_res : int, float, optional
Spatial resolution of the destination NetCDF file, defaults to 0.25 degree.
dest_temp_res : string, optional
Temporal resolution of the destination NetCDF file, possible values: (‘month’, ‘dekad’), defaults to dekad.
dest_start_date : datetime, optional
Start date of the destination NetCDF file, defaults to 2000-01-01.
Attributes
name (str) Name of the data source. filename (str) Structure/convention of the file name. filedate (dict) Position of date fields in filename, given as tuple. temp_res (str) Temporal resolution of the source. host (str) Link to data host. protocol (str) Protocol for data transfer. username (str) Username for data access. password (str) Password for data access. port (int) Port to data host. directory (str) Path to data on host. dirstruct (list of strings) Structure of source directory, each list item represents a subdirectory. begin_date (datetime) Date from which on data is available. variables (list of strings) Variables used from data source. nan_value (int, float) N a number value of the original data as given by the data provider. dest_nan_value (int, float, optional) NaN value in the final NetCDF file. tmp_path (str) Path where temporary files are stored. rawdata_path (str) Path where original files are stored. data_path (str) Path where resampled NetCDF file is stored. dest_regions (list of str) Regions of interest where data is resampled to. dest_sp_res (int, float) Spatial resolution of the destination NetCDF file. dest_temp_res (string) Temporal resolution of the destination NetCDF file. Methods
- download(download_path=None, begin=None, end=None)[source]¶
“Download data
Parameters: begin : datetime, optional
start date of download, default to None
end : datetime, optional
start date of download, default to None
- download_and_resample(download_path=None, begin=None, end=None, delete_rawdata=False, shapefile=None)[source]¶
Downloads and resamples data.
Parameters: download_path : str
Path where to save the downloaded files.
begin : datetime.date, optional
set either to first date of remote repository or date of last file in local repository
end : datetime.date, optional
set to today if none given
delete_rawdata : bool, optional
Original files will be deleted from rawdata_path if set True
shapefile : str, optional
Path to shape file, uses “world country admin boundary shapefile” by default.
- get_variables()[source]¶
Gets all variables from source given in the NetCDF file.
Returns: variables : list of str
Variables from source given in NetCDF file.
- read_img(date, region=None, variable=None)[source]¶
Gets images from netCDF file for certain date
Parameters: date : datetime
Date of the image.
region : str, optional
Region of interest, set to first defined region if not set.
variable : str, optional
Variable to display, selects first available variables if None.
Returns: img : numpy.ndarray
Image of selected date.
lon : numpy.array
Array with longitudes.
lat : numpy.array
Array with latitudes.
- read_ts(location, region=None, variable=None)[source]¶
Gets timeseries from netCDF file for a gridpoint.
Parameters: location : int or tuple of floats
Either Grid point index as integer value or Longitude/Latitude given as tuple.
region : str, optional
Region of interest, set to first defined region if not set.
variable : str, optional
Variable to display, selects all available variables if None.
Returns: df : pd.DataFrame
Timeseries for selected variables.
- resample(begin=None, end=None, delete_rawdata=False, shapefile=None)[source]¶
Resamples source data to given spatial and temporal resolution.
Writes resampled images into a netCDF data file. Deletes original files if flag delete_rawdata is set True.
Parameters: begin : datetime
Start date of resampling.
end : datetime
End date of resampling.
delete_rawdata : bool
Original files will be deleted from rawdata_path if set ‘True’.
shapefile : str, optional
Path to shape file, uses “world country admin boundary shapefile” by default.
