Summary
This chapter examines the data management in the FEWS Tools plugin for QGIS, including the different types of data formats used, downloading data, data availability, and data import/export using archives.
3.1. Data types
GeoCLIM uses four main data types: raster data in BIL format BIL (* .bil) and GeoTIF (*.tif or *.tiff), vector data in shapefile format (*.shp), tables in comma-delimited format (*.csv), and data Archives.
3.1.1. Characteristics of the raster dataset
A band interleaved by line (BIL) dataset contains two files: a (*.bil) file and a header file (*.hdr). The .bil file is a binary file that contains the pixel values (e.g., rainfall, temperature, etc.), while the HDR file contains the characteristics of the dataset, such as the geographic location, pixel size, and depth.
The header file is an ASCII text file; it can be generated or edited from a text editor (e.g., Notepad). For example, Figure 3‑1 shows that the header file contains information about the number of columns (ncols), number of rows (nrows), number of bits per pixel (nbits), and size of pixel (xdim and ydim), among others. Figure 3‑1 also shows the xdim and ydim values corresponding to the horizontal (x-dimension) and vertical (y-dimension) dimensions of a pixel with a size of 0.05 degrees, which is about 5 kilometers. The ulxmap and ulymap correspond to the x-axis and the y-axis coordinates of the center of the upper-left pixel of the raster image. There are additional keywords that the header could have (Table 3‑1) (ArcMap 10.3 Help, ESRI). Sometimes, if the header file is incorrect, you may need to modify it so that the data is read correctly by the program.
By default, the BIL dataset pixel type used is unsigned integers, unless the keyword pixeltype is used in the HDR file, and its value is signedint.
|
Keyword |
Acceptable Value |
Default |
|---|---|---|
|
nrows |
Any integer > 0 |
None |
|
ncols |
Any integer > 0 |
None |
|
nbands |
Any integer > 0 |
1 |
|
nbits |
1, 4, 8, 16, 32 |
8 |
|
pixeltype |
SIGNEDINT |
Unsigned Integer |
|
byteorder |
I = Intel; M - Motorola |
Same as host machine |
|
layout |
bil, bip, bsq |
bil |
|
skipbytes |
Any integer ≥ 0 |
0 |
|
ulxmap |
Any real nuber |
0 |
|
ulymap |
Any real number |
nrows - 1 |
|
xdim |
Any real number |
1 |
|
ydim |
Any real number |
1 |
|
bandrowbytes |
Any integer ≥ 0 |
Smallest integer ≥ (ncols x nbits) / 8 |
|
totalrowbytes |
Any integer ≥ 0 |
For bil: nbands x bandrowbytes; for bip: smallest integer ≥ (ncols x nbands x nbits) / 8 |
|
bandgapbytes |
Any integer ≥ 0 |
0 |
Table 3-1. The header file is composed of a series of keywords and their respective accepted values. Source: ArcMap 10.3 Help, ESRI.
An important keyword in the header file is the pixeltype since it defines the type of value, unsigned (+), or signed (+ or -) a pixel could have. For example, rainfall data could only have unsigned (+) values since precipitation is only positive. However, if you look at the header file for CHIRPS, it uses signed values (+ or -) since the nodata value is -9999. In the example in Figure 3-1, the pixeltype is missing, so the program assumes that the data is unsigned; in the case of CHIRPS, we would have to add a new line on the header file defining the pixeltype as signedint. Another keyword to keep in mind is the nbits because it indicates the number of bits per pixel or the depth of the raster image (e.g., nbits=16 bit means that a pixel in the raster dataset can have any one of 216 = 65536 unique values). Table 3‑2 (ESRI, Bit depth capacity for raster dataset pixels) shows a list of values a raster dataset could have depending on the pixel depth or nbits value.
|
Pixel Type |
Bit Depth |
Value range for a raster dataset |
|---|---|---|
|
unsigned |
1 bit |
0 to 1 |
|
unsigned |
2 bit |
0 to 4 |
|
unsigned |
4 bit |
0 to 16 |
|
unsigned |
8 bit |
0 to 255 |
|
signed |
8 bit |
-128 to 127 |
|
unsigned |
16 bit |
0 to 65535 |
|
signed |
16 bit |
-32768 to 32767 |
|
unsigned |
32 bit |
0 to 4294967295 |
|
signed |
32 bit |
-2147483648 to 2147483647 |
|
floating point |
32 bit |
-3.402823466e+38 to 3.402823466e+38 |
Table 3-2 The range of values a dataset could store depends on the nbits.
3.1.2. Characteristics of the GeoTIFF dataset
A GeoTIFF (*.tif or *.tiff) is a TIFF image that stores its geospatial information inside the file itself, as tags (TIFF stands for Tagged Image File Format). Unlike a BIL dataset, which relies on a separate header file, a GeoTIFF is self-contained: pixel values and all the dataset characteristics —dimensions, pixel size, location, pixel type, nodata value, and coordinate reference system— travel together in one file.
This is the main practical difference between the two formats. A BIL header can be missing keywords or be incorrect, forcing you to edit it by hand; in a GeoTIFF that information is written by the software that created the file and is normally read correctly without editing. In particular, the GeoTIFF explicitly records the pixel type (unsigned integer, signed integer, or floating point) and the nodata value, so there is no need for a pixeltype keyword and no risk of the program assuming the wrong type. Rainfall data such as CHIRPS, which uses −9999 as nodata, therefore requires no extra steps. The file also embeds the CRS (for CHIRPS, EPSG:4326 / WGS 84) and is often stored with lossless compression (LZW, DEFLATE), which reduces size without altering pixel values, whereas BIL files are uncompressed.
Two minor notes: a GeoTIFF may come with optional auxiliary files (a world file .tfw, statistics .aux.xml, or overviews .ovr), and files larger than 4 GB are written as BigTIFF, a variant that most current software reads transparently. To inspect any of these characteristics, open the layer properties in QGIS and go to the Information tab, which lists the dimensions, pixel size, data type, nodata value, and CRS.
3.1.3. Vector data
Another type of data used in fews_tools is vector data in shapefile format (*.shp). To get more information about how to open, create, or edit shapefiles in QGIS, see Intro to QGIS.
3.1.4. Tables
The GeoCLIM program uses tables in comma-delimited format (*.csv) as input and output data. For example, tables are inputs in the process of blending raster data with station values (BASIICS) or validating raster data. For the blending process, the CSV table must have columns for ID, latitude (lat), longitude (long), year, and time period (pentads, dekads, or months), such as the months of January-December in Figure 3-4. The ID, lat, lon, and year columns do not have to be in any order, and additional columns are permitted. However, the time-period columns need to be consecutive (Figure 3-2).
3.1.5. Climate data archives in FEWS Tools
A climate data archive is a compressed file with the ending ‘.climdata’ that contains a group of files of a specific climatic variable. The Archive contains the necessary information to define the data in the FEWS Tools plugin. This is an easy way to save a copy of the dataset, or part of it, that has been defined in the tool or to share data between users. See section 3.4 for instructions on how to create an Archive.
3.2. Download data
GeoCLIM allows you to download data for those datasets that include FTP information on their definition form. To download data, follow the steps below:
-
Click on the Download Data by Date icon on the main toolbar. See red box in Figure 3-3.
-
Select the dataset that you want. See Figure 3-4(1).
-
The Data Duration (2), and Parameter (3) should change accordingly.
-
Select the starting and ending date. See Figure 3-4.
-
Click OK to start the download. See Figure 3-4.
3.3. Data availability/export data
The View Available Data function, see red box in Figure 3‑5 facilitates the following:
-
Display a list of files in your dataset
-
Display a list of missing files in the timeseries
-
Deletes files in the dataset.
-
Exports the dataset or part of it into a GeoCLIM archive or in same file format.
To check the available datafiles in your dataset click on the View Available Data icon on the main toolbar.
The function will open the information for the default dataset. Select the desired dataset from the pulldown menu. The Available Rainfall Data tool also allows you to identify the missing data in the time series, Figure 3-6. Click on List Missing Data to obtain a list of missing files in the time series. Select the complete or part of the dataset and click on the Export button to save the selected files into a data archive that could be shared with other GeoCLIM users. A data archive is a compressed file that includes all the files, selected on the Available Rainfall Data tool, together with the data-definition information.
3.4. Create an Archive
To create an archive, follow the steps below:
-
Open the View Available Data tool.
-
Select a climate variable.
-
Select the entire or part of the time-series to be added to an archive.
-
Click Export.
-
Give a name to the new archive and click OK.
-
Select Include averages to include the averages for the selected periods in the archive.
-
Select the destination directory.
-
Click OK. A new file with
.climdataformat is created in the destination directory.
3.5. Importing archives
To import an GeoCLIM archive, follow the steps below:
-
Open the Import Climate Data Archives tool from the toolbar. See Figure 3-7.
-
Browse to the archive to be imported and click Import.
To ensure that the archive imported correctly, open the View Available Data function, and click on the dataset pulldown menu and select the new dataset.