Agroclimatological Tools Knowledge Base
English French Spanish
English French Spanish

Chapter 3: Data Management in the FEWS Tools Plugin

Summary

This chapter examines the data management in the FEWS Tools plugin for QGIS, including the different types of data formats used, downloading data, data availability, and data import/export using archives.

3.1. Data types

GeoCLIM uses four main data types: raster data in BIL format BIL (* .bil) and GeoTIF (*.tif or *.tiff), vector data in shapefile format (*.shp), tables in comma-delimited format (*.csv), and data Archives.

3.1.1. Characteristics of the raster dataset

A band interleaved by line (BIL) dataset contains two files: a (*.bil) file and a header file (*.hdr). The .bil file is a binary file that contains the pixel values (e.g., rainfall, temperature, etc.), while the HDR file contains the characteristics of the dataset, such as the geographic location, pixel size, and depth.

HDR file text.
Figure 3-1 Example of a HDR file.

The header file is an ASCII text file; it can be generated or edited from a text editor (e.g., Notepad). For example, Figure 3‑1 shows that the header file contains information about the number of columns (ncols), number of rows (nrows), number of bits per pixel (nbits), and size of pixel (xdim and ydim), among others.  Figure 3‑1 also shows the xdim and ydim values corresponding to the horizontal (x-dimension) and vertical (y-dimension) dimensions of a pixel with a size of 0.05 degrees, which is about 5 kilometers. The ulxmap and ulymap correspond to the x-axis and the y-axis coordinates of the center of the upper-left pixel of the raster image. There are additional keywords that the header could have (Table 3‑1) (ArcMap 10.3 Help, ESRI). Sometimes, if the header file is incorrect, you may need to modify it so that the data is read correctly by the program.

By default, the BIL dataset pixel type used is unsigned integers, unless the keyword pixeltype is used in the HDR file, and its value is signedint.

Keyword

Acceptable Value

Default

nrows

Any integer > 0

None

ncols

Any integer > 0

None

nbands

Any integer > 0

1

nbits

1, 4, 8, 16, 32

8

pixeltype

SIGNEDINT

Unsigned Integer

byteorder

I = Intel; M - Motorola

Same as host machine

layout

bil, bip, bsq

bil

skipbytes

Any integer ≥ 0

0

ulxmap

Any real nuber

0

ulymap

Any real number

nrows - 1

xdim

Any real number

1

ydim

Any real number

1

bandrowbytes

Any integer ≥ 0

Smallest integer ≥ (ncols x nbits) / 8

totalrowbytes

Any integer ≥ 0

For bil: nbands x bandrowbytes; for bip: smallest integer ≥ (ncols x nbands x nbits) / 8

bandgapbytes

Any integer ≥ 0

0

Table 3-1. The header file is composed of a series of keywords and their respective accepted values. Source: ArcMap 10.3 Help, ESRI.

An important keyword in the header file is the pixeltype since it defines the type of value, unsigned (+), or signed (+ or -) a pixel could have. For example, rainfall data could only have unsigned (+) values since precipitation is only positive. However, if you look at the header file for CHIRPS, it uses signed values (+ or -) since the nodata value is -9999. In the example in Figure 3-1, the pixeltype is missing, so the program assumes that the data is unsigned; in the case of CHIRPS, we would have to add a new line on the header file defining the pixeltype as signedint. Another keyword to keep in mind is the nbits because it indicates the number of bits per pixel or the depth of the raster image (e.g., nbits=16 bit means that a pixel in the raster dataset can have any one of 216 = 65536 unique values). Table 3‑2 (ESRI, Bit depth capacity for raster dataset pixels) shows a list of values a raster dataset could have depending on the pixel depth or nbits value.

Pixel Type

Bit Depth

Value range for a raster dataset

unsigned

1 bit

0 to 1

unsigned

2 bit

0 to 4

unsigned

4 bit

0 to 16

unsigned

8 bit

0 to 255

signed

8 bit

-128 to 127

unsigned

16 bit

0 to 65535

signed

16 bit

-32768 to 32767

unsigned

32 bit

0 to 4294967295

signed

32 bit

-2147483648 to 2147483647

floating point

32 bit

-3.402823466e+38 to 3.402823466e+38

Table 3-2 The range of values a dataset could store depends on the nbits.

3.1.2. Characteristics of the GeoTIFF dataset

A GeoTIFF (*.tif or *.tiff) is a TIFF image that stores its geospatial information inside the file itself, as tags (TIFF stands for Tagged Image File Format). Unlike a BIL dataset, which relies on a separate header file, a GeoTIFF is self-contained: pixel values and all the dataset characteristics —dimensions, pixel size, location, pixel type, nodata value, and coordinate reference system— travel together in one file.

This is the main practical difference between the two formats. A BIL header can be missing keywords or be incorrect, forcing you to edit it by hand; in a GeoTIFF that information is written by the software that created the file and is normally read correctly without editing. In particular, the GeoTIFF explicitly records the pixel type (unsigned integer, signed integer, or floating point) and the nodata value, so there is no need for a pixeltype keyword and no risk of the program assuming the wrong type. Rainfall data such as CHIRPS, which uses −9999 as nodata, therefore requires no extra steps. The file also embeds the CRS (for CHIRPS, EPSG:4326 / WGS 84) and is often stored with lossless compression (LZW, DEFLATE), which reduces size without altering pixel values, whereas BIL files are uncompressed.

Two minor notes: a GeoTIFF may come with optional auxiliary files (a world file .tfw, statistics .aux.xml, or overviews .ovr), and files larger than 4 GB are written as BigTIFF, a variant that most current software reads transparently. To inspect any of these characteristics, open the layer properties in QGIS and go to the Information tab, which lists the dimensions, pixel size, data type, nodata value, and CRS.

3.1.3. Vector data

Another type of data used in fews_tools is vector data in shapefile format (*.shp). To get more information about how to open, create, or edit shapefiles in QGIS, see Intro to QGIS.

3.1.4. Tables  

The GeoCLIM program uses tables in comma-delimited format (*.csv) as input and output data. For example, tables are inputs in the process of blending raster data with station values (BASIICS) or validating raster data. For the blending process, the CSV table must have columns for ID, latitude (lat), longitude (long), year, and time period (pentads, dekads, or months), such as the months of January-December in Figure 3-4. The ID, lat, lon, and year columns do not have to be in any order, and additional columns are permitted. However, the time-period columns need to be consecutive (Figure 3-2).       

Example csv file with time period columns in consecutive order from January to December.
Figure 3-2 GeoCLIM accepts tables in comma delimited format (CSV).

3.1.5. Climate data archives in FEWS Tools

A climate data archive is a compressed file with the ending ‘.climdata’ that contains a group of files of a specific climatic variable. The Archive contains the necessary information to define the data in the FEWS Tools plugin. This is an easy way to save a copy of the dataset, or part of it, that has been defined in the tool or to share data between users. See section 3.4 for instructions on how to create an Archive.

3.2. Download data

GeoCLIM allows you to download data for those datasets that include FTP information on their definition form. To download data, follow the steps below:

  1. Click on the Download Data by Date icon on the main toolbar. See red box in Figure 3-3.

The FEWS Tools toolbar with the Download Data by Date option highlighted.
Figure 3-3 To download data using fews_tools, make sure that the ftp information is included in the dataset definition. Click on the download icon and complete the form.
  1. Select the dataset that you want. See Figure 3-4(1).

  2. The Data Duration (2), and Parameter (3) should change accordingly.

  3. Select the starting and ending date. See Figure 3-4.

  4. Click OK to start the download. See Figure 3-4.

GeoCLIM Figure 3-6.png
Figure 3-4 Download data by date for a selected dataset. See global data information in Chapter 2 for details.

3.3. Data availability/export data

The View Available Data function, see red box in Figure 3‑5 facilitates the following:

  • Display a list of files in your dataset

  • Display a list of missing files in the timeseries

  • Deletes files in the dataset.

  • Exports the dataset or part of it into a GeoCLIM archive or in same file format.

To check the available datafiles in your dataset click on the View Available Data icon on the main toolbar.

The FEWS Tools toolbar with the View Available Data option highlighted.
Figure 3-5 To view the data available to run the GeoCLIM/GeoWRSI functions, click on the View Available Data icon and select the desired dataset.

The function will open the information for the default dataset. Select the desired dataset from the pulldown menu. The Available Rainfall Data tool also allows you to identify the missing data in the time series, Figure 3-6. Click on List Missing Data to obtain a list of missing files in the time series. Select the complete or part of the dataset and click on the Export button to save the selected files into a data archive that could be shared with other GeoCLIM users. A data archive is a compressed file that includes all the files, selected on the Available Rainfall Data tool, together with the data-definition information.

The View Available Data screen.
Figure 3-6 The View Available Data tool allows to list the data files in the time series for a selected climate dataset.

3.4. Create an Archive

To create an archive, follow the steps below:

  1. Open the View Available Data tool.

  2. Select a climate variable.

  3. Select the entire or part of the time-series to be added to an archive.

  4. Click Export.

  5. Give a name to the new archive and click OK.

  6. Select Include averages to include the averages for the selected periods in the archive.  

  7. Select the destination directory.

  8. Click OK. A new file with .climdata format is created in the destination directory.

3.5. Importing archives

To import an GeoCLIM archive, follow the steps below:

  1. Open the Import Climate Data Archives tool from the toolbar.  See Figure 3-7.

  2. Browse to the archive to be imported and click Import.

The FEWS Tools toolbar with the Import Climate Data Archives option highlighted.
Figure 3-7 FEWS Tools allows you to share with other users a complete or part of a dataset using a data archive.

To ensure that the archive imported correctly, open the View Available Data function, and click on the dataset pulldown menu and select the new dataset.