Release Notes for Data Sprint 21.10

Dated: 12 November, 2021

Introduction

The following list of new features, tasks and resolved bugs are included in the latest Sprint 21.10.2 release to FDW/FDE:

Features

Price availability validation https://fewsnet.atlassian.net/browse/DATA-972
Replace the Southern Africa Cross-Border Trade on Data Centre with a direct download from FDW https://fewsnet.atlassian.net/browse/DATA-1064
Operationalize FAOB Ingestion https://fewsnet.atlassian.net/browse/DATA-1110
Enhancements to Market Price Time Series extracts https://fewsnet.atlassian.net/browse/DATA-1129
Create specific viewsets for calculated data series https://fewsnet.atlassian.net/browse/DATA-1135
Add "Fare" to market products price_type https://fewsnet.atlassian.net/browse/DATA-1178
Use Direct ExchangeRate api end point when filtering/searching exchange rates other domain by country https://fewsnet.atlassian.net/browse/DATA-1180
Remove Copyright footer and add Rights Disclaimer page https://fewsnet.atlassian.net/browse/DATA-1181
Allow `NOT_COLLECTED` data points to be reported and imported in a generic way https://fewsnet.atlassian.net/browse/DATA-1201
Review Other domain end points to ensure they point to their specific end points for specific metadata https://fewsnet.atlassian.net/browse/DATA-1206

Tasks

Commodity Balances historic data upload https://fewsnet.atlassian.net/browse/DATA-820
Change drupal node title from Central America and Caribbean to Guatemala and Haiti https://fewsnet.atlassian.net/browse/DATA-1200

Bugs

The buttons “Download “ , “Browse” , “Chart” and the “Excel Web Query URL” are missing from the price data set page of the FDW https://fewsnet.atlassian.net/browse/DATA-1189
Upon creating calculated indices (https://fdw.fews.net/en/admin/price/calculatedindex/add/), the Owner dropdown field doesn’t populate the username of the user that logged in https://fewsnet.atlassian.net/browse/DATA-1190
New price data information for Zimbabwe is not recognized by the FDW https://fewsnet.atlassian.net/browse/DATA-1207

Features

Price availability validation https://fewsnet.atlassian.net/browse/DATA-972

This enhancement is about adding a validation error message, which is stated as: Availability should be set to 0 (Zero) or left empty for unavailable prices, for adding an availability value other than zero for the Not Available data point values.

Replace the Southern Africa Cross-Border Trade on Data Centre with a direct download from FDW https://fewsnet.atlassian.net/browse/DATA-1064

This feature enhancement allows the direct download of the Southern Africa Cross Border Trade data file from the data center on the FEWS NET website (https://fdw.fews.net/api/tradeflowquantityvalue/?dataset=SA XBT trade dataset&format=xlsx&fields=website) and the downloaded file contains the selected fields fit for the website extracts.

Operationalize FAOB Ingestion https://fewsnet.atlassian.net/browse/DATA-1110

This feature is about introducing a pipeline that imports FAOB data into FDW. While this feature enables the functionality, there will be future work to improve upon the user interface and allow the project to utilize the data more effectively.

Enhancements to Market Price Time Series extracts https://fewsnet.atlassian.net/browse/DATA-1129

This feature is about enhancing the time series extract of the market price data across multiple areas (for instance, https://fdw.fews.net/api/marketpricefacts/?format=html&timeseries=true&dataseries=6453165) which includes the following:
- Showing Country or Geographic Region for the national-level Exchange rates columns.
- Adding new metadata heading column after the last exchange rate and before the first secondary price index with the same values except using "base value" instead of "unit" and "name" instead of indicator_type.
- Updating the existing second metadata heading column that appears before the first exchange rate so that it uses "-" instead of "base period"; "rate type" instead of "indicator type".
- Removing prefix from the base value and base period metadata values.
- Using “Collected" instead of "Stored" as the "field_source” for the Exchange Rate and Secondary Price Index data series which are retrieved from the system.
- Using the actual indicator name of the exchange rates , such as e.g. "Domestic Currency per USD" instead of the generic "Exchange Rate".
- Rounding Exchange Rate Values to 8 decimal places.
- Populating the Latitude and Longitude metadata for the Market-level exchange rates.

Create specific viewsets for calculated data series https://fewsnet.atlassian.net/browse/DATA-1135

This feature is about implementing the calculatedpriceindex, calculatedpriceindexvalue, calculatedpriceindexvaluefacts , priceratio , priceratiovalue and priceratiovaluefacts APIs:
- https://fdw.fews.net/api/calculatedpriceindex/ ,
- https://fdw.fews.net/api/calculatedpriceindexvalue/
- https://fdw.fews.net/api/calculatedpriceindexvaluefacts/ ,
- https://fdw.fews.net/api/priceratio/ ,
- https://fdw.fews.net/api/priceratiovalue/ ,
- https://fdw.fews.net/api/priceratiovaluefacts/ (which return data series with the appropriate domain-specific metadata, for instance, https://fdw.fews.net/api/calculatedpriceindex/?format=xml&fields=simple and https://fdw.fews.net/api/priceratio/?format=xml&fields=simple).

Add "Fare" to market products price_type https://fewsnet.atlassian.net/browse/DATA-1178

This enhancement is about incorporating the Fare price type to the Price data domain. It is selectable under the price type dropdown list (i.e https://fdw.fews.net/en/admin/price/marketproduct/add/).

Use Direct ExchangeRate api end point when filtering/searching exchange rates other domain by country https://fewsnet.atlassian.net/browse/DATA-1180

This feature enhanced the search and filter capabilities of the Exchange Rate data series endpoint (/api/exchangerate) so that it searches all country currency assignments instead of just the country assigned to the Data Series.

Remove Copyright footer and add Rights Disclaimer page https://fewsnet.atlassian.net/browse/DATA-1181

This enhancement is about removing the current copyright footer and adding the right disclaimer page to the Help text found on the platform (i.e. https://fdw.fews.net/en/docs/home/data_disclaimer/data_disclaimer.html).

Allow `NOT_COLLECTED` data points to be reported and imported in a generic way https://fewsnet.atlassian.net/browse/DATA-1201

This feature is about allowing the generic import of data points with a Not Collected status which later can be replaced with the actual status and data values.

Review Other domain end points to ensure they point to their specific end points for specific metadata https://fewsnet.atlassian.net/browse/DATA-1206

This feature is about further enhancing the Other data domain (also known as the Semi-Structured domain) end point of the Secondary Price Index which now return data with the appropriate domain-specific metadata (i.e http://fdw.fews.net/api/priceindex/ and http://fdw.fews.net/api/priceindexvaluefacts/).

Tasks

Commodity Balances historic data upload https://fewsnet.atlassian.net/browse/DATA-820

This task was about uploading the commodity balances historic data and it was successfully loaded to the platform (i.e. https://fdw.fews.net/en/admin/balance/commoditybalance/).

Change drupal node title from Central America and Caribbean to Guatemala and Haiti https://fewsnet.atlassian.net/browse/DATA-1200

This task was about making Drupal node title change from Central America and Caribbean to Guatemala and Haiti on the http://fews.net website.

Bugs

The buttons “Download “ , “Browse” , “Chart” and the “Excel Web Query URL” are missing from the price data set page of the FDW https://fewsnet.atlassian.net/browse/DATA-1189

In the Price data set page of the FDW (i.e https://fdw.fews.net/en/admin/price/pricedataset/), the buttons Download , Browse , Chart and the Excel Web Query URL were missing and this was resolved.

Upon creating calculated indices (https://fdw.fews.net/en/admin/price/calculatedindex/add/), the Owner dropdown field doesn’t populate the username of the user that logged in https://fewsnet.atlassian.net/browse/DATA-1190

Upon creating calculated indices (https://fdw.fews.net/en/admin/price/calculatedindex/add/), the Owner dropdown field didn’t populate the username of the user that logged in and this was resolved.

New price data information for Zimbabwe is not recognized by the FDW https://fewsnet.atlassian.net/browse/DATA-1207

There was a problem in the system recognizing the price data for the country Zimbabwe and this was resolved and the metadata successfully loaded.