Release Notes for Data Sprint 21.04

Introduction

The following list of new features, tasks, and resolved bugs are included in the latest Sprint 21.04 release to FDW/FDE:

Features

Prototype of replacement MarketPriceResource - https://fewsnet.atlassian.net/browse/DATA-742

Vector Tile Renderer for DataPointViewSet subclasses - https://fewsnet.atlassian.net/browse/DATA-772
Update the urls and related viewsets to allow us to call the maps and shapefiles using url parameters rather than query parameters https://fewsnet.atlassian.net/browse/DATA-780
Market Price ingestion pipeline from Magpi to FDW https://fewsnet.atlassian.net/browse/DATA-795
Custom country filters for Trade domain https://fewsnet.atlassian.net/browse/DATA-857
Adding new phase description in the FDW https://fewsnet.atlassian.net/browse/DATA-860
Add availability_description as textual column in API output https://fewsnet.atlassian.net/browse/DATA-868
New data sources or viz in support of Country Case Study - https://fewsnet.atlassian.net/browse/DATA-871
Prototype: Reusable CHIRPS, NDVI, WRSI maps and charts https://fewsnet.atlassian.net/browse/DATA-872
Enhancements to Reusable visualizations https://fewsnet.atlassian.net/browse/DATA-880
Parameter Form Enhancements or Reusable Visualizations https://fewsnet.atlassian.net/browse/DATA-881
Update FDE Endpoint for Exchange Rate Values https://fewsnet.atlassian.net/browse/DATA-898
Improve ad-hoc shapefile calibration validation and help text - https://fewsnet.atlassian.net/browse/DATA-899
Disable zero value for Market Price https://fewsnet.atlassian.net/browse/DATA-908
Add Geographic_unit field to PriceIndex resource https://fewsnet.atlassian.net/browse/DATA-909
Add pct_change_from_one_year_ago to Exchange Rate Value Facts and Secondary Price Index extracts https://fewsnet.atlassian.net/browse/DATA-910

Tasks

Update Burkina Faso country status metadata https://fewsnet.atlassian.net/browse/DATA-892

Bugs

For Remote Monitoring Countries, the admin0 attributes are named inconsistently: fix the data and make resource case insensitive https://fewsnet.atlassian.net/browse/DATA-816
In DataCollection admin "Save and continue editing" and "Save and add another" buttons doesn't recognize uploaded data only "Save" button does https://fewsnet.atlassian.net/browse/DATA-827
UserViewSet should have custom permissions https://fewsnet.atlassian.net/browse/DATA-855
Price Domain Not Rendering Filters https://fewsnet.atlassian.net/browse/DATA-889
Sort out a bug with per_country_aliases in the Classified Products search https://fewsnet.atlassian.net/browse/DATA-915
"Docs" Button Missing from FDE https://fewsnet.atlassian.net/browse/DATA-921
On Trade Domain, Revert country names to full-length - currently are abbreviated https://fewsnet.atlassian.net/browse/DATA-926

Features

Prototype of replacement MarketPriceResources - https://fewsnet.atlassian.net/browse/DATA-742

This feature is a prototype for replacing the existing MarketPriceResource with a new MarketPriceResource that offers flexibility for recognizing data. In subsequent Sprints we can add new functionality as required to resolve the other tickets. Eventually we can deprecate LegacyMarketPriceResource (the previous version of this MarketPriceResources).

Vector Tile Renderer for DataPointViewSet subclasses - https://fewsnet.atlassian.net/browse/DATA-772

This feature is about applying the vector tiles using ST_AsMVT with an MVTTileRenderer for the DataPointViewSet endpoint which includes the necessary attributes for the interactive thematic maps.

Update the urls and related viewsets to allow us to call the maps and shapefiles using url parameters rather than query parameters https://fewsnet.atlassian.net/browse/DATA-780

The ipcmap , ipcpngmap and /api/ipcpackage API endpoints have been enhanced to allow the calling of maps and shapefiles using the URL parameters in addition to the query parameters, which make it clearer for the user what they are accessing, for instance;
https://fdw.fews.net/ipcmap/region/902/ML1/2020-02-01 ,
https://fdw.fews.net/ipcpngmap/region/902/ML1/2020-02-01 ,
https://fdw.fews.net/api/ipcpackage/region/902/ML1/2020-02-01

Market Price ingestion pipeline from Magpi to FDW https://fewsnet.atlassian.net/browse/DATA-795

This was a prototype evaluation to determine if and to what degree MagPi could integrate with FDW. The evaluation was focused on Market Price data ingestion from Haiti, and it was successful. More work on this feature is forthcoming.

Custom country filters for Trade domain - https://fewsnet.atlassian.net/browse/DATA-857

This feature enhances the filtering capability of the APIs (/api/tradeflowquantity/ and /api/tradeflowquantityvalue/) of the Trade data domain by applying the filters country_code, country and geographic_group across the three-country related fields, i.e. reporting_country, source_country and destination_country, for instance ;
(https://fdw.fews.net/api/tradeflowquantity/?country=ethiopia&geographic_group=14) and
(https://fdw.fews.net/api/tradeflowquantityvalue/?country_code=mw&geographic_group=14)

Adding new phase description in the FDW https://fewsnet.atlassian.net/browse/DATA-860

This feature is about adding the new description type Famine Likely for phase 5 IPC Classification data.

Add availability_description as textual column in API output - https://fewsnet.atlassian.net/browse/DATA-868

This feature is about enhancing the marketpricefacts API endpoint in order to show the availability_description as textual column in API output.

New data sources or viz in support of Country Case Study - https://fewsnet.atlassian.net/browse/DATA-871

This feature consisted of developing a web-based visualization prototype based on IMF IFS data, and it was made successfully and presented to the Early Warning Team.

Prototype: Reusable CHIRPS, NDVI, WRSI maps and charts https://fewsnet.atlassian.net/browse/DATA-872

This enhancement is about developing a prototype for reusable CHIRPS, NDVI, WRSI maps and charts, for illustrating potential new visualization capabilities.

Enhancements to Reusable visualizations https://fewsnet.atlassian.net/browse/DATA-880

This ticket included a number of small feature additions and bug fixes to the web-based reusable visualization prototypes mentioned.

Parameter Form Enhancements for Reusable Visualizations https://fewsnet.atlassian.net/browse/DATA-881

This included a number of feature additions to the parameter form for the reusable visualization prototype, such as:
- Visual indication of mandatory parameters,
- Remove border from individual parameters,
- Make get_parameters and get_parameter form a class method,
- Generate clean urls for the refreshable link,
- Use dataseries=1234&dataseries=5678 in the refreshable link for consistency with the API,
- Support the use of &_refresh=True in the refreshable URLs.

Update FDE Endpoint for Exchange Rate Values https://fewsnet.atlassian.net/browse/DATA-898

This enhancement is about changing the FDE API Endpoint of the for Exchange Rate Values from /api/datapointfacts to /api/exchangeratevaluefacts, for instance; (http://fdw.fews.net/api/exchangeratevaluefacts/?format=csv&dataseries=6466306&fields=simple)

Improve ad-hoc shapefile calibration validation and help text - https://fewsnet.atlassian.net/browse/DATA-899

This feature is about enhancing the ad-hoc shapefile calibration validation and help text of the (https://fdw.fews.net/en/spatial/adhocpopulation/ ) which easily helps to export the population and demography of the custom shapefiles.

Disable zero value for Market Price https://fewsnet.atlassian.net/browse/DATA-908

This enhancement is about adding a validation error message, i.e Market Price Datapoints cannot have a 0 (Zero) value , for when adding or uploading price data with a zero values.

Add Geographic_unit field to PriceIndex resource https://fewsnet.atlassian.net/browse/DATA-909

This feature is about introducing the field geographic_unit to the PriceIndexResource so that it can match up the FNIDs and names of the geograhic_unit column of the uploaded file.

Add pct_change_from_one_year_ago to Exchange Rate Value Facts and Secondary Price Index extracts https://fewsnet.atlassian.net/browse/DATA-910

To provide additional analytical capability, the API endpoints (i.e /api/exchangeratefacts and api/datapointfacts) has been enhanced to include pct_change_from_one_year_ago in the extracts for the Exchange Rate Value Facts field and a custom Price Index Facts; (i,e https://fdw.fews.net/api/exchangeratevaluefacts/?format=html&dataseries=3068019&dataseries=3068017&fields=simple and https://fdw.fews.net/api/datapointfacts/?format=html&dataseries=5404600).

Tasks

Update Burkina Faso country status metadata https://fewsnet.atlassian.net/browse/DATA-892

This task added the country Burkina Faso to the group FEWS NET Presence Countries(https://fdw.fews.net/en/admin/common/countrygroup/911/change/) and removed it as well from the group FEWS NET Remote Monitoring Countries(https://fdw.fews.net/en/admin/common/countrygroup/912/change).

Bugs

For Remote Monitoring Countries, the admin0 attributes are named inconsistently: fix the data and make resource case insensitive https://fewsnet.atlassian.net/browse/DATA-816

There was an issue in naming the admin0 attributes of the shapefile exported for RM countries and this was resolved.

In DataCollection admin "Save and continue editing" and "Save and add another" buttons doesn't recognize uploaded data only "Save" button does https://fewsnet.atlassian.net/browse/DATA-827

There was a problem in recognizing the uploaded data values if the user uses the buttons Save and continue editing and Save and add another of the data collection screen (i.e https://fdw.fews.net/en/admin/warehouse/datacollection/add/) and this was resolved and it is working as expected for all the data domains.

UserViewSet should have custom permissions https://fewsnet.atlassian.net/browse/DATA-855

There was an issue with access to the API, https://fdw.fews.net/api/user/, where the list of all fdw users were shown without logging into the system and this was resolved.

Sort out a bug with per_country_aliases in the Classified Products search https://fewsnet.atlassian.net/browse/DATA-915

There was a problem in searching of the classified products with per_country_aliases and this was resolved.

“Docs” Button Missing from FDE https://fewsnet.atlassian.net/browse/DATA-921

There was an issue of accessing the Mayan system due to the missing of the Docs button on the FDE and this was resolved. The button is returned and the functionality is working as expected.

On Trade Domain, Revert country names to full-length - currently are abbreviated https://fewsnet.atlassian.net/browse/DATA-926

In the trade data domain of the FDE (i.e. https://fdw.fews.net/data-explorer/#/trade), the reporting country, source, and destination were represented in abbreviated texts and this was corrected. They are now displayed in their full country names.