Release Notes for Data Sprint 20.11

Introduction

Date: 21 December 2020

The following list of new features, tasks, and resolved bugs are included in the latest Sprint 20.11.11 release to FDW/FDE:

Features

• Design and implement import of WorldPop demographic data https://fewsnet.atlassian.net/browse/DATA-479

• Create a PandasKMLRenderer https://fewsnet.atlassian.net/browse/DATA-546

• Writable API for DataCollection, DataCollectionPeriod and DataPoint https://fewsnet.atlassian.net/browse/DATA-552

• Managing weekly commodity availability data in the FDW https://fewsnet.atlassian.net/browse/DATA-568

• Enhanced Country API for easier maintenance of visualizations https://fewsnet.atlassian.net/browse/DATA-569

• Add verbiage on the login page indicating that we are currently not accepting new FDE account registrations https://fewsnet.atlassian.net/browse/DATA-599

• Replicate the "Options" tab functionality that is currently in the Price domain (FDE) onto the Trade Domain in FDE https://fewsnet.atlassian.net/browse/DATA-600

• Include a filter box for Flow Type on the Trade domain https://fewsnet.atlassian.net/browse/DATA-601

• Create a DataCollectionFilter and DataCollectionPeriodFilter and update serializers https://fewsnet.atlassian.net/browse/DATA-624

• IPC Map improvements https://fewsnet.atlassian.net/browse/DATA-625

• IPC phase data validations https://fewsnet.atlassian.net/browse/DATA-627

• Sending email notification to the user after Drupal publication https://fewsnet.atlassian.net/browse/DATA-630

Tasks

• Reconciling crop production indictors of the crop_region and crop_unit for Afmadow, Lower Juba, Somalia Crop Reporting Unit https://fewsnet.atlassian.net/browse/DATA-453

• Reload historic IPC data https://fewsnet.atlassian.net/browse/DATA-541

• Reload all Pre-2018 IPC data into the system https://fewsnet.atlassian.net/browse/DATA-597

Bugs

• Cross Border Trade data are not recognized from the data collection screen https://fewsnet.atlassian.net/browse/DATA-605

• Semi-structured i.e (WFP VAM, Afghanistan: Days Work Available ) uploading issue https://fewsnet.atlassian.net/browse/DATA-614

• Geographic filter of the FDE took longer minutes to load the data https://fewsnet.atlassian.net/browse/DATA-645

• The IPCPackageViewSet doesn't support KMU documents correctly https://fewsnet.atlassian.net/browse/DATA-705

• Market name not recognized in upload file https://fewsnet.atlassian.net/browse/DATA-709

Features

Design and implement import of WorldPop demographic data https://fewsnet.atlassian.net/browse/DATA-479

• This feature is about loading the WorldPop data into Amazon’s S3 bucket and enhancing the APIs to get a population data extract in a tabular format (as seen here: https://fdw.fews.net/api/geographicunit/?format=csv&unit_type=fsc_admin_lhz&country_code=KE&as_of_date=2020-12-1&fields=with_population) and with a demographic breakdown (as seen here: https://fdw.fews.net/api/geographicunit/?format=csv&unit_type=fsc_admin_lhz&country_code=KE&as_of_date=2020-12-1&fields=with_population ).

Create a PandasKMLRenderer https://fewsnet.atlassian.net/browse/DATA-546

• This enhancement is about allowing the KML format extract for the for DataPointViewSet, for instance, https://fdw.fews.net/api/cropproductionindicatorvalue/?format=kml&country_code=AF&cpcv2=R01.

Writable API for DataCollection, DataCollectionPeriod and DataPoint https://fewsnet.atlassian.net/browse/DATA-552

• This feature is about enhancing the operations of creating, reading, updating and deleting of the existing data collection, data collection period, data point API end points from the back end.

Managing weekly commodity availability data in the FDW https://fewsnet.atlassian.net/browse/DATA-568

• This feature is about allowing the import of the weekly commodity availability data into FDW and extract the data from the API end point of the marketpricefacts (i.e. https://fdw.fews.net/api/marketpricefacts/).

Enhanced Country API for easier maintenance of visualizations https://fewsnet.atlassian.net/browse/DATA-569

• This feature is about enhancing the country API (https://fdw.fews.net/api/country/) to include all the key fields which help for easier maintenance of the visualization.

Add verbiage on the login page indicating that we are currently not accepting new FDE account registrations https://fewsnet.atlassian.net/browse/DATA-599

• This feature is about introducing a verbiage on the login page indicating that we are currently not accepting new FDE account registrations (https://fdw.fews.net/accounts/signup/).

Replicate the "Options" tab functionality that is currently in the Price domain (FDE) onto the Trade Domain in FDE https://fewsnet.atlassian.net/browse/DATA-600

• This enhancement is about adding the Options tab functionality to the Trade data domain of the FDE (https://fdw.fews.net/data-explorer/#/trade).

Include a filter box for Flow Type on the Trade domain https://fewsnet.atlassian.net/browse/DATA-601

• This feature is about introducing Flow Type filter to the Trade data domain of the FDE (https://fdw.fews.net/data-explorer/#/trade).

Create a DataCollectionFilter and DataCollectionPeriodFilter and update serializers https://fewsnet.atlassian.net/browse/DATA-624

• This feature is about adding the filters datasourcedocument_name,datasourceorganization_name , data_domain , datasourcedocument_source_type, country_code ,collection_status for the APIs (https://fdwdev.fews.net/api/datacollection/ and https://fdwdev.fews.net/api/datacollectionperiod/) which enables user to get specific data.

IPC Map improvements https://fewsnet.atlassian.net/browse/DATA-625

• This enhancement is about improving the IPC map by supporting the remote monitoring countries and using an enhanced country API .

IPC phase data validations https://fewsnet.atlassian.net/browse/DATA-627

• This feature is about introducing validations for IPC phases data imports.

Sending email notification to the user after Drupal publication https://fewsnet.atlassian.net/browse/DATA-630

• This feature is about implementing email notification to the user who published the IPC classification data.

Tasks

Reconciling crop production indictors of the crop_region and crop_unit for Afmadow, Lower Juba, Somalia Crop Reporting Unit https://fewsnet.atlassian.net/browse/DATA-453

• This task was about reconciling the crop production indicators i.e (crop_region and crop_unit) by changing the geographic unit types of Afmadow & Hagar to crop_region unit type .

Reload historic IPC data https://fewsnet.atlassian.net/browse/DATA-541

• This task was about loading the historic FSO and FSOU data of the IPC Classification into FDW and the data was successfully loaded.

Reload all Pre-2018 IPC data into the system https://fewsnet.atlassian.net/browse/DATA-597

• This task was about loading pre-2018 IPC historical data into the FDW Production environment (https://fdw.fews.net/en/admin/warehouse/datacollection/?data_domain=IPCClassification) and the data was loaded successfully.

Bugs

Cross Border Trade data are not recognized from the data collection screen https://fewsnet.atlassian.net/browse/DATA-605

• There was an issue in loading cross border trade data from the data collection screen (https://fdw.fews.net/en/admin/warehouse/datacollection/add/) and this was resolved and the functionality is working as expected.

Semi-structured i.e (WFP VAM, Afghanistan: Days Work Available ) uploading issue https://fewsnet.atlassian.net/browse/DATA-614

• There was a problem in loading the Days Work Available data and this was resolved and the final template for uploading the data has also been shared.

Geographic filter of the FDE took longer minutes to load the data https://fewsnet.atlassian.net/browse/DATA-645

• There was an issue in loading the data values of the geographic filter of the crop data domain and this was resolved.

The IPCPackageViewSet doesn't support KMU documents correctly https://fewsnet.atlassian.net/browse/DATA-705

• The downloaded KMU shapefiles were not showing the correct documents and this was resolved and it is working as expected .

Market name not recognized in upload file https://fewsnet.atlassian.net/browse/DATA-709

• There was a problem in recognizing market in upload files and this was resolved and can now upload the files successfully.