Preparing Data for Upload
There are three categories of information that need to be checked when preparing data for upload.
Spreadsheets for most domains require a straight table with one data point per row. Certain domains, such as Prices, will load cross-tabs (i.e., with markets down the side and products across the top).
Column headings
Column headings must be recognized by FDW. When uploading ad hoc data, column names may need to be adjusted.
Important: If you regularly receive a document from a non-FEWS NET source that contains alternate column headings, do not edit the column headings every time. Instead, submit a help desk ticket so the Hub can add the headings as aliases to the system.
Source files uploaded to FDW must include properly defined column headings, including:
Column headings with correctly spelled metadata fields.
Column headings that contain metadata fields recognized by FDW.
A column heading for each column that contains data.
The examples below describe common errors related to column headings:
Scenario: Uploading price data using the import screen (https://fdw.fews.net/en/admin/price/marketprice/import/) results in the following error message:
‘country’ is not in list
Explanation: The country column heading is missing or spelled incorrectly in the source file.
Solution: Ensure the source file contains the corrected column heading.
Scenario: Uploading price data from the import screen (https://fdw.fews.net/en/admin/price/marketprice/import/) results in the following error message:
['Cannot identify a DataCollectionPeriod for every row.']
Explanation: The error indicates a mismatch in the column headings contained in the file and those recognized by FDW. For example, the data_source_document name or data_source_organization name in the file do not align with those in FDW. This error may also indicate that the file contains a blank column heading.
Solution: Ensure all column headings are titled correctly and there are no blank headings.
Metadata
Metadata values must be recognized by FDW. New metadata must be added through the data series or metadata importing screen.
Source files uploaded to FDW must include appropriate metadata information, including:
Metadata values that match those available in FDW.
Compatible metadata values for each record.
Metadata values free of extra characters or values that cannot be recognized.
The examples below describe common errors related to metadata:
Scenario: Uploading exchange rate data from the import screen (https://fdw.fews.net/en/admin/price/exchangeratevalue/import/) results in the following error message:
· Cannot identify Data Series for row {'datapoint_status': 'Collected', 'dataseries': None, 'market': None, 'datasourcedocument': <DataSourceDocument: FEWS NET, Democratic Republic of the Congo: Exchange rate, Democratic Republic of the Congo>, 'data_domain': 'ExchangeRate', 'from_currency': 'USD', 'to_currency': 'CDF', 'value': 2028.0, 'rate_type': 'Secondary', 'market_name': 'Uvira, Maendeleo', 'country': 'Congo, The Democratic Republic of the', 'data_source_organization': 'FEWS NET, Democratic Republic of the Congo', 'data_source_document': 'Exchange rate, Democratic Republic of the Congo', 'period_date': 44592.0}
Explanation: The market name in FDW and the file do not match.
Solution: Correct the market name in the file.
Scenario: Uploading cross border trade metadata (https://fdw.fews.net/en/admin/trade/tradeflowquantity/import/) results in 3 invalid results:
Explanation: The invalid results are caused by incompatible values across the flow_type, source_country, and destination_country fields. E.g., a flow_type value of export cannot be valid if the source_country and destination_country values are the same.
Solution: The values must be corrected to upload the data successfully.
Scenario: A user uploads IPC data (https://fdw.fews.net/en/admin/warehouse/dataupload/add/) and receives the following error message:
‘NoneType’ object is not iterable
Explanation: This error is caused by an extra character or value under the metadata columns. Solution: Removing the extra character or value will resolve the issue. You can use the TRIM function in Excel to remove extra spaces from text, leaving only a single space between words and no space characters at the start or end of the text.
Scenario: A user uploads price data from either the import screen (https://fdw.fews.net/en/admin/price/marketprice/import/) or data collection screen (https://fdw.fews.net/en/admin/warehouse/datacollection/add/) and receives a skipped result:
An example of receiving a skipped result from the import screen.
An example of receiving a skipped result from the data collection screen.
Explanation: The file contains a new metadata record not in FDW.
Solution: Check for new metadata information that needs to be added through the data series or metadata importing screen before uploading the file again.
An example of adding new metadata information on the Market Products Import screen. The preview screen shows one new record has been detected.
Data values
Data values must be accurately matched to their column headings.
Source files uploaded to FDW must include appropriate data values, including:
Data values that align with the correct metadata field in the column headings.
Data values that are within an expected range.
The examples below describe common errors related to data values:
Scenario: A user uploads price data from the data collection screen (https://fdw.fews.net/en/admin/warehouse/datacollection/add/) and receives the following error message:
Error in 3 market prices from uploaded file warehouse/data collection/filename.xlsxs
Similarly, uploading the same file from the import screen (https://fdw.fews.net/en/admin/price/marketprice/import/) results in the following error message:
‘retail’
Explanation: The file contains interchanged data values for the price_type and product_source columns. As the data values could not be matched to the metadata fields in the column headings, they could not be uploaded.
An example of an error with interchanged values causing an error upon upload.
Solution: Correcting the file so the data values align with the correct metadata field will resolve the issue.