Source Organizations and Source Documents

At its core, the FEWS NET Data Warehouse (FDW) model relies on Source Organizations and Source Documents to organize data provided on a regular basis. This page provides information about defining and managing Source Organizations and Source Documents in FDW.

Data sources in FDW

Source Organizations in FDW refer to the organizations that provide data, such as the National Drought Management Authority (NDMA), Kenya, or the International Organization for Migration (IOM).

Source Documents organize distinct sets of regularly collected data from a given Source Organization. Each set of data is uploaded under its own Source Document. The Source Document is assigned a Document Type that specifies which domain the data is in and additional attributes, such as whether it is observed or projected data.

Data Series are assigned an Indicator and a Geographic Unit in addition to a Source Document. They may contain other metadata as well.

Each data submission for a Source Document is a new Data Collection. The Source Document is typically on a regular collection schedule, for example, Weekly.

Example: FEWS NET is a Source Organization with multiple associated Source Documents. Price data collected by FEWS NET in South Sudan is uploaded to the Source Document named Famine Early Warning System (FEWS NET), South Sudan, Price. This Source Document falls under the Market Product data domain and a new Data Collection is added on a weekly basis. Data is added to multiple Data Series, each uniquely defined by a specific Market, Product, Unit, and Price Type.

Managing Source Organizations

Source Organizations are managed within Metadata Management > Warehouse > Source Organizations.

The Source Organizations admin screen includes the following options to create Source Organizations in FDW:

Add Source Organization: Used to manually add a Source Organization.
Import Source Organizations: Used to create a bulk upload of Source Organizations from a file.

Source Organizations are defined using the following fields:

Name: May use a shortened form of the organization name, e.g., WFP.

Full name: Can contain a longer version of the organization name alongside its abbreviation, e.g., World Food Programme (WFP).

Description: Allows for extra information or detail about the Source Organization.

Country: Select a Country if the data for the Source Organization is tied to one particular country.

Source Documents: Allows the management of Source Documents tied to the Source Organization.

Managing Source Documents

Source Documents are managed within Metadata Management > Warehouse > Source Documents.

The Source Documents admin screen includes the following options to create Source Documents in FDW:

Add Source Document: Used to manually add a Source Document.
Import Source Documents: Used to create a bulk upload of Source Documents from a file.

Source Documents are defined using the following fields:

Source Organization: Use the lookup to select the Source Organization.

Source Document: The Source Document name should describe the source of the data. Naming conventions vary by domain. For example:

Source Document names for the Market Product domain reference the Source Organization and Country, e.g., Famine Early Warning Systems Network (FEWS NET), Nigeria, Price (weekly).
Source Document names for the Acute Food Insecurity Classification specify the report type and country, e.g., Food Security Outlook, Afghanistan.
Source Documents for Spatial data are named in relation to the Document Type. For example, Administrative unit boundaries defined by Census use the following naming convention: Census admin boundaries, [Country].

When including a country in a Source Document name, ensure that it matches the English Name on the Country metadata record in FDW. This information can be found by accessing Metadata Management > Common > Countries from the FDW toolbar.

Document Type: Select the Document Type from the dropdown list.

Country: Select a Country if the data for the Source Document is tied to one particular country. Leave blank for regional or global data.

Description: Provide any extra information or detail, such as the name of the Source Document in the native language of the Country it belongs to.

Usage Policy: Each Source Document must be assigned a Usage Policy, which is used to enforce permissions determining who can access the data. See Categorization and Permissions for additional information. The Usage Policy field has the following options:

Restricted: The source data will only be visible to Users that have been specifically granted permission to use it (or who are Superusers).
- This Usage Policy is only used in rare cases where customized, restricted access to the data is warranted.
Public: The source data may be made available on the public website for analysis by anyone.
Early Warning Analysis Only: For Early Warning Analysis only.
FEWS NET Only: The source data may only be analyzed by FEWS NET staff. The results of the analysis may be published.

Usage Policy Reason: Select a Usage Policy Reason if the Usage Policy is not Public.

Usage Policy Justification: If the Source Document is not Public and requires a Usage Policy Reason, you must also provide a justification for the reasoning behind the selection of the specific Data Usage Policy Reason.

Usage Last Changed: Indicates when the Usage Policy was changed.

Usage Notes: Provide additional notes on the data usage as applicable. For example, data with a Public usage policy may specify that users are required to acknowledge the original data source.

License: Allows you to indicate a standard open data license that the data collected using this Data Source Document should abide. The License describes what rights the FEWS NET project has to the data collected using this Source Document. Descriptions of Creative Commons licenses can be found on the HDX website.

Current Schedule: Provide the details of the data upload source using the following fields:

Source Type: Select the Source Type that reflect the method used to ingest the data.
Web Service: If the data will be ingested automatically via an API, select the corresponding Web Service name.
Display Order: Used to set the display order of Market then Product.

Aliases: Allows you to create a list of alternative names or spellings for the Data Source Document name.

Published notifications: Allows you to add email addresses to receive notifications when Data Collections for this Source Document are Published. See Enabling Notifications for more information.

Under Review Notifications: Allows you to add email addresses to receive notifications when Data Collections for this Source Document are placed Under Review.

Schedule: Allows the addition of a Schedule, typically Weekly, Monthly, or Ad Hoc.

Editing a data source name

To update a data source name, the Source Document must be edited separately from the Source Organization.

To edit the Source Document:

Go to Metadata management > Warehouse > Source documents and select the record to open it.
Edit the Source Document field with the new name.

To edit the Source Organization:

Go to Metadata management > Warehouse > Source organizations and select the record to open it.
Edit the Name field with the new name.