Loading Baselines via Pipelines

This process is for bulk or individually uploading Baselines via pipelines. For a single Baseline upload via the database admin screen go to BSS Setup > Livelihood Zone Baselines > Add Livelihood Zone Baseline.

Loading process

The process of loading BSS is an iterative one. In general, the process is as follows:

1. Add the BSS file(s) to appropriate folders and files

2. Load metadata

3. Load Baselines

4. Load Communities

5. Load imported baselines

At each step, you must look for partitions that have failed and follow the troubleshooting process.

Add or update BSS file and associated BSS metadata

1. To add a new BSS to the database, first upload the file to the appropriate country folder inside the BSS folder in Google Drive. Do not add a shortcut to the owner’s version stored locally or on the partners' servers; the pipeline should read from a copy of the data on FEWS NET’s servers. This way we will always have an exact copy of what the owner gave to the Hub. 

2. Add the BSS path, starting with the country name, to the Folder Contents sheets in the BSS Metadata spreadsheet. Add the BSS path and any required metadata in the Metadata sheet. If the file is a new version of a previously uploaded BSS, ensure that the path is correct and matches in both locations.

3. If you are uploading a new version of a previously-uploaded spreadsheet, first delete the old version of the baseline and livelihood zones from the database. Ensure also that you delete references to this spreadsheet in the Corrections tab of the BSS Metadata Google sheet. This will ensure a clean upload without run errors. Cut and paste the corrections associated with the previous version to another file so as not to lose them if necessary. 

4. Files are locked as soon they are received by the Hub. However, all columns and rows should be unhidden and iterative calculations should be turned on before a BSS is loaded. If this is not the case, unlock the file, make those formatting changes, and re-lock the file.

Ensure the Pipeline metadata is updated (Load metadata)

Bulk uploading is done using Dagster. You can learn about the concepts Dagster uses in their documentation. For each Asset (e.g. bss_metadata , community_instances) we have multiple Partitions - one for each BSS. This approach allows us to process each BSS separately so that a single BSS with errors does not prevent other BSSs from being loaded.

1. Open Dagster using [database URL]/pipelines (e.g., headev.fews.net/pipelines/)

2. Go to Jobs > update_metadata > Launchpad > Launch run to load the new metadata.

Load the Baselines

This step loads BSS spreadsheet files into the database.

1. Go to Catalog > uploaded_baselines > Lineage.

2. Click Upstream > All > Materialize all.
   Note: There are 2 Materialize buttons. Materialize will only materialize the selected asset. Materialize all will materialize all assets shown in the lineage. In this case, make sure you are clicking Materialize all and not Materialize.

3. Select the partition(s) you wish to upload to the database. Deselect undesired partitions.
   Note: Avoid launching runs for hundreds of partitions if you only need one. You can cancel if you make a mistake, but it still wastes time and energy and can take some time and cleaning for the system to reset if there is a mistake or failure.

4. Click Launch Run when the appropriate number of partition runs matches your expectation. 

5. Once your run has been successful, double-check that the database is indeed using the new file by verifying that the file pathname the database is reading from is correct.

   1. Go to the database admin screen > BSS setup > Livelihood zone baselines.

   2. Search for the Livelihood Zone you're looking for and select the one for the appropriate reference year.

   3. In the first section you will find the label BSS Excel file:. Confirm that the path name matches the revised file path you added to the metadata sheet. If they do not match, you are probably trying to upload a new version of a prior BSS that was not fully deleted at all appropriate levels. Double-check the deletion process and start again.

Load Communities

This step loads the community and district names from the BSS WB sheet.

1. Go to Catalog > imported_communities > Lineage.

2. Select Upstream and a Graph depth of All.

3. Click Materialize all.

4. Select your desired partitions and click Launch.
   Note: For a full database re-upload, the number of partitions should equal the number of materialized partitions from uploaded_baselines; otherwise the number of partitions should match your expectations.

5. Once complete, go to Runs and click on the name of any failed runs to review any errors for troubleshooting.

6. Go to Jobs > update_metadata > Launchpad > Launch run to load the community aliases.

Load imported baselines

This step loads data from the BSS WB sheet.

1. Go to Catalog > imported_baseline > Lineage.

2. Select Upstream and a Graph depth of 4.

3. Click Materialize all.

4. Make sure all desired partitions are selected and click Launch.

5. Once complete, go to Runs and click on the name of any failed runs to review any errors for troubleshooting.

Next steps

Re-run livelihood_activity_label_recognition_dataframe and drop the link to the updated file in the Hub HEA Activity Group.

Once the BSS has been loaded, refer to the guidance on the Quality Assurance for BSS Loading page.