Household Economy Analysis (HEA) Livelihoods Explorer Database
Household Economy Analysis (HEA) Livelihoods Explorer Database
Breadcrumbs

Troubleshooting and Corrections

Troubleshooting process

Pipeline error messages usually indicate the cell or label causing the error. Use the troubleshooting table and/or check errors against the spreadsheet that contains the metadata and correct if necessary.

There are 4 ways to resolve an error or conflict:

  • Change the metadata (i.e., add an alias or product or change the instruction to the pipeline for recognizing the data) using the BSS Metadata spreadsheet. This is the primary way we aim to resolve issues. See the Metadata Reconciliation section for more info.

  • Change the BSS using a correction. This may need to be used when:

    • A cell has missing data or data in an incorrect format

    • Spelling and label formats are inconsistent

    • The BSS is logically inconsistent

  • Choose to retain the error. Sometimes errors signal a logical inconsistency in the data that don’t invalidate the conclusions but that may be useful in interpreting results. For example, assume there is a rowset of poor household summary values with 17 entered into the first row intending to refer to kg produced, but the first row does not have the quantity_produced attribute. Instead, the kg produced row follows and contains a value of 1, and the price row 1500 such that the formula in the income row returns 1500 instead of 25,500 (17*1500). It is not possible to correct this error without also changing the total income for the poor wealth group and their total food and income. For uploading historical baselines on behalf of BSS owners, we have opted instead to minimize changes to data. Retaining these errors allows users to understand the issue and more appropriately interpret the data.

  • Change the pipeline. Sometimes there are bugs or unique situations to accommodate by adapting the software. Regular expressions are part of the software that helps recognize labels and patterns in the BSS. A new regular expression may be appropriate if there is a common syntax that applies to multiple BSS and for which the things that change can change consistently. For example, many BSS have references like mangoes: kg collected or maize: kg produced. In such cases the syntax is [product]: [unit of measure] [quantity produced]. The name of the product varies, the colon signifies a separation between the product and the unit of measure, and quantity produced and quantity collected are both aliases for the attribute quantity_produced.

If an error can not be fixed or retained in one of the first three ways, you will need to create a ticket to get help to change the pipeline. Be sure to check existing tickets first to see if the issue has already been documented. 

Once errors have been corrected:

  1. To load the corrected errors, to go Assets > completed_bss_metadata > Lineage > Upstream > Materialize all.

  2. Go to Jobs > update_metadata > Launchpad > Launch run. On top pane, ensure that load_all_corrections and load_all_metadata are successful. load_all_metadata may take some time to complete.

  3. Re-run the step for the partitions that just failed.

  4. Once errors are addressed:

    1. Check the _instances assets (e.g., livelihood_activity_instances, wild_foods_instances) for unrecognized labels and address issues as needed.

      1. We only need to recognize labels for rowsets that contain data. That means we don’t need to fix unrecognized labels for rowsets that contain no data or only 0s. Note that an unrecognized label might show 0 datapoints and 0 summary_datapoints and still contain data in every cell in the rowset except for the first IsStart row. If an unrecognized label reports zero datapoints and zero summary_datapoints, check the BSS to ensure there is actually no data for that rowset. If there is no data, then no effort needs to be invested in making the pipeline read the label.

      2. We can also ignore unrecognized labels for rows that are summaries (indicated by the word total or subtotal) or section headings (e.g. Fishing). Even though those may have Data Points and summary data, we should ignore them to avoid double-counting.

    2. Check the _instances assets for errors and address issues as needed.

    3. Check the _activity_valid_instances assets for unrecognized labels and errors. Again, compare to the reference data sheet, label recognition dataframe, Dagster asset preview, and database to diagnose the error.

  5. Once the BSS is successfully loaded, follow the guidance found on Quality Assurance for BSS Loading.

Using the Corrections tab of the BSS Metadata spreadsheet

When loading using the pipeline, corrections should not be made in the original BSS file. Original files should be locked to edits and stored exactly as they were submitted.

To add a new entry to the Corrections tab:

  • Choose the baseline and worksheet where the correction needs to be made. Add the cell or cell range as applicable. Add the BSS path to column J.

  • The pipeline will not accept a correction to a cell that does not match the value that it reads from the BSS. Therefore, previous_value should correspond to the value within the original BSS as identified by the pipeline in the error message, and value should be the desired value.

  • Add the date, author, and a comment explaining the change. If the author is new to this process, ensure they have an associated account in the database.

If you are uploading a new version of a previously-uploaded spreadsheet, delete references to this spreadsheet in the Corrections tab of the BSS Metadata Google sheet. This will ensure a clean upload without run errors. Cut and paste the necessary corrections associated with the previous version to another file so as not to lose them if necessary. See also the instructions on Deleting Baselines and Livelihood Zones.

Closing out a BSS work session

  1. Avoid ending work with a failure in an update_metadata load. If you broke it, you’re likely to be best positioned to know what broke it and how to fix it. Leaving update_metadata failures incompletely resolved may cause difficulties for others working on the system.

  2. If you made a change that affected multiple BSSs (i.e., changing the status of a label affecting multiple BSSs from RegularExpression to Override), it is ideal to run all of the partitions for all assets affected by your changes so that the database does not become out of sync with the metadata. This may mean running assets all the way through from corrected_files to imported_* so the database will be in sync with both the metadata and the pipeline’s capacity (and across multiple baselines). This is why we regularly purge and update the database and sync the dev and stage databases.