Errors and how to correct them | FEWS NET Help Center

Unexpected or missing information

Error message example	Possible reason error occurred	Correction needed
`BSS ‘Nigeria-NG20-2022-08-31’ , cell ‘<Cell 'Data’ .AC219> , value found ' ' , expected '0’`	There is an empty cell, but the pipeline expects a zero.	Change the correction in the BSS Metadata corrections tab. Ensure the `previous_value` blank.
`ValueError: Found LivelihoodStrategy from row 9 without a section_header_label set`	There is a value in a cell that should be empty.	There is data in the row that initiates the rowset that needs to be removed or moved.
`Community lookup didn’t find any matches`	Communities have not yet been imported, there is a community alias that has not been identified in the Community Aliases sheet in BSS Metadata, or there is a mismatch in the data.	Check BSS community names in affected sheets. Correct any errors if necessary using Corrections for WB sheet or Community Aliases for other sheets. Reload load_all_corrections and/or load_community_aliases as needed. Re-run the failed asset upstream from corrected_files. If this doesn’t work, try re-running imported_communities upstream to corrected_files and then again re-run the failed asset upstream from corrected_files.
`Could not convert string to float: ‘mature’`	There is a string value where an integer is expected.	Check and correct the BSS. Rerun load_all_corrections. Re-run the failing asset upstream from corrected_files.
ValueError: Unexpected prior value in source BSS. BSS `Mali~ML10-YOR~2009-09-30`, cell `'Data'!I540`, value found `0.527835`, expected `0.53`	The previous_value correction was based on copying the human-visible percentage from the BSS rather than the calculated value from the formula in the BSS cell visible to the pipeline.	Replace the correction with the longer value. Note: if you are transposing values from one cell to another using corrections, use the longer value for both the previous_value and the destination cell value.
	There is an unexpected mismatch in the data.	Delete and reload the data From the Django admin screen, delete the mismatched data. Re-run all the steps starting with uploading the baseline.
`Value must be lower than max_value.`	Summary value does not fit within the specified range (i.e., summary value of 10 for range of 11-15).	Correct the range to include the summary value.
`Invalid percentage_kcals value -0.03913894324853229: Ensure this value is greater than or equal to 0.`	Negative kcal% value. These are always paired with an `Invalid percentage_kcals value` error.	Correct the kcal% value to 0 using -0.03913894324853229 as the previous_value. Specify in the correction note whether there is a visible error (i.e., production < quantity sold + quantity other use) or whether the error is unknown. Note: quantity_other_use is not usually used in any calculation, so if the quantity other use is in error, that’s not a big deal.
ValueError: Unexpected prior value in source BSS. BSS `Mali~ML10-YOR~2009-09-30`, cell `'Data'!I540`, value found `0.527835`, expected `0.53`	The previous_value correction was based on copying the human-visible percentage from the BSS rather than the calculated value from the formula in the BSS cell visible to the pipeline.	Replace the correction with the longer value. Note: if you are transposing values from one cell to another using corrections, use the longer value for both the previous_value and the destination cell value.

Issues with products

Error message example	Possible reason error occurred	Correction needed
`ClassifiedProductLookup found multiple ClassifiedProduct matches for some rows` `product_name product_id` `Shea nuts R01349HA` `Shea nuts R01499HA`	There are two different product IDs associated with Shea nuts.	See Removing Duplicate Products for detailed instructions. In the database, go to Metadata > Products. Find the product and delete the incorrect one. You may need to delete other attached metadata first. Run update_metadata. Return to pipelines and click Re-execute.
`Cannot determine product_id from lait achete' (poudre) for summary Food Purchase on row 494 for label ‘Lait achete’ (poudre): quantite' (kg)'`	`lait achete' (poudre)` is not in the Classified Products list.	Add an alias to the appropriate existing product in the ClassifiedProduct sheet of Reference Data or, if necessary, consult with the HEA data lead to determine whether a new product needs to be added to ClassifiedProduct. Re-load metadata.
`Cannot determine product_id from kg vendus oseille for summary CropProduction on row 389 for label ‘Autre culture de rente: kg vendus Oseille’`	“0seille” is an alias in Classified Products, so the pipeline is unable to identify what term is the product because “0seille” follows rather than precedes kg vendus.	Apply an activity label override specifying IsStart = TRUE, attribute = “quantity_sold,” product = “Roselle,” unit = “kg.” Re-load metadata.
	Missing payment_product.	Check the label in the label recognition dataframe to identify the implications of applying an override or changing the regex. If appropriate, apply the override or create a ticket to change the regex.

Issues with labels

Unrecognized labels

The status field in the Reference Data Label sheet is blank.

Add the appropriate status.

No regular expression exists that can read the data

The Reference Data Label sheet directs the pipeline to apply a Regular expression to read the data, but no regular expression exists that can read the data. This may happen because the syntax is unique. The Label sheet sometimes shows in the Notes column a regex that is or was reading the label. If the Notes column is empty, there may not be a regex reading the data. You can direct the pipeline to read a label appropriately either by:

Applying an Override to assign the appropriate values. Do this only if the Override is appropriate for all affected labels. You may need to visually inspect the BSS if there are only 1 or 2 for the label as identified in the Reference Data label sheet. If there are more than 2 unique BSSs identified in the Reference Data sheet, then check the appropriateness of your intended solution by identifying all relevant BSS by using the AllLabels sheet in the label recognition dataframe.
To override a label:
Change the status from Regular Expression to Override
- Apply the appropriate IsStart (=TRUE if the row starts a rowset, otherwise for rowset content rows or section headings or summary rows, IsStart=FALSE).
- Apply the appropriate attribute and product (and unit) or wealth characteristic identified by the label.
- Add a note in the Notes column stating that the status is Override and explaining why.
Create a ticket to request a change the regex (link to Epic HEA-812 Regex fixes).
- Do this when a regex exists and/or when a regex is an appropriate solution (i.e., the syntax is common, not unique, and the syntax has both an element of stability and a variable element. Example: Millet: kg produced and sorghum: kg produced are part of a kg produced regex pattern).
- When creating a ticket, provide the name and link for the example BSS and the relevant run. Use three backticks (```) to create a monospace code block that will allow you to paste the relevant parts of the error using monospace font so that the error message appears more readable. Attach a file with an extract of relevant AllLabels or LabelSummary.

Product, wealth characteristic, or alias is not in Reference Data

Check by searching ClassifiedProduct or WealthCharacteristic for the expected term. If an appropriate product or characteristic exists but the necessary alias does not, feel free to add a missing alias for any existing product or wealth characteristic using the “~” separator. First check that the alias is not already present in the worksheet before adding. Duplicate aliases will cause load failures.

If an entire product or wealth characteristic is missing, add it to the Reference Data spreadsheet in the appropriate tab.

For products check the FDW to see if an appropriate product exists there but is not yet in the HEA database product list. If the product exists in FDW, add it to the Reference Data sheet.
To create a new product, identify the closest, lowest-level CPC 2.1 code ending in 0; add the appropriate R, P, or L prefix for the HEA database code; and add a suffix H* (where * represents the appropriate letter in alphabetical order).

Duplicate labels

Error message example	Possible reason error occurred	Correction needed
`ValueError: Found duplicate entries in worksheet`	Duplicate labels exist in the relevant Reference Data *Label sheet.	Search for the label in the appropriate Reference Data *Label sheet. If it appears multiple times with the same BSSs referenced, then investigate which should be retained and which should be deleted. Delete the duplicate/ unnecessary label.
`ValueError: Lookup found multiple matches for some rows:`	Duplicate labels exist in the database due to a change in the reference data.	Delete the incorrect entries from the database. This may include deleting dependent values. For large deletions, it may be better to request deletion from the back end rather than the front end. Run update_metadata. If necessary reload deleted and dependent partitions/assets from instances to imported to ensure that the database is complete.
`dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "consolidated_fixture":` ... `The above exception was caused by the following exception: RuntimeError: LivelihoodStrategy 4 duplicates existing record with different values:`	There are multiple rowsets in the BSS referring to the same activity with no additional identifier or season to differentiate them. Common rowsets duplicated in Data and Data2 or Data3 are: Petty trade Firewood/charcoal Labor migration Handicrafts Other (if there are multiple rowsets in the same section containing data with no further clarification in the label)	It is preferable for the BSS owner to pick one “rowset” for the database to read as Petty trade. To solve without the owner, relabel the rowsets allowing the pipeline to recognize additional identifiers that differentiate the rowsets.

Error message example

Possible reason error occurred

Correction needed

ValueError: Found duplicate entries in worksheet

Duplicate labels exist in the relevant Reference Data *Label sheet.

Search for the label in the appropriate Reference Data *Label sheet. If it appears multiple times with the same BSSs referenced, then investigate which should be retained and which should be deleted. Delete the duplicate/ unnecessary label.

ValueError: *Lookup found multiple * matches for some rows:

Duplicate labels exist in the database due to a change in the reference data.

Delete the incorrect entries from the database. This may include deleting dependent values. For large deletions, it may be better to request deletion from the back end rather than the front end.

Run update_metadata.

If necessary reload deleted and dependent partitions/assets from *instances to imported* to ensure that the database is complete.

dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "consolidated_fixture":
...
The above exception was caused by the following exception: RuntimeError: LivelihoodStrategy 4 duplicates existing record with different values:

There are multiple rowsets in the BSS referring to the same activity with no additional identifier or season to differentiate them.

Common rowsets duplicated in Data and Data2 or Data3 are:

Petty trade
Firewood/charcoal
Labor migration
Handicrafts
Other (if there are multiple rowsets in the same section containing data with no further clarification in the label)

It is preferable for the BSS owner to pick one “rowset” for the database to read as Petty trade.

To solve without the owner, relabel the rowsets allowing the pipeline to recognize additional identifiers that differentiate the rowsets.

Other label issues

Error message example	Possible reason error occurred	Correction needed
`Found duplicate value ReliefGiftOther from row 556 for existing attribute strategy_type with value PaymentInKind from row 553`	An incorrect or incompatible strategy_type has been applied to a label.	If Reference Data indicates the label is governed by a regular expression, create a ticket for the developers to check the regular expression and resolve the error. If the label is governed by an Override, check that the strategy_type is not incorrectly assigned and change it if appropriate.
`ValueError: Found new section header 'ECALLEUSE DE POISSON' from row 183 without an income_subtotal_label set for the previous section with label 'AUTO-EMPLOI' from row 158`	The pipeline wasn’t able to read a summary row for the previous section	The pipeline understands all capital letters as an indicator of a section heading. There were several labels in this section with all capital letters. Once corrected to capital + lowercase letters, the pipeline was able to pass the asset.
`Found values kg without an identified attribute on row 507 for label ‘Autre: Arachide seche’'`	The label doesn’t specify the attribute for the row, which contains the values “kg.” Check the label_recognition datasheet to find out whether a regular expression identifies the label and the ClassifiedProduct sheet to ensure that “arachide seché” is referenced.	If the label_recognition_datasheet identifies a regular expression that reads the label, consider making a ticket to ask that the regular expression identify the attribute as activity_notes. If no regular expression recognizes the label, apply an override using the ActivityLabel sheet (IsStart = TRUE, attribute = activity_label, product = Groundnuts (dried)). Re-load metadata.

Error message example

Possible reason error occurred

Correction needed

Found duplicate value ReliefGiftOther from row 556 for existing attribute strategy_type with value PaymentInKind from row 553

An incorrect or incompatible strategy_type has been applied to a label.

If Reference Data indicates the label is governed by a regular expression, create a ticket for the developers to check the regular expression and resolve the error. If the label is governed by an Override, check that the strategy_type is not incorrectly assigned and change it if appropriate.

ValueError: Found new section header 'ECALLEUSE DE POISSON' from row 183 without an income_subtotal_label set for the previous section with label 'AUTO-EMPLOI' from row 158

The pipeline wasn’t able to read a summary row for the previous section

The pipeline understands all capital letters as an indicator of a section heading. There were several labels in this section with all capital letters. Once corrected to capital + lowercase letters, the pipeline was able to pass the asset.

Found values kg without an identified attribute on row 507 for label ‘Autre: Arachide seche’'

The label doesn’t specify the attribute for the row, which contains the values “kg.” Check the label_recognition datasheet to find out whether a regular expression identifies the label and the ClassifiedProduct sheet to ensure that “arachide seché” is referenced.

If the label_recognition_datasheet identifies a regular expression that reads the label, consider making a ticket to ask that the regular expression identify the attribute as activity_notes. If no regular expression recognizes the label, apply an override using the ActivityLabel sheet (IsStart = TRUE, attribute = activity_label, product = Groundnuts (dried)).

Re-load metadata.

Community names

For incorrect or missing names in the WB sheet, add a correction and an alias. For names in other sheets, just add an alias.

All names should start with a capital letter and be followed by lowercase letters. All words in proper place names should be capitalized. Ideally names will match, in order of priority, FDW, Google, or a UN source. If a community is referenced in multiple BSS, the same spelling should be used consistently across all BSS.

Seasons

Season aliases for milk production and butter production must match.

Seasons within a purpose (i.e. milk is a purpose and butter is a different purpose) cannot overlap months. Within a purpose, each month must be assigned to 1, and only 1, season.

Seasons may not use partial months.

To account for differences in seasonality across different geographies, a single country may specify different sub-geographies with different season-sets (i.e., “Ethiopia” vs. “Ethiopia, Somali” vs. “Ethiopia, Benihangul Gumuz”.)

The Seasonal Activity Type worksheet

Unlike the other assets and sheets a season label will either be recognized or it will not. There is no ability to override a label for a season.

Seasons for which is_key = FALSE will not show on the Explorer.

Payment in kind

Each rowset must have a unique [activity] product (the thing people are doing, like pre-harvest labor) named in the IsStart=TRUE row label and a payment product given in parentheses in the payment in kg per time IsStart=FALSE row label. This is a new rule we made that helps us to organize the data, but most historical BSS don’t work this way, so lots of corrections may be needed.

References to similar activities across multiple rowsets

The BSS can only assign 1 product + attribute pair (or 1 set of activity product + payment_product + attribute) per rowset.

Many BSS have multiple products or activity_product/payment_product combinations in 1 rowset. These are complicated cases.

Apply an override using the most general product corrections possible to allow the entry. This might require changes to multiple rowsets if the generalization would result in multiple rowsets with the same activity/payment_product. In principle, it is the BSS owner’s responsibility to ensure that their data is clean, clear, and uploaded accurately. We’re trying to balance the most accurate data upload possible with the most reasonable amount of our effort.

Example A: Rowset contains wealth group data for weeding+sorghum, sowing+rice, threshing+sorghum, labor+meals (meals with given as converted to kg cereals) (and summary values for labor+sorghum for VP and labor+rice for P). Correct the BSS labels to apply activity product “Casual labor” and payement_product “cereals.” While most of the types of labor referenced are pre-harvest labor, the “labor+meals” information could be non-agricultural; therefore we must assume the highest level of labor in the logic, which is casual labor. “Cereals” is a product in the HEA database, and a common generalization in HEA data collection. The assumed kcal value/kg of mixed cereals is 3500 (HEA Field Guide Template).

Example B: Rowset 1 with label “Labour: m d’o” and wealth group entries for pre-harvest labor+sorghum, weeding+millet, harvest labor+sorghum, casual labor+rice. A second rowset for labeled “Labour: land preparation” with payment in kind (millet) land only 1 entry of land preparation+millet for the same village as weeding+millet from the “Labor: m d’o” rowset. Merge the entries for weeding+millet and land preparation+millet and relabel the “labour: land preparation” rowset as “labour: pre-harvest labor.”

Other errors

Error message example	Possible reason error occurred	Correction needed
`UserLookup didn’t find any matches`	There are names listed in the Correction worksheet that do not exist in the database.	You need user accounts in your database for every name in the Correction worksheet in order to load the corrections. Open the BSS Metadata spreadsheet to the Corrections tab. Ensure that every name listed under the author column has an associated account in the database.
`Error applying correction to BSS ‘Niger~NEo3(DBP)~2015-06-03’ , worksheet ‘Data’ , range ‘AX’`	There is no reference row number.	Row number needs to be added to the Correction worksheet of the BSS Metadata spreadsheet. Run load_all_corrections. Rerun the failed asset upstream from corrected_files.
`duplicate key value violates unique constraint` or `key (id) = (...) already exists`	The pipeline ran a bunch of partitions simultaneously and assigned the same key (id) to different partitions/assets, causing all but the first user of the unique key (id) requirement to fail the asset.	Iteratively re-run failed partitions from the original run until no additional partitions pass.
`bulk_update() cannot be used with primary key fields`		Run `update_metadata` again. Click on `load_all_metadata` bar in the top pane. Use dropdown on Re-execute menu to choose Selected step.
The error is maintained after you make the proposed correction.	Google Sheets does not save the change until you exit or unselect the cell.	Click outside the cell with the change. Re-run the required assets.
The error is maintained after you make the proposed correction.	An unintended change to Reference Data or BSS Metadata column headings prevents the pipeline from reading the sheet. Changes are not being recorded by the database.	Verify that all column headings are original and haven’t been accidentally changed.

Issues with prior corrections

When re-uploading a baseline you may encounter an error about a correction that previously worked for which the expected value does not match the pipelines’ observed value. This may be caused by hyperlinks in labels changing Excel-saved values for the cell. Sometimes a version of the BSS might be saved that prioritizes the BSS’s memory of the hyperlink text, while sometimes a version of the BSS might be saved that actually prioritizes the label. While we aim never to change a BSS, we do need to unlock files to unhide rows/columns, and even this creates opportunities for this kind of error. In this case, revert the BSS in Google Sheets to the original version (ensure all rows and columns are visible and the name matches the BSS metadata expectations) and re-upload it. Then match the corrections needed to the re-uploaded, restored “original” baseline version.

Common errors with existing tickets

Payment in kind

There are also still some errors in how the pipeline is reading payment in kind that should be resolved with pending tickets/merges (see HEA-851).

“A Livelihood Strategy must have either a Product or an Additional Identifier for LivelihoodStrategy 47 from 'Data'!541:541:” This is because the pipeline isn’t recognizing the (activity) product alias in ClassifiedProduct.
“Missing mandatory foreign key payment_product_id for PaymentInKind 26 from 'Data'!B553:” This is because the pipeline isn’t recognizing the payment_product in the “payment in kg per time ([product])” row, either due to a syntax error or because the product isn’t listed as an alias in the ClassifiedProduct sheet.
“Unrecognized 'livelihood_strategy' foreign key ('ML10-YOR', '2009-09-30', 'PaymentInKind', '', '', '') for LivelihoodActivity 661 from 'Data'!B541:” and/or “Unrecognized 'livelihoodactivity_ptr' foreign key” I believe these are duplicate errors related to the errors above.
Despite the above, there may be some errors you can still see and fix:
1. Unique labels (check ActivityLabel or label recognition dataframe for only 1 BSS). These you can override with no risk to other BSS currently in the database (consider possible implications for future BSSs).
2. Unclear activity product names: apply a correction to the label or add the alias to the ClassifiedProduct list as appropriate
3. References to multiple similar activities throughout the rowset: Use the most common highest level activity product for the rowset. Example: the label says ag labor but the activity_notes reference weeding, land prep, and sowing then change the label to pre-harvest labor)

Milk

“Lactation days must be provided if there are milking animals for MilkProduction” ticket (HEA-1030)
“Daily production must be provided if there are milking animals” (HEA-1030)

Other Cash Income

“Income for 'Other Cash Income' must be payment per time * times per year for OtherCashIncome” This may be because of a rounding error ( HEA-986 , HEA-982 ) or because number of months ( HEA-982 ) or number of people ( HEA-1053 ) is not a whole number.
“Quantity produced for Other Cash Income must be payment per time * number of people * labor per month * months per year for OtherCashIncome” ( HEA-985 )

Other errors

“Expenditure for a Food Purchase must be quantity purchased multiplied by price for FoodPurchase 53” This may be due to a rounding error or also because number of months is not a whole number.