Errors and how to correct them | FEWS NET Help Center

Unexpected or missing information

Error message example	Possible reason error occurred	Correction needed
`BSS ‘Nigeria-NG20-2022-08-31’ , cell ‘<Cell 'Data’ .AC219> , value found ' ' , expected '0’`	A correction specifies that `previous_value` is an empty cell, but the pipeline found a “0” in the BSS.	Change the correction in the BSS Metadata corrections tab. Ensure the `previous_value` cell is blank.
`ValueError: Found LivelihoodStrategy from row 9 without a section_header_label set`	There is a value in a cell that should be empty.	There is data in the row that initiates the rowset that needs to be removed or moved.
`Community lookup didn’t find any matches`	Communities have not yet been imported, there is a community alias that has not been identified in the Community Aliases sheet in BSS Metadata, or there is a mismatch in the data.	Check that BSS community names in affected sheets are all the same (spelling, capitalization, punctuation including spaces). Correct any discrepancies if necessary using Corrections for WB sheet or Community Aliases for other sheets.
`Could not convert string to float: ‘mature’`	There is a string value where an integer is expected.	Correct the BSS using the corrections spreadsheet.
ValueError: Unexpected prior value in source BSS. BSS `Mali~ML10-YOR~2009-09-30`, cell `'Data'!I540`, value found `0.527835`, expected `0.53`	The previous_value correction was based on copying the human-visible percentage from the BSS rather than the calculated value from the formula in the BSS cell visible to the pipeline.	When making a correction of a percentage calculated by a formula, use as the `previous_value` the decimal value as reflected by Paste Special > Formula. If that option isn’t available, replace the correction with the value identified by the pipeline error rather than the rounded value shown by the BSS.
`Value must be lower than max_value.`	Summary value does not fit within the specified range (i.e., summary value of 10 for range of 11-15).	Correct the range to include the summary value.
`Invalid percentage_kcals value -0.03913894324853229: Ensure this value is greater than or equal to 0.`	Negative kcal% value. These are always paired with an `Invalid kcals_consumed value` error.	Correct the kcal% value to 0 using the invalid value as the `previous_value`. Specify in the correction note whether there is a visible error (i.e., production < quantity sold + quantity other use) or whether the error is unknown. Note: `quantity_other_use` is not usually used in any HEA calculation, so if the `quantity_other_use` is causing the error, correction is effectively unnecessary, though a BSS owner may correct the issue without affecting other income. If the error is that quantity sold > quantity produced, then 1) the BSS owner may correct the quantity produced or quantity sold data and related `strategy_type` and total income. If that is not possible, then 2) retain the error.

Issues with products

Error message example	Possible reason error occurred	Correction needed
`ClassifiedProductLookup found multiple ClassifiedProduct matches for some rows` `product_name product_id` `Shea nuts R01349HA` `Shea nuts R01499HA`	There has been a metadata change resulting in two different product IDs associated with Shea nuts.	See Removing Duplicate Products for detailed instructions. In the database, go to Metadata > Products. Find the product and delete the incorrect one. You may need to delete other attached metadata first. Run update_metadata. Return to the pipeline and re-execute the asset stream beginning from corrected_files.
`Cannot determine product_id from lait achete' (poudre) for summary Food Purchase on row 494 for label ‘Lait achete’ (poudre): quantite' (kg)'`	`lait achete' (poudre)` is not in the Classified Products list.	Add an alias to the appropriate existing product in the ClassifiedProduct sheet of Reference Data or, if necessary, consult with the HEA data lead to determine whether a new product needs to be added to ClassifiedProduct. Re-load metadata.
`Cannot determine product_id from kg vendus 0seille for summary CropProduction on row 389 for label ‘Autre culture de rente: kg vendus 0seille’`	“0seille” (with a zero rather than an “O”) is an alias in Classified Products. Therefore, the problem is that because “0seille” follows rather than precedes kg vendus, the pipeline is unable to parse the syntax.	Apply an activity label override specifying IsStart = TRUE, attribute = “quantity_sold,” product = “Roselle,” unit = “kg.” Re-load metadata.
`Missing mandatory foreign key payment_product_id for PaymentInKind`	Missing payment_product.	Check the label in the label recognition dataframe to identify whether an override will cause problems for other BSS reads. If appropriate, apply an override to specify the payment_product. If this would cause a problem for other BSSes, try changing the label such that the pipeline can read cleanly or create a ticket to change the regex.

Issues with labels

Unrecognized labels

No regular expression exists that can read the data

A regular expression may be unable to identify a product or additional identifier or an attribute because the syntax is unique or because the product is not listed as a product or alias in the ClassifiedProducts sheet of Reference Data. The Label sheets (e.g., ActivityLabel) sometimes show a regex that is or was reading the label in the Notes column. If the Notes column is empty, there may not be a regex reading the data. You can direct the pipeline to read a label appropriately either by:

Changing the status from Regular Expression to Override in the relevant Label sheet of Reference Data
- Do this only if the Override is appropriate for all affected labels. If there are only 1 or 2 BSSes you may identify them from the relevant Label sheet. If there are more than 2 unique BSSs, then check the appropriateness of the proposed Override by identifying all relevant BSS by using the AllLabels sheet of the livelihood_activity_recognition_dataframe file.
- In the Label sheet:
  - Add the label to the sheet if it is missing. Ensure that the label is in all lowercase. The pipeline is case-agnostic, but the database interprets capitals and lowercase as different instances.
  - Change the label status to Override.
  - Apply the appropriate IsStart value. IsStart = TRUE if the row starts a rowset or is a section heading, otherwise for rowset content rows or summary rows, IsStart = FALSE.
  - Apply the appropriate attribute and product (and unit) or wealth characteristic identified by the label.
  - Add a note in the Notes column stating that the status is Override and explaining why.
Correcting the BSS.
- Change the label in the BSS for a label that can be recognized by regular expression and reload it from original_files downstream to uploaded_baselines OR
- Change the label in the BSS Metadata Corrections sheet to one that can be recognized by regular expression.
Creating a ticket to request a change the regex.
- Do this when a regex exists and/or when a regex is an appropriate solution (i.e., the syntax is common, not unique, and the syntax has both an element of stability and a variable element. Example: Millet: kg produced and sorghum: kg produced are part of the “kg produced” regex pattern: [product]: [unit] [attribute]).
- When creating a ticket, provide the name and link for the example BSS and the relevant run. Use three backticks (```) to create a monospace code block that will allow you to paste the relevant parts of the error using monospace font so that the error message appears more readable. Attach a file with an extract of relevant AllLabels or LabelSummary from the livelihood_activity_label_recognition_dataframe sheet.

Product, wealth characteristic, or alias is not in Reference Data

Check by searching ClassifiedProduct or WealthCharacteristic for the expected term.

If an appropriate product or characteristic exists but the necessary alias does not (i.e., different language or spelling), feel free to add a missing alias (lowercase) for any existing product or wealth characteristic using the “~” separator. First check that the alias is not already present in the worksheet before adding. Duplicate aliases will cause load failures.

In a few cases, a new or differentiating product may be merited instead of adding an alias. This includes:

People produce multiple varieties of a product in the same location.

The product is likely to be used in multiple countries, across multiple baselines.

The different varieties are relevant to different production techniques or vulnerabilities.

Examples

We do not have different products in HEA for short-cycle varieties of cereals. We can make this distinction if necessary using additional identifiers.
Flood-recession sorghum is not a different product than rainfed sorghum. If the distinction is necessary for the livelihood zone, it can be made using additional identifiers.
Different end uses of an item do not merit different product. For example, bananas used for beer production are not different in terms of production process, vulnerability, or kcals than those used for consumption. If the distinction must be made, it can be done using an additional identifier.
On the other hand, many wild foods have multiple edible parts that may be available at different times of year and may have different caloric content. For example, both leaves and fruit are available at different times of year and have different nutritional content. These are two products that may be worth differentiating.

If an entire product or wealth characteristic is missing, add it to the Reference Data spreadsheet in the appropriate tab.

For products, check the FDW to see if an appropriate product exists there but is not yet in the HEA database product list. If the product exists in FDW, it may be added directly to the Reference Data sheet.
If the product does not exist in FDW, create a new product by identifying the closest, lowest-level CPC 2.1 code ending in 0; add the appropriate R, P, or L prefix for the HEA database code; and add a suffix H* (where * represents the next letter in alphabetical order).

Example

If there were a need to add Government employment as a separate product, it would fall under S85H.

Product code	Product name
S85HA	Casual labor
S85HB	Skilled labor
S85HC	Other self-employment
S85HD	Formal employment

Formal employment is the last currently named item as S85HD. Government employment would need to be added as S85HE.

Duplicate labels

Error message example	Possible reason error occurred	Correction needed
`ValueError: Found duplicate entries in worksheet`	Duplicate labels exist in the relevant Reference Data *Label sheet.	Search for the label in the appropriate Reference Data Label sheet. If the same label appears multiple times in the Label sheet with the same BSSs referenced, then investigate which should be retained and which should be deleted. Delete the duplicate/ unnecessary label.
`ValueError: Lookup found multiple matches for some rows:`	Duplicate labels exist in the database due to a change in the reference data.	Delete the incorrect entries from the database. This may include deleting dependent values. For large deletions, it may be better to request deletion from the back end rather than the front end. Run update_metadata. If necessary reload deleted and dependent partitions/assets from *instances to imported_baseline to ensure that the database is complete.
`dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "consolidated_fixture":` ... `The above exception was caused by the following exception: RuntimeError: LivelihoodStrategy 4 duplicates existing record with different values:`	There are multiple rowsets of the same strategy_type in different sheets of the BSS referring to the same activity with no additional identifier or season to differentiate them. Common rowset start labels duplicated across Data and Data2 or Data3 are: Petty trade Firewood/charcoal Labor migration Handicrafts Other (if there are multiple rowsets in the same section containing data with no further clarification in the label)	It is preferable for the BSS owner to pick one rowset for the database to read, either merging duplicate rowsets, or clarifying the differentiation by adding a product-relevant additional identifier. To solve without the BSS owner, relabel the rowsets allowing the pipeline to recognize additional identifiers that differentiate the rowsets, for example, “Petty trade - Data2!”.
`dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "imported_baseline":` … `RuntimeError: Problem installing fixture '/tmp/tmpwxqi3_33.verbose_json': Failed to save Livelihood Product Category (ML15, 2020-09-30, P, FoodPurchase, 2, P23520, , ) {'id': None, 'baseline_livelihood_activity': 1870776, 'basket': 2, 'percentage_allocation_to_basket': 0.5}` … `django.db.utils.IntegrityError: Could not load baseline.LivelihoodProductCategory(pk=None): duplicate key value violates unique constraint "baseline_livelihoodprodu_baseline_livelihood_acti_bce1483a_uniq" DETAIL: Key (baseline_livelihood_activity_id)=(1870776) already exists.`	Additional symptoms: Duplicate key error does not resolve with iterative runs. Duplicate value error also appears in relevant `*valid_instances` asset(s). Likely cause: Duplicate labels in BSS, often including hyperlinks. Common rowset start labels duplicated in Data include: Main cereal - Belg: kg produced Enset/cassava: no. local meas. Main cashcrop kg sold Sugar purchase: kg	Investigate the BSS for duplicate labels. Investigate the various `*valid_instances` assets for culprits. Replace duplicate labels with labels reflecting the appropriate `Product` / `additional_identifier` and reload the BSS or the data.

Other label issues

Error message example	Possible reason error occurred	Correction needed
`Found duplicate value ReliefGiftOther from row 556 for existing attribute strategy_type with value PaymentInKind from row 553`	An incorrect or incompatible strategy_type has been applied to a label.	If Reference Data indicates the label is governed by a regular expression, create a ticket for the developers to check the regular expression and resolve the error. If the label is governed by an Override, check that the strategy_type for the section is not incorrectly assigned and change it if appropriate.
`ValueError: Found new section header 'ECALLEUSE DE POISSON' from row 183 without an income_subtotal_label set for the previous section with label 'AUTO-EMPLOI' from row 158`	The activity label in the BSS is in all capital letters.	The pipeline understands all capital letters as an indicator of a section heading. There were several labels in this section with all capital letters. Once corrected to sentence case, the pipeline was able to pass the asset.
`Found values kg without an identified attribute on row 507 for label ‘Autre: Arachide seche’'`	The label doesn’t specify the attribute for the row, which contains the unit values “kg.”	Change the BSS label to include an appropriate term to indicate the attribute name_of_measure such that the contents of the row may be appropriately recognized. Alternatively, check the label_recognition sheet to investigate BSSes sharing this label and consider applying an override for the label if appropriate.
`Invalid expenditure value Arachide: “Arachide” value must be a float.`	The label does not specify an attribute or the implied attribute requires an integer. The label row or rowset start row contains strings referencing the product (or a unit of measure) instead.	Strings specifying the product should be redundant as the label should specify the product. These may be deleted by correcting the BSS. Alternatively, an Override may be applied to specify the attribute activity_notes, if investigation reveals that no other BSS would be adversely affected.

Community names

For incorrect or missing names in the WB sheet, correct the BSS directly and reload it OR add a correction and an alias to BSS Metadata. For incorrect names in other sheets, just add an alias.

All names should start with a capital letter and be followed by lowercase letters. All words in proper place names should be capitalized. Ideally names will match, in order of priority, FDW, Google, or a UN source. If a community is referenced in multiple BSSes, the same spelling should be used consistently across all BSSes.

Season definitions

Season definitions are used to differentiate products for which different BSSes use the same season name but which may be produced or harvested in different months of the year.

Season aliases for milk production and butter production must match.

Within a purpose, each month must be assigned to a season, and may be assigned to only 1 season. Months cannot overlap seasons within the same purpose.

Seasons may not use partial months.

To account for differences in seasonality across different geographies, a single country may specify different sub-geographies with different season-sets (i.e., “Ethiopia” vs. “Ethiopia, Somali” vs. “Ethiopia, Benishangul Gumuz”.)

The Seasonal Activity Type worksheet

Unlike the other assets and sheets a season label will either be recognized or it will not. There is no ability to override a label for a season.

Seasonal_activity_types for which is_key = FALSE will not show on the Explorer.

Seasonal_activity_types for which has_product = TRUE must have a product in a prior row.

Payment in kind

Each rowset must have a unique [activity] product (the thing people are doing, like pre-harvest labor) named in the IsStart=TRUE row label and a payment_product given in parentheses in the payment in kg per time row. This helps organize the data. However, since most historical BSS don’t work this way, so lots of corrections may be needed.

References to different activities within the same rowset

The BSS can only assign 1 product + attribute pair (or, in the case of payment in kind, 1 set of activity product + payment product + attribute) per rowset.

Many BSS have multiple products or activity_product/payment_product combinations in 1 rowset. If the product in the label is generally accurate and recognized by the pipeline, and if the pipeline recognizes the rowset attribute as activity_notes, then the variation will be recorded in the database (but will not be visible in the Explorer). If the rowset’s starting row attribute is recognized as product_name or there is no attribute recognized and there are multiple products in the rowset, then the case is complicated.

The preferred solution would be for the BSS owner to correct the data, ensuring that each rowset contains data for only one product (or only one product + payment_product pair for the payment in kind strategy_type). This may require many corrections across multiple rowsets to reorganize datapoints and to clarify labels, products, and possibly additional identifiers.

When the BSS owner is unavailable, we must balance the most accurate data upload possible with the most reasonable amount of non-owner effort. The most expedient solution in such cases is to correct the label(s) such that they refer to the most general product possible that incorporates the varying data and ensure variations in the rowset are specified using an additional_identifier.

Example: Generalizing product labels

Rowset labeled “Autre achat:” contains various products mentioned in different wealth groups, including maïs, pain, spaghetti, and galette. While separating each product into its own rowset would be preferred, there may not be enough rows available in Excel to do this, and the effort required to implement such changes at data ingestion may be high or may not be possible without the authority of the BSS owner. In such cases, the most efficient solution is to relabel the row. You could:

Option 1: reload using the pipeline:
- Relabel the row as “Autre achat - mais, pain, spaghetti, galette”
- Add the label “autre achat - mais, pain, spaghetti, galette” to the ActivityLabel sheet as an Override specifying the product “Other food purchase,” the attribute activity_notes, and the additional_identifier “mais, pain, spaghetti, galette.”
- Reload metadata and rerun the pipeline for imported_baselines upstream to All -2 (livelihood_activity_instances tier).
Option 2: reload using the database:
- Relabel the row directly in the BSS file as “Autre achat - mais, pain, spaghetti, galette”
- Delete the prior version of the baseline from the database (livelihoods.fews.net/admin Livelihood Zone Baselines > [Select BSS] > Delete selected Livelihood Zone BSSes > Go > Yes, I’m sure)
- Reload BSS from original_files downstream to uploaded_baselines

Example: Merging entries

Rowset 1 with label “Labour: m d’o” has wealth group entries for:

pre-harvest labor+sorghum
weeding+millet
harvest labor+sorghum
casual labor+rice

Rowset 2 with label “Labour: land preparation” has only 1 entry of land preparation+millet for the same village as weeding+millet from the “Labor: m d’o” rowset.

In this case, because weeding and land preparation often rely on similar pre-harvest assumptions and because the distinction is not made for any other village/wealth group combination, it is appropriate to merge the land preparation+millet value from the second rowset into that of the weeding+millet value in the previous rowset. Delete the single land preparation+millet datapoint from the second rowset. This will allow the pipeline to read the second rowset as empty without losing the wealth group focus group datapoint values. The label of the first rowset implying manual or casual labor and the items in the rowset are is broad enough to include pre-harvest, harvest, and non-agricultural labor.

Other errors

Error message example	Possible reason error occurred	Correction needed
`UserLookup didn’t find any matches`	There are names listed in the Author column of the Corrections worksheet that do not exist in the database.	You need user accounts in your database for every name in the Correction worksheet in order to load the corrections. Open the BSS Metadata spreadsheet to the Corrections tab. Ensure that every name listed under the author column has an associated account in the database.
`Error applying correction to BSS ‘Niger~NEo3(DBP)~2015-06-03’ , worksheet ‘Data’ , range ‘AX’`	There is no reference row number.	Row number needs to be added to the Correction worksheet of the BSS Metadata spreadsheet.
`duplicate key value violates unique constraint` or `key (id) = (...) already exists`	The pipeline ran a bunch of partitions simultaneously and assigned the same key (id) to different partitions/assets, causing all but the first user of the unique key (id) requirement to fail the asset.	No changes needed; this is not an error in the BSS or data. Iteratively re-run failed partitions from the original run until no additional partitions pass.
`bulk_update() cannot be used with primary key fields`		Run `update_metadata` again. Click on `load_all_metadata` bar in the top pane. Use dropdown on Re-execute menu to choose Selected step.
The error is maintained after you make the proposed correction.	Google Sheets does not save the change until you exit or unselect the cell.	Click outside the cell with the change. Re-run the required assets.
The error is maintained after you make the proposed correction.	An unintended change to Reference Data or BSS Metadata column headings prevents the pipeline from reading the sheet. Changes are not being recorded by the database.	Verify that all column headings are original and haven’t been accidentally changed.

Issues with prior corrections

When re-uploading a baseline you may encounter an error about a correction that previously worked for which the previous_value in the Corrections sheet and the pipeline’s observed value no longer match. This may be caused by hyperlinks in labels changing Excel-saved values for the cell. Sometimes a version of the BSS might be saved that prioritizes the BSS’s memory of the hyperlink text, while sometimes a version of the BSS might be saved that actually prioritizes the human-visible label. While we aim never to change a BSS, we do need to unlock files to unhide rows/columns, and even this creates opportunities for this kind of error. In such cases, begin the BSS upload to Google Sheets and the database again.

Download the original version of the BSS from Google Sheets.
Open the original version in Excel. Ensure all rows and columns are visible, that the name matches the BSS metadata expectations, and that iterative calculations is turned on. Save As .xlsx if needed
Re-upload the revised file to Google Drive as a new version of the previously-loaded file.
Re-upload the file to the database by running original_files downstream to uploaded_baselines; then load_all_corrections; then imported_communities upstream to corrected_files; then imported_baselines upstream for All assets -1 (excluding corrected_files).

In this process, corrections, particularly those that involve hyperlinks or #REF! or #ERROR may need to be changed. Simply change the previous_value entries in the BSS Metadata Corrections sheet to match the value the pipeline encounters.

Common errors with existing tickets

Payment in kind

There are also still some errors in how the pipeline is reading payment in kind that should be resolved with pending tickets/merges (see HEA-851 ).

“A Livelihood Strategy must have either a Product or an Additional Identifier for LivelihoodStrategy 47 from 'Data'!541:541:” This is because the pipeline isn’t recognizing the (activity) product alias in ClassifiedProduct.
“Missing mandatory foreign key payment_product_id for PaymentInKind 26 from 'Data'!B553:” This is because the pipeline isn’t recognizing the payment_product in the “payment in kg per time ([product])” row, either due to a syntax error or because the product isn’t listed as an alias in the ClassifiedProduct sheet.
“Unrecognized 'livelihood_strategy' foreign key ('ML10-YOR', '2009-09-30', 'PaymentInKind', '', '', '') for LivelihoodActivity 661 from 'Data'!B541:” and/or “Unrecognized 'livelihoodactivity_ptr' foreign key” I believe these are duplicate errors related to the errors above.
Despite the above, there may be some errors you can still see and fix:
1. Unique labels (check ActivityLabel or label recognition dataframe for only 1 BSS). These you can override with no risk to other BSS currently in the database (consider possible implications for future BSSs).
2. Unclear activity product names: apply a correction to the label or add the alias to the ClassifiedProduct list as appropriate
3. References to multiple similar activities throughout the rowset: Use the most common highest level activity product for the rowset. Example: the label says ag labor but the activity_notes reference weeding, land prep, and sowing then change the label to pre-harvest labor)

Milk

“Lactation days must be provided if there are milking animals for MilkProduction” ticket ( HEA-1030 )
“Daily production must be provided if there are milking animals” ( HEA-1030 )

Other Cash Income

“Income for 'Other Cash Income' must be payment per time * times per year for OtherCashIncome” This may be because of a rounding error ( HEA-986 , HEA-982 ) or because number of months ( HEA-982 ) or number of people ( HEA-1053 ) is not a whole number.
“Quantity produced for Other Cash Income must be payment per time * number of people * labor per month * months per year for OtherCashIncome” ( HEA-985 )

Other errors

“Expenditure for a Food Purchase must be quantity purchased multiplied by price for FoodPurchase 53” This may be due to a rounding error or also because number of months is not a whole number. HEA-983 , HEA-1144 , HEA-1339