Bad Science Behind the Recent Attack

Download a pdf of this article here

On January 17, Professor Bellemare, an economist at the University of Minnesota, published an opinion piece in the NY Times claiming that his research showed a correlation between the number of farmers’ markets and the number of food-borne illnesses in a state.

Even taking Bellemare’s claim at face value, it simply doesn’t mean a lot. Correlation does not mean causation. There are numerous examples of meaningless correlations, such as a graph that was developed showing that the rise in autism correlates with the rise in consumption of organic foods – although no one would contend that there’s a logical connection between those two trends.

Moreover, the way Bellemare found this supposed correlation was deeply flawed. The only data used were the number of farmers’ markets in each state and the number of foodborne outbreaks and illnesses in each state. His team made no attempt to identify how many of those outbreaks or illnesses had known causes – thus ignoring the fact that thousands of these illnesses were clearly attributed to contamination in the mainstream food supply, not farmers’ markets.

They also lacked data on multistate outbreaks – which often involve hundreds if not thousands of illnesses – for more than half of the years they studied. And since those large multistate outbreaks have all been due to large-scale production and distribution, not farmers’ markets, that gap seriously skewed the data.

An online search revealed references to only three outbreaks of foodborne illnesses that have been attributed to food sold at farmers’ markets, anywhere in the country over the last several years. Contrast that to the more than 18,000 outbreaks listed in the CDC database, or the almost daily reports of recalls and illnesses stemming from the food supply in general.   Bellemare’s claims defy common sense based on an extremely poorly designed and faulty study of correlation.

While promoting his findings in the newspaper, Bellemare has not actually released his most recent research. He states that it is an “updated version” of the working paper he self-published in 2015, which had numerous problems:

• Internally contradictory results: While claiming that he found a positive correlation between the number of farmers’ markets in a state and the number of foodborne illnesses, the author admits that he found precisely the opposite – a negative correlation – between the number of farmers’ markets that accept SNAP (food stamps) and the number of foodborne illnesses.
  • If food sold at farmers’ markets truly posed a significant risk of foodborne illness, that would be true whether or not the market accepted food stamps. The exact same food safety laws and regulations apply to farmers’ markets that accept SNAP and those that do not. Many of the exact same farmers and food producers sell at both types of markets.
  • The author’s only explanation is to assume that people on SNAP would be eating at riskier locations normally: “[T]his might mean that when a new market opens that accepts SNAP benefits, the customers it draws in were more likely to suffer from food-borne illness when they shopped elsewhere.” But he provides zero data or facts to support this hypothesis. Instead, he seems to be appealing to the stereotype that markets that accept food stamps must be located in poor areas (which is not consistently true), and that poor people are eating food from unsafe sources.
  • The far simpler and more logical theory is that these contradictory results indicate that the entire correlation is meaningless.

• Exaggeration of the effect, if any: Even if the correlation actually existed, and even if it reflected a causal relationship, the magnitude would be tiny. The 2015 paper states: “On average, 10 more farmers market per million individuals in the average state-year over that period are associated with one more reported outbreak of any kind of food-borne illness. …. On average, 10 more farmers market per million individuals in the average state-year over that period are associated with 14 to 20 additional reported cases of any kind of food-borne illness.”
  • That means that if Bellemare’s research actually showed a causal relationship and not just correlation (which it does not), if we added 260 more farmers’ markets in Texas, then we might see between 14 and 20 more cases of foodborne illness a year – an average of less than one illness per market every ten years.
  • This very tenuous, small relationship is apparent in looking at the charts in the article posted at http://marcfbellemare.com/wordpress/wp-content/uploads/2015/07/BellemareKingNguyenFarmersMarketsJuly2015.pdf To any normal person, the data do not show any relationship.

• Extremely weak and incomplete data:
  • The study looked only at the number of farmers’ markets in each state. Yet some farmers’ markets have only a few dozen customers each week, while others have thousands of customers each week. The simple number of farmers’ markets in a state does not provide any real data on how many people are buying food at those markets.
  • While the study covered 8 years, the authors only had data on multistate outbreaks for 3 of those years. Multistate outbreaks often involve hundreds, if not thousands, of individuals, which could significantly change the number of illnesses reported from a state. It is not reasonable to look for a correlation to the number of illnesses in a state when the actual numbers of illnesses in each state is only available from the CDC for less than half of the years considered.
  • The authors relied on the CDC database to find the numbers of outbreaks and illnesses in each state. In more than half of the outbreaks (9,789 out of 18,213) listed in the CDC database, there was a specific food or foods that caused the outbreak. Where those foods came from – whether a grocery store, farmers’ markets, or other venue – could have been ascertained by inquiring with the health departments. That would have provided concrete data as to whether any of these outbreaks were attributable to farmers’ markets, yet the authors chose to ignore that minimally rigorous approach to the question.

• Failure to consider more likely explanations for the correlation.
  • Bellemare does not look at other potential causes that are likely to show a similar correlation, such as increasing numbers of salad bars and packaged vegetables sold at grocery stores, fast food chains, restaurants and hospitals. The interest in fresh foods and healthier eating, which is one of the major drivers of the growth in the number of farmers’ markets, has also driven a significant increase in sales of fresh and ready-to-eat produce at these higher-risk locations in the same time frame.
  • As just one timely example, the same week the opinion piece was published, Dole closed one of its plants because authorities connected pre-packaged salads from that facility to foodborne illnesses in eight states, including at least one death.

The flaws in the research lead to the question: why make such claims? There’s an old adage: follow the money. Bellemare directs the Center for International Food and Agricultural Policy. The Center is funded by an endowment from Cargill and ADM, whose control over our agricultural and food system is increasingly threatened by the growing consumer demand for local and sustainably raised foods. (http://www.cifap.umn.edu/about/goals/goals/goals/history)