Measuring Seattle’s Soda Tax: This is Why That’s So Difficult

A few weeks ago the Seattle City Auditor issued a new evaluation report on the city’s Sweetened Beverage Tax, the latest (and perhaps the final) in a series since the tax went into effect in January 2018. And consistent with earlier reports, the findings continue to show that the tax has done little to reduce consumption of sweetened beverages.

The report, commissioned on behalf of the city by King County Public Health and implemented by a research group at University of Washington, attempts to assess the impact of the tax and the programs that it has been funding on “norms and attitudes” of Seattle residents toward sugary beverages.

These types of studies are notoriously difficult to do well, since they require answering a counterfactual: how does the situation today compare to what would have happened if Seattle hadn’t passed a soda tax? Following a best practice, the researchers attempted to answer this by constructing a “synthetic Seattle” from other cities with comparable demographics (but no soda tax), and then conducting a “difference in differences” study comparing how things changed in Seattle compared to how they changed in the “synthetic Seattle.” Residents were polled in late 2017, just before the tax went into effect, and then again two years later in late 2019.

The UW research team constructed their synthetic Seattle using four other cities:  Minneapolis, MN; Rockville, MD; Bethesda, MD; and Arlington, VA. This is a curious compilation: Minneapolis is understandable, up to a certain point, as it is often compared to Seattle for similar educational attainment, earnings, and political leanings even though its economic base differs substantially, but the three small D.C. suburbs — all a fraction of the size of Seattle — have far less obviously in common with our city.

In response to an email inquiry, Professor Jesse Jones-Smith, who heads the UW research team, provided the following explanation:

“In choosing the comparison cities, we used key sociodemographic characteristics from the 605 “census places” (which is a census defined geography) and filtered for places that were similar on race/ethnicity,  per capita income, and percent of the population with a Bachelor’s degree. Based on previous literature, we hypothesized that these demographics would be important determinants of norms and attitudes. Seattle is pretty different from many large cities in its racial composition, having a higher percentage of White people compared to many West Coast cities. We therefore first looked for cities with similar racial composition. The was no single place that did not already have a tax or was not actively considering a tax (Portland was actively considering a tax when we were choosing comparison areas) that was a perfect match for Seattle on these three demographics. Minneapolis has a much lower per capita income (~34K vs 55K in Seattle for the year we pulled the Census data). However, we thought politically and culturally, Minneapolis was a good match, and it was a good match on the % of the population who are White. We would disagree that the DC suburbs we used in the sample have a very different economic base as you suggest. The census data indicated that economic indicators for the DC suburbs was actually a good matched to Seattle.  Arlington, VA was very well matched on % White and % with a Bachelor’s degree; but their per capita income was a little higher than Seattle (65K vs 55K). Rockville/Bethesda also made the short list for being well matched on all three characteristics, with per capita income closest to Seattle (48K vs 55K) and similar % with Bachelor’s degree. Population density of Arlington, Minneapolis and Seattle are all very similar (all around 8700), while Rockville/Bethesda is lower (~4500).

In addition to demographic characteristics, we were advised by multiple experts to pick at least two comparison areas that when averaged “look” like Seattle, since it is very difficult to find excellent matches for places like Seattle. This is similar to the idea of the Synthetic Control Method where multiple cities are used to construct a similar “synthetic version” of the city with the policy intervention. The idea behind using two places is not to look at them separately as comparison areas, but rather to look at them together. Unfortunately, for outcomes like Norms and Attitudes, there was no pre-existing administrative data to match on, so we matched on key demographics that are thought to influence these outcomes. You can see from the table of demographics in the report that indeed, the combination of these comparison areas is pretty well matched to Seattle demographics, with the exceptions elaborated on in the report.  Additionally, given the potential for mass media campaigns to influence norms and attitudes, it was important to select comparison areas that were well outside of the Seattle media market.”

“Pretty well matched” is pretty well subjective, and a table in the report providing demographic information suggests several ways in which the matching is at least a bit off. It also suggests another methodology issue with the study, in that more subjects were polled by phone in Seattle than in Minneapolis and in both cities there was a substantial shift to web-based polling for the 2019 round of surveys. Smith-Jones explained how her research team adjusted the raw results to compensate for this:

“We address this in two ways. Since you are reading the report in detail, you’ll note that we use propensity scores to weight the results such that people in Seattle time 2, Comparison time 1 and Comparison time 2 are weighted to look as similar as possible on demographics and survey mode (phone or web) as the people in Seattle at time 1. This is the recommended approach with repeated cross-sectional data in order to address any compositional differences over time. The propensity score weights upweight people who “look” most like those in Seattle at baseline on demographics. The weights are created with respect to all demographic and mode characteristics at once, but would serve to potentially down-weight the responses of people who responded by web and up-weight the response of those responded by phone in the second wave, in addition to those in the comparison area at time 1. In addition, we use what is called a ‘doubly robust’ approach and also control for all of these variables additionally in our statistical models.  The point of this is to control for any residual differences in the outcomes that are not well-controlled by the propensity score matching.”

This is not a terribly satisfying answer because re-weighting based on propensity scores will tend to amplify sampling biases when the sample size is small — and in this study the sample size is relatively small.

There are some other issues with the study that we will discuss later, but for the moment let’s get to some of the key findings of the report:

• Support for a tax on sweetened beverages declined to approximately the same extent in both Seattle and the comparison cities, suggesting that living with the tax did not noticeably change attitudes towards it.
• More than 90% of those surveyed had heard of the tax, and 47% said that they had heard negative messaging about it, compared to 29% in the comparison cities.
• 32.3% of survey participants in Seattle reported drinking less sugary beverages, compared to 36.1% of those in the comparison cities. Of those in Seattle who reported drinking less, 46% of them said it was because of the soda tax.
• The research team broke out those surveyed into “lower income” and “higher income,” where lower income is anyone making less than 260% of the federal poverty level. It found a handful of statistically significant differences in views on the tax and its impacts, as well as on the health impacts of consuming sugary beverages.
• Among those with higher incomes, high consumption of sugary beverages increased slightly in Seattle and decreased a few percentage points in the comparison cities — leading to a “difference in differences” increase of 5.3% in Seattle compared to the other cities.
• Among those with lower incomes, high consumption decreased slightly in Seattle but jumped a whopping 15 points in the comparison cities, giving a “difference in differences” decrease of 16.9% in Seattle.
• Among those with higher income, high consumption actually increased in Seattle and decreased in the comparison cities.

The 15-point increase in consumption in the comparison cities is a red flag that something is amiss here; it is implausible, to say the least, that consumption would increase that much over only two years — and especially at a time when thee are signs that nationally the consumption of sugary beverages is steadily trending down. Buried in the report’s appendix are some of the raw numbers that help us to understand this anomaly better. For the low-income respondents, in Seattle 19.9% were high consumers in 2017 and 18.0% were in 2019; in the comparison cities, 10.5% were high consumers in 2017 and 25.5% were in 2019.

A couple of observations. First, again it’s extremely unlikely that the statistic for the comparison cities actually increased by 15 percentage points in just two years; that strongly suggests that either the 2017 figure or the 2019 one (or both) are inaccurate.  Second, the fact that Seattle was at 19.9% in 2017 while the comparison cities were at 10.5% argues that they are not, in fact, a good point of comparison. When asked to defend this, Professor Jones-Smith replied:

“The difference-in-difference methods are often used in cases where starting levels are different. The statistical assumptions are not dependent on having similar starting levels. The (untestable) assumption that is required for these to be valid estimates of the effect of the tax is that the comparison area is a good counterfactual for what would have happened in Seattle had the tax not passed. We did our best to pick a good counterfactual by matching on the characteristics that we think determine trends in sugary beverage intake and norms and attitudes, as noted above.

Jones-Smith fails to recognize a weakness of the difference-in-difference approach: while it can be independent of starting levels, that is only true to the extent that there is room for the numbers to move in both directions. The comparison cities started at 10.5% in 2017; one could argue that is near or at the practical “floor” for how low high consumption could actually drop, and it had no room to go down further — or at the very least the pressures that would cause it to do so are very different than those that would cause Seattle to drop below 19.9%.

Jones-Smith had no specific explanation as to why the comparison cities would have such an anomalous increase of 15 points other than to say that it was “unexpected,” but she did emphasize that as a result she didn’t believe there was strong evidence of a reduction in consumption in Seattle:

“We agree that it’s possible the results are not indicative of a wholesale decline in SSB intake. We point out in the report that the declines are only seen among lower income residents (Seattle time 1 vs time 2) and that the large difference-in-difference is driven by a large increase in high consumers in the comparison area, which was unexpected. Because the declines in Seattle comparing time 1 to time 2 are statistically significant and because the trend in the lower income residents in the comparison area was up and not down, our assessment is that is suggestive, but not strong, evidence of a decline in intake among lower income consumers. We are careful to say the above in the report and to report what the results are, but we also report that we don’t think this is strong evidence of a large decline.”

But the consumption numbers for lower-income respondents in the comparison cities are consistent with their responses to other questions as well, particularly those related to the health impacts of consuming sugary beverages. Those believing that they cause “serious health problems” dropped ten points (from 90.3% to 80.0); 7 points for dental health problems; eight points for diabetes, and 8.5 points for heart disease. In addition, the 2017 baseline figures are substantially higher than those for Seattle but the 2019 numbers are close to identical. Together that evidence suggests that there was a significant sampling bias in the 2017 survey as conducted in the comparison cities — a much more plausible explanation than that 10% of people in the comparison cities simply forgot what they knew two years earlier about the health effects of soda.

But there’s another message hidden in the raw numbers for the lower-income respondents in both Seattle and the comparison cities. In both 2017 and 2019, nearly 90% of the respondents acknowledged that drinking sugary beverages causes dental health problems, obesity and diabetes. People aren’t drinking sugary beverages out of ignorance; it’s an informed choice on their part, for better or worse (mostly worse). They know what they’re doing to themselves.

And yet Seattle City Hall stubbornly refuses to admit that the sweetened-beverage tax isn’t having the desired effect, despite years of studies all pointing to that conclusion.  When the tax was originally passed, all parties acknowledged that it was a highly regressive tax, effectively taxing poor people to pay for programs for poor people. Proponents justified it by arguing that it was a “two for one” that would directly reduce consumption of sugary beverages as well as pay for programs, and it was tolerated by the low-income communities being taxed because the money raised was to be invested back in those communities. The good news is that the bulk of the money has indeed been invested into Seattle’s low-income communities; the bad news is the false promise of reduced consumption.

While no one in City Hall will utter the words “consumption has not decreased,” they are budgeting the tax revenues with that understanding. In 2018, the first year of the tax, it brought in $23 million in revenues, far above the expected target; in 2019, the figure increased to $24.3 million. Last year with the shutdown of restaurants, tax revenues dropped to $15.7 million. But the current forecast for this year has revenues rebounding to $20.8 million, and the most recent projection for 2022 is $22.3 million, closing in on pre-COVID levels. No one is predicting any sustained reduction in tax revenues due to a decrease in consumption. When asked to comment on its sweetened-beverage tax revenue forecast, here is the response from the City Budget Office:

“To address first the overarching question of the apparent contradiction of rising revenue expectations against the policy expectations of steadily reducing consumption of sugar-sweetened beverages, from a revenue standpoint, prior assumptions were that population growth, job growth and income growth would, in the short- to medium-term, outpace declines due to changes in consumption behavior and price growth.  This assumption has yet to be fully tested and observed, because we’ve been way-laid by the pandemic and stay/work-at-home policies.  These virus-related policy and health risk effects have significantly reduced consumption volume in Seattle. Due to restaurant and other venue closures, reported concentrate ounces (used to produce fountain drinks) fell approximately 52% in 2020 to 200.5 million ounces from 414.1 million ounces in 2019.  Ready-to-Drink ounces, in contrast, also decreased in 2020, but only by approximately 9.25% or 78.2 million ounces.  Clearly, for Seattle residents who work in Seattle, some substitution between buying a soft drink at one’s downtown Seattle lunch place could occur with buying soft drinks for home consumption at the grocery store, but it is hard to make up for the loss of suburban commuters’ consumption.

A medium-term forecast (perhaps 2023-2027) would maintain this basic modest growth in revenue assumption, even as some people reduce consumption.  For the immediate period of 2021-2022, however, this notion of modest, steady growth is being swamped by larger effects from employment growth and income availability in a time of changing unemployment insurance benefits and federal stimulus, as well as, the re-opening of tourism, restaurants and return-to-office activities.  These changes, should combine, if they occur, to result in larger increases in consumption that could begin to approach pre-pandemic levels.  The forecast just presented on August 17, is based on economic data as of July and assumes general forward progress against the Corona virus and COVID, including the steady reopening of events, venues, tourism, office-work, etc.  Under such assumptions there would be expected larger than trend growth increases in consumption in Seattle, primarily as office workers return.  As we described, however, there are substantial risks to this forecast, primarily related to the pandemic and our continued response to it.

We have no consistent source of price information, so generally assume general inflation.  Trying to project near-term consumption is the overwhelming driving factor in our forecasts at this time.”

The Sweetened Beverage Tax Community Advisory Board, which is tasked with recommending how the tax revenues should be spent, recently sent a letter to the Mayor and City Council with its recommendations for the upcoming 2022 budget. It too acknowledges and is planning on sustained consumption, evidenced by the fact that it is recommending to stop setting aside money to pay for job retraining programs for beverage-industry employees who might lose their jobs through beverage industry downsizing. However, it also recommends discontinuing funding for any further evaluations of the impacts of the sweetened beverage tax — essentially preventing any future reminders that the tax isn’t working. In fact, the board plays games with dates to try to end the five-year commitment to funding evaluation in the tax ordinance one year early; the ordinance clearly states that the commitment is for five years starting with the first year that the tax is collected (2018), but the Board writes:

In 2017, the Seattle Office of the City Auditor established a contract with Public Health – Seattle & King County (PHSKC) to complete the evaluation outlined in the ordinance. PHSKC worked with the University of Washington and Seattle Children’s Institute to design and conduct the evaluation. Funding began in 2017 so researchers could collect baseline (pre-tax) measures. From the CAB’s perspective, 2021 is the fifth year of funding for the SBT evaluation (2017, 2018, 2019, 2020, 2021) and the financing obligations have been met.

It also suggests that in the future if City Hall ever revisits evaluating the impacts of the tax, it should focus on the funded programs, “rather than the tax’s effectiveness.”

The CAB recommends the City consider a follow-up evaluation of the tax in the future, perhaps 10 years after the SBT was adopted (i.e. in 2027) to examine the impact of the tax policy. Additionally, we recommend future evaluations of the tax require matching funds from academia. Five years into the tax, investing in evaluations that consider the impact of SBT-funded programs, as we outlined in Recommendation 3, rather than the tax’s effectiveness, is better aligned with the CAB’s accountability core value.

Sadly, this fulfills city hall skeptics’ worst predictions about tax-and-spend politicians: once the money is rolling in, forget about the original intent of the tax and just focus on spending the money.

To be clear, there is nothing inherently wrong with the programs that are being funded with sweetened-beverage tax revenues (though it’s perhaps questionable to try to run education and counter-marketing programs to teach people things they already know about the health effects of sugary beverages). But at this point, all that’s left of the original proposal is a tax on poor people to pay for programs for poor people. That is not good policy, and it’s certainly not “progressive” policy. But there are community-based organizations with close ties to City Hall that are now dependent upon that revenue stream, and programs such as childcare that are the pet projects of certain elected officials and now major beneficiaries of the tax revenues. For all the complaints from the Mayor and City Council about Seattle’s regressive tax structure, they seem more than happy to leave this highly regressive piece of it intact despite its failed promise.

The story previously ran in SCCInsight, Kevin’s proto-Seattle City Council Blog