"Poll" showing 73 percent approval for Mayor Lee was flawed

Mayor Ed Lee appears larger than life in the USF "poll."
Tim Daw

There was a poll conducted in late November by the University of San Francisco, the results of which were released in conjunction with the San Francisco Chronicle, claiming that 73 percent of San Franciscans approve of Mayor Ed Lee's performance.

It didn't take long for Lee's supporters to begin touting the figure as fact; soon after the poll appeared on SFGate.com on Dec. 9, the results wallpapered the comment section of the Guardian's website as the answer to any criticism of Mayor Lee, his policies, or the city’s eviction and gentrification crises. 

After all, it was a big number that seems to suggest widespread support. But closer analysis shows this "online poll" wasn’t really a credible poll, and that number is almost certainly way over-inflated. [Editor's update 1/13: The authors of this survey contest the conclusions of this article, and we have changed the word "bogus" in the original headline to "flawed." The issue of the reliability of opt-in online surveys is an evolving one, so while we stand by our conclusions in this article that the 73 percent approval figure is misleading and difficult to support, we urge you to read Professor Corey Cook's response here and our discussion of this issue in this week's Guardian.]

The problems with the USF “poll” are numerous, but the most glaring of those issues has to do with its lack of random selection. According to the New York Times Style Guide, a poll holds value in what's called a "probability sample," or the notion that it represents the beliefs of the larger citizenry.

The USF poll registered responses from 553 San Franciscans. That number itself isn't the issue, or it wouldn't be if those 553 individuals were procured through a random process. But they weren't, and it wasn't even close.

The survey participants were obtained via an "opt-in" list that, according to David Latterman — a USF professor, co-conductor of the poll, and downtown-friendly political consultant — meaning that anyone who participated in this particular poll had previously stated they were willing to participate in a poll. This phenomenon is known as self-selecting.

"We work with a rather large national firm and they have a whole series of opt-in panels," Latterman told the Guardian. "So they've got lists of thousands of people who have basically said, ‘Yes, we'll take a poll.’ And the blasts go out to these groups of people."

That means that even prior to conducting the poll, results had already been tailored toward a certain set of citizens and away from anything that could be classified as "random." And even the Chronicle acknowledged in the small type that “Poll respondents were more likely to be homeowners,” further narrowing the field down to one-third of city residents, and generally its most affuent third.

Even if pollsters could match the demographics of the polled with the "true demographics" as Latterman called them, it still wouldn't address the issue of self-selection. But that's not all: The list of "opt-in" participants, which was acquired through a third party vendor, according to Latterman, only contained English-speaking registered voters. And anyone contacted was contacted via email, another red flag in the world of accurate of polling data.

Interestingly, the USF “poll” also found that 86 percent of respondants said that lack of affordability was a major issue in the city, while 49.6 percent of that same group considered housing developers to be most at fault for the astronomical real estate prices. So, to recap: This poll, touted by many people as gospel in the comment section of this site, found that while the City is totally unaffordable, the man in charge of the City is barely culpable for that situation, and he remains incredibly popular.

According to the NYT Style Guide, "Any survey that relies on the ability and/or availability of respondents to access the Web and choose whether to participate is not representative and therefore not reliable." 

Uh oh. 

Russell D. Renka, professor of Political Science at Southeast Missouri State, conveyed far stronger feelings on the matter in his paper "The Good, the Bad, and the Ugly of Public Opinion Polling," saying that a self-selected sample "trashes the principle of random selection… A proper medical experiment never permits someone to choose whether to receive a medication rather than the placebo."

Strike two.

He then writes, "Any self-selected sample is basically worthless as a source of information about the population beyond itself."

Strike three.

So then why were such frowned-upon methods used in this poll?

Latterman attributes the tactics to many things, but mostly to the rapidly changing technological landscape of San Francisco, coupled with the high costs of alternative methods and a large renters market. 

"San Francisco is a more difficult model," Latterman said. "So Internet polling has to get better, because phone polling has gotten really expensive."

But even if Internet polling needs to improve, it is still important to prominently note that in original source material, lest you give folks the wrong ideas. Or even just misinformed ones. Unless what you’re trying to present is less about polling that trying to sell San Franciscans on the idea that Mayor Lee enjoys widespread support.






I agree. There's nothing "moderate" or "liberal" about tearing apart a city with rampant, pro-corporate gentrification, giving corporate welfare to predatory tech companies, and forcing longtime residents out in the attempt to give the city a major "make over" (to make the city the opposite of what it was: a city for the wealthy) and abandon its wonderful radical, Bohemian past. Lee is a conservative and so is the D8 supervisor. They both gift wrap themselves in the word "moderate." That's nothing but BS/PR.

Posted by Nonpartisan Guest on Jan. 12, 2014 @ 9:43 pm

far more about you than it says about Ed Lee.

Posted by Guest on Jan. 13, 2014 @ 7:31 am

I understand that. 2015 is an off-year election so it could easily depend upon the very low voter turnout. We saw how the very low voter turnout rejected Lee's favorite project: 8 Washington project, this past November. Would those same voters elect Tom Ammiano mayor? We'll see.

Posted by Guest on Jan. 11, 2014 @ 9:49 pm
Posted by Guest on Jan. 12, 2014 @ 4:33 pm

It's never happened, so no one should even try.

Posted by Greg on Jan. 12, 2014 @ 5:36 pm

tendency to be branded a loser and it makes it harder next time. Nixon came back, but not many do.

Ammiano is a genial guy but I do not think he can beat a popular incumbent. He's also getting on a bit and the city has never elected a gay mayor.

Posted by Guest on Jan. 12, 2014 @ 6:02 pm

Harvey Milk, Jerry Brown, Alan Grayson, Barack Obama, Paul Wellstone, Bill Clinton, Ed Rendell, Dennis Kucinich, Antonio Villaraigosa, Tom Bradley, Harry Reid, Barbara Mikulski, Bernie Sanders... to name but a few. Some lost very badly, like Milk, Obama, and Rendell. Some like Bernie Sanders were even perennial candidates on the fringe. And Ed Lee may not be so popular come 2015.

Posted by Greg on Jan. 12, 2014 @ 9:51 pm
Posted by Guest on Jan. 13, 2014 @ 8:08 am

Cynthia McKinney is another one who you probably find laughable, who lost, badly, and then went on to win the next election for her old seat. The fact that even candidates you hate, or find laughable, can do this, just goes to show that no one should be ruled out. As for Ed Lee, if his numbers keep dropping this way, I wouldn't be surprised if he declines to run in 2015.

Posted by Greg on Jan. 13, 2014 @ 2:55 pm

I see no reason why he wouldn't do that again.

Posted by Guest on Jan. 13, 2014 @ 3:05 pm

Which is exactly why I see no reason why someone who is a total unknown today, couldn't win the mayor's race against a well-known candidate like Ed Lee 2 years from now.

Posted by Greg on Jan. 13, 2014 @ 3:26 pm
Posted by Guest on Jan. 13, 2014 @ 3:37 pm

" and the city has never elected a gay mayor."

Well we don't know that do we? The city has never elected an openly gay mayor. This time could be a first. One would think that with gay marriage legal in many places (including CA) that Tom's sexual orientation would be irrelevant. It should be irrelevant/moot. No one obsesses over the sexual orientation of Newsom or the conservative/corporatist piece of work in there now. Why is a Queer guy's sexual orientation more important than anyone else's?

Posted by Nonpartisan Guest on Jan. 12, 2014 @ 9:57 pm

Maybe not direct personal prejudice. But a question mark over their ability at the highest level.

For much the same reason, we have never elected a female or gay President or Governor.

Posted by Guest on Jan. 13, 2014 @ 8:05 am
Posted by Guest on Jan. 12, 2014 @ 8:07 am


Posted by Guest on Jan. 13, 2014 @ 12:16 am


Posted by Guest on Jan. 13, 2014 @ 12:17 am

Reply to: Posted by Guest on Jan. 11, 2014 @ 5:49 pm

The BG clearly has your support since you live here daily (how many hits do you give them a day refreshing your browser to be the first to write your little scripted one-liners on any article they publish?), so you are part of that "minority support" you go on about. Ever think of that? And the fact that you feel you must continue shilling and serving as a hack for conservative Lee is a sad sign of despair and desperation on your part.

(Will this comment get through the filter by placing it down here?...it wouldn't let me reply underneath the comment)

Posted by Guest on Jan. 13, 2014 @ 12:20 am
Posted by Guest on Jan. 13, 2014 @ 8:07 am

If every1 in #sf could vote,the city would be more right wing as many former victims of communism (i.e China,Cuba,Russia) are conservatives, some extreme conservatives. Lol the hippies would be gone and definitely the blacks or what is left of them anyways, but would out city become cheaper? Overwhelmingly right wing cities and states are indeed more affordable than democratic ones. Texas and the entire south for that matter. So it is something to think about although it may take another 25 years before enough of these immigrants have their voting rights and out number the hippies that control the city now

Posted by Guest on Jan. 13, 2014 @ 11:04 am

I doubt it will ever become a Republican town, but we probably will also never see a left-wing mayor.

Put another way, we will stay socially liberal but become more conservative economically.

Posted by Guest on Jan. 13, 2014 @ 11:47 am

lets call a duck a duck....they fucking LIED ABOUT THE POLL. If any of you actually think that the information we get from university..ANY university..is the complete truth...well you all have some growing up to do...drying behind the ears.....wake it up people...!!

Posted by the Hun on Jan. 19, 2014 @ 10:21 am

i.e. they all show a majority approval for Lee. You would need to come up with something special to show otherwise.

Posted by Guest on Jan. 19, 2014 @ 10:48 am

Is any poll credible, I've yet to see any credible poll, for anyone

Posted by Guest on Jan. 19, 2014 @ 1:01 pm

happens to agree with what they personally believe.

So the SFBG finds that no poll is "credible" because they all show majority support for Ed Lee, whom the SFBG hates.

But of course the polls are credible because they broadly all agree with each other, and all are consistent with Ed Lee's easy election victory.

Posted by Guest on Jan. 19, 2014 @ 1:57 pm

i never believe in spell casting, until when i was was tempted to try it. i and my husband have been having a lot of problem living together, he will always not make me happy because he have fallen in love with another lady outside our relationship, i tried my best to make sure that my husband leave this woman but the more i talk to him the more he makes me fell sad, so my marriage is now leading to divorce because he no longer gives me attention. so with all this pain and agony, i decided to contact this spell caster to see if things can work out between me and my husband again. this spell caster who was a woman told me that my husband is really under a great spell that he have been charm by some magic, so she told me that she was going to make all things normal back. she did the spell on my husband and after 5 days my husband changed completely he even apologize with the way he treated me that he was not him self, i really thank this woman her name is Dr Aluta she have bring back my husband back to me i want you all to contact her who are having any problem related to marriage issue and relationship problem she will solve it for you. her email is traditionalspellhospital@gmail.com, she is a woman and she is great. wish you good time.

Posted by MCCART on Jan. 19, 2014 @ 10:52 pm

So...it may be true that Lee has plenty of support in the SF. I am really not surprised. USF (Which by the way is a white, rich kids school) does a "poll" that supports the strongly pro-business Lee.

Downtown business and real estate interests, (Dan White's ghost is laughing) along with the Silicon Valley and start up bubbles have created a housing boom/bubble in SF that changed the demographics of San Francisco from a working and middle class community to a wealthy hipster dominated "Green Zone" verging on the "Closed Cities" of perpetually corrupt Russia.

If it weren't for the strong tradition of activism and rent control SF would be a gated playground for the rich like Dubai. And that is on the way.

Posted by Deej on Feb. 18, 2014 @ 1:11 pm

categorization of it as being a "white school" (as if the word "white" was some kind of insult) is disturbingly racist.

And that devalues whatever your point was here - it's not clear.

Posted by Guest on Feb. 18, 2014 @ 1:18 pm

Post new comment

The content of this field is kept private and will not be shown publicly.