Leading Questions, Logical Fallacies, and Barking Dogs

Long after Breakfast at Tiffany’s and before the A-Team, George Peppard played freelance investigator Thomas Banacek for two seasons on television. We loved the show. Banacek used get involved with a convoluted case, then quote some Polish proverb to explain his solution process. I have no idea if any of these proverbs were real, but I always remember this one, “If you’re not sure that it’s potato borscht, there could be children working in the mines,” and his translation, “What you don’t know can hurt you a lot.” Many discussions of gun rights vs gun control lead to a citation of a statistic, or a series of statistics in support of the argument. The first thing to consider is not whether these statistics support your worldview, but whether they reflect reality. Why? Because what you don’t know can hurt you a lot. Or the non-Polish corollary, it isn’t what we don’t know that gives us trouble, it’s what we know that ain’t so.

First consider the methodology behind the statistics. Good pollsters know that bad questions can skew the results:

The wording of a question is extremely important. We are striving for objectivity in our surveys and, therefore, must be careful not to lead the respondent into giving the answer we would like to receive. Leading questions are usually easily spotted because they use negative phraseology. As examples:

Wouldn’t you like to receive our free brochure?

Don’t you think the Congress is spending too much money?

Pollsters with an agenda thrive on leading and loaded questions, and know that the answer to a question can sometimes be influenced by the preceding questions. So if I wanted more no answers to a question about gun control, I’d probably ask questions about home defense first. If I wanted more yes answers, I’d probably ask about murder/suicides and spree killings beforehand.

The extreme form of leading questions are push polls, as described on Wikipedia:

Perhaps the most famous use of push polls is in the 2000 United States Republican Party primaries, when it was alleged that George W. Bush’s campaign used push polling to torpedo the campaign of Senator John McCain. Voters in South Carolina reportedly were asked “Would you be more likely or less likely to vote for John McCain for president if you knew he had fathered an illegitimate black child?” The poll’s allegation had no substance, but was heard by thousands of primary voters. McCain and his wife had in fact adopted a Bangladeshi girl.

We have often read that greater ownership of guns corresponds with less crime in a given place over a given period. Therefore, we are told, ipso facto More Guns = Less Crime. We’ve also read the opposite.

Ipso facto is Latin for, “by that fact itself.” The classic logical fallacies are also expressed in Latin, one of them being post hoc ergo propter hoc. That means, “after this, therefore because of this.” Let’s look at some examples that logical fallacy:

After Brad Kozak practiced at the gun range, he started shooting more accurately, therefore his range practice made him a better shot.

After Brad Kozak started blogging at TTAG, more people started reading TTAG, therefore Brad’s writing led to TTAG’s increased readership.

After Brad Kozak joined the NRA, the NRA endorsed Harry Reid, therefore Brad’s joining influenced the NRA’s endorsement.

Although the first seems likely, the second possible and the third absurd, none of these statements are intrinsically logically sound because they don’t prove causation – only correlation. Brad may have also had corrective eye surgery, or gotten a better weapon to become a better shot. People may be reading TTAG because of other writers. And the NRA probably didn’t consider Brad’s feelings when they endorsed Reid. When given such statisics, we need to know whether the statistician eliminated other possible causes and established a clear connection between the events.

There’s another problem with statistics – we don’t understand the math very well. Suppose you’re watching someone’s dog. They tell you that Fido barks constantly when hungry, but when you feed him well, he barks less five percent of the time. When Fido is barking, what is the likelihood that he is hungry?

I put the answer at the end of this blog. I barely remember Prob/Stat, but this seems like an easy puzzle. We’re given 100% barking when hungry and only 5% otherwise, so if Fido is barking it seems obvious that there is 95% probability that he is hungry rather than just ornery. Right?

Odds Are, It’s Wrong. Science fails to face the shortcomings of statistics

It’s science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation. Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions. Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.

Replicating a result helps establish its validity more securely, but the common tactic of combining numerous studies into one analysis, while sound in principle, is seldom conducted properly in practice.

Experts in the math of probability and statistics are well aware of these problems and have for decades expressed concern about them in major journals. Over the years, hundreds of published papers have warned that science’s love affair with statistics has spawned countless illegitimate findings. In fact, if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature.

“There is increasing concern,” declared epidemiologist John Ioannidis in a highly cited 2005 paper in PLoS Medicine, “that in modern research, false findings may be the majority or even the vast majority of published research claims.”

Ioannidis claimed to prove that more than half of published findings are false, but his analysis came under fire for statistical shortcomings of its own. “It may be true, but he didn’t prove it,” says biostatistician Steven Goodman of the Johns Hopkins University School of Public Health. On the other hand, says Goodman, the basic message stands. “There are more false claims made in the medical literature than anybody appreciates,” he says. “There’s no question about that.”

Nobody contends that all of science is wrong, or that it hasn’t compiled an impressive array of truths about the natural world. Still, any single scientific study alone is quite likely to be incorrect, thanks largely to the fact that the standard statistical system for drawing conclusions is, in essence, illogical. “A lot of scientists don’t understand statistics,” says Goodman. “And they don’t understand statistics because the statistics don’t make sense.”

And what about Fido?

Answer: That probability cannot be computed with the information given. The dog barks 100 percent of the time when hungry, and less than 5 percent of the time when not hungry. To compute the likelihood of hunger, you need to know how often the dog is fed, information not provided by the mere observation of barking.

OK, that makes sense. If I rarely feed Fido, he’ll probably be barking almost 100% of the time because he’s hungry, and he always barks when he’s hungry. So my 95% guess would be close.

But if I always keep Fido well fed, he’ll only be barking about 5 or 10% of the time, and rarely because he’s hungry. So my 95% guess would be dead wrong. And sometimes—especially when you use stats effectively to justify a position on gun control—being dead wrong can get you dead.