PDA

View Full Version : Statistics Done Wrong


traynor
04-29-2015, 11:27 AM
"What goes wrong most often in scientific research and data science? Statistics.

Statistical analysis is tricky to get right, even for the best and brightest. You'd be surprised how many pitfalls there are, and how many published papers succumb to them. Here's a sample:

Statistical power. Many researchers use sample sizes that are too small to detect any noteworthy effects and, failing to detect them, declare they must not exist. Even medical trials often don't have the sample size needed to detect a 50% difference in symptoms. And right turns at red lights are legal only because safety trials had inadequate sample sizes.
Truth inflation. If your sample size is too small, the only way you'll get a statistically significant result is if you get lucky and overestimate the effect you're looking for. Ever wonder why exciting new wonder drugs never work as well as first promised? Truth inflation.

The base rate fallacy. If you're screening for a rare event, there are many more opportunities for false positives than false negatives, and so most of your positive results will be false positives. That's important for cancer screening and medical tests, but it's also why surveys on the use of guns for self-defense produce exaggerated results.

Stopping rules. Why not start with a smaller sample size and increase it as necessary? This is quite common but, unless you're careful, it vastly increases the chances of exaggeration and false positives. Medical trials that stop early exaggerate their results by 30% on average."

Statistics Done Wrong: The Woefully Complete Guide
http://www.statisticsdonewrong.com/

Free.

DeltaLover
04-29-2015, 12:33 PM
"What goes wrong most often in scientific research and data science? Statistics.

Statistical analysis is tricky to get right, even for the best and brightest. You'd be surprised how many pitfalls there are, and how many published papers succumb to them. Here's a sample:

Statistical power. Many researchers use sample sizes that are too small to detect any noteworthy effects and, failing to detect them, declare they must not exist. Even medical trials often don't have the sample size needed to detect a 50% difference in symptoms. And right turns at red lights are legal only because safety trials had inadequate sample sizes.
Truth inflation. If your sample size is too small, the only way you'll get a statistically significant result is if you get lucky and overestimate the effect you're looking for. Ever wonder why exciting new wonder drugs never work as well as first promised? Truth inflation.

The base rate fallacy. If you're screening for a rare event, there are many more opportunities for false positives than false negatives, and so most of your positive results will be false positives. That's important for cancer screening and medical tests, but it's also why surveys on the use of guns for self-defense produce exaggerated results.

Stopping rules. Why not start with a smaller sample size and increase it as necessary? This is quite common but, unless you're careful, it vastly increases the chances of exaggeration and false positives. Medical trials that stop early exaggerate their results by 30% on average."

Statistics Done Wrong: The Woefully Complete Guide
http://www.statisticsdonewrong.com/

Free.

Other things that might be responsible for wrong statistical analysis:
p { margin-bottom: 0.1in; line-height: 120%; } - Garbage in garbage out

- Wrong null hypothesis

- Lack or wrong data normalization

crestridge
05-01-2015, 07:46 PM
You are so correct; however, in addition, the use of 1st true positives and 2nd true positives in association with false positives, smooths out the exaggeration. But the key is to place a reasonable probability to your own estimate, then obtain further evidence apart from your estimate, and then find a "devil's advocate" (so to speak), (a proper false positive estimate), then do the math with all 3 factors, which produces a single probability.

Usually our own estimates are to high, so we need other opinions which may be more negative. The combinations probably give a more realistic view of our predictions.

We must realize a lot of the things that happen on the track are out of our control, and these impact our analysis.