Misleading Studies: Bias and Hype

Hot Topics

Bias In Research Articles

 

Most published articles are taken at face value and lauded for the treasured secrets they purport to unveil. Some studies and topics result in a veritable media frenzy that uncritically accepts the study as if they were receiving tablets carved in stone from on high.

Merely applying the principles of “evidence based” research is not sufficient to be certain that a study is of value. The threshold is high for a study to count as acceptable and truly high level evidence. In medicine the “gold standard” is considered to be a prospective, blinded, clinical trial. But study design is key and is often wrong. The methods of selecting subjects, choosing the proper variables, and controlling them carefully is often flawed. Very often a “preliminary study” that has barely any subjects, in many cases 8, 12, 14 or 22, is taken to be a revelation of vital and unparalleled information. In fact those studies, as the authors usually note in print, if not on TV, are merely suggestions for future studies to more thoroughly investigate their proposals. No one should ever take a study of 22 people or 14 individuals and believe that the information is no germane to the billions of people who have not been studied. You don’t need to study a billion people to have good data, but statistical methods have a “power” measurement which the low numbers in preliminary studies do not pass muster.

Sometimes though, even what seem to be the best designed studies have problems. Human beings, their health, their activity are difficult to study. The science of biomechanics has hardly scratched the surface of what there is to know about walking, running, let alone serving a tennis ball, pitching a baseball or throwing a winning touchdown (rather than an interception). We can ask Mark Sanchez about his perception of when his passes work and why some of them don’t work at all, but he probably couldn’t tell us the answer and the answer doesn’t lie in his arm. Tim Tebow may help him figure it out. (Now why does Tebow have a hot web site, and Sanchez just that Wikipedia entry?)

Tebow-Sanchez BFF
Tebow-Sanchez BFF

There has been a trend lately that has focused on whether shoe suggestions or other changes at the level of the foot can either predict or change injury rates. One study titled “Is Your Shoe Rx Evidence Based” was not a study at all, but a several page editorial. An interesting study  showed that those who were forced to wear shoes they didn’t choose but which were chosen for them not on the basis of comfort or past injury history, but on their foot type showed that those who were forced to wear motion control shoes merely because they had flat feet did not fare well. Most runners will try to continue running in shoes that they have successfully run in previously unless media hype, marketing, or peer pressure leads them to try something different. This is like one of Newton’s laws ” A body continues moving at constant speed and direction unless acted upon by an outside force”.

My general recommendations on shoes follow ( I couldn’t resist this tangent but I have often avoided trying to be doctrinaire on the issue of shoes.):

New Rules of shoes

  • wear what you’ve found keeps you from getting injured
  • wear what you’ve worn previously that is comfortable and you’ve run successfully in
  • don’t wear more shoe than you need
  • don’t wear less shoe than works for you
  • there is little to no reason to select a shoe based on cushioning (overhyped in the past by shoe manufacturers marketing campaigns and by the media)
  • for most people a middle of the road shoe will work fine (light weight stability)
    • (neutral/cushioned means not much, but mushy)
    • motion control is too rigid for most, but ok for a few special needs individuals
    • motion control shoes are very boxy, and actually may be fit and feel better with an orthotic for those who have been injured and find that they are helpful (of course the common wisdom is no orthotics in motion control shoes, but common knowledge is quite common but not always knowledge)
  • stability shoes stake out a wide middle ground from not much stability to a lot, bordering on motion control
  • minimalist shoes may work quite well for you but transition carefully and slowly, very, very slowly
    • anecdotal stories indicate that some people find their hips and sometimes knees feel better in the minimalist shoe
    • breaking the rules of the “terrible too’s” of too much, too soon, too often, too fast with too little rest will get you injured very quickly no matter what any study shows
  • changing your foot strike is now considered by the minimalist proponents to be a step in the right direction
    • my long term ideal foot strike is midfoot rather than forefoot or heel
    • some minimalist “gurus” (okay, one) have termed an acceptable heel strike as a “proprioceptive heel strike” – a made up term, but acknowledging the reality that somewhere approaching 90% of the top 10 finishers in last year’s Boston Marathon were either heel or mid-foot strikers.
    • barefoot and usually minimalist shoe runners will do often do better with a forefoot strike
    • don’t forget to transition slowly (repeat this mantra 10 times slowly)
  • some have complained of 12 mm lifts in the heel, the manufacturers have responded
    • one popular shoe went from 12 mm heel to ball incline to a decrease of 33% down to 8 mm
    • many other shoes have changed similarly
      • stability shoes have sometimes become light weight trainers to the dismay of runners who have have used them for up to 17 previously successful iterations yet to the joy of others

And most important of all, something is right for every one. Not everyone attends the same place of worship and some don’t go at all. Evolutionary biologist Richard Dawkins skips it entirely. Not everyone wears a minimalist shoe, and many don’t wear any at all. Whatever it is that works for you, do it. Make sure you get your exercise, and for us runners, we know that we are all getting the best possible exercise. Swimmers, cyclists, tennis players and those who play ultimate frisbee may view it differently.

Now the point of this really isn’t the media and the changes in shoe recommendations which are now in many respects better than before, the point of this discussion is “how do we know we know something can a journal article ‘prove’ it”. So far as we were saying, studies have not been able to tell us how to avoid running injuries. That has been so for the past 30 or more years. Acccording to the literature shoes don’t prevent injury, orthotics don’t prevent injury, and even a graduated running program doesn’t prevent injury. Let’s look at that last statement, theoretically proved by a journal article “a graduated running program does not prevent injury”. That’s shocking. We don’t hear that study regularly quoted, but we did hear it a bit when it first came out a few years ago. The study is called No Effect of a Graded Training Program on the Number of Running-Related Injuries in Novice Runners: A Randomized Controlled Trial. The methodology needs a bit of examination. The study compared two groups training for a 4 mile race: one used a “standard 8 week training program” and the other used a “graded 13 week program” of 10% mileage increases per week. It found there was no difference in injury rate between the two groups. (RRI – running related injury which stopped training for 1 week or more). The injury rates in the control group and study group varied between 20.3% to 20.8%.

Does this study match the real world manner in how people run. Most people run and plan to continue to run, and not stop at the end of a study. So the goal for many may be to reach a certain level and then sustain it. This level may be at the 2 to 4 mile mark or it may be 6 to 10. Many of those running a marathon will do that as a rare event and not sustain training runs of 18 to 20 miles as a weekly routine. So 4 miles may be a reasonable distance for most runners to achieve on at least one training run per week. The premise as I see it is what happens when you reach a 4 mile run distance and then sustain that distance weekly for a year. In the case of this study it was not sustained. Those who reached the limit of a 4 mile run after 8 weeks stopped running. Those who reached it in 13 weeks kept running for a total of 62.5% longer in weeks run (5/8 – 5 weeks longer than 8 weeks = 13 weeks). That is an impressively longer number of weeks to run for at this modest distance. It isn’t comparing a 8 week versus a 26 week marathon training program. It is comparing modest distances that one should be able to achieve without great difficulty somewhere within that 8 to 12 week time frame. After reaching that modest distance though, I’d expect the runners to keep running and make the end point the same for both groups.

Experimenter Bias

Another reason for study #FAIL

Nature magazine had an article this past month entitled Beware the creeping cracks of bias

This essay purports that bias is systematic and even worse, it is not random. Random bias would average out, non-random bias is self-sustaining and deluding.

“Alarming cracks are starting to penetrate deep into the scientific edifice. They threaten the status of science and its value to society. And they cannot be blamed on the usual suspects — inadequate funding, misconduct, political interference, an illiterate public. Their cause is bias, and the threat they pose goes to the heart of research.

Bias is an inescapable element of research, especially in fields such as biomedicine that strive to isolate cause–effect relations in complex systems in which relevant variables and phenomena can never be fully identified or characterized. Yet if biases were random, then multiple studies ought to converge on truth. Evidence is mounting that biases are not random. A Comment in Nature in March reported that researchers at Amgen were able to confirm the results of only six of 53 ‘landmark studies’ in preclinical cancer research”

Cultural factors can pull research into repeated errors:

“How can we explain such pervasive bias? Like a magnetic field that pulls iron filings into alignment, a powerful cultural belief is aligning multiple sources of scientific bias in the same direction. The belief is that progress in science means the continual production of positive findings. All involved benefit from positive results, and from the appearance of progress. Scientists are rewarded both intellectually and professionally, science administrators are empowered and the public desire for a better world is answered. “

Research attempting to carefully limit variables does not reflect the real world very well:

“Researchers seek to reduce bias through tightly controlled experimental investigations. In doing so, however, they are also moving farther away from the real-world complexity in which scientific results must be applied to solve problems.”

An important conclusion of the author is that “a biased scientific result is no different from a useless one.”

“A biased scientific result is no different from a useless one. Neither can be turned into a real-world application. So it is not surprising that the cracks in the edifice are showing up first in the biomedical realm, because research results are constantly put to the practical test of improving human health. Nor is it surprising, even if it is painfully ironic, that some of the most troubling research to document these problems has come from industry, precisely because industry’s profits depend on the results of basic biomedical science to help guide drug-development choices.”

One outcome of bias in studies is that the public will realize that the scientists are not living up to their rules on objectivity. They may also realize that the media and journalists are too readily jumping on the hype bandwagon with little critical thought (WMDs in Iraq anyone? Tom Friedman and almost everyone else?). Critical report is, one could say, critical. Examine thoroughly every aspect of each new study. Don’t take anything for granted. Pressure those writing about a topic to understand what they are writing about. Too often a trend starts with a meme that goes out and dominates all only to be found that it wasn’t really a panacea. The statement “Throw away your high tech running shoes and you will never have another running injury” was made on national television about 2 years ago by an un-named friend wasn’t true at the time and still isn’t true. It only works if you begin swimming and give up other sports. The statement made that “run a marathon and you’ll be immune from heart disease for 5 years” was made by the late Dr. Thomas Bassler. It wasn’t true then and isn’t true now. both were catchy phrases and were cited by the media’s hyperbole machine.

Bias may guide the questions asked that research is supposed to answer. Bias may guide the methodology and the conclusions. The longing for a good quote and the “new” may lead to a worthless and untrue meme dominating the public spaces for years until it is corrected. Not long ago the media claimed that HMOs were going to save the American health care system. So between bias, studies that don’t live up or properly measure their claims and the media – we all have quite a job to do in determining what is real and what works for us. We are an experiment of one. We sometimes need someone to help interpret the results of the experiment.

And someday perhaps we’ll put into context Thomas Kuhn and the “Structure of Scientific Revolution”

Recommended Reading

Ida Buist, Steef W. Bredeweg, Willem van Mechelen, Koen A. P. M. Lemmink, Gert-Jan Pepping, and Ron L. Diercks
No Effect of a Graded Training Program on the Number of Running-Related Injuries in Novice Runners: A Randomized Controlled Trial. Am J Sports Med January 2008 36 33-39; published online before print October 16, 2007, doi:10.1177/0363546507307505

Daniel Sarewitz Beware the creeping cracks of bias . Nature 485, 149 (10 May 2012) doi:10.1038/485149a

John Horgan What are Science’s Ugliest Experiments Scientific American May 14, 2012. (I could quibble with the choices: the Tuskegee study was omitted, but mentioned, while the more recently newsworthy Guatemalan study was listed. You’ll get the message though.)

 

Just Give Me Some Truth: John Lennon