Bad Science: Quacks, Hacks, and Big Pharma Flacks

"This is really one of the best books I have ever read. It’s been hugely successful in the UK, and has now been published in the US as well. I thought it was just going to be Ben laying into silly tabloid newspaper stories about how everything in the world either cures cancer or causes it. And he does do that. But if you go through the book as a science writer, which I am (I’m writing about the science of economics, and what economists do) I started becoming increasingly ashamed of myself, at the standards that Ben holds up for writing about what’s going on in academia, versus the standards that I was holding myself up to. You read a paper, you understand a paper, you write a paper up. Ben is saying, “What about the contrary papers? What about the research that contradicts it? How is it that you found this particular piece of research? Have you looked widely enough? Have you thought about publication bias?” He is really thinking about all the different ways we can deceive ourselves that something is true when it really isn’t. Ben is just a master of that. It changed the way I thought about my own writing and it changed the way I thought about the world. It’s also become particularly relevant because economists have got much more interested in using the methodology of clinical trials. I mentioned Esther Duflo earlier, who is the author of a new book called Poor Economics . She won the Bates Clark medal for her use of randomised controlled trials in economics – the kind of trials doctors use to test new drugs. Economists are now using it to test different sorts of social policy. They’re tremendously inventive and I write about some of them in my new book, Adapt . For example, there is one trial currently going on to test the effectiveness of post-conflict reconstruction projects. There are two million people in the control group and two million in the treatment group. This project is being conducted in the Democratic Republic of Congo, in places that don’t exist on any map, that you have to wade up to your chest in swampy water for hours to reach. The ambition of some of these experiments is just incredible. It’s a really important change in the social sciences and reading Ben’s book is a real complement to it. It’s not enough just to run a randomised trial, you have to understand all the other stuff that is going on as well. That’s what makes medicine such an effective academic discipline. Ben is not defending the medical establishment against all comers. He’s explaining the scientific method and how it is used and where it falls short. When I talk to Ben about it, he says: “Whenever I look at how it works in other fields, I conclude that it’s bad in medicine, but it’s worse everywhere else.” Whether we’re talking about conflicts of interest policy, or trial registries, which make sure that trials with inconvenient results don’t just disappear, whatever we’re talking about, the doctors are ahead of the rest of us. But they haven’t got it entirely right. That’s one very important direction for economics. Also, reading Ben’s work made me think more deeply about a lot of the very cool results that are coming out of behavioural economics. You’ve got a good researcher who runs an experiment and produces a result and it’s published in a peer-reviewed paper. Previously I would have said, “OK, fine, that’s good enough for me.’’ But now I think, “Well, hang on, how many experiments have been done that didn’t get published? What’s the publication bias? What are the systems we have in place? What’s the bias in the media towards reporting the interesting results versus the boring results?” You do find the same few studies reported again and again in the media. They become iconic, when in fact they’re not really representative of what’s going on. Ben really got me to think hard about that process. I’m much more of a microeconomist, and all the books I’ve recommended are in some way related to microeconomic issues because I do feel microeconomics has made more progress in what is a very difficult field of study. But to defend the macroeconomists: you’re right they haven’t been able to give us a simple answer to the question, “Does fiscal stimulus work?” But the whole idea behind fiscal stimulus is to use the power of government to force unused resources into action. You simply cannot, in an abstract way, say whether it works or it doesn’t because so much depends on questions like, “What are the unused resources in the economy? What kind of sectors are they in? How open is the economy? Is it connected to other economies? Does it trade a lot? Does it not trade a lot? How deep is the recession? What kind of stimulus are we talking about? Is this writing cheques to people? Is it cutting taxes? Or launching spending programmes? What kind of spending programmes? Are we building roads or are we doing something else?” There are so many different ways it could be done. There will never be a single answer, because it will always depend on context. Also, whenever it’s been tried in the past, it’s been in a chaotic environment where lots and lots of other things are going on. You can’t create those experiments in a macro environment. I think they have made progress. You would have hoped they had made more, but you have to respect the difficulty of the question that is being asked. You said it’s a simple question. Sure it’s simple, in that it doesn’t involve that many words. But that doesn’t mean it’s easy to answer. Maybe you should just create a few economies, run some treatments and run some controls. Going back to Cory Doctorow’s book – and this is going to sound really, really weird and far out, but I believe it’s true – these online computer games are getting sophisticated enough now, and complex enough, with lots and lots of players, and real economies inside them, that you can start running experiments in these virtual economies to see how they work. That is something some economists are interested in doing. Maybe World of Warcraft will one day give us the answer to some of these questions you want answers to."

"Bad Science by Ben Goldacre. It’s partly an amalgamation of his columns in The Guardian, which dissect unsubstantiated claims that the media and businesses make on the basis of pseudo-science. It’s just a brilliant book, and he’s a fearless defender of science. He’s been on the wrong end of libel actions and so on, but he’s always right about the science, and about the statistics. One part of his book that I have a particular interest in is where he deals with this phenomenon that’s known as the ‘seductive allure of neuroscience’. This is the idea that for some reason the addition of ‘neurosciency’ words, brain words, make people more convinced about a certain product or finding. There was an experiment done a couple of years ago in which they explained to volunteers various different psychological phenomena. Some explanations of these phenomena were good, some were bad – so, they might say: ‘Researchers say that this phenomenon happens because…’ And then there were identical explanations but with random ‘brain words’ inserted, like ‘Brain scans indicate that this phenomenon happens because of the frontal lobe circuitry known to be involved’. The effect was that the participants generally rated the bad explanations as less satisfactory than the good explanations, except when they had neuroscience words in: which dramatically increased the satisfactory nature of the bad explanations. What it shows is that if you add a few brain words people are more likely to be convinced by something. It explains things like the success of products like ‘Baby Einstein’ and Brain Gym – which is a mainstream educational tool that’s sold for use in schools and used by many teachers, which makes these absolutely crazy claims about how the brain works and what you can do to improve things like attention and concentration in the classroom. We are weirdly impressed by science. Not explicitly, but the only way to deal with this is to understand more about science and the scientific method. I think he would argue generally that knowledge of the scientific process is very poor in this country and that’s why people are so convinced by, for example, the MMR-autism story. It’s a real shame – this kind of an education would absolutely empower us to make the right decisions but we don’t have it."