The fifth in our series on this year’s group projects by Sci Com students – this week, it’s the turn of Stephen McGann, Emma Houghton-Brown and Haralambos Dayantis.
***
Do scientists see the world as objectively as they like to think?
Arguments have raged for years between those who regard science as an entirely objective discipline, and certain social scholars that believe science is subject to the same cognitive biases as every other human enterprise. At times, these debates have become less than polite.
In 1996, physicist Alan Sokal perpetrated an infamous academic hoax against a postmodernist journal called Social Text. Sokal submitted a paper for publication with the lavish title “Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity.” It was filled with the kind of popular academic jargon familiar to postmodernist scholars. To the delight of the publishers, the eminent physicist seemed to be embracing a socially constructed view of his discipline. They duly published.
A month later, Sokal announced that his paper was a fake – a stitched-together collection of nonsense to demonstrate the lack of intellectual rigour amongst postmodernists. The journal editors were furious, accusing Sokal of a lack of ethics. Many scientists were delighted. The affair became a major episode in what is now dubbed the ‘Science Wars.’
We became interested in Sokal’s particular objectives for the hoax. He claimed it was an ‘experiment’ to test whether editors would ‘publish an article liberally salted with nonsense if it (a) sounded good and (b) flattered the editors’ ideological preconceptions.’
This gave us a mischievous idea. What would happen if we took Sokal’s broad premise and turned it around onto scientists? Could we make scientists believe a hoax TV news story because it (a) employed familiar TV conventions and (b) it presented a flattering narrative of a lone scientist battling corrupt authority?
We set about constructing a four-minute TV news item about a visiting Japanese scientist called Shigeyuki Kagoshima, whose important climate-saving research had been thwarted by a cynical Chinese corporation. We studied science news clips on television to mimic common devices such as lab presentations and interview conventions. We presented our film to science undergraduates at Imperial College as a genuine news piece – and tested whether our audience could detect the content as fake. Finally, we revealed our hoax – and asked them for their reactions.
The piece was shot by professional filmmaker Mark McGann of Drama Direct. Practically all elements are fictional – the Chinese company does not exist, and Dr Kagoshima is played with wonderful authenticity by Shigeyuki Koide, the former science editor of Japanese newspaper Yomiuri Shimbun. In fact, the only real scientist was physics professor Jenny Nelson. But would the science audience be able to tell what was real and what wasn’t?
Our results were fascinating. We showed our film to 23 Imperial College science undergraduates attending a Science Communication class:-
We told them we had made a film to expose a scientific controversy, and wanted their input on the ethical issues. None of the students noticed anything amiss. They accepted the ‘facts’ at face value, and discussed the ethics of the fictional scandal at length. A third of the students even felt strongly enough about the issue to raise it outside of class.
When we revealed our deceit, they were understandably cautious. But thankfully they embraced our experiment (rather than our necks), and generously participated in a survey to assess which elements of the film they thought were real or faked. The results were better than we’d hoped for. For instance, when asked to guess the real-life careers of the participants Jenny Nelson and Shigeyuki Koide, only a minority identified Nelson as a scientist, while a large majority thought Koide was a genuine physicist – and none identified him as a journalist!
Our hoax seemed a great success. Even those trained to think scientifically could be duped in a Sokal-like manner by familiar conventions, expectations and prejudices. But what does it all really prove?
Perhaps it’s simply that a hoax is not really an experiment. It’s a rhetorical tool – founded on a preconception of a target’s weaknesses. Like Sokal’s hoax, our fake was a form of entrapment – moulding evidence to most expose what we assumed were our target’s vices. Ironically, by attempting to expose social constructivism in others, both Sokal’s hoax and ours demonstrated the subjective thinking of the perpetrator.
We were left with a renewed respect for the Royal Society’s famous motto, ‘Nullius in Verba’ – ‘take nobody’s word for it’.
Not even our own.
***
Stephen McGann, Emma Houghton-Brown and Haralambos Dayantis are currently studying for an MSc in Science Communication at Imperial College, London
Refractive Index 
Well yes, the above is an illustration of why the scientific method was introduced in the first place…
I find the assignment of career pretty interesting after the fact! Would love to see that explored further.
Is the fact that no-one questioned the video that significant though? The video is well-presented and has no immediately obvious (to me, at least, not working in any of the related fields) scientific holes. Doesn’t this just prove it’s easy to lie to someone?
Are you able to provide a list of the clues the students could have used to determine that it was a fake? Otherwise how is it a result? You can always lie to people who have no alternative channel for verification and are in a context where they are predisposed to believe you.
Hi Michael,
Actually, there were a few potential ‘holes’ that they could have spotted! But what was fascinating was that even though some students DID see that we were in a completely unsuitable lab for the device we showed, and that the solar device was actually a cheap piece of electrical equipment stolen from a nearby parts bin, these students told us how they allowed themselves to ignore such details because other elements persuaded them of wider authenticity.
I think what was more interesting for us about the video questioning was what took place *after* the room knew that there were fabricated elements. Although there was now the kind of increased attention that comes from knowing that one is looking at an illusion – it was still difficult for most to discern the boundaries of the deceit, and therefore raises questions to me about what an ‘obvious hole’ would look like, and whether the fact that something is sufficiently ‘well-presented’ should – in itself – be mitigation for any of us.
As we ourselves said, we doubt how far any hoax can be said to ‘prove’ any point. But we think that it perhaps gives a glimpse of the extent to which we employ associative trust in our everyday observations – even those of a scientific nature.
Showing the pholtovoltaic output of an ordinary 7-segment LED display chip pretty much marks it as fake to me.
I would also be disinclined to believe anyone who said they could double the output of a PV device. I might believe a few percentage points but doubling is way too much of a stretch.
Two clues to the veracity of the piece.
Hard science undergraduates rarely take sociology courses like “Science Communication” and even if they did they would not yet be experts in any hard science field, let alone an expert in the topic of the spoof as the reviewers were in Sokol’s experiment. So the primary flaw is that the population sample is wrong for the experiment.
This experiment is hardly the mirror of Sokal’s with expert reviewers being deceived in their own field. These social scientists did not set up a proper experiment to prove their thesis that hard scientists could also be deceived by convincing mumbo jumbo in their own field. Yes these students were as gullible to cargo cult science as the sociologists that Sokal spoofed, but that is not what the experiment sets out to prove so it is an irrelevant finding even if it is interesting.
All that these experimenters show is that they were incapable of constructing a proper experiment for their own thesis.
The score after two rounds: Hard science 2, Soft science 0
Rich, here’s one:
Is it necessary to wear a lab coat, purple gloves and goggles to test a small solar cell?
Hi Rich,
I agree – one can always construct an elaborate lie for others – that’s why we ultimately questioned the premise of Sokal’s – and our own – hoax. However, I would perhaps want to unpick the idea of a ‘belief predisposition context’ for the receivers of a communication. Where do we set the boundary for this predisposition? And – having passed it – does it then absolve the ‘victims’ of a hoax or fraud from any further discriminatory responsibility for what they have erroneously perceived? If a lie looks real enough, are we simply blameless to believe it? What do we mean by ‘real enough’ – and who decides? Should we not also question the perceptions, assumptions, associative trust heuristics and media mechanisms that allow us all to be so persuaded in the first place?
I’m reminded of the Andrew Wakefield case. Wakefield’s work was initially published in the Lancet – and only much later proven to be seriously fraudulent. Does the Lancet have nothing to learn from this case, because Wakefield himself did not provide sufficient clues for determining his own fraud at the time? Or did Wakefield’s job at the Royal Free Hospital provide sufficient ‘belief predisposition context’ to absolve science of all responsibility for this serious failure of the peer-review process? And where does that leave the public and press who are highly predisposed to believe such a reputable journal like the Lancet? Does the Lancet make clear to the public its *own* predisposition contexts?
Hi Kierliddle,
Yes – and as illustrated by Ivan Oransky’s wonderful RetractionWatch website – http://retractionwatch.wordpress.com/ I suggest it’s a knowledge production process that is not entirely immune to being deceived by human deceit backed by apparent credibility.
Among the “potential holes” was one that stood out. “Dr. Kagoshima” (brilliantly portrayed) and the reporter look at the readout. She says, “over twice the energy.” But the meter reads in millivolts, a sure sign that it’s measuring not energy, but potential. To establish that the energy is really more than double would require a more complex apparatus, knowledge that I’d bet not one in a hundred journalists would possess. Engineering students should all have had that lapse at least gnawing at their minds.
The title reads “Do scientists see the world as objectively as they like to think?”. Did you actually use scientists, or science students? If students, is the title appropriate, and what did we learn about scientists?
Hi Sean,
A really interesting comment in the light of our early research. You seem to accept – like many do! – that Sokal’s hoax was indeed a a valid “experiment”, whereas others – ourselves included – are “incapable of constructing a proper experiment” like Sokal to test a social scientific thesis.
But *was* Sokal’s hoax a valid experiment?
Stephen Hilgartner wrote a fascinating paper on this very issue. It’s a good read if you get the chance:
http://www.jstor.org/discover/10.2307/689833?uid=3738032&uid=2&uid=4&sid=56295493223
In this paper, Hilgartner compares the experimental methods behind Sokal’s hoax with a similar hoax perpetrated by William M. Epstein, a social researcher. He points out that Sokal’s hoax was *highly* unscientific: for instance(see p512), Epstein tested his hoax on a carefully selected list of 146 journals. Sokal?Just a single journal. Epstein performed a full statistical and qualitative analysis of results. Sokal? No analysis – just his own judgement. He simply declared his success in the media a month later!
Yet Hilgartner’s key point is this. The relatively unknown Epstein was castigated for his “unethical behaviour,” whereas Sokal – an eminent physicist – was widely praised. So where did Sokal’s credibility come from? It clearly wasn’t his methodology – but seems to be related to his *reputation*, rather than his methods. And this is, ironically, a rather social scientific finding!
Secondly, Hilgartner argues that Sokal had no academic expertise in social science, and yet assumed that his rather unscientific approach was perfectly sufficient given his expertise in another field of study. This too suggests interesting things about the social aspects of the Sokal affair.
So I might suggest that the 2-0 victory you claim for “Hard science” is a little optimistic. There seems to be very little that is ‘hard science’ about the Sokal affair – nor indeed our rather humble project! 😉 That it could be considered so in Sokal’s case I find rather interesting.
Hi dp,
Actually, both science students and a practicing scientist were involved in the hoax, although the scientist – the wonderful Professor Jenny Nelson – was not involved in the later film showing. In Professor Nelson’s case, this involved some deception on our part – and we were very mindful of our ethical responsibilities. The interview with her in the film was conducted without her knowledge of our true purpose. So after this was done, and before proceeding further, we came clean and sought her consent to continue. To our great relief, she responded with good humour, generosity and curiosity. She seemed interested in the extent to which she had suspected something odd about the interview, and yet had continued – trusting to the wider credibility of the students and the situation. She subsequently gave her permission to show the film, and enjoyed our account of reactions – particularly the wide identification of her by undergraduates as an arts professional, and not a scientist!
So what did we learn about scientists? Well, our single scientist taught us something very heartening. That the curiosity which drives her eminent scientific work is strong enough to embrace questions regarding the wider societal signals that may influence what even an accomplished scientist accepts as ‘real’. And that this question can be embraced with an openness matched by the comments of the students we showed the film to. I was left with a feeling that the mind of a scientist – aspiring or established – can be a wonderfully interesting and flexible place to live in.
The biggest difference between the Sokal “experiment” and this one is that Sokal did his experiment on experts in the field of postmodernism, while you did your experiment on undergraduate students who have not even completed their degree, much less practiced their profession and been subjected to the rigors of the scientific method (assuming that they are real scientists or engineers, and not political, social, or some other pseudo-science).\\BTW, what sort of “science” were these students studying – environmental science, electrical engineering, chemistry, or what? It makes a difference – the softer sciences are almost as bad as poly-sci, while engineers have to build stuff that works, or else people die.
Perhaps the large sign saying ‘Eyewash’ might have been a clue?
And the leap from a meter showing ‘volts’ to a commentary discussing ‘energy’. Related, but different things..
@ Steve McGann
In reply to Sean you pose the rhetorical question: “But *was* Sokal’s hoax a valid experiment?”
Ah yes, ‘valid’! A word beloved of the social constructionist bunch. But even so, you don’t answer your question. You do tell us that some fellow named Hilgartner reckoned Sokal’s hoax was highly unscientific but you don’t say why. (BTW I don’t want to pay the 19$ to read Hilgartner’s paper.) By your:
“Epstein tested his hoax on a carefully selected list of 146 journals. Sokal?Just a single journal. Epstein performed a full statistical and qualitative analysis of results. Sokal? No analysis – just his own judgement. He simply declared his success in the media a month later!”
you imply that Sokal’s hoax was statistically inadequate in some way and that that establishes its unscientificness (that word ought to exist). Leaving aside the question of there be *anything* at all scientific in the social so-called sciences or how one could tell, perhaps you could explain why statistics has to be brought into this in the first place. If you do, then while you’re at it maybe you could tell us what *minimum* sample size Sokal should have used along with its *scientific* derivation.
But moving on, as they say, and skipping past the irony, as you call it, of Hilgartner’s key point (a ‘scientific’ one no less, about social judgements being affected by perceived social status — well, quelle surprise!) you then move on to:
“Secondly, Hilgartner argues that Sokal had no academic expertise in social science, and yet assumed that his rather unscientific approach was perfectly sufficient given his expertise in another field of study.”
So Hilgartner *argues* that Sokal had no academic expertise in social science, does he? Wow! Impressive. (But then seeing as Sokal is a physicist that’s hardly surprising. In fact, even if he did have such expertise, I’d imagine he’d prefer to keep it very quiet — after all, he’s got a respectable reputation as a physicist to consider.) But Sokal freely admits that he has no such expertise, or least one can reasonably infer that from what I recall of his revelation following his researches prompted by his reading of Gross & Levitt’s ‘Higher Superstition’, so why the need for an *argument* to establish a rather boring and very simple *fact*?
By the way, despite Sokal’s claim and what he says about it himself, I see it as wholly irrelevant whether his spoof is regarded as ‘scientific’, ‘valid’ or even an ‘experiment’; it was a *hoax*, and it worked a treat.
As for:
“Sokal? No analysis – just his own judgement. He simply declared his success in the media a month later!” [SM]
As a boy I fired a stone at a window with my catapult to see if I could break it. I saw it break. It broke. No analysis – just my own judgement. As far as I can tell, Sokal’s judgement there is as good as mine.
Look, never mind all that yakety yak about ‘science’, ‘scientific’, ‘valid’, ‘experiment’, ‘statistical’ or ‘analysis’, just get your boys to pull off hoax like Sokal’s — get them to spoof up a physics paper and get it published some half decent physics journal, then you can crow.
HI John,
There’s a “social constructionist bunch?” Oh dear…. I’m imagining a chapter of academic Hells Angels riding around terrorising scientists by demanding validity at knife-point! 😉
As to the concept of logical validity – are you sure that this fine positivist principle is a concept beloved only by social constructionists?
“I see it as wholly irrelevant whether his spoof is regarded as ‘scientific’, ‘valid’ or even an ‘experiment’; it was a *hoax*, and it worked a treat.”
Which is entirely your prerogative! It is your subjective personal choice to define this hoax as a ‘spoof’, a ‘joke’, or anything else – and disregard Sokal’s own claims to it being an ‘experiment’. However, you have – as far as I know – made no wider claims in media as to the academic significance of your personal ‘spoof’. Sokal did. If the authors of the journal affected by his ‘experiment’ defined it as ‘unethical’, would their view have equal weight to Sokal’s? To yours? Less weight? More? Whose definition is correct – and who decides?
“As a boy I fired a stone at a window with my catapult to see if I could break it. I saw it break. It broke. No analysis – just my own judgement. As far as I can tell, Sokal’s judgement there is as good as mine”
I think this example illustrates the matter rather well. What Hilgartner is arguing, I believe, is that Sokal ‘threw a stone’ at one cultural studies journal. The hoax ‘broke a window’ – in that it achieved its specific objective by making that journal publish that hoax. He threw it. It broke.
But what – exactly – was ‘broken’? Just that window? Or every window in the entire edifice of cultural studies? Sokal seemed to be suggesting that his experiment exposed fundamental flaws in an entire discipline – with many dozens of journals and many different beliefs and methods.
My catapult breaks a window. From this I can determine that my catapult breaks that window. Are you saying that from this one act I can safely generalise that this catapult breaks every window?
“Just get your boys to pull off a hoax like Sokal’s — get them to spoof up a physics paper and get it published some half decent physics journal, then you can crow.”
Perhaps you could explain to me who ‘my boys’ might be? (my project colleague Emma is most certainly not a boy ;-)) And where – and about what – am I ‘crowing’?
The Science Communication course at Imperial requires that all students are graduates in a scientific or technical field. Students come from a range of disciplines – and also from different academic levels. This includes those who hold doctorates and have worked as scientists – and, yes, physicists. A particularly rewarding aspect of the course for me is to watch able scientists applying their minds to wider questions regarding science and its role in society. To these colleagues, the idea of critiquing a discipline that they cherish seems a natural extension to the curiosity that led them to pursue science in the first place.
Or does this appetite for reflexive enquiry mean they must now purchase a motorcycle and join a ‘social constructionist bunch’? 😉
Hi Sean
All participants in the sociological experiment actually were “hard science undergraduates” at Imperial College. They were a range of electrical engineers, mechanical engineers, physicists, biologists, etc… I have a full spreadsheet detailing the background of every participant if you’re interested. They were not renowned experts in the topic of the spoof, but that was never the intention.
You say:
“Yes these students were as gullible to cargo cult science as the sociologists that Sokal spoofed, but that is not what the experiment sets out to prove so it is an irrelevant finding even if it is interesting.”
It actually pretty much is what we set out to prove. This blog post is a condensed version of the full research document, in which our aims were set out more clearly. Steve has already answered a lot, and I agree that our experiment was no less “proper” than Sokal’s.
It was always intended to be illustrative, rather than rigorously “scientific”. That was never the point.