A Mindful Sceptic's Guide to Scientific Claims

Simple Ways to Think Clearly About Complex Evidence

In a recent article on Mindful Sceptics, I described why the truth about scientific authority was uncomfortable.

The Uncomfortable Truth About Scientific Authority

Because of how our brains work, we don’t have to believe what science tells us, which is handy when science generates objective evidence against a value we hold.

If we don’t like it or don’t want to agree with it for any reason, evidence is easily dismissed without hurting our sense of self. This happens even when we claim respect for science and the scientific method.

To help counter the tendency to dismiss legitimate science, here are five questions to ask yourself to avoid such rampant acts of dissonance.

Let's start with an example.

What do you make of this picture? It is a pattern that an ecologist might call a fenceline contrast.

The vegetation seems different from one side of the fence to the other.

The fence cuts across any likely natural contrasts in the soil and topography, making it tempting to assume that the grassland management has something to do with the difference.

Perhaps there is less grazing, some irrigation or fertiliser additions, and maybe even some herbicide application for weed control.

The assumption is that there is a human-induced cause.

The owner of the field on the right greets you with a kind smile. He says he’s proud of the increase in grass growth because a year ago, he converted his farm to an eco-retreat and removed all the cattle.

There is the reason.

Alright it is a reason, but is it the reason?

How do you tell?

After 40 years as an environmental scientist and consultant, I've developed a framework for evaluating claims like this.

It just takes a quick answer to five essential questions that have saved me from blind acceptance and reflexive scepticism.

Here is the list…

• What kind of evidence are we looking at?

• Who's making the claim?

• What's the context?

• Where are the gaps?

• What are the values at play?

And now, in more detail…

1. What Kind of Evidence Are We Looking At?

Not all evidence is created equal.

There is a hierarchy in the strength and reliability of different types of scientific evidence, with higher levels typically representing more rigorous, objective, and trustworthy findings, such as systematic reviews and randomised controlled trials, while lower levels include less reliable evidence, like expert opinions or case reports.

Here is a typical hierarchy that every mindful sceptic should know:

Systematic reviews sit at the top because they combine and analyse multiple research studies to give us the big picture. Think of them as the greatest hits album of scientific evidence.

Single studies come next. These are valuable but limited, like judging a band by one song.

Expert opinions follow next and are helpful but potentially biased, like taking music recommendations from a friend who only listens to jazz.

Personal anecdotes (and single photographs of fields) land at the bottom as compelling but unreliable, like claiming a song is universally loved because you and your friends enjoy it.

2. Who's Making the Claim?

This isn't about credentials alone. I've met brilliant citizen scientists and questionable professors.

Instead, ask:

• Do they follow the scientific method?

• Are they transparent about their methods?

• Do they acknowledge uncertainties?

• What's their stake in the outcome?

For example, a study funded by renewable energy companies deserves the same scrutiny as one supported by oil companies. Good science stands up to examination regardless of its source.

3. What's the Context?

That fenceline image is tempting, given that the context includes a specific and drastic management intervention of destocking to establish an eco-retreat.

But the evidence is weak. A picture may be worth a thousand words, but are they meaningful?

More context for the evidence would help.

Context questions to ask:

• What's the scale of the evidence?

• Are there geographical or temporal limitations?

• What assumptions underlie the conclusions?

• What alternatives were considered?

4. Where Are the Gaps?

Every claim, scientific or otherwise, has limitations and uncertainties.

A mindful sceptic looks for:

• Acknowledged uncertainties

• Alternative explanations

• Missing data

• Untested assumptions

In our example, although fenceline contrasts are widely used in applied ecology to compare managed and unmanaged areas, they often provide weak evidence due to confounding factors like differences in soil, microclimate, and historical land use, which may bias conclusions about the impact of management practices.

5. What Are the Values at Play?

Now, if you only have time to ask one of the five questions, this is the one.

Evidence is supposed to be objective and aims to minimise personal bias. Still, determining what is considered objective often involves underlying values, such as fairness, accuracy, and impartiality.

As a human or two are required to generate the evidence, they are riddled with values that easily leak into the scientific process.

Even the most robust science exists within the context of human values.

In our fenceline contrast, the landholder on the right-hand side wants more grass, diversity, and critters on the land that was once pasture. He has a value set.

Science can tell us what is but not what ought to be. Consider these value-laden questions:

• What trade-offs are involved?

• Who benefits, and who bears the costs?

• What priorities are being assumed?

• What definitions are being used?

What about the fenceline contrast

So, let's put the fenceline contrast to the five questions.

• What kind of evidence are we looking at?

• Who's making the claim?

• What's the context?

• Where are the gaps?

• What are the values at play?

All the evidence we have is an image of what looks like a difference in vegetation on either side of a stock-proof fence. An image without accompanying metadata is very low on the evidence hierarchy.

The claimant is a landholder who has just made a drastic land use change from livestock to ecotourism.

Context information is limited other than drastic management intervention of removing livestock.

There are gaps you can drive a truck through, not limited to when the photo was taken, limited data, multiple alternative explanations for the observation and a heap of untested assumptions.

Values are everywhere.

It is easy to conclude that any scientific claim about the contrast across the fence is weak, almost irrelevant, given the available information. Indeed, there is no evidence for any position, only conjecture.

Even the best science operates within human values. Recognizing this doesn't weaken evidence. It helps you evaluate it more thoughtfully.

Being a mindful sceptic isn't about rejecting or accepting all scientific claims. It's about engaging thoughtfully with evidence while recognising its power and limitations.

Next time you encounter a scientific claim, take a moment to run it through these five questions. You might be surprised at what you discover.

There were no before and after surveys to validate the fenceline image, but the only wild platypus I have ever seen in my 28 years in Australia was in a stream at the foot of the field on the right!

---