Russell Newton dives into how to evaluate data and statistics you can actually trust, ensuring you don’t fall for misleading claims. Learn the key indicators that reveal whether a statistic is trustworthy or manipulated, and how to think critically about data claims. By applying these insights from Russell’s expertise as an audiobook narrator and personal development coach, you’ll be better equipped to navigate the sea of information, spot statistical manipulation, and make informed decisions.

At Russell Newton’s channel, we dive deep into personal development through the lens of audiobooks and life wisdom. As a professional narrator, Russell brings clarity to complex topics like trust in statistics and how to critically evaluate data claims to avoid misleading or fake statistics. Join us for insights on statistical literacy and understanding data that truly reflects the truth in numbers.

In high school, I took two career aptitude tests.
The first one plotted my results on a chart, and I landed in the exact middle, which meant it couldn’t point me toward any career.
Now the second one, it printed a career right on my report, dental hygienist.
So, which one do you believe?
Most of us would go with the second one, it’s specific.
It came out on an official looking report,
It gave me an actual answer, but it never told me how it got there.
And that’s the thing about numbers and test results.
Now, smart people will trust bad ones all the time.
It’s not because they’re careless.
A number that’s consistent, neatly labeled, and handed to you by someone credible, it just feels true.
And here’s another example.
I had a guest Rick Collaton from on social skills coaching podcast.
He was talking about his sleep numbers.
Listen to how he puts it.
I still don’t get the amount of deep sleep.
I get enough that I feel very rested.
I’m just not impressed with the numbers.
I rarely get an hour of deep sleep and that’s a little troubling.
I can ask my eight sleep.
I can ask my aura ring or my Garmin.
They’re all going to tell me I didn’t have enough deep sleep, but maybe I’m wired just a little bit.
He feels rested, but three devices say he isn’t.
And it’s the numbers that bothered him.
Now, here’s what this episode is going to show you.
One, a number can be perfectly consistent, still measure the wrong thing.
Reliable and valid are two different questions.
Two, most bad numbers, they don’t come from the liars.
They come from small samples and correlations mistaken for causes.
3.
Being smart that doesn’t protect you.
Bias and overconfidence work on everyone.
And 4.
A short set of questions you can ask of any number to catch most of it.
Let’s start with the two questions people blur together.
In his most recent book, Statistics Simplified, Albert Rutherford draws the line.
Two terms matter for any data.
Reliability is whether the results can be reproduced.
Would another study on the same subject get the same answer?
Validity is whether the data actually proves the conclusion, and neither is ever guaranteed, even in scientific studies.
Take SAT scores, widely accepted data.
Their validity has been hotly debated, especially since COVID-19.
The College Board says scores are strongly predictive of college performance and retention.
And a 2020 Forbes headline defends them.
Six arguments against the SAT and why they don’t hold up was the name of their article.
Most people still believe higher scores go with better college performance, but there’s no consensus either way.
The way through is to read many sources, look at the data yourself if you can, and not believe what you hear until you have fact checked it or gotten it from a source you know is reliable.
Now, take away the study, just a claim.
Kerry Prigeant on the Science of Self podcast talks about what a claim needs before it means anything.
Here’s how he puts it.
There’s a special kind of declaration called assessments and assessments is most of what we live in.
The good, bad, right, wrong, like it don’t like it makes sense.
Don’t make sense.
Agree to disagree to that.
So assessments about this chair.
It’s a, it’s a comfortable chair.
Depends on your standards for comfort.
It’s a good looking chair.
depends on your standards for what’s good looking.
He’s old.
Well, if you’re comparing to people, a lifespan of, say, dogs, yes, I’m extraordinarily old.
You compare to a lifestyle of what some of these sea turtles, they live into their hundreds.
Now I’m about middle age.
Am I tall?
I’m six foot two.
Compared to the standard of pygmies, no, I’m a freaking giant.
Compared to the standard of the
the average NBA player too short for the game.
So again, whatever your assessment is, there always has to be some standards that you’re comparing it to, and there has to be some actions.
Like you could say I was tall or not tall if you had not a six foot two, and you didn’t know what standards you’re comparing it to.
A claim needs a standard.
A study needs both.
Would it come out the same twice?
And does it show what it says it shows?
pass one and fail the other, and the number still looks official.
Consistent?
That does not mean that it’s correct.
Sometimes it’s worse.
The number’s been dressed up to look solid.
Now, Rutherford’s book, Fooled, opens with one of the worst cases on record.
In 1998, Andrew Wakefield published a Lancet paper suggesting a possible link between the MMR vaccine and autism.
It covered just 12 children and presented them as developing normally before the vaccine.
Some had been recruited through legal efforts against vaccine manufacturers and Wakefield had been paid by the lawyers involved.
A person who was supposed to be credible used that credibility to fabricate evidence.
Our job is to look past the costume and ask not just whether someone sounds credible, but whether their evidence is.
Now, how did that get through?
Well, let’s go back to statistics simplified.
Wakefield’s fraud wasn’t recognized right away by the Lancet or the medical community.
He was eventually removed from Britain’s General Medical Council, and Wikipedia now opens his entry with a British fraudster, discredited academic, anti-vaccine activist, and former physician.
What exposed his financial conflicts and the extent of the fraud was a dogged journalist, Brian Deere,
The authors argue it wasn’t one bad apple, but a system’s failure that let flawed research gain traction.
Fould has more.
Three ways evidence gets dressed up.
We learned this as children.
The evidence for Santa and the Tooth Fairy was overwhelming, until the handwriting on the gift bag looked like dads.
As adults, we meet new Santas in lab coats, suits, and letters after a name.
And Wakefield is not an isolated case.
And one falsified paper in a respected journal can create panic with horrifying consequences.
So look at the naked evidence, how things were actually measured.
And you don’t need to be a statistician to catch fabricated data.
A too-good-to-be-true instinct is usually enough.
If 20 people respond about a new iPhone and all give nearly identical praise, be suspicious because real answers are messy.
Accurate numbers can mislead too.
Colgate’s more than 80% of dentists recommend Colgate
That came from a survey that let dentists recommend more than one brand.
But the ad made it sound like 80% chose Colgate over competitors.
That is cherry picking.
And one question catches it.
What are they not showing me?
Now, visual evidence can be staged, too.
In 2006, a Reuters photo of bombed out Beirut was approved by editors and spread worldwide.
But some bloggers noticed the smoke pattern repeated perfectly, and real smoke billows in different directions.
In one study, people spotted altered images only about 66% of the time, so even careful viewers get fooled.
We don’t need to become experts at spotting fakes.
We need to compare and question, which is how the Reuters photo was caught.
Look past the costume.
ask for the evidence.
Wakefield had a motive, but most bad numbers don’t.
Statistics simplified on the mistakes nobody plans.
To learn whether most Americans prefer mittens or gloves, you survey a sample, a smaller group that represents the whole.
If it isn’t representative, you can’t draw accurate inferences
Survey a playground, and 90% may say mittens.
But if 90% of your respondents are under five, you’ve learned about small children, not the country.
A good sample is proportional.
Every subgroup appears in the same proportion it does in the population.
Even excellent studies carry sampling error, the unavoidable gap between sample and population.
Polls can also fail through who answers.
in the famously wrong survey before the 1936 election, most of the sample never responded.
So the answers came from people who felt more passionately about responding.
One theory about a later election points to the same non-response bias.
The people who didn’t answer the polls were mostly Trump supporters.
Some of those polled may also have said they supported Clinton, but never voted.
Now, from the same book, a different accident.
Sometimes we find a correlation where there is none, most likely a fluke.
Tyler Viggins’ example is the correlation between yogurt consumption and Google searches for I can’t even.
How it happens?
gather a lot of data, dredge it by correlating one variable against every other just to see what sticks, then calculate the p-value to see whether the connection is statistically significant.
The p-value is the probability of getting the same results by chance.
Dredging until you find something significant, whether or not it makes sense, is called p-hacking.
Vigin’s spurgus correlations show it’s possible given enough data.
And dangerous, because the results can be incredibly misleading.
So I ask the plain question, where did the number come from?
I asked my guest Zoe Sharif on social skills coaching and here’s the answer I got.
Are these studies or analytics that you’ve developed off your client base or off other studies?
Where are these, the numbers you’re referring to, where are they coming from?
Yeah, I shared 300 surveys.
So I had, it started with 80 surveys and this was for a group project when I was in my data analytics course.
Once I really started writing and I was ready to publish, I realized that 80 is simply not good enough.
So I hired a company to find survey takers in the age groups where I was missing out on and also relationship statuses that I was missing out on.
So I ended up having a more equal distribution.
I believe that the total amount of surveys is 300 right now.
All to say there’s room, there’s so much room to understand this analyze it.
In fact, my
hope is that someday a graduate student or a graduate program will take this on and really delve deep into the science of what I’ve created.
So for now, it’s 300 surveys.
So 300 surveys have provided a preliminary analysis and overview and validation of what relationships need to make it work and also what the factors are with a relationship breakdown.
But yeah, so that’s what I mean by analytics.
I noticed what she did.
She told me where the numbers came from, how it grew, and what it can’t yet say.
That’s what to ask for of any number.
No fraud, no villain.
So why do smart people buy it anyway?
The brain treats stereotypes and biases as the efficient way to think.
Research, skepticism, and asking questions take far more effort.
Good reasoning also comes in degrees.
And often what looks like a good critical argument is really weak reasoning dressed up as thorough.
The first step is acknowledging when evidence is flawed and when it is strong, whether it’s yours or someone else’s.
A non-critical source relies on feelings, trusts information that may not be reputable, and leans one way.
It’s red flags, no other sources, or all its data from one place or one person.
A weak, critical approach catches inaccuracy but accepts the main assumptions.
You check to link sources or ask a friend where they heard something.
A source can look like it has many sources that all trace back to the same small, likely biased group.
A strong critical approach examines the assumptions and biases in the information and in the sources it cites, then does its own research to see whether the same results come up.
A strongly critical source draws on a variety of places and takes an objective stance.
Even then, humans make errors in reasoning
Let me read this from the book.
It’s the Linda problem from Think Critically.
Rutherford describes her like this.
Linda is a 31-year-old, single, outspoken, and very bright.
She majored in philosophy.
As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.
Now, choose one about Linda.
Linda is a bank teller, or Linda is a bank teller and is active in the feminist movement.
The two specifics are always less probable than one, yet most people pick the second because the brain saves energy by filling gaps with a story that fits.
Those who got it right asked, where is the bank teller part coming from?
Most of us rely on stereotypes even if we think we don’t, and critical thinkers learn which ones they use so they can catch the brain taking the quick route.
The brain also overrates its memory, which is why we still trust eyewitnesses.
Bayesian analysis shows how to weigh them.
Say a hit-and-run witness says the car was blue.
15% of the city’s cars are blue and 85% green, and the witness identifies colors correctly 80% of the time.
The witness would say blue in 29 of 100 accounts.
So when this witness says blue, they’re right only 41% of the time.
That’s far more context than taking them at their word.
Now from the same book, Chapter 4, Kahneman’s Two Speeds of Thinking.
Because of survival instincts, our brains want fast decisions and to avoid effort.
Kahneman calls the shortcut mode fast thinking, or System 1, instant, unconscious, automatic, emotional, intuitive.
Slow thinking, System 2, is conscious, rational, and deliberate.
Fast thinking checks in with your baseline of normality, but believes things without evidence.
Slow thinking examines more deeply, but drains energy and time.
Slow thinking goes against human nature.
It asks us to try to disprove our own beliefs and accept the unfamiliar.
So we usually bring it out only when something unexpected happens.
The shortcuts aren’t wrong in general.
Nobody needs deep analysis to pick lunch.
Slow thinking belongs where we’re likely to err, such as when we have little information and might jump to conclusions.
And just because a few people believe something, even the loudest or the closest to you doesn’t make it true.
Overconfidence is the related trap.
Overly confident in good results, we take ill-advised risks and don’t prepare for failure.
And we aren’t good at judging our own abilities.
Now, the reverse can also happen.
After a failure, the brain overcorrects, making decisions too cautious or too hazardous, driven by emotion rather than logic.
Rutherford discusses this in his book, Mental Models, where he names three traps that you can check for.
confirmation bias, which favors information that supports what we already believe and dismisses what contradicts it.
If you believe remote work kills productivity, ask what someone smart who disagrees with you would say.
Anchoring relies too heavily on the first piece of information, even if it’s arbitrary.
The author’s son saw a jacket marked down from $400 to $200, and the $400 became his reference point, so $200 felt like a deal.
The jacket was probably never worth $400.
Similar ones sell for $120.
The check?
What would I think if I hadn’t seen the original price?
The availability heuristic makes anything vivid, recent, or emotional feel common.
After a plane crash makes the news, some people avoid flying entirely, while others fixate on the survivor in seat 11A and request that seat for safety.
Both groups still drive to the airport, though the car ride is statistically far more dangerous than the plane ride.
The check
Is this actually frequent or just trending?
Comparing gut reactions with data pulls the spotlight from drama back to reality.
Carry Prijjan again on how we hold a belief once we have it.
Here’s how he puts it.
accumulates, and you begin developing some assessments about the world early on, and then you spend the rest of your life gathering evidence that those assessments are correct.
The problem is we forget they’re just assessments.
It’s just a story we told ourselves.
We hold them like they’re the truth, not just the truth, but the truth in capital letters.
And those are generally the places, especially the ones we hold very dear to our hearts.
Those are the places where people usually suffer.
Here’s another guest from the podcast, Dr. Carmen Bell Ross on why you can’t catch it yourself in her own words.
The thing, we all have blind spots.
You know why?
Because we only see the world through the lens that we can see the world through, right?
We only see ourselves through the lenses that we can see ourselves through.
And so just being able to acknowledge, yes, I am human.
And so therefore by default, I’ve got some blind spots somewhere.
Now being smart doesn’t turn the shortcuts off.
Everyone has blind spots, and that includes you.
So you can’t trust the feeling of being right.
You need a checklist.
Peter Hollins, in his book, Rapid Knowledge Acquisition, calls it prompt.
Six letters, six questions.
Presentation.
Is the information laid out clearly enough that you can find what you need?
and presentation can also deceive.
Many fake news sites mimic official-looking governmental bodies, and a quick look shows the information may be a fringe group’s unfounded opinion.
R is relevance.
Does it apply to your situation?
A scholarly article on an agricultural technique may turn out to have been written for a different country with a different climate and geography.
Objectivity.
Is it neutral?
Many people think they can spot bias when they’re really just good at agreeing with people they already agree with and never notice they’ve taken in only part of the truth.
Be especially careful with politics, history, culture, ethics, any controversial topic where large amounts of money are involved.
Challenge your own assumptions and ask whether what you’re reading is fact or opinion.
M, in prompt, is method.
Don’t assume all published material follows sound methods of argument.
P, second P, is provenance.
Consider the source.
Credentials do not make an author correct or free of bias, but it is extra trust gained when an author is transparent about who they are, what their interests are, and what qualifies them.
Look at who published something and why.
A pamphlet on the health benefits of milk written by a doctor, employed by a major dairy company, and published by a research group that’s really a dairy lobby in disguise.
Take that with a pinch of salt.
And finally, tea timeliness.
Information that is correct and fair one moment may not be a minute later.
Some knowledge holds for decades, but on topics like the economy, a pandemic, cryptocurrency values, climate change, political events, you need current information.
Note when something was published, and remember that newer information may have made it less valuable.
Classic material can still help you interpret current events, and one more, and it’s the hardest.
Dr. Bell Ross, on which question to ask?
Here’s what she had to say.
When you ask someone to tell you about yourself, you don’t necessarily know the answers, right?
As a general rule, I love to only ask the questions where I think I know the answers.
Know the answer.
Yes, because there’s a certain amount of courage that comes along with saying, I don’t know the answer.
I have no idea what the answer may be, and I may not even be prepared for what this answer may be.
I don’t feel that courageous all the time.
Sometimes I do and when I’m feeling more courageous, I ask those kinds of questions because I don’t know the answers and I am interested to get a perspective that’s not my own, right?
Now back to my two tests from high school.
One put me uselessly in the middle of some charts.
The other printed dental hygienist on my report and never said why.
Rick Colleton has the same problem with a different number.
He feels rested, three devices say he isn’t, and the number still bothers him.
So this was never about being smart.
Before you trust a number, ask what was measured, who says so, and where it came from.
And one last thing.
There’s a line about this that gets credited to Einstein.
It isn’t actually his, it comes from a sociologist William Bruce Cameron in a 1963 textbook.
The mix-up is its own small example.
The most repeated line about measurement carries a label nobody checked.
As William Bruce Cameron put it, not everything that can be counted counts.
And not everything that counts can be counted.