The reliability of various news media was a focus of discussion during the 2016 and 2018 elections. Various news sources have been criticized as biased, or even “fake”. A lot of analysis indicated that the opposing sides of the political debate were consuming very different information. Is there a way to improve how we consume news?
Most people know that some news sources are better than others. But when you ask anyone which are the good ones and bad ones, their list maps pretty reliably to a list of “good ones with whom I agree with politically” versus “bad ones on the other side of the political spectrum.” This isn’t good for news-gathering. It’s too easy to allow confirmation bias to blind us to diverse viewpoints. It’s also not a good way to convince other people that your news sources are better than their own.
Can we come up with a more neutral framework for evaluating news sources? I have political leanings, just like everyone, but I also like to read analysis and opinions from other viewpoints, so I spend a lot of time evaluating journalism from people who don’t agree with me politically.
How to identify crappy news sources
And I’ll say this– I don’t have a method for identifying good news sources. However, I do have some confidence that I can easily identify bad news sources and commentary. So my approach is to eliminate obviously bad sources, and give the rest the benefit of the doubt. I think it’s a fair assumption that if a news piece doesn’t have any clearly misleading stuff, it’s worth a read.
Five red flags for any news source
So let’s use that framework and start with some behaviors that I consider to be telltale signs that a news site, television station, newspaper, blog, etc is bad.
- Assertions are unsupported with data or citations.
- Supporting data exists, but is limited or cherry-picked for no clear reason. A popular formulation is to say, “Federal Policy on X should be Y. In the State of [some random state], when Y policy was implemented, Z happened.” Was that the only state with Y policy? Or was that 1 of 3 states that implemented the policy, and that state was the only one with results that support the author’s preferred conclusion?
- Correlation data is improperly interpreted, especially if there is nothing addressing potential confounding factors. Most partisan gun crime statistics fall into this category, on both sides of the debate. Same for lots of health food claims.
- Obvious signs of amateur production quality, like text in ALL CAPS or goofy color palettes (hot pink text on a black background, etc). Frequent spelling errors.
- An underlying worldview of large-scale conspiracy.
Cherry picking: my favorite disqualifier
More than anything, cherry picking is the feature that I most commonly prune a news source for. When you look at a supporting fact for an argument, ask yourself, is this data broadly representative of the policy in question? For example, in an article about federal marijuana policy, let’s say the author cites a figure on violent crime in low-income pot users ages 16-24. Why did the author select this specific slice of data? Why not some broader measure like violent crime rates for users versus non-users?
If there is no apparent reason for the specificity, it is always because the broad data did not support the author’s prior belief. This simultaneously shows us that the source 1) has an intention to persuade, not inform, and 2) is willing to engage in deception to achieve persuasion.
What factors shouldn’t disqualify a source?
Let’s list some factors that I do not think should eliminate a news source from your rotation.
- An overt partisan bias. It’s OK for a news source to have a horse in the race. However, in my experience, news sources that lean very far one way or the other tend to be low-quality. And, if you prefer to get your news from only one source, it shouldn’t be a partisan one.
- The medium. Good TV exists. Good Twitter exists. Good Facebook exists. Just acknowledge that some media, Facebook in particular, are engineered to activate and feed your own existing bias.
- The age of the source. Lots of blogs are new and good. Some print newspapers are old and bad.
So what should we do?
The best bet is to read a lot of different news sources, continuously add new sources to the mix, and continuously remove sources that don’t meet our quality standards. And even better, acknowledge that we are all subject to cognitive and emotional biases. Then read all news with a healthy measure of humility.