Don't Read This if you Trust Me: The Pitfalls of Trusted Sources

Keith Kloor reports that Daniel Kahan recently "said that 'people misinform themselves.' What did he mean by this? Well, people have go-to sources for issues they don’t have time (or the inclination) to research. Your go-to source on a contentious issue–such as climate change or GMOs–is likely to share your values. That affinity is what makes the source trustworthy to you. But that doesn’t mean your trusted source is necessarily going to provide you with correct information."

I disagree for two reasons: 1) That's not quite what Kahan is concerned about, and 2) I think there are some information sources that are able to resist ideological decisions -- and we would do well to turn to them in times of misinformation.

1)  Kahan has an excellent blog where he tries to explain his often counter-intuitive research.  For example: a study he conducted evaluating the relationship between numeracy and ideology.  He looked at a person's ability to detect statistical covariance in case studies that were value-neutral versus case-studies about hot-button topics like abortion and gun control.

Not surprisingly, people had a harder time correctly interpreting data about hot-button topics.  To be specific, people failed to properly analyze data when it conflicted with their ideology.  

Kahan likes to say that "critical reasoning is being used opportunistically."  And he goes on to point out that more proficient people (i.e. more proficient at value-neutral numeracy tasks) are more polarized than less proficient people, not because they are more biased (although this may be true) but because they are better at fitting the evidence to their existing ideological biases.  Importantly, this effect appears to be equivalent on both sides of controversial topics.  Neither liberals nor democrats have a monopoly on crazy baseless beliefs.

2)  This brings me to my second point.  Perhaps there are a group of people who are not liberal or conservative; people who do not have strongly-held opinions about anything apart from what the evidence provides.  Probably more people would self-describe themselves in this group than can actually live up to this standard, but still.  It seems to me that this would be the ideal of a dispassionate, objective observer.  A true scientist.  And if our go-to sources are value-less, or better stated, if our go-to sources hold objective knowledge as their highest value, than we are justified in turning to them for information.  Doesn't mean they can't be wrong, but if they have the characteristics I mentioned previously, then at least they are thoughtful, transparent, and open to conflicting data.

Presumbably Keith would support this second point, if he wants us to keep reading his blog!  However,  Keith Kloor goes on to point out that even trustworthy sources can hold fallacious viewpoints: "Groups like Greenpeace and thought leaders such as Michael Pollan, Vandana Shiva, and Bill Nye have enormous clout in their respective spheres. "  These people and groups earned this clout by speaking truth to power.  But that doesn't mean all of their opinions are objectively justified.  People can be rational about some topics, but irrational about other topics! 

How To Find The Truth (on the Internet)

I recently read about two different meta-review techniques: the Total Evidence Approach and the Quality Analysis Method, and that got me thinking about information processing and knowledge creation in our information-saturation internet-era.  How do we find the truth on the internet?

"In the total evidence approach (Kluge 2004; Sherman et al. 2008) all information is considered and data are not weighed by quality of evidence. Although the total evidence approach is subject to the biases and errors of individual studies, we deemed it preferable to the alternative “quality analysis” method (Sherman et al. 2008) in part because of the difficulty of objectively evaluating the relative validity and quality of the widely heterogeneous data sets that we reviewed."  Source.

I think we can all agree that a total evidence approach isn't going to work very well on the internet: there is simply way to much junk to try to average out truth from the hubbub.  But how to engage a quality analysis?  Michelle Nijhuis describes an iterative process of fact-checking in journalism, wherein she continually seeks out new sources to comment on and counterbalance other sources, until ideally, after an infinite(!) number of steps, truth is reached asymptotically.  But she admits this approach is time-intensive and unwieldy.  Furthermore, this approach can lead to the problem of "false objectivity":  journalists actively obscure truth when they try to objectively treat controversial issues by giving crackpots equal weight with experts and scientists. 

Instead,  I use a balance of evidence approach to truth-finding in science debates: I read widely and then select trustworthy sources.  If a person or organization publishes unsupported or erroneous information, I tend not to give them a second chance.  Also, sources that don't include open debates are outed as substandard information sources and discarded from the analysis.  A process of winnowing results, and after several months (years) of research, only the best sources are left standing.

Typically, the best sources are: experts who publish in-depth analyses of primary sources (e.g. journal articles).  They are open to quality comments from a range of voices, and their work is therefore continually self-correcting. 

Interestingly, I often prefer blogs over than traditional journals in my research.  Bloggers can attain higher standards of truth than the peer-review system.  Journals are slow to correct mistakes and often don't include enough discussion to reveal divergent viewpoints.  

Levels of Abstraction

Hong Lei's talk on 1/24/07 brought up the old issue of whether we're smart enough to understand our own minds, the self-servingly appellated "most complex object in the universe". All of his statistics are motivated to try to capture the a priori differences observed in the spike waveforms. But just because we can see a difference doesn't mean our mathematics can [accurately] capture it. For example, his algorithm for labelling bursts uses an arbitrary cutoff and binning, so that some information is lost going from the original analog to the digital output. Again, when he measures constancy of spike number in bursts, there is an arbitrary distinction between "same number" and "different number" [with no regard for spacing of spikes]. But information is lost in each analog-to-digital conversion: information is lost at each analytical step. Today in Genetics class Dr. Pierson cautioned that we can make theories about biological mechanisms, but inevitably the biochemistry is more complex and nuanced. Eventually you are just taking averages of averages of averages. This is the danger in statistics of, say, finding the standard deviation of standard deviations.