Potential flaws in unconscious bias tests

An interesting article by John Tierney notes controversy over the way researchers have been using split-second reactions on a computer test (the Implicit Association Test, or I.A.T.) to diagnose an epidemic of racial bias. Critic Hart Blanton notes:

One can decrease racial bias scores on the I.A.T. by simply exposing people to pictures of African-Americans enjoying a picnic...Yet respondents who take this test on the Web are given feedback suggesting that some enduring quality is being assessed...People receiving feedback about their ‘strong’ racial biases are encouraged in sensitivity workshops to confront these tendencies as some ugly reality that has meaning in their daily lives. But unbeknownst to respondents who take this test, the labels given to them were chosen by a small group of people who simply looked at a distribution of test scores and decided what terms seemed about right. This is not how science is done.

However,

In a new a meta-analysis of more than 100 studies, Dr. Greenwald, Dr. Banaji (two of the leading I.A.T. researchers) and fellow psychologists conclude that scores on I.A.T. reliably predict people’s behavior and attitudes, and that the test is a better predictor of interracial behavior than self-description. Their critics reach a different conclusion after reanalyzing the data in some of those studies, which they say are inconsistent and sometimes demonstrate the reverse of what has been reported. They have suggested addressing the scientific dispute over bias — and the researchers’ arguments about the legal implications for affirmative-action policies — by having the two sides join in an “adversarial collaboration.”

One critic, Philip Tetlock, a psychologist at the University of California, Berkeley, said he had found prominent research groups and scholars willing to mediate joint experiments. But so far nothing has happened — and each side blames the other. Dr. Greenwald says he tried proposing a joint experiment to Dr. Tetlock only to have it rejected. Dr. Tetlock says that he tried a counterproposal and offered to work out a compromise, but that the I.A.T. researchers had refused two invitations to sit down with independent mediators.

After all the mutual invective in the I.A.T. debate, maybe it’s unrealistic to expect the two sides to collaborate. But these social scientists are supposed to be experts in overcoming bias and promoting social harmony. If they can’t figure out how to get along with their own colleagues, how seriously should we take their advice for everyone else?