Rothman's brief reviews of a number of other books debating progress and optimism versus pessimism about the future make his article well worth reading. I'll pass on the end of the article (note especially the last paragraph), which points to a book by Hans Rosling:
In “Factfulness: Ten Reasons We’re Wrong About the World—and Why Things Are Better Than You Think,” the Swedish global-health statistician Hans Rosling, who wrote the book with his son and daughter-in-law, tries to find such a picture. Most depictions of the world, Rosling thinks, are either too optimistic or too pessimistic; if they don’t succumb to despair, they seem to look too quickly away from suffering. Rosling adopts a mantra—“Bad and better”—to avoid these extremes. “Think of the world as a premature baby in an incubator,” he suggests:
The baby’s health status is extremely bad, and her breathing, heart rate, and other important signs are tracked constantly so that changes for better or worse can quickly be seen. After a week, she is getting a lot better. On all the main measures, she is improving, but she still has to stay in the incubator because her health is still critical. Does it make sense to say that the infant’s situation is improving? Yes. Absolutely. Does it make sense to say it is bad? Yes, absolutely. Does saying “things are improving” imply that everything is fine, and we should all relax and not worry? No, not at all. Is it helpful to have to choose between bad and improving? Definitely not. It’s both. It’s both bad and better. Better, and bad, at the same time. . . . That is how we must think about the current state of the world.
Rosling’s image captures many of the perplexities of our collective situation. We desperately want the baby to survive. We also know that survival doesn’t guarantee happiness. The baby is struggling, and suffering, and will continue to do so; as a result, we’re more likely to be happy for her than she is to be happy for herself. (Pinker, similarly, is happier for us than we are.) It’s possible, moreover, that she’ll be saved only temporarily. No one is ever truly out of the woods.
In the meantime, the baby’s survival depends on the act of diagnosis. Until her ailments are identified, they can’t be cured. Problems and progress are inextricable, and the history of improvement is also the history of problem-discovery. Diagnosis, of course, is an art in itself; it’s possible to misunderstand problems, or to overstate them, and, in doing so, to make them worse. But a world in which no one complained—in which we only celebrated how good we have it—would be a world that never improved. The spirit of progress is also the spirit of discontent.