Weapons of Math Destruction

I do a fair amount of modeling at work, nothing as consequential as a credit score or a sentencing algorithm, more like predicting failure rates on parts, but the underlying instinct is the same one O’Neil is going after here, which is that a model is only as honest as the assumptions baked into it, and most people never see the assumptions.

Her core move, calling out the opacity as the actual danger rather than the math itself, is the part that’s stuck with me since finishing it. A bad model that everyone can inspect gets fixed. A bad model nobody can see into just keeps running, and the people it’s grinding through have no path to even understand why they got the outcome they got, let alone appeal it. The teacher-evaluation chapter got under my skin the worst, a system so noisy that a good teacher gets fired based on what’s essentially statistical noise dressed up as objectivity.

It’s not a technical book, if you wanted equations you won’t find many, and I know some readers wanted more rigor and found it a little magazine-styled in its prose. I didn’t mind that. The examples do the work, hiring algorithms, insurance pricing, predictive policing, and by the fourth or fifth one you start seeing the same shape everywhere: a proxy standing in for something that would be illegal to measure directly, doing the same damage with better PR.

The one thing I wanted more of was what to actually do about it, beyond “demand transparency,” which is true but a little thin as a call to action for anyone who isn’t a regulator. Still, this is the book I’ve handed to the most people this year, including two coworkers who build the kind of systems this book is warning about and didn’t love hearing it.

Five stars, and I mean it as a warning as much as a recommendation.