J.F. Martel on Machines vs. Fractals, Black Box Personhood, and Navigating The AI Fairy Tale

Halfway through one of my favorite sci-fi novels, Charles Stross’ Accelerando, we tune in to the members of an interstellar first contact mission as they pass the time debating whether the Technological Singularity has happened yet. Spoiler alert: all of them are uploaded minds appearing in a consensus VR environment as various post-human avatars, riding inside a computer the size of a grain of rice on a craft the size of a soda can. To readers it seems like a satire: what, if not this, would it take to convince you we’re over the rainbow? But good science fiction provokes us to question the present, and so we must ask: what are we waiting for? Are we still moderns? Is this still Western civilization? Should we be looking forward to the age of machine superintelligence, or has it already happened, like physicist Cosma Shalizi argues in his blog post “The Singularity in Our Past Light-Cone”?

Here’s a clip from that piece:

Exponential yet basically unpredictable growth of technology, rendering long-term extrapolation impossible (even when attempted by geniuses)? Check.Massive, profoundly dis-orienting transformation in the life of humanity, extending to our ecology, mentality and social organization? Check.Annihilation of the age-old constraints of space and time? Check.Embrace of the fusion of humanity and machines? Check.Creation of vast, inhuman distributed systems of information-processing, communication and control, "the coldest of all cold monsters"? Check; we call them "the self-regulating market system" and "modern bureaucracies”.

Maybe we ought to consider, like Bruno Latour, that We Have Never Been Modern. Or maybe, as Federico Campagna suggests in Prophetic Culture, each era’s inhabitants identify as “modern” and project the “likely story” produced by their process of “worlding” to imagine futures that recede like mirages or rainbows as we approach the horizon of our understanding? By the time we arrive, we have transformed and the mysteries of the ancient and future are conserved. Some Indigenous cultures believe that all animals identify as “people” — perhaps every world is mundane to its native observers, and yet all of them arise out of chaos and ineffability. Science can’t answer some questions because it depends on replicability and provisional consensus, and some questions ultimately force us out of attempts to get everything to make sense and into contemplative surrender to our own cognitive limits (no matter how much we augment ourselves).

Science will, of course, continue. As Ted Chiang wrote twenty five years ago in his short story “Catching Crumbs from The Table”, advancements in AI and biotechnology could foreseeably “[leave] journals to publish second-hand accounts translated into human language… Journals for human audiences were reduced to vehicles of popularization, and poor ones at that, as even the most brilliant humans found themselves puzzled by translations of the latest findings… Some left the field altogether, but those who stayed shifted their attentions away from original research and toward hermeneutics: interpreting the scientific work of metahumans.”

In 2025, living through the superexponential evolution of machine intelligence, this story hits close to home. What will we do when all breakthroughs are made by black box AI systems whose logic and insights evade us? We already have to take large language models on faith, doing our best to conserve a modest sliver of understanding as we resign ourselves to the practical benefits of successful but illegible prediction. But given that scientific progress has largely advanced through t