The Modular Future of Data Centers is Coming to a Parking Lot Near You

Agentic AI is driving the use of modular data centers, according to Tony Grayson, former President of Compass Datacenters, and now President and General Manager of Northstar Federal and Northstar Enterprise & Defense, where he has innovated modular data centers designed to bring flexibility and efficiency to a market in which technology evolves faster than traditional large-scale data centers can keep up with.

Grayson is a well-known leader within the data center world. In a lively conversation with his friends and industry sojourners Phillip Koblence and Nabeel Mahmood, Grayson talks about the frustrating history and bright future of modular data centers and the AI applications that are finally driving demand for smaller data centers that can fit in parking lots. 

As the former Commanding Officer of a US Navy submarine, Grayson shares keen insights into mission-critical energy, having been responsible for, essentially, a "submerged nuclear reactor" in which even small mistakes can lead to fatalities.

Among the key takeaways of Cool Vector's conversation with Tony Grayson:

• Modular data centers are set to dominate deployment timelines. Grayson argues that modular deployments are becoming essential as compute demand from AI workloads outpaces hyperscale construction timelines. Instead of multi-year builds, modular units can be deployed in “three to six months,” making them critical for real-time AI inference such as fraud detection and language translation. “What modular data centers give you is a very, very quick time in the market at a very good cost on something that’s easily upgradeable,” says Grayson.

• Distributed compute is replacing "bigger is better" economics. Scale for its own sake is no longer aligned with technology or economics. Customers want smaller, controllable AI environments—sometimes literally “in the parking lot”— rather than massive centralized builds. “It’s not a how big can you build? It’s how can you build a hundred thousand of these things just for one platform and place them all around the world?” says Grayson.

• Nuclear power may lose admirers after overpromises. While nuclear power is again being floated as a solution for energy-hungry data centers, Grayson—drawing on his background as a submarine commander—warned of overpromising. He noted that advanced reactor projects face long regulatory and technical hurdles, predicting a backlash if expectations are not met. “I am worried that nuclear is gonna get a bad name when all these Gen 4s who promised delivery in a couple years never happen,” he says.

• Decomposable infrastructure will displace GPUs. Grayson highighted the looming shift from GPU-dominated architectures to decomposable infrastructure and custom ASICs, which can outperform GPUs at lower cost. This evolution will fundamentally reshape facility design and economics over the next five years. “You are getting AI right now, which is agentic machine to machine stuff," says Grayson. "In a couple years, they’re gonna have decomposable infrastructure where basically you’re separating your CPU, your memory, your storage with optical, and then we’re gonna build a data center for that,” says Grayson

• Europe enforces sustainability as U.S. lags behind. Sustainability remains a patchy priority, with European regulators pushing strict standards while U.S. operators often give it “lip service.” Time-to-market and ROI remain dominant drivers, even if climate goals are compromised. “I think Europe is super sustainable. In the US, it’s more lip service than anything else right now,” says Grayson.

Follow Cool Vector on LinkedIn:  https://www.linkedin.com/company/cool-vector-media/