Distributed Computing - Sustainable Carbon-Aware and Water-Efficient LLM Scheduling in Geo-Distributed Cloud Datacenters

Hey everyone, Ernis here, ready to dive into some fascinating research that hits close to home – literally, close to our home, planet Earth! We're talking Large Language Models, or LLMs, like ChatGPT, CoPilot, and Gemini. You know, the AI helpers we're all getting used to?

Now, we've heard a lot about how much it costs to train these massive AI brains. It takes a ton of processing power, and that translates to a lot of energy. But what about after they're trained? What about every time we ask them a question?

Well, it turns out that the environmental cost of just using LLMs – what the researchers call the "inference phase" – is actually a much bigger deal than we thought. Get this: some studies estimate that the cost of running these models in terms of energy, water, and carbon emissions can be 25 times higher per year than the initial training costs!

Think of it like this: building a car takes a lot of resources, but driving it around for years and years burns a whole lot more fuel. LLMs are similar. All those back-and-forth questions we ask every day add up, and the impact on the environment is significant.

Specifically, the researchers point out that for every 20 to 50 questions we ask an LLM, about 500 milliliters of fresh water is used. That's like drinking a bottle of water! Multiplied by billions of queries, that's a serious drain. And it all contributes to a giant carbon footprint.

So, what can we do about it? That's where this research comes in. The team developed a new framework they call SLIT. What SLIT does is it tries to optimize four key things at the same time:

• Quality of service: How quickly you get your answer (time-to-first-token).
• Carbon emissions: The amount of greenhouse gasses released.
• Water usage: The amount of fresh water consumed.
• Energy costs: The financial cost of the electricity used.

The heart of SLIT is a clever algorithm that's like a super-smart scheduler. It uses machine learning to figure out the best way to run LLMs across different data centers around the world, taking into account things like local energy prices, carbon intensity of the electricity grid, and even the availability of water.

It's like finding the optimal route for a delivery truck, but instead of packages, we're talking about AI queries, and instead of gas mileage, we're talking about sustainability.

This is really important because, as LLMs become more and more integrated into our lives, we need to think about their environmental impact. This isn't just a problem for tech companies; it's a problem for all of us.

Here’s a key quote from the paper:

Such a framework will become increasingly vital as LLMs proliferate.

This research offers a glimmer of hope. By optimizing the way we run LLMs, we can reduce their carbon footprint and water usage without sacrificing performance.

So, here are a couple of things I'm pondering after digging into this research:

• If frameworks like SLIT become widespread, how much of a difference could they really make in reducing the overall environmental impact of AI? Is it enough, or do we need even more radical solutions?
• As users, do we have a responsibility to be more mindful of how often and for what purposes we use LLMs? Could we, say, group our queries to reduce the overall load?

Let me know what you think! I'm really curious to hear your thoughts on this. Until next time, keep learning, and keep questioning!

Credit to Paper authors: Hayden Moore, Sirui Qi, Ninad Hogade, Dejan Milojicic, Cullen Bash, Sudeep Pasricha