October 16, 2022 • 49 mins
Summary
The generation, distribution, and consumption of energy is one of the most critical pieces of infrastructure for the modern world. With the rise of renewable energy there is an accompanying need for systems that can respond in real-time to the availability and demand for electricity. FlexMeasures is an open source energy management system that is designed to integrate a variety of inputs intelligently allocate energy resources to reduce waste in your home or grid. In this episode Nicolas Höning explains how the project is implemented, how it is being used in his startup Seita, and how you can try it out for your own energy needs.
Announcements
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great!
- When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Your host as usual is Tobias Macey and today I’m interviewing Nicolas Höning about FlexMeasures, an open source project designed to manage energy resources dynamically to improve efficiency
Interview
- Introductions
- How did you get introduced to Python?
- Can you describe what FlexMeasures is and the story behind it?
- What are the primary goals/objectives of the project?
- The energy sector is huge. Where can FlexMeasures be used?
- Energy systems are typically governed by a marketplace system. What are the benefits that FlexMeasures can provide for each side of that market?
- How do renewable sources of energy confuse/complicate the role that the different stakeholders represent?
- What are the different points of interaction that producers/consumers might have with the FlexMeasures platform?
- What are some examples of the types of decisions/recommendations that FlexMeasures might generate and how to they manifest in the energy systems?
- What are the types of information that FlexMeasures relies on for driving those decisions?
- Can you describe how FlexMeasures is implemented?
- How have the design and goals of the system changed/evolved since you started working on it?
- What are the interfaces that you provide for integrating with and extending the functionality of a FlexMeasures installation?
- What are the operating scales that FlexMeasures is designed for?
- What are the most interesting, innovative, or unexpected ways that you have seen FlexMeasures used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on FlexMeasures?
- When is FlexMeasures the wrong choice?
- What do you have planned for the future of FlexMeasures?
Keep In Touch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Unknown (00:13):
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode.
With their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform,
including simple pricing, node balancers, 40 gigabit networking, and dedicated CPU and GPU instances.
(00:39):
And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover.
Go to python podcast.com/linode
today to get a $100 credit to try out their new database service, and don't forget to thank them for their continued support of this show.
The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. SelectStar's
(01:04):
data discovery platform solves that out of the box with a fully automated catalog that platform solves that out of the box with a fully automated catalog that includes lineage from where the data originated all the way to which dashboards rely on it and who is viewing them every day.
Just connect it to your DBT,
Snowflake, Tableau, Looker, or whatever you're using, and select star will set everything up in just a few hours.
Go to python podcast.com/selectstar
(01:26):
today to double the length of your free trial and get a swag package when you convert to a paid plan.
Your host as usual is Tobias Macy. And today, I'm interviewing Nicholas Hoehning about Flex Measures, an open source project designed to manage energy resources dynamically
Nicholas, can you start by introducing yourself?
Yes. Hi, Tobias. I'm Nicholas. I'm German
(01:48):
and living in the Netherlands
since 15 years. I've been a programmer for a bit more than 20 years
using Python for a long time. I've been working on energy since 12 years, I think,
from that vintage point.
And do you remember how you first got introduced to Python?
1st, I was I had been using
(02:09):
very little c and Visual Basic. I've been working on
access
databases
and and word
macros.
At at some point, I did then, go to university.
During my bachelor's, I got to work on a start up. It was a neuroscience
startup
that is still around. It's called Iquant.
(02:30):
The idea was to
transfer neuroscience
models of visual attention
into a web application so that you can, you know, basically upload your
ad and find out
where people might look first.
And we do we use a lot of Python there, and I had to you know, I was sitting at lunch, reading the standard Python tutorial so I could get to work on that startup, still remember.
(02:55):
In terms of the Flex Measures project that you are working with, can you describe a bit about what it is and some of the story behind how it came to be and why you decided that this dynamic
energy demand response system is something that you wanted to spend your time and energy on?
Yeah. Sure. And I'd love to yeah. It has become my life topic.
I started a PhD
(03:16):
around this topic in 2009
when
it wasn't absolutely clear that we will get the kind of energy transition that we are basically in right now,
driven by, you know, lots of renewables
being the major source at some point.
Flex measures, basically, an energy management system.
(03:37):
It helps you manage your energy. In the last couple of years, energy management has meant that you manage
the data around your energy, metering data.
You put it on NICE dashboards and you do billing around it. You know, that's also significant work to be done. Flex Measures focuses more on
the energy flexibility
(03:57):
and optimizing
when energy is consumed during the day
because that's an a very important piece in the energy transition.
As any any expert will tell you, we'll need to get to this kind of optimization,
at some point.
Otherwise, we will not get the fit between
available renewable energy and the actual consumption to have a stable energy system. So flex measures helps you find the best times
(04:25):
for your flexible energy assets. Let's say, a battery. That's the best example of a flexible energy asset.
When should that battery be charging or discharging?
The other examples for flexible assets are heating. You can heat before
somewhat sometime before you're really home, for instance.
Heat pumps are a big topic.
(04:46):
The industry, you have a lot of potential
to run energy intensive processes at a slightly different time. And Flex Measures really helps you
model all your energy data in real time and make that decision.
You mentioned that a number of the other energy management systems are focused more on a kind of 1 way path of information where you're just retrieving data from the system and then using that to build some analysis, but that doesn't get acted upon in any sort of automated fashion. I'm wondering if you can just give a bit more of an overview of what the space looks like for these energy management systems and if there are any kind of alternatives to flex measures or if it's something that is relatively unique in the space.
(05:27):
There are some projects
that have on the road map to also do this kind of real time optimization,
but you got a lot of inertia in these projects as well in companies. So I see a lot of companies who
had to start basically going to customers who have not even any metering
installed whatsoever
other than having a main meter. And the main meter's data
(05:50):
is not real time as well.
So these companies started with bringing in their Raspberry Pis or whatever it is,
becoming good at that, starting to measure, becoming good at visualizing information, becoming good at doing them some some aspects of billing maybe.
And to do this step into data science optimization,
I see
(06:10):
I see the difficulties in some organizations to to that extra step. So
these people are happy
to get some other project and get some other help with that. I don't know all the projects out there, of course. So Flex Measures is open source, and another open source project that I know about is just called OpenEMS.
That's a Java project in,
originated in Germany
(06:32):
that's, built a quite interesting foundation around it. But also there, there's a big focus in inertia on getting the metering
and visualization
correctly done.
And there's not so much support yet to
optimize the actual
what actually happens and and influence that, and we have decided to only focus on that.
(06:54):
As far as the
primary focus and goals of Flex Measures, I'm wondering if you can just talk to some of
the kind of outcomes that you're aiming for and
some of the,
I guess, target audience for the project.
As a big goal, the big hairy goal is that Plaxminder should really help to
(07:15):
speed up the energy transition and, you know, become 1 of the standard pieces of climate tech,
infrastructure tech that you would also always consider when you start a new
modern energy project where
your real time optimization
is a part of the solution.
The main way I think Flex Measures can help is really being a developer friendly tool,
(07:38):
solve a lot of these
things that take you weeks of work, get your data model right, get your data integrations right,
get some monitoring
right,
decide how to implement where your algorithms should be run,
have a queue for that, you know, processing queues.
There's all these things that developers know
will keep you
(07:59):
busy before you get to the real
use case, before you really help your customer, or you are implementing something very
shortly, very quick hack.
See that happening a lot. Other people asking their cousin's neighbor to have a simple algorithm put somewhere, but this is about, you know,
real time data pipelines who have to be kept alive and be checked on health. Yeah. I think we succeed if developers around the world who I expect in the next 10 years to
(08:27):
be looking around how they should approach the similar similar challenges to always consider flex measures.
And that can be in a start up somewhere.
That can be in
a larger energy company
where they have already their ICT team, but, haven't solved or started to do this yet.
And and I think
(08:48):
this is a problem that happens in all the the the parts of the energy system. For me,
there's 2 big parts. That's industry and,
what's usually
called smart city at the moment. So anything in the the built environment, those are 2 huge sectors of the energy sector, which in itself is 1 of the biggest
sectors in the world that needs to undergo the most rapid transition you could imagine. The other plans are 15 year plans basically to basically change
(09:16):
how everything works.
So we really hope flex measures can help get that super big vision done in time.
Do you also see flex measures as something that an individual homeowner might use as part of managing their energy systems, particularly as more people invest in solar and small scale wind?
Yes. I want to talk about a small project we've been doing. So
(09:41):
besides working with customers or with companies that can scale up innovation.
We've seen that this is still slow. Right? So these companies all have to think differently before they haven't been involved with energy so far. It's a long process. So we have, on the sides, created a living lab with just 1 household
with someone who's also very enthusiastic
(10:02):
about, specifically,
vehicle to grid charging.
They bought the exact car and the exact charger that can do that already, the 1 that is allowed to do that in the Netherlands.
And we've really implemented
the automated optimization
of the charging and discharging in in his house. And for that, we needed to actually do something in the house.
Flex measures is basically supposed to run-in the cloud.
(10:25):
It's dockerized. It can also run locally.
But it basically makes decisions. It doesn't really show you something.
So I want to pitch a very cool project that's called Home Assistant.
It's also open source. And it's also originated in the Netherlands,
but they have a
company in the US also now. Very interesting
group of people. Basically, in the home automation
(10:47):
sector with their software.
We use Home Assistant together with Flex Measures to optimize the charging, discharging, and and taking into account the the solar panel production to really create a cool project. We also open sourced our Home Assistant plug in.
So that's out there. We were on Dutch TV and a couple of people wanted to use it as well. And now there's this week, the first 2 others who've implemented it in their house. So now we have 3.
(11:15):
But for us, as a company, that's not the focus, but it's where you can really do something right now and really show something. And there's always these, enthusiasts
that just want to build something in their house like that.
As far as the overall question
of energy systems, I mean, that can be
it's a very broad term with a lot of different,
(11:35):
I guess, energy sources and use cases
kind of embedded in it, where 1 element of it is the electrical grid. Another is the usage of fossil fuels for transportation and shipping.
There's home heating. There's heating at, you know, industrial scale for large factories.
And the majority of those,
(11:56):
I guess, ecosystems of energy usage are governed largely by different market forces where there is the, you know, energy producer, which is becoming a bit of a fuzzy line now with more homes and residential and industrial installations having their own energy generation through things like solar and wind. And then there's the energy consumer and the distribution system.
(12:18):
And the pricing of those different resources are governed by, you know, some level of dynamic usage where, you know, there are peak loads, and so that can increase energy prices for certain periods.
And I'm wondering
what you see as some of the benefits that Flex Measures can provide to
this marketplace ecosystem, particularly given the fact that it's not as clear cut now as it once was?
(12:41):
No. You're absolutely right. It's the vast,
sometimes confusing
system. I like the analogy that because somehow it's all connected and it has to happen, in real time and the frequency of the grid,
as far as electricity is concerned is held in some
synchronization.
We've created the largest man made machines with the grids. So you got the US grid, the European grid, and the Asian grid, basically, as the largest 3 machines you could even imagine where your house is a part of it, or the electric assets,
(13:13):
and it happens in real time. Different markets
buy and sell and allocate energy for different horizons.
And FlexMaster is certainly
not very useful in the range of, let's say, you're buying your energy a couple months ahead and buying these big budgets.
Also, I wouldn't say it's usable in the second range. There's some balancing markets to get this frequency challenge on the grid, right, where you have to react within a second.
(13:41):
But everything in between,
where you have to react basically during the day
or you think in the evening about your plan the next day,
that's where Flex Measures,
is really useful.
And as I said, it's really on the electricity
range
where we want to use it.
So if you're a consumer,
(14:02):
you can use flex measures to use greener electricity, but it will help the grid stability. But I think mostly what people care about, of course, is to to have their energy bill in check. The spread on market prices has gone up.
Energy bills
are rising a lot. So there's a lot of urgency, I think. Now there's
clarity that this is very important for them to achieve.
(14:24):
And
from a grid perspective,
if you're a network operator,
it also is great if a lot of
decentralized decisions are happening
that help the grid stability.
We we have to follow what renewables are doing to a big extent more and more.
We cannot influence when we get that solar power or wind power, but we can influence
(14:47):
when we use power. And if a lot of decentralized
decisions are being made to shift their consumptions
towards when renewable energy is really there. We get a more stable grid.
And
the only other solution for network operators would be to, you know, double or triple every cable in size.
So
this is a big societal
cost saving we can achieve with installing
(15:09):
better decision making in in the nodes, which turns how the energy system works on its head. Right? So still now in the last decades, we have optimized a few power plants per country
and just forecasted what consumers will be doing.
And now that partly turns on its head. It's very interesting.
Digging a bit more into that question of renewables
(15:30):
and their broader distribution, particularly with residential installations as part of that mix, I'm curious
what you see as some of the
broader impact on the way that the kind of marketplace economy around electrical grids in particular
is likely to shift as the power generation and power distribution
(15:51):
becomes
bidirectional
on a more routine basis.
1 thing we see in the Netherlands, and I think that the Netherlands are a bit ahead of the curve there, is
more dynamic prices.
I think you can choose between
5 suppliers now that will buy
power on the day ahead market, which has 24
(16:12):
different
slots. But then they will just
give you those
24 prices.
So the the the afternoon
before, you will know what your 24 prices will be the next day. And if you have,
you know, some optimized
optimized system like what we did, for instance, in the vehicle to grid project,
that's going to turn out quite well for you then. You will not consume energy when those prices are high. Of course,
(16:39):
what's interesting is that this might create more of a divide between those people that can afford that
that has this cool
infrastructure plus software, and they get their average energy bill.
Well, they keep it low, let's say.
And who's going to pay a lot? That's probably going to if everything goes like this, it's going to be the poor people.
(17:00):
So this is outside of the technical
realm, also very interesting
political discussion where we still have some time. And I'm also involved in some of these discussions to discuss how we can
achieve
a more just way to do that.
What we really need is not for individual homeowners to
invest in their own house and make it smarter.
(17:22):
I like it. I'm enthusiastic about it when it happens, but what we really need is this scale.
We really need to do these things for everybody on a big scale.
So for instance, big housing
corporations
already know, given their ESG
strategy,
they have to renovate
at least 4% of the whole portfolio every year. And every year that they don't do that, it's just getting to it, getting to be more pressing, like,
(17:50):
5%, 6%.
And then, yeah, you need to
make a whole neighborhood or a whole big flat
react to flexibility
demands from the grid.
And that can be very exciting that you then, for instance, have 1 giant heat pump for a whole flat. And especially if you get to bigger numbers,
there will be
(18:11):
a better way to react.
For instance, it's almost impossible
to forecast for a single house when you will be home, when will you cook.
You can have a little bit of a idea. But if you forecast a 100 consumers,
you get a much better data
forecast, which you can really work with. So I think
(18:32):
that's really what I need to see, these these bigger scaled approaches.
And, yeah, I hope Flex Measures can be can be helpful with that.
Digging into some of the way that Flex Measures
operates, I'm wondering if you can talk to the sources of data that you're looking at, some of the ways that you manage that integration
and processing,
(18:53):
and the types of interaction that different kind of users or operators might have with Flex Measures and how much of it is just you kind of set it and forget it? Yes. Sure.
So my simple picture of flex measures is data in, scheduling out.
Data in means certainly metering data.
You want to know what all your assets
(19:14):
did in the past. It can be very useful to, for instance, for a battery, know the possible state of charge it has now.
It can help to forecast what will happen in the near future.
We also read in prices,
market prices, for instance, the next day,
weather forecasts.
So you can then forecast your solar generation if that's on your roof.
(19:38):
Sometimes the occupants'
plans are also data that you need to take in. So in the vehicle to grid, living lab in this application,
The occupants' data might be that they say they will need the car for a longer trip
tonight at 6 PM. That's important data that influences
the scheduling because the car should be a 100% full by then.
(20:01):
Or in a factory,
they also have new constraints every day, how much
processing work they need to be doing.
That's,
about the data. And you said earlier that set and forget might be
an interesting concept. I really agree.
For,
most people
who will live in modern energy environments
(20:24):
with with some optimization
happening,
they should not even
care about that during a normal day.
So in the vehicle to grid project that we've been doing, that really was also design goal.
It worked.
And from then on, it just quietly
optimized every day what the battery was doing.
We did, however, in that project, think you might want an override button. You might not want to
(20:49):
meddle with what exactly the optimization is doing, but
it's nice, of course, in an optimized system to have the ability to say, just stop what you're doing or
the other button says,
charge the car now.
I don't care about the price or whatever.
The idea is that these buttons are almost never used, but they should be there. So I think that end users who live in these
(21:13):
built environments should really not be involved that much,
and it's more the
servicing
around that. So Flex Measures
needs to be hosted, and we need to
check that that data is always flowing
and check with every error that might
come up in in the scheduling.
(21:35):
And you have some companies that you call energy service companies.
So a lot of them, for instance, have these more traditional
energy management systems I talked about earlier.
And
I think that they would be users of Flex Measures.
If we don't do it for them, they would really
build it into their monitoring pipeline,
have a KPI and reporting pipeline to regularly check if that's happening correctly.
(22:00):
1 thing that's nice to check is, the universal
smart energy framework.
It's called Yusef.
My cofounder was on the design committee, and they talked about the different roles in modern smart energy system.
You mentioned the prosumer already.
I mentioned the energy service companies, and you also have companies they're called aggregators.
(22:22):
They
aggregate the energy flexibility
from
a lot of customers
into their virtual power plant and and just trade that flexibility
on markets.
I think you could build energy aggregator using FlexNetters,
but that's not what we do. We use it for
optimization behind your meter.
(22:43):
As far as the actual
implementation
of Flex Measures, I'm wondering if you can talk to some of the
design considerations that went into it, some of the challenges of being able to
flexibly manage scale of inbound data and the types of requests that you're getting.
And also, I'm interested in maybe some of the capabilities of being able to federate across different installations of flex measures and, you know, route different decision requirements or different data points based on different kind of inbound information?
(23:16):
So from the beginning,
a few years ago,
I think, I didn't mention that before. But the first
project where Flex Measures really got its first development
time, that was with a South Korean customer called A1 Engineering.
They wanted to do simulation
first of a of a system with a lot of smart charging,
(23:37):
but then also move to
real time operation in a later phase. So this is
the main design idea
that has been there from the start that you can use
something
in reality in real operation,
but it's also useful
initially in a phase where you are simulating what might actually happen.
And that's still very relevant.
(23:59):
So we're doing a simulation with a bigger real estate company right now. That's their first step. I think that that main design goal still stands.
Now I haven't used multiple Flex Measures nodes and then federated the decisions yet. So I think
that could be built, like, a lower level of Flex Measures instances, basically, governing,
(24:20):
smaller installations.
And another 1,
maybe trying to aggregate. As I mentioned earlier, the it's a concept of the flexibility aggregator
and aggregate across
those behind the meter installations that could be a higher level of extensions
instance. Yeah. Now we haven't done that yet.
We consciously
(24:40):
can decide not to, you know, let the volume of data grow too large.
So for instance,
you might run into projects where they have 10 second
interval readings.
But to make the kind of decisions that Flex Measures needs to make, let's say, on an hourly basis or 15 minute basis,
you don't need to read in the finely
(25:03):
structured data. It would be best if that other system gives us already aggregated
data. For instance, in in 5 minute steps, that'll be fine. And then the amount of data isn't growing that much,
but we're already looking into that. So we
our data is stored in, Postgres.
Of course, that we also made that choice because you have so much support for
(25:27):
growing bigger with your databases and have all these clusters, clustering strategies, and sharding.
And, also, of course, there's interesting
technology built upon Postgres. I think 1 of called Timescale DB that really focuses only on the time series
aspect.
And 1 thing that we considerably
put some effort in is to model
(25:50):
this data because it comes, as you said, from different sources.
There might be a company that
installed the sensors
at the factory or is responsible
at the houses to measure what's happening.
So weather forecasting
service. So we use, open weather map.
Flex measures can read data from there.
(26:10):
The European transmission system operators have a service
in open transparent API as well. So you got multiple sources.
They all send you data that might be
about different things.
It has
different time zones.
It's crucial to maybe know when they know about that.
So not only the time that they're talking about,
(26:32):
1 point in time with 1 measurement,
has 1
date time, of course, about the time that you're talking about. But we're also interested in the daytime when they said this would happen.
If it's before the time, this is a forecast. If it's after the time, it's a measurement
that can be quite useful when you write optimization algorithms.
(26:55):
All this led us to write our own,
basically,
time series data model. We called it Timely Beliefs.
It's actually an extension
on pandas. So what it makes possible is that you have a pandas data frame
or basically a class that extends from Pandas data frame, and we call it the beliefs data frame.
(27:16):
We wrote a lot of functionality around that that you can, you know, merge these data frames or change the resolution or whatever. It's a very interesting project.
They took a lot of our time, and now we're profiting from that within Flex Measures to really understand all the time
what that data really means.
As far as the kind of architecture of the project, digging a bit more into that, I'm also interested in understanding the
(27:39):
kind of extension points that you've built to be able to allow
different users of Flex Measures to adapt to different data sources or
being able to extend the types of decisions and machine learning capabilities that they want to use or the types of predictions that they want to generate
and maybe a bit more of the kind of data integration and kind of predictive capabilities that you've built into it? Flex Measures is extensible
(28:05):
with plugins
done last year, and we are using that ourselves
because often
every project has these differences,
different data sources, or or needs.
Often, it's also the visualization
that you want to adapt to a customer.
That's very important
to the whole architecture that you can add your own
(28:26):
API endpoints even,
your own visualization, and and we have a command line interface.
You can even add your own
command
line interface commands.
For instance, 1 that regularly
gets data from some other open API,
and you want to run that hourly in a Chrome job as a Flex Measures command.
(28:47):
Flex Measures actually is built on Flask,
and Flask has some nice capabilities for that.
Of course, for APIs, but also the it integrates with click. So command line interfaces are, nice to write.
And with the Flask blueprint
technology, it was
quite a joy to write our 1st plug in
(29:08):
architecture. So what you do in the plug in is mostly
add a blueprint
to flex measures.
Flask has a CLI
command, and we renamed that to flex measures.
So,
yeah, it all ties back into what I said earlier. We want flex measures to be usable by software developers. And then,
of course, APIs are super important, but also the command line interface, I think, is is super important. We wrote a
(29:36):
tutorial how you would use Flex Measures within a few minutes, you know, just to show you
within a few minutes what you might be doing,
which is based on Docker. So you don't need to install all these things. Just pull the Docker image. And then that tutorial is
a few
calls to the command line interface
about modeling that couple assets that you have a battery, for instance, loading in some data
(30:00):
from a CSV file and asking Flex Measures to compute a great
optimized schedule for the next day. That's all command line interface command. That's what I really like.
Optimization
in the background
happens
I think we use pyomo.
It's also a very powerful
library for for these kind of things.
(30:21):
And we used there was some sklearn
that we used. But recently,
we've picked up sktime,
which is really
a package focusing on time series
forecasting, doing smart things around that, and it's very nicely done, it seems.
As you have been
building Flex Measures and
(30:41):
iterating on the problem space and also with the
donation of it to the LF Energy
Foundation.
I'm wondering what are some of the ways that the design and goals and scope of the project have changed or evolved since you first started working on it? I said earlier that,
actually,
the overall design goal to make both simulation and optimized real time operation possible from the same code base is still intact.
(31:08):
I liked open source in Flex Measures. It it happened roughly,
well, less than 2 years ago. I we hit that button in GitHub
in a couple months later, donated to the Linux Energy Foundation.
I liked that that pushed us to even more transparency
and even more having a focus on code quality
to make sure that
(31:29):
Blake 8,
MyPy,
and Black always run. It's always looking good.
And documentation
got a lot of boost.
And, also,
in the meantime, we did once make big changes to the data model. When we worked for industry customers,
we noticed that our initial data model was a bit too simplistic.
(31:51):
I think we modeled
power,
weather data, and prices.
And if you go to industry, you quickly realize they want to measure a lot of other things that all have
some influence on your
outcomes.
For instance, you might be measuring steam flow,
and
that wasn't possible. So we took a lot of time to make our data model more generic. Now we can model anything. Another interesting aspect of the kind of open sourcing question, and it's addressed a little bit by your donation of the project to the LF Energy Foundation. But I also know that you are running a business where Flex Measures serves as a kind of central portion of what you're providing
(32:32):
at CITA. And I'm curious if you can talk to some of the ways that you think about
which pieces of your work you want to put into the open source Flex Measures project and what are some of
the other kind of capabilities or layers that you're building on top of that that are part of the kind of commercial and business focused offering?
It's a very interesting time to have a commercial open source
(32:54):
start up.
I was recently involved in an accelerator,
on the US side.
I had to attend at night, always at 11 PM.
But
there's really some investors in the US
that are looking for open source companies to invest in
because you got, in the last couple of years, some really interesting examples,
(33:15):
mostly on the database
developer tooling side.
So, obviously, in the energy sector, this is quite new. But it's clear that you can
run a business even if you're
very open.
LIMX Energy Foundation advised to use permissive licenses. So in this case, we use Apache 2.
You could also use MIT, for instance,
(33:38):
over a Copyleft licenses because the idea is to just, you know, open the doors wide to
growing the community,
letting other people use the tooling you've you write. Even if they don't re even tell you about it, they should have no doubts that they can use it freely
because you want to just grow, grow, grow, grow. That's after the Linx Energy Foundation told us that they use those licenses.
(34:01):
I also heard it from this venture capitalist accelerator program. It's a very interesting
vision on innovation, on open innovation that this might be 1 of the future scenarios that comes true. Right? That the pool of open source
innovation becomes
as big in other sectors has become in databases
and developer
(34:22):
tooling. Nonetheless, well, we have made Flex Measures possible to create plug ins.
And
oftentimes, the customer says, well, if you're developing something here about our processes, I don't want that to be public. Or, if I pay you to do something, I don't want it to be public. And with the plug ins, that's very possible, of course. Then you have
Flex Measures as an open platform.
(34:43):
And, also, Timely Believes is open,
and we occasionally
open up other projects that we work on. For instance, I mentioned that, Home Assistant plug in in the in the vehicles to grid
project that's open.
I think we have a couple
closed repositories
lying around right now with work for customers.
It's something that's maybe not ready for the world.
It will always happen.
(35:05):
I'm also interested in understanding a bit about given the kind of predictive and machine learning focused capabilities
that are part of the Flex Measures platform
and the fact that you are
building
predictions about and
energy capacity,
(35:25):
what are some of the
challenges that you face in terms of accounting for whether to bias towards false positives or false negatives, and what levels of uncertainty you're willing to accept in some of those predictions and decisions that you
make? Yeah. That is a very interesting topic to go into.
It's difficult to answer
(35:45):
without having even more experience. We had
a project trying to find out if
forecasting should be done with something very sophisticated. You know? You can use linear regression to forecast, let's say, the solar production,
RIMO models,
but But you can also use a neural network.
We found that the the neural network isn't
(36:07):
really better
that much better, and you don't know what's going on. And just recently,
I saw a paper come by
that some researchers even confirmed that. So for time series forecasting,
currently, there doesn't seem to be a reason to become very sophisticated.
What we did do was
we saw that
(36:28):
there is algorithms
which tell you about the uncertainty
that lies in the prediction. So it's mostly,
ensemble
algorithms like random forest.
They have, you know, a lot of forecast and then make a a mean forecast, but they tell you how far all of the others
were distributed.
(36:49):
And
that information can potentially, of course, be useful for exactly your question that you know that some forecasts are
shaky,
and you might decide
for a scheduling algorithm that there's a policy of not acting in those situations.
And that's something I didn't mention about the timely beliefs data model. 1 column in the belief data frame is the uncertainty. So how certain
(37:15):
forecast, for instance, would be. And it reminds me of a setting we have in a scheduling algorithm right now.
For the battery charging,
1 property you need then is, round trip efficiency.
So
you can discharge
a battery.
Let's say, you discharge 1 kilowatt hour
and then you charge that back.
(37:36):
Let's say the other 1 around. You charge it
first and then you discharge it,
and you've made some arbitrage because your 24 hourly prices had this nice difference
or that some difference. But the battery is not 100% efficient,
so you end up maybe that it's 80% efficient,
could become 90% in some batteries.
So you're losing some of that power in that process.
(37:58):
And
it's very important to model that so that the scheduling algorithm then sometimes does nothing.
Price difference difference is too low,
then it doesn't make sense to do all that. So sometimes the schedule is to do nothing even if there are some price differences. And I see the
policy of doing nothing if the uncertainty is too high, and that's a similar thing.
(38:20):
In your work of building flex measures
and working with different companies to help implement it and
providing it to the community and working with people who are using it for their own purposes. What are some of the most interesting or innovative or unexpected ways that you've seen flex measures applied?
There's not that much,
but there was a project where flex measures wasn't even being used for any scheduling,
(38:43):
which is
basically the end goal of using Flex Measures is to have a schedule and
then change the times when you do use energy.
And in that case, it was a project where it was
just so useful to
use Flex Measures to just be a plain energy management system,
getting all this data together with data pipelining being done quite fast,
(39:07):
but then not doing forecasting but in doing something else. There was a different algorithm that was basically some advanced analytics.
And it didn't you have to use electricity. It was about gas consumption in in industry.
So that is possible, but that's not where I want to take it.
In your work of building the project and building a business around it and donating it to the community, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
(39:32):
Something that you have to learn on your own is how long it takes to integrate
data
that constantly flows and that has these
intricacies that I mentioned earlier about time series
that you could get slightly wrong. It could be
you get the time zone slightly wrong or
somebody
sends you data that isn't localized.
(39:55):
You might
be 1 hour off half of the year.
Your whole schedule will not be good.
They also might, send you data that gets the daylight saving time wrong, and then there might be 2 hours in the year where you fail when daylight saving time is switching back and forth.
And, yeah, the sheer amount
(40:16):
of
attention and work that gets into that
is is quite staggering. We had an industry client with a partner who does the censoring,
and they didn't have the API ready for us. So
it was
not what I had expected in 2021 from that kind of player. What we could get from them was emails with Excel spreadsheets,
(40:39):
And we decided that they send us 3 per day. So every 8 hours, we have we wrote a Flexner's plug in that has its own command line interface command to just handle that email
and get the data in.
Yeah. You can imagine that that prolongs that kind of pilot
considerably.
Yeah. It's definitely remarkable
(40:59):
how much
of modern
data engineering and data management still relies on CSVs and FTP.
Yes.
Absolutely. And
if you look at the energy sector as a whole that these kind of experiences,
1 dimension
towards the realization that there will not be 1 end to end solution that can be used across the whole country or across a continent that that will just take over. We have to
(41:27):
build a lot of
unique end solutions.
Let's call them apps
that work differently in different energy systems, countries,
cultural contexts,
sectors.
And I think
flex measures can be a basis for all of that, but you always need to
put something on top that really fits
(41:49):
to that situation.
For people who are interested in being able
to gain some more insight and potentially act on the different aspects of energy usage, whether it's at the residential or industrial or grid scale? What are the cases where Flex Measures is the wrong choice?
I've mentioned that earlier, but there are some people who have to act within seconds.
(42:10):
And that's usually if you're really taking part in this first
reserve mechanism
reacting to balancing problems,
frequency things.
And
we have put a lot of focus on flex measures to be developer friendly and correct about the data so that developers really understand what they're building.
(42:31):
And we haven't made it, you know, a Rust project
that lets you do all that within seconds.
As you continue to build and iterate on the project
and continue to iterate on the business that you're building on top of it, what are some of the things you have planned for the near to medium term or any particular problem areas that you're excited to dig into?
Yeah. So first, about the FlexVicious project,
(42:54):
I think
what we really could do with that project is to support
the 2 groups of users that I see
in what they are
doing as a as a first steps. So I see the first group as being the 90%
users
who have a use case that
really happens
(43:15):
a lot in the world, and Flex Measures should support out of the box
that the optimization
for that use case can be done within, you know, a few hours.
Data integration plus having that optimization work should be a few hours. So that that's a a little bit of a benchmark.
Let's say, as an example,
somebody has solar on the roof and there's somehow
(43:38):
a battery, a home battery, or an electric car,
and they have a dynamic tariff.
That might happen a lot in the next years across the world, something like that.
And that should just work. And,
basically, that does work now. What we have to do is documentation,
write an excellent tutorial
(43:58):
so that
everybody could do that.
And there's probably a list of 4 or 5 more of these
situations
that should just work.
And the other group of users are the power users
that really have a unique problem and may come with their unique
idea of solving it.
(44:19):
And then I'm talking mostly about the algorithms.
Right now, you can already
plug in your own scheduling algorithm.
Or to be honest, at the time of recording, that is a PR I'm
having reviewed.
But that's really important for some people. That's researchers,
certainly.
That's their whole line of work to come up with better algorithms.
(44:42):
Maybe find that neural network algorithm that really does make the difference.
But you see that also in some companies really have
certainly an industry that just have their unique challenge. And they should be using flex measures,
you know, as is and then just change that 1 thing that there's
a common interface
with how
(45:03):
the question to make a new schedule is described.
But they could, in the back, just switch out the algorithm that is used to make the schedule.
I think these 2 groups of users deserve
each of their own dedicated
tutorials by us to really make it clear, to make it a joy to do so much as our work. I mentioned 2 sectors that we're working on.
(45:24):
The industry
world is very interesting and challenging,
but it's also very diverse.
Right now, those environments,
what excites me the most, and I'm I'm talking to
companies there that really understood the scale
that I mentioned earlier, they have renovate or we modernized the whole portfolio
within a very short amount of years. So scale is their big topic.
(45:48):
And the third thing I didn't mention is microgrids.
So microgrid is your own little electricity grid
that is connected
maybe to the main grid
or not,
completely run on its own. And that's a situation
that we might see all across the world. So there might be
more
self serving neighborhoods
(46:11):
that are not completely
dependent on the grid in Western countries, but certainly in Africa, that's something that's just the 1 way to go for a lot of regions.
There are some interesting companies there that I hope to do more with in the near future because Flex Measures should be able to
basically
run a small village.
You might think there's too many assets, but what really matters is the flexible assets. So that's where
(46:37):
that defines how much flex flex measures has to do, how many flexible things are there.
Emobility
is making huge steps in Africa, certainly.
We're
hoping to be involved in a project in Victoria Lake where all the fishers get upgraded to electric boats, for instance,
And they use energy for productive uses,
(46:59):
like cooling the fish they caught,
water purification.
That's all flex it's all something you can shift around in the day. And then flexiators can really help, for instance, in the design phase already to not buy oversized
equipment.
I don't think you will become rich in Africa, but that's something I'd really like to be involved in. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks.
(47:28):
This week, I'm going to choose the show She Hulk from the Disney plus. Just a fun and goofy show. Some interesting storyline there, so I've been enjoying watching that. So I recommend that for people who are looking for something to do in the evening. And with that, I'll pass it you, Nicholas. Do you have any pics this week?
Yeah. I'm watching a very nice German series on Netflix,
(47:49):
and it's called KLEO,
k l e o.
It has a bit of kill bill
visuals because it's an ex
agent from the East German
Ministry of Defense who got thrown into jail. And she
comes out of jail right when the wall has been torn down and tries to find out what happened, who did it.
(48:10):
It's not just Kill Bill, but it's also charming and funny.
Thank you very much for taking the time today to join me and share the work that you're doing on flex measures. It's definitely a very interesting project in a very important area of effort. So I appreciate all the time and energy that you and the other contributors are putting into that. And I hope you enjoy the rest of your day. Thanks, Tobias. It was a joy to be here.
(48:34):
Thank you for listening. Don't forget to
Visit the site at pythonpodcast.com
to subscribe to the show, sign up for the mailing list, and read the show notes. And if you learned something or tried out a project from the show, then tell us about it. Email host hostpythonpodcast.com
(48:58):
with your story.
And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.
Hello, and welcome to podcast dot in it, the podcast about Python and the people who make it great. When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode.
With their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform,
including simple pricing, node balancers, 40 gigabit networking, and dedicated CPU and GPU instances.
(00:39):
And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover.
Go to python podcast.com/linode
today to get a $100 credit to try out their new database service, and don't forget to thank them for their continued support of this show.
The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. SelectStar's
(01:04):
data discovery platform solves that out of the box with a fully automated catalog that platform solves that out of the box with a fully automated catalog that includes lineage from where the data originated all the way to which dashboards rely on it and who is viewing them every day.
Just connect it to your DBT,
Snowflake, Tableau, Looker, or whatever you're using, and select star will set everything up in just a few hours.
Go to python podcast.com/selectstar
(01:26):
today to double the length of your free trial and get a swag package when you convert to a paid plan.
Your host as usual is Tobias Macy. And today, I'm interviewing Nicholas Hoehning about Flex Measures, an open source project designed to manage energy resources dynamically
Nicholas, can you start by introducing yourself?
Yes. Hi, Tobias. I'm Nicholas. I'm German
(01:48):
and living in the Netherlands
since 15 years. I've been a programmer for a bit more than 20 years
using Python for a long time. I've been working on energy since 12 years, I think,
from that vintage point.
And do you remember how you first got introduced to Python?
1st, I was I had been using
(02:09):
very little c and Visual Basic. I've been working on
access
databases
and and word
macros.
At at some point, I did then, go to university.
During my bachelor's, I got to work on a start up. It was a neuroscience
startup
that is still around. It's called Iquant.
(02:30):
The idea was to
transfer neuroscience
models of visual attention
into a web application so that you can, you know, basically upload your
ad and find out
where people might look first.
And we do we use a lot of Python there, and I had to you know, I was sitting at lunch, reading the standard Python tutorial so I could get to work on that startup, still remember.
(02:55):
In terms of the Flex Measures project that you are working with, can you describe a bit about what it is and some of the story behind how it came to be and why you decided that this dynamic
energy demand response system is something that you wanted to spend your time and energy on?
Yeah. Sure. And I'd love to yeah. It has become my life topic.
I started a PhD
(03:16):
around this topic in 2009
when
it wasn't absolutely clear that we will get the kind of energy transition that we are basically in right now,
driven by, you know, lots of renewables
being the major source at some point.
Flex measures, basically, an energy management system.
(03:37):
It helps you manage your energy. In the last couple of years, energy management has meant that you manage
the data around your energy, metering data.
You put it on NICE dashboards and you do billing around it. You know, that's also significant work to be done. Flex Measures focuses more on
the energy flexibility
(03:57):
and optimizing
when energy is consumed during the day
because that's an a very important piece in the energy transition.
As any any expert will tell you, we'll need to get to this kind of optimization,
at some point.
Otherwise, we will not get the fit between
available renewable energy and the actual consumption to have a stable energy system. So flex measures helps you find the best times
(04:25):
for your flexible energy assets. Let's say, a battery. That's the best example of a flexible energy asset.
When should that battery be charging or discharging?
The other examples for flexible assets are heating. You can heat before
somewhat sometime before you're really home, for instance.
Heat pumps are a big topic.
(04:46):
The industry, you have a lot of potential
to run energy intensive processes at a slightly different time. And Flex Measures really helps you
model all your energy data in real time and make that decision.
You mentioned that a number of the other energy management systems are focused more on a kind of 1 way path of information where you're just retrieving data from the system and then using that to build some analysis, but that doesn't get acted upon in any sort of automated fashion. I'm wondering if you can just give a bit more of an overview of what the space looks like for these energy management systems and if there are any kind of alternatives to flex measures or if it's something that is relatively unique in the space.
(05:27):
There are some projects
that have on the road map to also do this kind of real time optimization,
but you got a lot of inertia in these projects as well in companies. So I see a lot of companies who
had to start basically going to customers who have not even any metering
installed whatsoever
other than having a main meter. And the main meter's data
(05:50):
is not real time as well.
So these companies started with bringing in their Raspberry Pis or whatever it is,
becoming good at that, starting to measure, becoming good at visualizing information, becoming good at doing them some some aspects of billing maybe.
And to do this step into data science optimization,
I see
(06:10):
I see the difficulties in some organizations to to that extra step. So
these people are happy
to get some other project and get some other help with that. I don't know all the projects out there, of course. So Flex Measures is open source, and another open source project that I know about is just called OpenEMS.
That's a Java project in,
originated in Germany
(06:32):
that's, built a quite interesting foundation around it. But also there, there's a big focus in inertia on getting the metering
and visualization
correctly done.
And there's not so much support yet to
optimize the actual
what actually happens and and influence that, and we have decided to only focus on that.
(06:54):
As far as the
primary focus and goals of Flex Measures, I'm wondering if you can just talk to some of
the kind of outcomes that you're aiming for and
some of the,
I guess, target audience for the project.
As a big goal, the big hairy goal is that Plaxminder should really help to
(07:15):
speed up the energy transition and, you know, become 1 of the standard pieces of climate tech,
infrastructure tech that you would also always consider when you start a new
modern energy project where
your real time optimization
is a part of the solution.
The main way I think Flex Measures can help is really being a developer friendly tool,
(07:38):
solve a lot of these
things that take you weeks of work, get your data model right, get your data integrations right,
get some monitoring
right,
decide how to implement where your algorithms should be run,
have a queue for that, you know, processing queues.
There's all these things that developers know
will keep you
(07:59):
busy before you get to the real
use case, before you really help your customer, or you are implementing something very
shortly, very quick hack.
See that happening a lot. Other people asking their cousin's neighbor to have a simple algorithm put somewhere, but this is about, you know,
real time data pipelines who have to be kept alive and be checked on health. Yeah. I think we succeed if developers around the world who I expect in the next 10 years to
(08:27):
be looking around how they should approach the similar similar challenges to always consider flex measures.
And that can be in a start up somewhere.
That can be in
a larger energy company
where they have already their ICT team, but, haven't solved or started to do this yet.
And and I think
(08:48):
this is a problem that happens in all the the the parts of the energy system. For me,
there's 2 big parts. That's industry and,
what's usually
called smart city at the moment. So anything in the the built environment, those are 2 huge sectors of the energy sector, which in itself is 1 of the biggest
sectors in the world that needs to undergo the most rapid transition you could imagine. The other plans are 15 year plans basically to basically change
(09:16):
how everything works.
So we really hope flex measures can help get that super big vision done in time.
Do you also see flex measures as something that an individual homeowner might use as part of managing their energy systems, particularly as more people invest in solar and small scale wind?
Yes. I want to talk about a small project we've been doing. So
(09:41):
besides working with customers or with companies that can scale up innovation.
We've seen that this is still slow. Right? So these companies all have to think differently before they haven't been involved with energy so far. It's a long process. So we have, on the sides, created a living lab with just 1 household
with someone who's also very enthusiastic
(10:02):
about, specifically,
vehicle to grid charging.
They bought the exact car and the exact charger that can do that already, the 1 that is allowed to do that in the Netherlands.
And we've really implemented
the automated optimization
of the charging and discharging in in his house. And for that, we needed to actually do something in the house.
Flex measures is basically supposed to run-in the cloud.
(10:25):
It's dockerized. It can also run locally.
But it basically makes decisions. It doesn't really show you something.
So I want to pitch a very cool project that's called Home Assistant.
It's also open source. And it's also originated in the Netherlands,
but they have a
company in the US also now. Very interesting
group of people. Basically, in the home automation
(10:47):
sector with their software.
We use Home Assistant together with Flex Measures to optimize the charging, discharging, and and taking into account the the solar panel production to really create a cool project. We also open sourced our Home Assistant plug in.
So that's out there. We were on Dutch TV and a couple of people wanted to use it as well. And now there's this week, the first 2 others who've implemented it in their house. So now we have 3.
(11:15):
But for us, as a company, that's not the focus, but it's where you can really do something right now and really show something. And there's always these, enthusiasts
that just want to build something in their house like that.
As far as the overall question
of energy systems, I mean, that can be
it's a very broad term with a lot of different,
(11:35):
I guess, energy sources and use cases
kind of embedded in it, where 1 element of it is the electrical grid. Another is the usage of fossil fuels for transportation and shipping.
There's home heating. There's heating at, you know, industrial scale for large factories.
And the majority of those,
(11:56):
I guess, ecosystems of energy usage are governed largely by different market forces where there is the, you know, energy producer, which is becoming a bit of a fuzzy line now with more homes and residential and industrial installations having their own energy generation through things like solar and wind. And then there's the energy consumer and the distribution system.
(12:18):
And the pricing of those different resources are governed by, you know, some level of dynamic usage where, you know, there are peak loads, and so that can increase energy prices for certain periods.
And I'm wondering
what you see as some of the benefits that Flex Measures can provide to
this marketplace ecosystem, particularly given the fact that it's not as clear cut now as it once was?
(12:41):
No. You're absolutely right. It's the vast,
sometimes confusing
system. I like the analogy that because somehow it's all connected and it has to happen, in real time and the frequency of the grid,
as far as electricity is concerned is held in some
synchronization.
We've created the largest man made machines with the grids. So you got the US grid, the European grid, and the Asian grid, basically, as the largest 3 machines you could even imagine where your house is a part of it, or the electric assets,
(13:13):
and it happens in real time. Different markets
buy and sell and allocate energy for different horizons.
And FlexMaster is certainly
not very useful in the range of, let's say, you're buying your energy a couple months ahead and buying these big budgets.
Also, I wouldn't say it's usable in the second range. There's some balancing markets to get this frequency challenge on the grid, right, where you have to react within a second.
(13:41):
But everything in between,
where you have to react basically during the day
or you think in the evening about your plan the next day,
that's where Flex Measures,
is really useful.
And as I said, it's really on the electricity
range
where we want to use it.
So if you're a consumer,
(14:02):
you can use flex measures to use greener electricity, but it will help the grid stability. But I think mostly what people care about, of course, is to to have their energy bill in check. The spread on market prices has gone up.
Energy bills
are rising a lot. So there's a lot of urgency, I think. Now there's
clarity that this is very important for them to achieve.
(14:24):
And
from a grid perspective,
if you're a network operator,
it also is great if a lot of
decentralized decisions are happening
that help the grid stability.
We we have to follow what renewables are doing to a big extent more and more.
We cannot influence when we get that solar power or wind power, but we can influence
(14:47):
when we use power. And if a lot of decentralized
decisions are being made to shift their consumptions
towards when renewable energy is really there. We get a more stable grid.
And
the only other solution for network operators would be to, you know, double or triple every cable in size.
So
this is a big societal
cost saving we can achieve with installing
(15:09):
better decision making in in the nodes, which turns how the energy system works on its head. Right? So still now in the last decades, we have optimized a few power plants per country
and just forecasted what consumers will be doing.
And now that partly turns on its head. It's very interesting.
Digging a bit more into that question of renewables
(15:30):
and their broader distribution, particularly with residential installations as part of that mix, I'm curious
what you see as some of the
broader impact on the way that the kind of marketplace economy around electrical grids in particular
is likely to shift as the power generation and power distribution
(15:51):
becomes
bidirectional
on a more routine basis.
1 thing we see in the Netherlands, and I think that the Netherlands are a bit ahead of the curve there, is
more dynamic prices.
I think you can choose between
5 suppliers now that will buy
power on the day ahead market, which has 24
(16:12):
different
slots. But then they will just
give you those
24 prices.
So the the the afternoon
before, you will know what your 24 prices will be the next day. And if you have,
you know, some optimized
optimized system like what we did, for instance, in the vehicle to grid project,
that's going to turn out quite well for you then. You will not consume energy when those prices are high. Of course,
(16:39):
what's interesting is that this might create more of a divide between those people that can afford that
that has this cool
infrastructure plus software, and they get their average energy bill.
Well, they keep it low, let's say.
And who's going to pay a lot? That's probably going to if everything goes like this, it's going to be the poor people.
(17:00):
So this is outside of the technical
realm, also very interesting
political discussion where we still have some time. And I'm also involved in some of these discussions to discuss how we can
achieve
a more just way to do that.
What we really need is not for individual homeowners to
invest in their own house and make it smarter.
(17:22):
I like it. I'm enthusiastic about it when it happens, but what we really need is this scale.
We really need to do these things for everybody on a big scale.
So for instance, big housing
corporations
already know, given their ESG
strategy,
they have to renovate
at least 4% of the whole portfolio every year. And every year that they don't do that, it's just getting to it, getting to be more pressing, like,
(17:50):
5%, 6%.
And then, yeah, you need to
make a whole neighborhood or a whole big flat
react to flexibility
demands from the grid.
And that can be very exciting that you then, for instance, have 1 giant heat pump for a whole flat. And especially if you get to bigger numbers,
there will be
(18:11):
a better way to react.
For instance, it's almost impossible
to forecast for a single house when you will be home, when will you cook.
You can have a little bit of a idea. But if you forecast a 100 consumers,
you get a much better data
forecast, which you can really work with. So I think
(18:32):
that's really what I need to see, these these bigger scaled approaches.
And, yeah, I hope Flex Measures can be can be helpful with that.
Digging into some of the way that Flex Measures
operates, I'm wondering if you can talk to the sources of data that you're looking at, some of the ways that you manage that integration
and processing,
(18:53):
and the types of interaction that different kind of users or operators might have with Flex Measures and how much of it is just you kind of set it and forget it? Yes. Sure.
So my simple picture of flex measures is data in, scheduling out.
Data in means certainly metering data.
You want to know what all your assets
(19:14):
did in the past. It can be very useful to, for instance, for a battery, know the possible state of charge it has now.
It can help to forecast what will happen in the near future.
We also read in prices,
market prices, for instance, the next day,
weather forecasts.
So you can then forecast your solar generation if that's on your roof.
(19:38):
Sometimes the occupants'
plans are also data that you need to take in. So in the vehicle to grid, living lab in this application,
The occupants' data might be that they say they will need the car for a longer trip
tonight at 6 PM. That's important data that influences
the scheduling because the car should be a 100% full by then.
(20:01):
Or in a factory,
they also have new constraints every day, how much
processing work they need to be doing.
That's,
about the data. And you said earlier that set and forget might be
an interesting concept. I really agree.
For,
most people
who will live in modern energy environments
(20:24):
with with some optimization
happening,
they should not even
care about that during a normal day.
So in the vehicle to grid project that we've been doing, that really was also design goal.
It worked.
And from then on, it just quietly
optimized every day what the battery was doing.
We did, however, in that project, think you might want an override button. You might not want to
(20:49):
meddle with what exactly the optimization is doing, but
it's nice, of course, in an optimized system to have the ability to say, just stop what you're doing or
the other button says,
charge the car now.
I don't care about the price or whatever.
The idea is that these buttons are almost never used, but they should be there. So I think that end users who live in these
(21:13):
built environments should really not be involved that much,
and it's more the
servicing
around that. So Flex Measures
needs to be hosted, and we need to
check that that data is always flowing
and check with every error that might
come up in in the scheduling.
(21:35):
And you have some companies that you call energy service companies.
So a lot of them, for instance, have these more traditional
energy management systems I talked about earlier.
And
I think that they would be users of Flex Measures.
If we don't do it for them, they would really
build it into their monitoring pipeline,
have a KPI and reporting pipeline to regularly check if that's happening correctly.
(22:00):
1 thing that's nice to check is, the universal
smart energy framework.
It's called Yusef.
My cofounder was on the design committee, and they talked about the different roles in modern smart energy system.
You mentioned the prosumer already.
I mentioned the energy service companies, and you also have companies they're called aggregators.
(22:22):
They
aggregate the energy flexibility
from
a lot of customers
into their virtual power plant and and just trade that flexibility
on markets.
I think you could build energy aggregator using FlexNetters,
but that's not what we do. We use it for
optimization behind your meter.
(22:43):
As far as the actual
implementation
of Flex Measures, I'm wondering if you can talk to some of the
design considerations that went into it, some of the challenges of being able to
flexibly manage scale of inbound data and the types of requests that you're getting.
And also, I'm interested in maybe some of the capabilities of being able to federate across different installations of flex measures and, you know, route different decision requirements or different data points based on different kind of inbound information?
(23:16):
So from the beginning,
a few years ago,
I think, I didn't mention that before. But the first
project where Flex Measures really got its first development
time, that was with a South Korean customer called A1 Engineering.
They wanted to do simulation
first of a of a system with a lot of smart charging,
(23:37):
but then also move to
real time operation in a later phase. So this is
the main design idea
that has been there from the start that you can use
something
in reality in real operation,
but it's also useful
initially in a phase where you are simulating what might actually happen.
And that's still very relevant.
(23:59):
So we're doing a simulation with a bigger real estate company right now. That's their first step. I think that that main design goal still stands.
Now I haven't used multiple Flex Measures nodes and then federated the decisions yet. So I think
that could be built, like, a lower level of Flex Measures instances, basically, governing,
(24:20):
smaller installations.
And another 1,
maybe trying to aggregate. As I mentioned earlier, the it's a concept of the flexibility aggregator
and aggregate across
those behind the meter installations that could be a higher level of extensions
instance. Yeah. Now we haven't done that yet.
We consciously
(24:40):
can decide not to, you know, let the volume of data grow too large.
So for instance,
you might run into projects where they have 10 second
interval readings.
But to make the kind of decisions that Flex Measures needs to make, let's say, on an hourly basis or 15 minute basis,
you don't need to read in the finely
(25:03):
structured data. It would be best if that other system gives us already aggregated
data. For instance, in in 5 minute steps, that'll be fine. And then the amount of data isn't growing that much,
but we're already looking into that. So we
our data is stored in, Postgres.
Of course, that we also made that choice because you have so much support for
(25:27):
growing bigger with your databases and have all these clusters, clustering strategies, and sharding.
And, also, of course, there's interesting
technology built upon Postgres. I think 1 of called Timescale DB that really focuses only on the time series
aspect.
And 1 thing that we considerably
put some effort in is to model
(25:50):
this data because it comes, as you said, from different sources.
There might be a company that
installed the sensors
at the factory or is responsible
at the houses to measure what's happening.
So weather forecasting
service. So we use, open weather map.
Flex measures can read data from there.
(26:10):
The European transmission system operators have a service
in open transparent API as well. So you got multiple sources.
They all send you data that might be
about different things.
It has
different time zones.
It's crucial to maybe know when they know about that.
So not only the time that they're talking about,
(26:32):
1 point in time with 1 measurement,
has 1
date time, of course, about the time that you're talking about. But we're also interested in the daytime when they said this would happen.
If it's before the time, this is a forecast. If it's after the time, it's a measurement
that can be quite useful when you write optimization algorithms.
(26:55):
All this led us to write our own,
basically,
time series data model. We called it Timely Beliefs.
It's actually an extension
on pandas. So what it makes possible is that you have a pandas data frame
or basically a class that extends from Pandas data frame, and we call it the beliefs data frame.
(27:16):
We wrote a lot of functionality around that that you can, you know, merge these data frames or change the resolution or whatever. It's a very interesting project.
They took a lot of our time, and now we're profiting from that within Flex Measures to really understand all the time
what that data really means.
As far as the kind of architecture of the project, digging a bit more into that, I'm also interested in understanding the
(27:39):
kind of extension points that you've built to be able to allow
different users of Flex Measures to adapt to different data sources or
being able to extend the types of decisions and machine learning capabilities that they want to use or the types of predictions that they want to generate
and maybe a bit more of the kind of data integration and kind of predictive capabilities that you've built into it? Flex Measures is extensible
(28:05):
with plugins
done last year, and we are using that ourselves
because often
every project has these differences,
different data sources, or or needs.
Often, it's also the visualization
that you want to adapt to a customer.
That's very important
to the whole architecture that you can add your own
(28:26):
API endpoints even,
your own visualization, and and we have a command line interface.
You can even add your own
command
line interface commands.
For instance, 1 that regularly
gets data from some other open API,
and you want to run that hourly in a Chrome job as a Flex Measures command.
(28:47):
Flex Measures actually is built on Flask,
and Flask has some nice capabilities for that.
Of course, for APIs, but also the it integrates with click. So command line interfaces are, nice to write.
And with the Flask blueprint
technology, it was
quite a joy to write our 1st plug in
(29:08):
architecture. So what you do in the plug in is mostly
add a blueprint
to flex measures.
Flask has a CLI
command, and we renamed that to flex measures.
So,
yeah, it all ties back into what I said earlier. We want flex measures to be usable by software developers. And then,
of course, APIs are super important, but also the command line interface, I think, is is super important. We wrote a
(29:36):
tutorial how you would use Flex Measures within a few minutes, you know, just to show you
within a few minutes what you might be doing,
which is based on Docker. So you don't need to install all these things. Just pull the Docker image. And then that tutorial is
a few
calls to the command line interface
about modeling that couple assets that you have a battery, for instance, loading in some data
(30:00):
from a CSV file and asking Flex Measures to compute a great
optimized schedule for the next day. That's all command line interface command. That's what I really like.
Optimization
in the background
happens
I think we use pyomo.
It's also a very powerful
library for for these kind of things.
(30:21):
And we used there was some sklearn
that we used. But recently,
we've picked up sktime,
which is really
a package focusing on time series
forecasting, doing smart things around that, and it's very nicely done, it seems.
As you have been
building Flex Measures and
(30:41):
iterating on the problem space and also with the
donation of it to the LF Energy
Foundation.
I'm wondering what are some of the ways that the design and goals and scope of the project have changed or evolved since you first started working on it? I said earlier that,
actually,
the overall design goal to make both simulation and optimized real time operation possible from the same code base is still intact.
(31:08):
I liked open source in Flex Measures. It it happened roughly,
well, less than 2 years ago. I we hit that button in GitHub
in a couple months later, donated to the Linux Energy Foundation.
I liked that that pushed us to even more transparency
and even more having a focus on code quality
to make sure that
(31:29):
Blake 8,
MyPy,
and Black always run. It's always looking good.
And documentation
got a lot of boost.
And, also,
in the meantime, we did once make big changes to the data model. When we worked for industry customers,
we noticed that our initial data model was a bit too simplistic.
(31:51):
I think we modeled
power,
weather data, and prices.
And if you go to industry, you quickly realize they want to measure a lot of other things that all have
some influence on your
outcomes.
For instance, you might be measuring steam flow,
and
that wasn't possible. So we took a lot of time to make our data model more generic. Now we can model anything. Another interesting aspect of the kind of open sourcing question, and it's addressed a little bit by your donation of the project to the LF Energy Foundation. But I also know that you are running a business where Flex Measures serves as a kind of central portion of what you're providing
(32:32):
at CITA. And I'm curious if you can talk to some of the ways that you think about
which pieces of your work you want to put into the open source Flex Measures project and what are some of
the other kind of capabilities or layers that you're building on top of that that are part of the kind of commercial and business focused offering?
It's a very interesting time to have a commercial open source
(32:54):
start up.
I was recently involved in an accelerator,
on the US side.
I had to attend at night, always at 11 PM.
But
there's really some investors in the US
that are looking for open source companies to invest in
because you got, in the last couple of years, some really interesting examples,
(33:15):
mostly on the database
developer tooling side.
So, obviously, in the energy sector, this is quite new. But it's clear that you can
run a business even if you're
very open.
LIMX Energy Foundation advised to use permissive licenses. So in this case, we use Apache 2.
You could also use MIT, for instance,
(33:38):
over a Copyleft licenses because the idea is to just, you know, open the doors wide to
growing the community,
letting other people use the tooling you've you write. Even if they don't re even tell you about it, they should have no doubts that they can use it freely
because you want to just grow, grow, grow, grow. That's after the Linx Energy Foundation told us that they use those licenses.
(34:01):
I also heard it from this venture capitalist accelerator program. It's a very interesting
vision on innovation, on open innovation that this might be 1 of the future scenarios that comes true. Right? That the pool of open source
innovation becomes
as big in other sectors has become in databases
and developer
(34:22):
tooling. Nonetheless, well, we have made Flex Measures possible to create plug ins.
And
oftentimes, the customer says, well, if you're developing something here about our processes, I don't want that to be public. Or, if I pay you to do something, I don't want it to be public. And with the plug ins, that's very possible, of course. Then you have
Flex Measures as an open platform.
(34:43):
And, also, Timely Believes is open,
and we occasionally
open up other projects that we work on. For instance, I mentioned that, Home Assistant plug in in the in the vehicles to grid
project that's open.
I think we have a couple
closed repositories
lying around right now with work for customers.
It's something that's maybe not ready for the world.
It will always happen.
(35:05):
I'm also interested in understanding a bit about given the kind of predictive and machine learning focused capabilities
that are part of the Flex Measures platform
and the fact that you are
building
predictions about and
energy capacity,
(35:25):
what are some of the
challenges that you face in terms of accounting for whether to bias towards false positives or false negatives, and what levels of uncertainty you're willing to accept in some of those predictions and decisions that you
make? Yeah. That is a very interesting topic to go into.
It's difficult to answer
(35:45):
without having even more experience. We had
a project trying to find out if
forecasting should be done with something very sophisticated. You know? You can use linear regression to forecast, let's say, the solar production,
RIMO models,
but But you can also use a neural network.
We found that the the neural network isn't
(36:07):
really better
that much better, and you don't know what's going on. And just recently,
I saw a paper come by
that some researchers even confirmed that. So for time series forecasting,
currently, there doesn't seem to be a reason to become very sophisticated.
What we did do was
we saw that
(36:28):
there is algorithms
which tell you about the uncertainty
that lies in the prediction. So it's mostly,
ensemble
algorithms like random forest.
They have, you know, a lot of forecast and then make a a mean forecast, but they tell you how far all of the others
were distributed.
(36:49):
And
that information can potentially, of course, be useful for exactly your question that you know that some forecasts are
shaky,
and you might decide
for a scheduling algorithm that there's a policy of not acting in those situations.
And that's something I didn't mention about the timely beliefs data model. 1 column in the belief data frame is the uncertainty. So how certain
(37:15):
forecast, for instance, would be. And it reminds me of a setting we have in a scheduling algorithm right now.
For the battery charging,
1 property you need then is, round trip efficiency.
So
you can discharge
a battery.
Let's say, you discharge 1 kilowatt hour
and then you charge that back.
(37:36):
Let's say the other 1 around. You charge it
first and then you discharge it,
and you've made some arbitrage because your 24 hourly prices had this nice difference
or that some difference. But the battery is not 100% efficient,
so you end up maybe that it's 80% efficient,
could become 90% in some batteries.
So you're losing some of that power in that process.
(37:58):
And
it's very important to model that so that the scheduling algorithm then sometimes does nothing.
Price difference difference is too low,
then it doesn't make sense to do all that. So sometimes the schedule is to do nothing even if there are some price differences. And I see the
policy of doing nothing if the uncertainty is too high, and that's a similar thing.
(38:20):
In your work of building flex measures
and working with different companies to help implement it and
providing it to the community and working with people who are using it for their own purposes. What are some of the most interesting or innovative or unexpected ways that you've seen flex measures applied?
There's not that much,
but there was a project where flex measures wasn't even being used for any scheduling,
(38:43):
which is
basically the end goal of using Flex Measures is to have a schedule and
then change the times when you do use energy.
And in that case, it was a project where it was
just so useful to
use Flex Measures to just be a plain energy management system,
getting all this data together with data pipelining being done quite fast,
(39:07):
but then not doing forecasting but in doing something else. There was a different algorithm that was basically some advanced analytics.
And it didn't you have to use electricity. It was about gas consumption in in industry.
So that is possible, but that's not where I want to take it.
In your work of building the project and building a business around it and donating it to the community, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
(39:32):
Something that you have to learn on your own is how long it takes to integrate
data
that constantly flows and that has these
intricacies that I mentioned earlier about time series
that you could get slightly wrong. It could be
you get the time zone slightly wrong or
somebody
sends you data that isn't localized.
(39:55):
You might
be 1 hour off half of the year.
Your whole schedule will not be good.
They also might, send you data that gets the daylight saving time wrong, and then there might be 2 hours in the year where you fail when daylight saving time is switching back and forth.
And, yeah, the sheer amount
(40:16):
of
attention and work that gets into that
is is quite staggering. We had an industry client with a partner who does the censoring,
and they didn't have the API ready for us. So
it was
not what I had expected in 2021 from that kind of player. What we could get from them was emails with Excel spreadsheets,
(40:39):
And we decided that they send us 3 per day. So every 8 hours, we have we wrote a Flexner's plug in that has its own command line interface command to just handle that email
and get the data in.
Yeah. You can imagine that that prolongs that kind of pilot
considerably.
Yeah. It's definitely remarkable
(40:59):
how much
of modern
data engineering and data management still relies on CSVs and FTP.
Yes.
Absolutely. And
if you look at the energy sector as a whole that these kind of experiences,
1 dimension
towards the realization that there will not be 1 end to end solution that can be used across the whole country or across a continent that that will just take over. We have to
(41:27):
build a lot of
unique end solutions.
Let's call them apps
that work differently in different energy systems, countries,
cultural contexts,
sectors.
And I think
flex measures can be a basis for all of that, but you always need to
put something on top that really fits
(41:49):
to that situation.
For people who are interested in being able
to gain some more insight and potentially act on the different aspects of energy usage, whether it's at the residential or industrial or grid scale? What are the cases where Flex Measures is the wrong choice?
I've mentioned that earlier, but there are some people who have to act within seconds.
(42:10):
And that's usually if you're really taking part in this first
reserve mechanism
reacting to balancing problems,
frequency things.
And
we have put a lot of focus on flex measures to be developer friendly and correct about the data so that developers really understand what they're building.
(42:31):
And we haven't made it, you know, a Rust project
that lets you do all that within seconds.
As you continue to build and iterate on the project
and continue to iterate on the business that you're building on top of it, what are some of the things you have planned for the near to medium term or any particular problem areas that you're excited to dig into?
Yeah. So first, about the FlexVicious project,
(42:54):
I think
what we really could do with that project is to support
the 2 groups of users that I see
in what they are
doing as a as a first steps. So I see the first group as being the 90%
users
who have a use case that
really happens
(43:15):
a lot in the world, and Flex Measures should support out of the box
that the optimization
for that use case can be done within, you know, a few hours.
Data integration plus having that optimization work should be a few hours. So that that's a a little bit of a benchmark.
Let's say, as an example,
somebody has solar on the roof and there's somehow
(43:38):
a battery, a home battery, or an electric car,
and they have a dynamic tariff.
That might happen a lot in the next years across the world, something like that.
And that should just work. And,
basically, that does work now. What we have to do is documentation,
write an excellent tutorial
(43:58):
so that
everybody could do that.
And there's probably a list of 4 or 5 more of these
situations
that should just work.
And the other group of users are the power users
that really have a unique problem and may come with their unique
idea of solving it.
(44:19):
And then I'm talking mostly about the algorithms.
Right now, you can already
plug in your own scheduling algorithm.
Or to be honest, at the time of recording, that is a PR I'm
having reviewed.
But that's really important for some people. That's researchers,
certainly.
That's their whole line of work to come up with better algorithms.
(44:42):
Maybe find that neural network algorithm that really does make the difference.
But you see that also in some companies really have
certainly an industry that just have their unique challenge. And they should be using flex measures,
you know, as is and then just change that 1 thing that there's
a common interface
with how
(45:03):
the question to make a new schedule is described.
But they could, in the back, just switch out the algorithm that is used to make the schedule.
I think these 2 groups of users deserve
each of their own dedicated
tutorials by us to really make it clear, to make it a joy to do so much as our work. I mentioned 2 sectors that we're working on.
(45:24):
The industry
world is very interesting and challenging,
but it's also very diverse.
Right now, those environments,
what excites me the most, and I'm I'm talking to
companies there that really understood the scale
that I mentioned earlier, they have renovate or we modernized the whole portfolio
within a very short amount of years. So scale is their big topic.
(45:48):
And the third thing I didn't mention is microgrids.
So microgrid is your own little electricity grid
that is connected
maybe to the main grid
or not,
completely run on its own. And that's a situation
that we might see all across the world. So there might be
more
self serving neighborhoods
(46:11):
that are not completely
dependent on the grid in Western countries, but certainly in Africa, that's something that's just the 1 way to go for a lot of regions.
There are some interesting companies there that I hope to do more with in the near future because Flex Measures should be able to
basically
run a small village.
You might think there's too many assets, but what really matters is the flexible assets. So that's where
(46:37):
that defines how much flex flex measures has to do, how many flexible things are there.
Emobility
is making huge steps in Africa, certainly.
We're
hoping to be involved in a project in Victoria Lake where all the fishers get upgraded to electric boats, for instance,
And they use energy for productive uses,
(46:59):
like cooling the fish they caught,
water purification.
That's all flex it's all something you can shift around in the day. And then flexiators can really help, for instance, in the design phase already to not buy oversized
equipment.
I don't think you will become rich in Africa, but that's something I'd really like to be involved in. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And so with that, I'll move us into the picks.
(47:28):
This week, I'm going to choose the show She Hulk from the Disney plus. Just a fun and goofy show. Some interesting storyline there, so I've been enjoying watching that. So I recommend that for people who are looking for something to do in the evening. And with that, I'll pass it you, Nicholas. Do you have any pics this week?
Yeah. I'm watching a very nice German series on Netflix,
(47:49):
and it's called KLEO,
k l e o.
It has a bit of kill bill
visuals because it's an ex
agent from the East German
Ministry of Defense who got thrown into jail. And she
comes out of jail right when the wall has been torn down and tries to find out what happened, who did it.
(48:10):
It's not just Kill Bill, but it's also charming and funny.
Thank you very much for taking the time today to join me and share the work that you're doing on flex measures. It's definitely a very interesting project in a very important area of effort. So I appreciate all the time and energy that you and the other contributors are putting into that. And I hope you enjoy the rest of your day. Thanks, Tobias. It was a joy to be here.
(48:34):
Thank you for listening. Don't forget to
Visit the site at pythonpodcast.com
to subscribe to the show, sign up for the mailing list, and read the show notes. And if you learned something or tried out a project from the show, then tell us about it. Email host hostpythonpodcast.com
(48:58):
with your story.
And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.
Popular Podcasts
United States of Kennedy
United States of Kennedy is a podcast about our cultural fascination with the Kennedy dynasty. Every week, hosts Lyra Smith and George Civeris go into one aspect of the Kennedy story.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com