Extend Kepler to provide real time carbon metrics

[adrianco]

I’m working with the Green Software Foundation and have proposed that we define a real time carbon metric standard. This seems like it could layer onto Kepler or be an extension of Kepler functionality. We’d blend the energy and attribution model of Kepler with carbon intensity data sources to produce a time series of carbon that maps to the energy data from Kepler. Is there interest in this, and should it be part of Kepler, or a separate project in your opinion?

[metacosm]

That's something I'd be interested in working on. Of note, Kepler already provides estimates of carbon consumption based on "standard" carbon factors. Ideally, we'd need real time sources along with precise enough geolocation of workloads to provide accurate carbon consumption metrics. Which carbon intensity factors are you considering, @adrianco?

[adrianco]

Thanks this is helpful. What I'm trying to do is come up with a clear spec for the real-time info that would be useful to people building carbon aware/optimizing apps (as opposed to monthly accounting kind of apps) and to work backwards to a common spec for the info we need from cloud providers, then to make it into a standard definition via the GSF and hassle all the cloud providers to get them to agree to make the info available. It may take a while, but a clear ask that customers can unite around is how I think we can get the clouds to give us what we need.

[metacosm]

That's the kind of approach I was thinking about as well. Hopefully, we can gather enough clout so that the hassle of cloud providers would be fruitful… 😉

[caldeirav]

The idea with kepler is also to open source the instrumentation and work with the community to make it a common CNCF standard. I think a common standard in instrumentation can really help ensuring we get consistency in the real-time info coming from the cloud providers. This does not preclude at all a standard definition via the GSF (we did not plan to work on this with kepler and it would be really helpful to work on that front too).

[rootfs]

@adrianco This is definitely an exciting project. There is some early experimentation that uses Kepler and a carbon intensity reporter to get carbon emission per workload. https://www.youtube.com/watch?v=dd-6rqp_qlA.

I understand there is a framework provided by GSF to track carbon intensity. A potential near term solution is to use the GSF webapp API with Kepler metrics and explore how to integrate the two data sources. wdyt?

[caldeirav]

@adrianco for information the POC for real-time carbon reporting is built on top of OS-Climate Data Commons (using a real-time data federation pattern on top of Prometheus, which still needs some technical work to optimize performance). Here is the reference: #210

[adrianco]

Thanks Vincent - good to see you again, this is very helpful.

[rootfs]

I have prototyped this concept using a carbon intensity API from Singularity. A preliminary dashboard is below. We can use real-time carbon intensity and Kepler energy metrics to quantify carbon footprint from workload, tenant, and clusters.

[adrianco]

Thanks, this is very cool.

[jbu]

#675 now exits, more as a thought experiment and prompt for discussion than anything. The point being that while reporting power usage of nodes and containers, and then combining that with public grid co2 intensity is excellent,

• a large enough data centre might have storage or generation attached so the co2 intensity differs from the public grid
• if we move up the serverless stack then more relevant questions become things like 'what's my co2 output per vcpu-second?, per message?' etc.

So can we give these sort of metrics as well?

[adrianco]

I've written a draft PRFAQ for a real time feed, including the idea that this could be used by Kepler in cloud deployments. I'm currently sharing it privately with cloud providers to get their feedback. Is there someone familiar with how Kepler is implemented that could find time to discuss this with me? My next step (soon) is to make it a public proposal via the GSF.

[rootfs]

That's great! Can you share with me via [email protected]? Thanks

[metacosm]

Would like to read it as well, if possible.

[husky-parul]

+1. Please add me as well. Thanks @adrianco

[adrianco]

The proposal has been shared publicly at the GSF Green-Software-Foundation/standards-wg#67
The most up to date text of the pr-faq is in this pull-request Green-Software-Foundation/pr-faqs@887177b

Is there a Kepler project meeting that I could attend to discuss this with the team, or should we schedule a specific meeting with interested parties? I want to make sure that the MEC-RT schema (which we haven't defined yet) feeds into Kepler in the right way.

[rootfs]

This is cool! @adrianco do you like to join the twice monthly community meeting? The schedule is here. You can add this discussion to the meeting agenda

[adrianco]

The GSF standards working group has recommended that this become a project, and it will be discussed by the GSF steering committee next week, if approved, a project manager and funding/support will be assigned. The next step will be to come up with a definition of the schema, and I think it should be based on OpenMetrics. However the first thing is to understand what existing frameworks like Kepler use as their naming and units for metrics, so that the realtime cloud data feeds in naturally. Where can I find the definitions Kepler uses for host level metrics?

[rootfs]

Thank you @adrianco for the update! This is very exciting! Please find the kepler metrics here. We are more than happy to discuss it in our next releases.

[adrianco]

Thanks, I'd just found that page.

[eilamt]

Hi Adrian, I think it is a great idea!

1. Your approach will work without requiring too many changes to Kepler for any Cloud Service where the granularity of sharing is that of containers (as opposed to software-as-a-service specific multi-tenancy model which can happen in legacy apps).
2. If we get a time-series for CI then it is a simple matter to combine energy and CI (two time series multiplications)
3. PUE is a convoluted topic, but, ignoring for a sec the limitations, we need a time-series of the overhead factor for cooling/power-distribution. E x PUE x CI
4. One more thing that we ought to think about is the management overhead of the platforms. For example, a service such as Lambda, has multiple management servers in each location that are always running irrespectively of any workloads. There is also the overhead of Continuous DevOps for every service that we may want to factor-in. These can be included as yet another overhead factor such as PUE. I call it platform_service_operational_overhead.
5. How to split each machines IDLE power? and other overheads? across the tenants? GHG protocol (specifically, the guidance document for ICT services, chapter number 4) provides good principles here, that we should definitely follow!
6. Same as support both msr and sysfs to read rapl #5 but for other overheads, that are NOT client load correlated - we should split across the tenants according to the same GHG principles.
7. How much averaging do we want to do within the scope of a service across the containers vs. actually tracking for each container the entire physical provenance? the later is more difficult! In particular, migrating VMs or Containers will have to be factored in, in the later case.
8. As you already eluded to, we must get the power sensing from the cloud provider, in the PM level, or, if VMs are used the VM level. IDLE power of PM must be split across VMs based on their size (eg, number of vcpus). If we do not have this, we have a big gap!
9. I explained some of it in a paper I wrote and presented in HotCarbon just a couple of weeks ago (although the topic is sustainable AI it is actually much more general) --- https://hotcarbon.org/2023/pdf/a13-eilam.pdf. Another paper, that my team wrote, is dealing with the question of inconsistencies across clouds, and going to the open question on whether anything can be estimate without cloud corporation (APIs): https://hotcarbon.org/2023/pdf/a9-arora.pdf

Sorry I missed the community meeting,
but I am definitely interested to continue the conversation!
This is GREAT that you are pushing this standard across the exa-scalers.
I am also trying to get IBM Cloud to join.

[adrianco]

This is now a project at the GSF https://github.com/Green-Software-Foundation/real-time-cloud - we are starting out by mapping the end to end flow of data including through Kepler on a miro board which can be viewed here. If someone from the project would like to help document the inputs and outputs from Kepler I'm happy to give you write access. https://miro.com/app/board/uXjVM1o59N4=/?share_link_id=828805085816

[rootfs]

thanks @adrianco this is a good visual, can you give me access to the miro board? My miro id is [email protected].

[husky-parul]

@adrianco Can you please extend the access to me as well? [email protected]

thanks

[eumel8]

Stumbled across this thread a few days ago, which brought up the initial idea of providing customers of our managed Kubernetes platform with information about how much carbon their workload is producing due to current energy generation. The data centers in our infrastructure are all in Germany, so we use real-time data from electricity generation in Germany.
The idea is for the customer to adapt their workload to the generation of green energy as much as possible. At the very least, the customer should be made aware of this topic. The project can be accessed under CaaS Carbon Footprint. Thank you for the existence of Kepler - a milestone on the way to measuring power consumption of virtual machines, ultimately in the Kubernetes cluster. Pioneering work.

[jochen-schuettler]

I was going to propose to couple electricitymaps with kepler for this, but it seems you already beat us to the task; using an official free source for the same data, if I understand correctly.

[eumel8]

Yes, it's free with a limit of 400 requests per user per minute, which is not a problem while fetch all data in one request and puzzle then the metric in the exporter.

Electricitymaps looks also interesting, a little more commercial. Entsoe is provided by the electricity providers, especially for the EU market. Data are also used by Fraunhofer Institut, e.g. https://energy-charts.info

[marceloamaral]

@eumel8 is this dashboard available? Is it only for EU?

@rootfs What about integrate it with Kepler operator?

[eumel8]

@marceloamaral The dashboard is here and here
Entsoe is like the name say (European Network of Transmission System Operators for Electricity) for the EU market.

[rootfs]

I have a PoC of integrating Singularity carbon intensity data (mostly North American data) to Kepler here.