<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gailin Pease on Medium]]></title>
        <description><![CDATA[Stories by Gailin Pease on Medium]]></description>
        <link>https://medium.com/@gailin.pease?source=rss-c705923da308------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*E8Z6cknYE-uGLuaz</url>
            <title>Stories by Gailin Pease on Medium</title>
            <link>https://medium.com/@gailin.pease?source=rss-c705923da308------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:10:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gailin.pease/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introducing CarbonFlow: tracing GHG emissions from source to consumption]]></title>
            <link>https://medium.com/singularity-energy/introducing-carbonflow-tracing-ghg-emissions-from-source-to-consumption-abef2e6fa739?source=rss-c705923da308------2</link>
            <guid isPermaLink="false">https://medium.com/p/abef2e6fa739</guid>
            <category><![CDATA[renewable-energy]]></category>
            <category><![CDATA[climate-change]]></category>
            <category><![CDATA[utilities]]></category>
            <category><![CDATA[electricity]]></category>
            <dc:creator><![CDATA[Gailin Pease]]></dc:creator>
            <pubDate>Mon, 01 Apr 2024 13:21:57 GMT</pubDate>
            <atom:updated>2024-04-01T13:21:57.215Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9bPqDYp4OVJk54y1" /><figcaption>Emission rates across a model of the U.S. transmission grid, approximating the behavior of the 2020 grid. The darkest colors indicate a consumed emission rate of 2,000 lbs CO2/MWh, while the lightest colors indicate a consumed emission rate of 0 lbs CO2/MWh.</figcaption></figure><p>Electricity generation is responsible for 25% of total U.S. greenhouse gas (GHG) emissions (<a href="https://www.epa.gov/ghgemissions/sources-greenhouse-gas-emissions">EPA</a>). Although that pollution is emitted at power plants, the power produced by a plant can be consumed by loads hundreds of miles away. Singularity’s CarbonFlow technology, demonstrated here on a model of the U.S. grid, enables granular tracing of emissions from source to consumption and improves on previous lower resolution approaches. CarbonFlow provides a powerful new tool for understanding how emissions flow across the grid and the emissions associated with electricity consumption.</p><p>Previous approaches to calculating consumed emissions use import and export data between grid operators, along with information about the fuels used to generate electricity within each grid operator’s territory, to estimate region-wide consumed emission rates (see, for example, <a href="https://gridemissions.jdechalendar.su.domains/#/">Chalendar et.al’s work</a>). There are many advantages to this — the data needed is freely available in near real time, and important large-scale inter-region trends are captured.</p><p>However, regional data can’t tell us how consumed emission rates vary within regions. Even at the regional level, regional consumed emission data is not always accurate, because it assumes that power is uniformly mixed within regions and does not account for how power flows on each transmission corridor may affect consumed emissions. To account for these important granular effects, CarbonFlow uses power flow tracing to identify consumed emission rates at each line and load on an electrical grid.</p><p>One context where CarbonFlow can be valuable is in determining emissions factors for emission accounting laws such as New York City’s Local Law 97, which requires large power users to reduce the emissions from their power consumption. NYC is a large load center within NYISO, and relies heavily on power imported both from other parts of the state as well as surrounding regions. Even in a state like New York with aggressive clean energy targets, consumed emission rates vary within the state, and NYISO-wide numbers can be misleading. In this blog post, we’ll discuss two important trends affecting NYC’s electricity emission rate:</p><ol><li><strong>New York’s consumed emission rates vary significantly by county</strong>. The average emission rate in NYC over a modeled year is 1.5 times the statewide average.</li><li><strong>Temporal patterns in emission rate vary across the grid</strong>. Within NYC, the highest emission hour in a modeled year has an emission rate 1.6 times the lowest emission rate hour.</li></ol><p>The data in this blog comes from a simulated model of the U.S. transmission system constructed by Breakthrough Energy (<a href="https://arxiv.org/pdf/2002.06155.pdf">Xu et al., 2020</a>), which was designed to approximate and validated against the 2020 U.S. grid, including regional temporal and spatial patterns. The emissions data in this model shouldn’t be thought of as matching to a specific transmission bus and time on the real grid, but instead as giving us insight the overall behavior of the 2020 grid. Singularity is working with grid operators to apply CarbonFlow to real grid topology and generation and load data.</p><h3>Regional grid variability in NYISO</h3><h4><strong>Insight 1: Consumed emissions intensity varies across the NYISO grid.</strong></h4><p>When we break down average consumed emission rates by each county in New York state, shown below, we see wide variation. In the southern part of the state, relatively high emission power plants combine with coal imports from Pennsylvania to result in higher average consumed emission rates than the state as a whole. In upstate New York, where there is more hydro generation, the opposite is true. (The model we use is limited to the U.S., but in the real grid, hydro imports from Quebec likely lead to even cleaner power in northern New York.)</p><p>In New York City (“Zone J”, for readers familiar with NYISO’s load zones), emissions are high throughout the year compared to the NYISO-wide average. This difference has significant implications for emissions accounting, because electricity users in New York City using a state-wide emission rate would be undercounting their emissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SkF3K--LN5adre_D" /></figure><p>When we zoom in, we see that even within the counties making up NYC (Bronx, New York (Manhattan), Kings (Brooklyn), Queens, and Richmond (Staten Island)), average emission rates vary widely. In the 2020 model used here, these trends were driven largely by the Indian Point nuclear plant north of NYC, which stopped operating in 2021. The northeastern part of the city (Bronx and Queens) received more nuclear power from Indian Point, which drove lower average emission rates relative to the southwestern part of the city (Manhattan, Brooklyn, and Richmond), which received more fossil power from Pennsylvania. The decommissioning of Indian Point has led to increased natural gas generation in the lower Hudson Valley north of NYC, so these spatial patterns have shifted in the years since 2020, with all boroughs in NYC relying more on fossil fuel power.</p><p>While revealing, these annual averages don’t tell the whole story. Next, we’ll dive into daily and seasonal trends across New York and within NYC.</p><h4><strong>Insight 2: Temporal patterns of consumed emission rate vary across New York</strong></h4><p>The average emission rate of consumed electricity across NYISO varies with a predictable daily pattern, shown in the figure below, where each day is represented by a column with hours represented vertically. Emission rates are low overnight when load is low, rise with load in the mornings, drop with solar generation mid-day, and rise sharply in the evening as decreasing solar production aligns with the evening spike in load as people get home from work. There is also a strong seasonal trend, with summers having higher emission electricity as high demand for cooling drives more generation from high emissions plants.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9XiR9pYeW9ePD4My" /></figure><p>With CarbonFlow, we can zoom in from this grid-wide picture to look at smaller regions or even specific transmission substations (each transmission substation feeds a distribution network that connects to homes, businesses, and other electricity consumers). When we do this, we see that the New York-wide temporal pattern conceals sub-regional variation.</p><p>The daily pattern and regional patterns in NYC are similar to the state-wide patterns, with dirtier summertime electricity and an evening peak in emission rate. The temporal variability is smaller in magnitude than the state-wide trend, because NYC’s fuel mix is more consistent over the day and year than the state as a whole.</p><p>Finally, we can zoom in even further, to boroughs within NYC, with the Bronx and Staten Island shown in Figure 2. At this level, because we’re using a synthetic model, the trends we see are less likely to match the patterns we’d see in real data. However, our work with actual transmission system data supports the overall insight that temporal trends can vary even across small spatial scales like New York City’s neighborhoods.</p><p>Although NYC as a whole has relatively uniform emission rates over the model year, the Bronx in this 2020 model actually had more temporal variability than the state overall. Electricity consumers in the Bronx consumed carbon free energy (mostly from Indian Point nuclear plant, which ceased operation in 2021) overnight during much of the year but saw a relatively large shift to higher emission electricity during summer afternoons. In contrast, Richmond county (Staten Island), on the southern side of the city, has consistently high emission rates and lower temporal variability than the city-wide trend.</p><h3>How CarbonFlow works: tracing electricity from generation to load</h3><p>Now that we’ve demonstrated what CarbonFlow can do, we’ll provide a brief overview of how it works. CarbonFlow is a power flow tracing algorithm, an approach with a long history of applications in market design and power systems research. Figure 3 shows how this differs from previous approaches, allowing a uniquely granular view into the transport and consumption of electricity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N5iuGNW_jZeN2tya" /></figure><p>We start with a model of the topology of the electrical grid, including the locations and physical parameters of its lines, transformers, and other features, along with data describing the generation and load at each transmission bus.</p><p>We assign each MWh generated on the grid an emission rate, from 0 lbs CO2/MWh at clean energy plants to 2000 lbs CO2/MWh or more at oil and coal plants. The locations where these emissions are generated are important for understanding pollution and related equity concerns, but they don’t show which end users are actually consuming electricity and therefore responsible for the associated emissions.</p><p>Consumed emission rates fill this gap, telling us what emissions were associated with the generation of electricity consumed at a given point on the grid. As generators and loads change their production and demand, power flows across transmission lines in constantly varying configurations. To calculate consumed emission rates, we need to model the dynamics of the entire system, including the electrical parameters of its lines, transformers, and other components, and map emissions data onto the electrical grid. We can then trace the power from each plant, and its associated emissions, as it travels to the loads it serves, and report the emission rate at each of those loads.</p><p>Although CarbonFlow gives us a consumed emission rate at each load, we can aggregate it to more user-friendly scales, like the counties discussed in this blog post.</p><h3>Applications of CarbonFlow</h3><p>CarbonFlow can be applied to use cases across carbon accounting, policy, and grid decarbonization applications. Singularity is working with partners including MISO and others on use cases including:</p><ol><li><strong>Location-based Scope 2 Emissions accounting. </strong>Data about the emissions intensity of consumed electricity allows consumers of electricity both big and small to account for the proportion of generated emissions they are responsible for, often called “location-based” carbon accounting. CarbonFlow could enable more granular and accurate location-based Scope 2 accounting, and also help inform more accurate residual mix estimates for market-based accounting. MISO worked with Singularity to explore nodal-level consumed carbon emissions via CarbonFlow for the first time.</li><li><strong>Quantifying clean energy deliverability. </strong>CarbonFlow can trace not only emissions but also, as demonstrated above, key drivers of emissions, including fuel mix and even power from individual generators. We’ve published a <a href="https://zenodo.org/record/8117207">preprint</a> introducing how tracing of individual generators can help answer whether a generator’s power is deliverable to a specific load, which informed our <a href="https://singularity.energy/boundaries-report">whitepaper</a> on how to define deliverability in clean energy markets.</li><li><strong>Data-driven insights for policymaking. </strong>Because nodal CarbonFlow results can be aggregated to any geographic resolution, they open new opportunities for accurate consumption-based insights at the city-level, county-level, state-level, or other administrative boundaries that don’t necessarily align with grid or utility boundaries. Such results could tell policymakers more about total consumed emissions within their jurisdiction, or transfers of emissions with neighboring jurisdictions.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abef2e6fa739" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/introducing-carbonflow-tracing-ghg-emissions-from-source-to-consumption-abef2e6fa739">Introducing CarbonFlow: tracing GHG emissions from source to consumption</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualizing electricity grid emissions data in Python]]></title>
            <link>https://medium.com/singularity-energy/visualizing-electricity-grid-emissions-data-in-python-778ff295e684?source=rss-c705923da308------2</link>
            <guid isPermaLink="false">https://medium.com/p/778ff295e684</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[renewable-energy]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Gailin Pease]]></dc:creator>
            <pubDate>Wed, 07 Feb 2024 16:32:27 GMT</pubDate>
            <atom:updated>2024-02-07T16:32:27.875Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://singularity.energy/">Singularity</a>, we develop tools to measure and analyze emission rates across the electrical grid. The amount of input data can be very large (the Eastern Interconnect, the transmission system that delivers electricity to customers from Maine to Florida to the Dakotas, has more than 50,000 connection points) and multidimensional, varying in time and space.</p><p>Over time we’ve developed a set of visualizations that help us quickly understand what’s happening in systems like this. They don’t tell us everything, but they let us get a big picture view, and help identify the places we need to explore further.</p><h3>The tools</h3><p>Before we can start visualizing electrical grid data, we need the right tools to structure it.</p><p>To work with electrical grid data, we use <a href="https://pypsa.readthedocs.io/en/latest/">PyPSA</a>, an open source power systems analysis library. PyPSA networks use Pandas dataframes to structure the data for each electrical grid component, which allows us to easily combine grid data from a PyPSA network with emissions data from other sources.</p><p>To format emissions data in this blog post, we’ll use Pandas. In reality, we’re often working with multiple types of data on the grid simultaneously, including fuel mix, CO2, and CO2 equivalent (a measure of the total warming potential of emitted CO2, CH4, and N2O), and we use <a href="https://xarray.dev/">xarray</a>, a multidimensional extension of Pandas.</p><p>Finally, we’ll need visualization tools. We use PyPSA’s matplotlib-based network visualization functions to visualize spatial patterns on the transmission grid topology. For our other plot styles (like the choropleths and heatmaps we show in this blog post), we use Plotly.</p><p>In this blog post, we’ll be visualizing a model of the 2020 US grid from Breakthrough Energy. You can read more about their grid modeling work <a href="https://breakthroughenergy.org/our-work/grid-modeling/">here</a>.</p><h3>The visualizations</h3><h4>1. Consumed emission rates on the transmission grid</h4><p>We start by visualizing consumed emission rates — that’s the emission rate associated with the electricity that a load, like a lightbulb or a data center, is using. We have a PyPSA network net and consumed_emissionsdata in units of lbs CO2e/MWh with a time index and bus names as columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x4ky0DMOqdf7D9EB" /></figure><p>We define a custom colormap that ranges in shade (light to dark) and from green to red to communicate increasingly emissions-intensive electricity:</p><pre>from matplotlib.colors import LinearSegmentedColormap<br>matplotlib_cmap = LinearSegmentedColormap.from_list(<br>   &quot;carbon&quot;,<br>   [&quot;#c2ffcd&quot;,&quot;#cdf0a5&quot;,&quot;#d8e07c&quot;,&quot;#e3d054&quot;,&quot;#edc02b&quot;,&quot;#c89421&quot;,&quot;#a26716&quot;,&quot;#7d3b0b&quot;,&quot;#570e00&quot;],<br>)<br>matplotlib_cmap.set_bad(&quot;lightgrey&quot;)</pre><p>Finally, we use PyPSA’s built in plot function to visualize the carbon intensity of each transmission system bus at one time, adjusting line_widths and bus_sizes as needed depending on the size of the network:</p><pre>this_time = consumed_emissions.index[0]<br>max_color=2000<br><br>net.plot(<br>       bus_cmap=matplotlib_cmap,<br>       bus_colors=consumed_emissions.loc[this_time, net.buses.index].values,<br>       bus_norm=matplotlib.colors.Normalize(vmin=0, vmax=max_color, clip=False),<br>       line_colors=&quot;lightgrey&quot;,<br>       line_widths=0.5,<br>       bus_sizes=.005,<br>       title=this_time.strftime(&quot;%h %d %H:00 UTC&quot;)<br>   )</pre><p>This gives us an image like this, showing each bus on the transmission grid:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xUEOJ_nmTMsR7rK5" /></figure><p>If we’re interested in more than one timestamp, we can save images for multiple snapshots and animate them using the imageio library:</p><pre>images = []<br>for f in image_file_names:<br>   images.append(imageio.imread(f))<br>imageio.mimsave(&quot;eastern_movie.gif&quot;, images, loop=0)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4VP4Ih6tXtnYap2W" /></figure><p>This shows us that during this day, there are high emission rates in the Midwest and lower emission rates in eastern and western regions of the grid. But there’s so much data here that it’s difficult to interpret overall trends, so our next step is to aggregate these results to policy-relevant spatial scales.</p><h4>2. Summarizing spatial trends</h4><p>Spatial averages of consumed emission rates can be a useful way to summarize trends. In the electricity world, there are spatial divisions with practical meaning: states have emission targets and regulations, and Balancing Authorities (BAs) control generation within their regions. However, it’s worth using caution when using spatial averages. A spatial average over a large region, like a state, will usually conceal more granular trends that would appear if you averaged over smaller regions, like counties.</p><p>To look at a month-long, BA-level average of the data we looked at above, we first calculate BA-wide consumed emission rates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/0*40T4nrMDmPD9XR6o" /></figure><p>Next, we use geopandas to read in a shapefile describing BA boundaries and merge in the emission rate data:</p><pre>bounds = gpd.GeoDataFrame.from_file(&quot;data/gis/Control__Areas/Control__Areas.shp&quot;)<br><br>bounds = bounds.merge(<br>   ba_emission_rates,<br>   how=&#39;left&#39;,<br>   left_on=&quot;NAME&quot;,<br>   right_index=True)<br><br># Remove any BAs outside of the eastern interconnect where we don&#39;t have data<br>bounds = bounds[~bounds[&quot;mean&quot;].isna()]<br><br>bounds = bounds.set_index(&quot;NAME&quot;)<br><br># Convert to lat/long coords for plotting<br>bounds = bounds.to_crs(&quot;WGS84&quot;)</pre><p>Finally, we can use Plotly’s built-in choropleth function to color each shape by its mean emission rate. Plotly can take a custom colorscale from a list of colors, and we use the same custom colorscale we used in the transmission system visualization. We add some custom styling to adjust the projection and add state and country boundaries, which make the map more interpretable.</p><pre>fig = px.choropleth(<br>   bounds,<br>   geojson=bounds.geometry,<br>   locations=bounds.index,<br>   color=&#39;mean&#39;,<br>   labels={&#39;mean&#39;:&#39;Consumed&lt;br&gt;emission rate&#39;},<br>   range_color=(0, 2000),<br>   color_continuous_scale=[&quot;#c2ffcd&quot;,&quot;#cdf0a5&quot;,&quot;#d8e07c&quot;,&quot;#e3d054&quot;,&quot;#edc02b&quot;,&quot;#c89421&quot;,&quot;#a26716&quot;,&quot;#7d3b0b&quot;,&quot;#570e00&quot;],<br>   width=700,<br>   height=450,<br>   title=&quot;August 2020&quot;<br>  <br>)<br><br><br>fig.update_layout(<br>       geo=dict(<br>           scope=&#39;north america&#39;,<br>           showland = True,<br>           landcolor = &quot;rgb(250, 250, 250)&quot;,<br>           subunitcolor = &quot;rgb(217, 217, 217)&quot;,<br>           countrycolor = &quot;rgb(217, 217, 217)&quot;,<br>           countrywidth = 0.5,<br>           subunitwidth = 0.5,<br>           resolution=50,<br>           showsubunits=True,<br>           projection_scale=2,<br>       )<br>   )<br><br><br><br><br>fig.update_geos(fitbounds=&quot;locations&quot;)</pre><p>The resulting image, below, shows that emission rate averages vary much less than the nodal rates we visualized above (the colorscales are the same in both). We can also see some trends in the averages that were less obvious in the highly granular data: the Midwest consumes on average higher emission power, while the Northeast has relatively low emission power. There’s also a pocket of particularly high emission power in South Carolina which might be worth further exploration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ieHqniQO0M06pUBLu29R5A.png" /><figcaption>Average consumed emission rate for each eastern interconnect balancing authority in August of a modeled year.</figcaption></figure><h4>3. Visualizing temporal trends</h4><p>So far, we’ve seen a granular visualization of emission rates at each transmission system bus at each hour, and a choropleth that captures average spatial trends. But what about easily viewing trends in time? For this, we use day-hour heatmaps.</p><p>First, we calculate consumed emission rates at some spatial scale (here, across New York):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*pWJv0bFBH02HFdQ8" /></figure><p>Next, we reshape the data so it’s indexed by hour (in Eastern time) and date:</p><pre>emission_rates[&quot;time_EST&quot;] = emission_rates.time.dt.tz_localize(&quot;UTC&quot;).dt.tz_convert(&quot;US/Eastern&quot;)<br>emission_rates[&quot;Hour (Eastern time)&quot;] = emission_rates.time_EST.dt.hour<br>emission_rates[&quot;Date&quot;] = emission_rates.time_EST.dt.date<br>emission_rates = emission_rates.drop_duplicates(subset=[&quot;Hour (Eastern time)&quot;,&quot;Date&quot;], keep=&quot;first&quot;) # Fall backward timezone change results in duplicated hour<br>emission_rates = emission_rates.pivot(columns=&quot;Date&quot;, index=&quot;Hour (Eastern time)&quot;, values=&quot;rate&quot;)</pre><p>This results in data that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5-__9Nsexm6zhLcp" /></figure><p>We can then visualize the data array as an image:</p><pre>fig = px.imshow(emission_rates, color_continuous_scale=carbon_colors,<br>        title=f&quot;New York State&quot;, zmin=0, zmax=1300, width=800, height=400)<br><br><br>fig.update_xaxes(<br>   dtick=&quot;M1&quot;,<br>   tickformat=&quot;%b&quot;)<br><br><br>fig.update_layout(<br>   coloraxis_colorbar=dict(<br>       title=&quot;Consumed rate&lt;br&gt;(lbs CO2e/MWh)&quot;,<br>   ),<br>)</pre><p>Which results in a plot like the one below. We can see a seasonal trend with higher emission electricity in the summer, and a daily trend with higher emission electricity in the evenings and mornings. The same precautions we mentioned when looking at the choropleth plot above are relevant here: the temporal trends we see at the New York-wide level might be different than the trends in a specific county. It’s always worth looking at temporal trends at a variety of spatial resolutions to get the full picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*56qMPDQc_U459FqI" /></figure><h3>Diving deeper</h3><p>The visualizations listed here are a good starting point, but they usually raise more questions than they answer. Once we’ve used these visualizations to get a broad picture of emission rates on the grid, we dive deeper into specific regions to answer our customer’s questions, and other tools can become useful. For example, bar and line charts can be useful to compare emission rates across regions, since they put emission rates on the y-axis instead of on the color axis. We also look at features like fuel mix and power flow to understand the drivers of emission rate trends.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=778ff295e684" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/visualizing-electricity-grid-emissions-data-in-python-778ff295e684">Visualizing electricity grid emissions data in Python</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LFEnergy Summit: Leveraging Technology to Realize Grid Decarbonization]]></title>
            <link>https://medium.com/singularity-energy/lfenergy-summit-leveraging-technology-to-realize-grid-decarbonization-fc157f9ce80?source=rss-c705923da308------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc157f9ce80</guid>
            <category><![CDATA[electricity]]></category>
            <category><![CDATA[climatetech]]></category>
            <category><![CDATA[emissions]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[renewable-energy]]></category>
            <dc:creator><![CDATA[Gailin Pease]]></dc:creator>
            <pubDate>Thu, 20 Jul 2023 16:52:14 GMT</pubDate>
            <atom:updated>2023-07-20T16:53:34.250Z</atom:updated>
            <content:encoded><![CDATA[<h3>LF Energy Summit: Leveraging Technology to Realize Grid Decarbonization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*sf6UFKrgYgmU2bWMM6bzLw.jpeg" /><figcaption>Our panel appearance at LF Energy. Photo from <a href="https://live.staticflickr.com/65535/52944523501_0d939816df_c.jpg">LF Energy</a>.</figcaption></figure><p>Last month, I had the exciting opportunity to represent Singularity Energy at the Linux Foundation Energy Summit in Paris, France. I was able to present some electricity delivery research that our team has been working on over the last few months (watch the panel <a href="https://www.youtube.com/watch?v=yCILZ7Fe--g&amp;list=PLKyFf1J9XkpuhYr-WmOHfHgHYC7jA-ohT&amp;index=20">here</a>). We also got to hear from experts across software, power systems operators, and academia about everything they’re doing to further grid decarbonization. Here are some of the most exciting trends I noticed during the conference:</p><p><strong>Power system operators are seeing the benefits of using open software</strong>.</p><p>Throughout Europe, power system operators at all scales, from distribution operators in individual cities to operators in charge of continent-scale coordination, are starting to use open software. This shift brings exciting opportunities for collaboration and innovation. Two examples of this open software adoption on display at the LF Energy Summit were SOGNO and PowSyBl:</p><ul><li><a href="https://www.linkedin.com/in/antonellomonti/">Antonello Monti</a>, professor at RWTH Aachen, shared how the open <a href="https://www.sogno-energy.eu/">SOGNO</a> platform has been deployed in Rome’s distribution grid to provide flexible demand. Deployment on a real grid is no easy feat: it requires interfacing with legacy data systems, providing a high level of reliability, and enabling a market for grid actors to buy and sell demand flexibility. But along with the challenges of implementation come opportunities, like a new project where the SOGNO platform could be adopted for use in EV charging hotspots. Check out the slides <a href="https://static.sched.com/hosted_files/lfenergysummit2023/8b/SOGNO_LFESummit2023.pdf?_gl=1*1gh5b7w*_ga*MTQ2NzAxODU3Ni4xNjgwMTk3NjM1*_ga_XH5XM35VHB*MTY4OTM2MTk5OS4xLjEuMTY4OTM2MjE5OC4wLjAuMA..">here</a>.</li><li><a href="https://www.linkedin.com/in/nicolasomont/">Nicolas Omont</a>, from Artelys, shared how Europe’s Regional Coordination Centers (RCCs), which are responsible for ensuring safe, coordinated grid operations across Europe’s many grid operators, are using the open source <a href="https://www.powsybl.org/">PowSyBl</a> to run security analysis of the European system. Check out his slides <a href="https://static.sched.com/hosted_files/lfenergysummit2023/48/Artelys%20-%20LFESummit%20-%202023-06-01.pdf?_gl=1*b6f414*_ga*MTQ2NzAxODU3Ni4xNjgwMTk3NjM1*_ga_XH5XM35VHB*MTY4OTM2MTk5OS4xLjEuMTY4OTM2MjE5OC4wLjAuMA..">here</a>.</li></ul><p><strong>Open software for power systems is good and getting better.</strong></p><p>Power systems are an essential backbone of our world. That means that real-world deployment of open software like SOGNO and PowSyBl is only possible if the tools are reliable, scalable, and secure. The Linux Foundation provides administrative and financial structure for the open source projects it hosts, enabling them to achieve the reliability essential to real-world deployment.</p><p>The Linux Foundation’s growing collection of energy-related projects cover everything from grid modeling and operational tools like PowSyBl and SOGNO to data standards like the Carbon Data Specification that aim to make carbon data intelligible across tools and platforms. The diversity of LF Energy projects illustrates an important fact of the power system software landscape: decarbonizing the grid is complicated, and there’s not one tool that can do it all. Instead, the LF Energy Summit showed how powerful it can be to have a suite of tools and a community of experts applying those tools to real-world grid decarbonization.</p><p><strong>Sometimes it’s about choosing the right tool for the job, not making the right tool for all jobs</strong></p><p>At Singularity, some of our work requires running simulations of power grids. There are lots of tools to do this work, from the expensive proprietary solutions often used by grid operators in the U.S. to research tools like PyPSA used in university labs and designed to be easy-to-use. The wealth of options invites a simple question: what’s the difference, and which one is the right one to use?</p><p>The LF Energy Summit reframed this abundance of tools for me. There isn’t one right tool for all projects — instead, it’s all about choosing the right tool for your specific use case. PyPSA is a research tool whose simplified models are perfect for capacity expansion modeling, which can answer big questions about how to best achieve grid decarbonization in the decades to come. PowSyBl, on the other hand, is built for deployment in power systems. It might not be as easy to model the impact of a still theoretical battery technology (a perfect use case for PyPSA!) but it <em>is</em> possible to study, with high accuracy, the security of the entire European transmission system.</p><p><strong>Research is essential for making informed decisions.</strong></p><p>At the LF Energy Summit, I presented some of Singularity’s recent research on a panel hosted by Google’s Hallie Cramer, alongside Ana Radovanovic (Google) and Iegor Riepinwith (TU Berlin). You can check out the full presentation <a href="https://www.youtube.com/watch?v=yCILZ7Fe--g&amp;list=PLKyFf1J9XkpuhYr-WmOHfHgHYC7jA-ohT&amp;index=20">here</a>.</p><p>Singularity has been using our CarbonFlowTM technology to trace the physical delivery of power through the transmission system from generators to loads. We’ve found that most generation is consumed close to where it’s generated, but that the locations of big load centers can change patterns of delivery. These limited locations of delivery have implications for electricity procurement as electricity consumers try to reduce their emissions. It was exciting to talk on the panel with Iegor, who uses capacity expansion modeling to study the impact of procurement decisions on grid decarbonization, about how different electricity procurement guidelines based on location or time-matching might impact the system as a whole.</p><p>Throughout the panel, one theme kept coming up: to decarbonize the grid, we need robust data and research to ensure we’re making effective changes. The hard work of all the attendees at the LF Energy Summit is helping us make data-driven strides towards a cleaner future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc157f9ce80" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/lfenergy-summit-leveraging-technology-to-realize-grid-decarbonization-fc157f9ce80">LFEnergy Summit: Leveraging Technology to Realize Grid Decarbonization</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Research computing in a startup with Celery]]></title>
            <link>https://medium.com/singularity-energy/research-computing-in-a-startup-with-celery-ed934064e4b7?source=rss-c705923da308------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed934064e4b7</guid>
            <category><![CDATA[renewable-energy]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[electricity]]></category>
            <dc:creator><![CDATA[Gailin Pease]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 13:06:45 GMT</pubDate>
            <atom:updated>2023-03-16T13:06:45.320Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*-Gaxvmz5u95OBglDgZd3zA.png" /></figure><h3>Why Celery?</h3><p>At Singularity, we work with large and complex data about the electricity grid, and we need to be able to quickly iterate on our analysis and modeling. Sometimes this data is huge; a recent project requires us to run multiple models, each producing more than 500 GB of data, and synthesize the results into actionable insights.</p><p>Here’s how we use <a href="https://docs.celeryq.dev/en/stable/getting-started/introduction.html">Celery</a> to make that possible.</p><p>In an academic research setting, computationally intensive research tasks like this would usually be handled on a dedicated research computing cluster, with a job manager (for example, <a href="https://slurm.schedmd.com/documentation.html">Slurm</a>) and networked storage. Jobs too large to run on a single machine could use a message passing interface (for example, <a href="https://mpi4py.readthedocs.io/en/stable/intro.html#what-is-mpi">MPI</a>). We needed similar capabilities, but because we’re a startup, we also wanted a system that we could eventually transform into a production system.</p><p>Enter Celery. We use Celery in our production applications to run slower jobs in the background, where they won’t affect the user experience. Celery makes it easy to use a distributed task queue, so it’s well suited to exactly this task, providing a simple interface for assigning work to many machines. We wanted to make Celery work for our research as well. In some cases, we were able to directly take our production Celery infrastructure. In other cases, our research needs required us to build new patterns to make Celery work for us.</p><h3>Making Celery work for us</h3><h4>Celery with ECS</h4><p>We use <a href="https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/sqs.html">SQS as our Celery message broker</a>. This means that we can leverage the same infrastructure management tools for research computing as we use in production, including Github actions and Terraform. Separately managing a research computing cluster, even one <a href="https://aws.amazon.com/blogs/hpc/introducing-aws-parallelcluster-3/">hosted by AWS</a>, would have introduced overhead, as each team member working on our research code would have to learn another set of tools.</p><p>A key benefit of deploying research tasks using Celery is the ability to run enormous jobs in parallel. Since we don’t want to pay for the large instances needed to run research tasks all the time, we use <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html">autoscaling rules</a> to create more instances as needed, and remove them when a job is finished. Currently, we’re using AWS’s built in CPU-based autoscaling rules. In the future, we plan to write custom autoscaling rules, which will allow us to scale based on queue size and customize scaling to meet our research needs in a cost-effective way.</p><h4>Capturing shared dispatch functionality</h4><p>With many research jobs running remotely and in parallel, some of which are dependent on others, it’s important to have consistent rules around logging and error handling. To handle this, we were able to adapt a pattern from our production infrastructure (which demonstrates how useful it can be to use the same tools across our organization!).</p><p>If we wrote error handling and logging in each task, it might look like this (in pseudocode):</p><pre>def my_task(model_name, weather_year):<br>  init_logger()<br> <br>  try:<br>    run_analysis()<br>  except MaxRetriesExceededError:<br>    # Log info about retry exceeded task<br>  except NetworkRelatedError:<br>    # Log and raise error so Celery retries<br>  except DeterministicMathError:<br>    # Log error but don’t retry, since this won’t be fixed<br>  except Exception:<br>    # Log and raise any other error</pre><p>In most cases, the types of errors and their handling are consistent across tasks. The errors we see in a research environment and the way we want to handle them are different than in our production environment — in a research environment, we expect jobs to fail, perhaps because a matrix is not invertible or some data was zero that shouldn’t have been. We don’t want to re-try, because we know these cases would always fail. We can capture all of this logic in a wrapper:</p><pre>def resilient_job(fn):<br>  @wraps(fn)<br>  wrapper(*args, **kwargs):<br>    init_logger()<br> <br>    try:<br>      fn(*args, **kwargs)<br>    except MaxRetriesExceededError):<br>      # Log info about retry exceeded task<br>    except NetworkRelatedError):<br>      # Log and raise error so Celery retries<br>    except DeterministicMathError):<br>      # Log error but don’t retry, since this won’t be fixed<br>  return wrapper<br><br>@worker.task()<br>@resilient_job<br>Def task(model_name, weather_year):<br>  run_analysis()</pre><h4>Managing file storage in Celery</h4><p>Research file storage has to be extremely flexible. We don’t know at the start of a project what our data access patterns will be; often, we don’t even know what types of data we’ll end up producing and consuming. As we experiment, we want to be able to quickly create, inspect, and consume new types of data.</p><p>Our first approach to this was simple: in each Celery task, we simply downloaded the data we needed and uploaded the results at the end. In pseudocode, this looks like:</p><pre><br>def task(model_name, weather_year):<br>  download(model_name)<br>  download(weather_year)<br> <br>  elec= run_analysis()<br><br>  write(elec)<br>  upload(elec)<br><br>  clean_up(model_name)<br>  clean_up(weather_year)</pre><p>While this works the first time, once you’ve scaled to half a dozen jobs, many of which use the same inputs and outputs, you end up with a lot of repeated upload and download code, much of which contains hard coded logic about where data is stored.</p><p>We address this problem by using a wrapper that declares what files a task will output and detects from task parameters what files it needs:</p><pre>def job_file_supplier(job_creates):<br>  def wrap_job(fn)<br>    @wraps(fn)<br>    wrapper(*args, **kwargs):<br>      for file_needed in kwargs:<br>        download(file_needed) <br>    fn(*args, **kwargs)<br>    for created in job_creates:<br>      upload(created)   <br><br>    for needs in kwargs: <br>      clean_up(needs)<br>  return wrap_job<br><br>@job_file_supplier(elec_data)<br>def task(model_name, weather_year):<br>  elec = run_analysis()<br>  write(elec)</pre><p>Now the wrapper captures file upload and download consistently across tasks. As long as function argument declarations encode necessary files consistently across tasks, each data type can be handled consistently by the wrapper. Of course, this is still not as robust as a production database, but it allows us flexibility. In our research, that flexibility is a necessity, but it doesn’t need to come at the cost of unmaintainable, hardcoded file access patterns.</p><h3>What’s next</h3><p>In this post we’ve covered how we’re currently supporting our research computing needs — but at a startup like Singularity, our requirements are constantly changing, and so our systems will continue adapting. If you’re interested in what we’re doing, or if you think you could help us do it better, <a href="https://www.linkedin.com/company/singularity-energy/">keep up to date with when we’re hiring</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed934064e4b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/research-computing-in-a-startup-with-celery-ed934064e4b7">Research computing in a startup with Celery</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Validating real-time electricity emissions rates with an hourly historical benchmark]]></title>
            <link>https://medium.com/singularity-energy/validating-real-time-electricity-emissions-rates-with-an-hourly-historical-benchmark-a9990a2c9049?source=rss-c705923da308------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9990a2c9049</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[emissions-reduction]]></category>
            <category><![CDATA[climate-change]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[electricity]]></category>
            <dc:creator><![CDATA[Gailin Pease]]></dc:creator>
            <pubDate>Tue, 20 Sep 2022 12:36:37 GMT</pubDate>
            <atom:updated>2022-09-21T19:09:54.574Z</atom:updated>
            <content:encoded><![CDATA[<h3>Validating near real-time electricity emissions rates with an hourly historical benchmark</h3><p>Electricity generation is responsible for <a href="https://www.epa.gov/ghgemissions/sources-greenhouse-gas-emissions">one-quarter of carbon emissions in the U.S.</a>, so many companies, cities, and policymakers who care about climate change have focused their efforts on reducing emissions from their electricity use. Real-time data about the emissions intensity of grid electricity, such as those available through Singularity’s <a href="https://carbonara.singularity.energy/">Carbonara</a> platform, is one important tool that empowers consumers to understand their emissions and make timely decisions about their electricity use to decrease emissions.</p><p>In one use case, Singularity and Sense collaborated to demonstrate that carbon-aware decision making can <a href="https://sense.com/whitepapers/Sense-EV-Carbon-Research.pdf">reduce the carbon impact of electric vehicle charging by up to 43%</a>. Accurate, real-time emissions data also enables more timely accounting of scope 2 emissions, which will be increasingly important for enforcement of emissions regulations and reporting of corporate climate risk. Realizing these benefits depends on the accuracy of real-time emission rate estimates, but how confident should we be in their quality?</p><p>Singularity Energy has performed what we believe is the first validation of near real-time hourly emissions data against a historical benchmark. This validation was made possible by Singularity’s recently released <a href="https://singularity.energy/open-grid-emissions">Open Grid Emissions Initiative</a> (OGE), an open-source dataset of historical, validated grid emissions that we believe represents the most accurate, comprehensive, and high-resolution grid emissions dataset for the U.S. available to date. <strong>Our analysis revealed that Singularity’s near real-time emissions data is accurate for the vast majority of electricity consumers in the United States.</strong> In the regions where we found issues with near real-time data, the comparison has already informed directions for future work on improving near real-time data in those regions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ashST1RZMuCZGC7o" /><figcaption>Figure 1: In the New York ISO (NYISO), real-time emission rates agree in trend and magnitude with the benchmark provided by OGE. However, there is a larger difference in magnitude of emission rates in the summer (right panel) than in the winter and spring (left panel), indicating that changes in emission rates throughout the year may not be fully captured by real-time data.</figcaption></figure><h3><strong>Why might real-time estimates differ from benchmark data?</strong></h3><p>To understand why real-time grid data may differ from benchmark data, it’s useful to understand where real-time emission intensities come from. Singularity multiplies near real-time net generation data for each regional generation fleet (natural gas, coal, wind, etc) by an annual-average, fleet-specific generated emission factor from the most recent year of the EPA’s eGRID database, resulting in emissions intensities available as soon as generation data is published (hence our use of “near real-time” to refer to the data compared here; though the same insights apply to true real-time data calculated using the same methodologies). In contrast, the Open Grid Emissions benchmark emissions data is mostly from measured hourly plant-level emissions supplemented by reported fuel consumption data.</p><p>There are several reasons why near real-time emission rate estimates may be inconsistent with benchmark data:</p><ol><li>The near real-time net generation data reported by grid operators may sometimes be incomplete, with grid operators missing or mis-reporting some fraction of generation on their grid.</li><li>The broad fuel categories used to describe real-time generation data may include several energy sources with different emission rates. For example, biomass, geothermal, and waste would all be grouped into the “other” in EIA’s Hourly Electric Grid Monitor, one of the main sources of real-time generation data.</li><li>The emission factor we use to convert generation to emission rates is specific to the generators of each fuel type in each region, but is an annual-average value. Using a constant value doesn’t allow us to capture seasonal or daily variation in fleet emission intensity.</li><li>We source the emission factors from eGRID, which is released with a 1–2 year lag. This means that the emission factor will not reflect recent changes in each fleet, for example, if older, dirtier plants of a fuel type have been retired.</li></ol><p>Real-time data is both vital for decision making and inherently limited by the data available in real time. Without comparing to a validated benchmark dataset, we can only guess how much the limitations of real-time data affect the quality of that data. Without an accurate picture of data quality, decision makers may have limited confidence in real-time data which results in less effective decisions. We hope that this validation increases confidence in real-time emissions reductions and informs improvements of real-time data.</p><h3><strong>How did we validate real-time estimates?</strong></h3><p>Accurate real-time emissions data should reflect both the hourly profile of emissions and the total magnitude of emissions. For applications like load shifting and energy storage, an accurate hourly shape is the most important feature of real-time data, since it represents which hours are relatively cleaner or dirtier than others. For scope 2 carbon accounting, an accurate emission magnitude is the most important feature, so that grid emissions can be correctly allocated.</p><p>To determine if a real-time dataset meets these criteria, we need an hourly picture of true grid operations, made freely available for the first time by Singularity Energy’s Open Grid Emissions Initiative. The first release of this dataset covers 2019 and 2020, and builds on decades of work in understanding the electric grid, including the EPA’s work on the<a href="https://www.epa.gov/egrid"> eGRID emissions database</a> and Catalyst Cooperative’s work to make U.S. energy data uniform and accessible via<a href="https://catalyst.coop/pudl/"> PUDL</a>. To learn more, and to hear about use cases outside of validation, check out our <a href="https://medium.com/singularity-energy/introducing-the-open-grid-emissions-initiative-42f68f3b3f49">announcement blog post</a>.</p><p>Figure 1 compares near real-time and benchmark data for 2020 in NYISO, the grid operator serving New York state. The shapes of the two profiles match well throughout the year, but the magnitude is different, with larger magnitude errors in near real-time data visible throughout the summer. To capture these differences, we use several metrics to evaluate how well near real-time estimates match OGE benchmark data in magnitude and shape.</p><p>To evaluate how accurate the hourly magnitude of the data is, we calculate median absolute error and median absolute percentage error, which reflect how far each hourly estimate is from the benchmark value. To evaluate how closely the shape of real-time data matches the benchmark, we calculate the correlation of the two time series across the entire year. A correlation coefficient of 1 indicates that the shapes match exactly, and a coefficient of -1 means that the shapes are opposite (e.g. near real-time data shows that emissions are relatively high when they are actually relatively low). A coefficient of zero means that there is no relationship between the two shapes. Ideal real-time data would have median absolute and percentage error close to zero and correlation close to 1.</p><p>For this comparison, we used near real-time emission rates calculated from EIA-930 generation data using Singularity’s methodology. We use EIA-930 data because it is the only public near real-time data source available across the entire continental U.S. In some regions, more accurate and/or more timely real-time data may be available directly from grid operators; however, the same methodological limitations will affect real-time data regardless of source. We evaluate near real-time data after outlier detection via a rolling filter, since some regions report extreme values during periods of missing data that have an outsized effect on correlation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DwNyv1SS5Eb_IGUU" /><figcaption>Figure 2: Two of the metrics we use to evaluate the accuracy of real-time data plotted against BA size. Real-time data in larger BAs generally has better accuracy than in smaller BAs, both in magnitude (difference) and shape (correlation). Ideal real-time data would have correlation of 1 and difference 0 when compared to benchmark data (vertical lines).</figcaption></figure><h3><strong>Results of our validation</strong></h3><p>When we compared Singularity’s real-time grid emissions data to benchmark OGE data, we found that the near real-time data agreed with the benchmark in most hours and regions. We found that emissions intensities match well in larger balancing authorities (BAs), including California ISO (CISO), the Midwest ISO (MISO), and the New England ISO (ISNE).Figure 2 demonstrates this trend, with larger BAs having consistently high correlation and low differences between near real-time and benchmark data. This means that most electricity consumers can feel confident when making decisions using real-time grid data.</p><p>The absolute average difference in emission rate across all U.S. BAs (weighted by BA generation) is 39.5 lb CO2/MWh, only 6.8% of the average benchmark emission rate. The average difference is negative, meaning that on average, near real-time estimates are slightly cleaner than benchmark data, although this varies by BA (see Table 1). The average correlation across BAs (weighted by BA generation) is .88, indicating that the shape of real time data matches well with benchmark data. Many large BAs actually perform better than these averages (Table 1). When making decisions based on real-time data, customers should consider real-time data quality in their specific region.</p><p>Two large BAs, PJM and BPAT, have relatively worse metrics, indicating that they are further in shape (PJM) and magnitude (BPAT) from benchmark data. In the case of PJM, a large grid operator in the mid-Atlantic, the relatively low correlation is due to low-quality near real-time data in January, 2020. Excluding January 2020, the correlation between real-time and benchmark data in PJM is .94. In BPAT, a grid operator in the Northwest with primarily hydro generation, consistently underestimated near real-time emission rates are caused by a single coal power plant. Although this plant supplies power throughout the year, especially during fall and winter, BPAT reports no coal generation in near real-time data, artificially lowering real-time emission rate estimates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AUGhiMcyO7ynqRa-CgbsSQ.png" /><figcaption>Table 1: Data quality metrics comparing Singularity’s real-time and benchmark OGE data for the 12 largest BAs by annual generation (find data for all BAs at the bottom of this post). Negative rate differences indicate that real-time data underestimates emissions, while positive differences indicate that real-time overestimates emissions. Ideal real-time data would have a rate difference of zero and a correlation of 1. All comparisons are made using generated emission rates over 2020.</figcaption></figure><p>In small balancing authorities, the picture is more mixed. Smaller BAs have more frequent and longer reporting gaps in near real-time data, perhaps because they have fewer resources to dedicate to reporting. In addition, there are inherent limitations to applying real-time methodologies in a small region. The same sources of noise present in all real-time data cause larger problems in smaller BAs. Temporal and inter-plant variation in emission rates may be more important in small BAs, where only a few plants are operating in each hour. In large BAs, generation from a fuel type represents generation over dozens of plants in each hour, so the emission rate over those plants will remain close to the average emission rate for that fuel type. In a BA with only a handful of plants of each fuel type, variation in the emission rates of each plant has a larger impact on the average emission rate.</p><h3><strong>Future directions</strong></h3><p>Singularity continually seeks to improve our real-time data, and we are excited about the opportunities Open Grid Emissions provides to further that goal. In order to improve the quality of the near real-time net generation data that we use for this calculation, we’re working with the EIA to improve the data balancing authorities submit to EIA’s<a href="https://www.eia.gov/electricity/gridmonitor/"> Hourly Electric Grid Monitor</a>. Our work on Open Grid Emissions has revealed some cases where balancing authorities submit data with the wrong time stamps, or submit transmission data incompatible with their generation and demand data. By working directly with the EIA to fix these data issues, we ensure that all users will have the best possible near real-time data, enabling us to work together towards grid decarbonization.</p><p>This validation also revealed opportunities to improve the fleet emissions factors that we are using to convert net generation to emissions. For example, for some fleets, there is strong seasonal or hourly variation in the carbon intensity of generation, suggesting that we could potentially improve the accuracy of our real-time estimates by using month-specific or monthly time-of-day specific fleet emission factors from OGE. Figure 3 compares the benchmark hourly natural gas emission rate to eGRID’s annual natural gas emission rate, used to calculate real-time emission rates. There is significant variation in the benchmark natural gas emission rate, with consistently higher rates than eGRID during the summer, which may explain some of the discrepancy between real-time and benchmark data shown in Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m39v-ysdaUD7hjTS" /><figcaption>Figure 3: Hourly vs. annual natural gas emission rates in NYISO, the New York ISO. Hourly emission rates have significant variability around eGRID’s annual average emission rate, used to calculate real-time emission rates. Large peaks (eg, May 21, 2020) are likely due to plant start-up.</figcaption></figure><p>Furthermore, because the OGE dataset uses higher-resolution input data than eGRID, we’ve found that even on an annual basis, the OGE fleet emissions factors may be more accurate than those reflected in eGRID. Moving forward, we may switch to using OGE data as the source of these fleet emission factors.</p><p>In the coming months, Singularity will be working to incorporate these insights into our real-time and near real-time data offerings. As the Open Grid Emissions initiative continues, we will continue to look for opportunities to improve our data, helping our customers work towards their climate goals.</p><p><strong>Appendix: full comparison of results across all balancing authorities</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Z5HuKgo09KkwyzIlFwcag.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fR78lXRgPWP0a3Dfsjpn6g.png" /><figcaption>Table 2: Results across all balancing authorities.</figcaption></figure><p>Table notes:</p><p>1. HGMA and HST are not included because they have zero 2020 generation in OGE data.</p><p>2. JEA, EEI, and NSB are not included because their real-time emission rates are not available from Singularity’s API.</p><p>3. CHPD, GCPD, DOPD, WAUW, SEPA, YAD, WWA, GWA, CPLW, and HGMA are hydro-only balancing authorities with 0 lb/MWh CO2 emission rates in OGE. In some cases, real-time emission rates are higher because the BA reports non-hydro generation to EIA-930. Because the emission rate in these BAs is zero, we cannot calculate a correlation or percentage difference.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9990a2c9049" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/validating-real-time-electricity-emissions-rates-with-an-hourly-historical-benchmark-a9990a2c9049">Validating real-time electricity emissions rates with an hourly historical benchmark</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>