<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Drew Hill on Medium]]></title>
        <description><![CDATA[Stories by Drew Hill on Medium]]></description>
        <link>https://medium.com/@hilldrew?source=rss-c97cdb307306------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*wmy6DSG98-RCg3_Om8WhXA.jpeg</url>
            <title>Stories by Drew Hill on Medium</title>
            <link>https://medium.com/@hilldrew?source=rss-c97cdb307306------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:10:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hilldrew/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Gaining clarity into low-cost sensors]]></title>
            <link>https://medium.com/ramboll-shair/gaining-clarity-into-low-cost-sensors-4e4d2ec3ca6a?source=rss-c97cdb307306------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e4d2ec3ca6a</guid>
            <category><![CDATA[environment]]></category>
            <category><![CDATA[air-pollution]]></category>
            <category><![CDATA[public-health]]></category>
            <category><![CDATA[sensors]]></category>
            <category><![CDATA[air-quality]]></category>
            <dc:creator><![CDATA[Drew Hill]]></dc:creator>
            <pubDate>Wed, 18 Dec 2019 22:47:12 GMT</pubDate>
            <atom:updated>2019-12-20T23:03:18.503Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>By: </strong><a href="https://www.linkedin.com/in/kate-lewis-03388298/"><strong>Kate Lewis</strong></a><strong>, </strong><a href="https://medium.com/u/c7e57f72f9e0"><strong>Libby Koolik</strong></a><strong>, </strong><a href="https://medium.com/u/15174d94e0a5"><strong>Julia Luongo</strong></a><strong>, and </strong><a href="https://medium.com/u/c97cdb307306"><strong>Drew Hill</strong></a></p><p>Low cost sensor-based monitors can provide insight into fluctuations in air quality at a local scale. The reliability and utility of these insights, however, can vary substantially between sensor makes, models, and types.</p><p>The Shair team is exploring the ability of low-cost PM2.5 sensors to reproduce measurements from regulatory grade equipment. Regulatory grade equipment describes devices that are recognized by the Environmental Protection Agency (EPA) as the national standard for consistent, reliable air quality data — such as Beta-Attenuation Monitors (BAM). So why doesn’t everyone use a regulatory monitor to measure local air quality?</p><p>These devices and their maintenance are very expensive — approximately $15,000 to $60,000 per monitor, upwards of $200,000 for initial site development for EPA compliance, and a considerable amount of employee time in operation and compliance-related expenses according to a recent presentation by a member of the Bay Area Air Quality Management District. It is not economically feasible to implement an EPA-compliant BAM on every street corner. Low-cost sensors that perform well against regulatory grade equipment open the door for everyone, including those in disadvantaged communities, to access real-time, local air quality information.</p><p>The Shair team is currently working with <a href="http://www.groundworkrichmond.org/">Groundwork Richmond</a> to install 50 such low-cost sensor-based monitors across the <a href="https://opendata.ci.richmond.ca.us/stories/s/jqp5-q3g4">City of Richmond</a> as part of a Community Air Grant funded by the <a href="https://ww2.arb.ca.gov/homepage">California Air Resources Board</a> (CARB). Before we outfit this monitoring network and begin to make air quality information available to the public, we want to know exactly how the low-cost monitors perform, especially in relation to the more-expensive equipment traditionally relied upon for regulatory and health science purposes.</p><p>To do so, we partnered with the <a href="https://medium.com/u/9f1c03cd92f6">Clarity Team</a>, which that produces the low-cost sensors deployed by <a href="http://www.groundworkrichmond.org/air-rangers.html">Groundwork Richmond</a> across the City of Richmond, and CARB’s Advanced Monitoring Techniques Section to test Clarity Node-S units against the tried and trusted BAM. Along the way, we had the opportunity to perform a collocation analysis with several other low-cost sensor-based monitors. A traditional collocation analysis places one or more monitors in the same location as a regulatory grade monitor in order to assess and, often, correct for differences.</p><p>On May 24, 2019, the Shair team deployed 43 Clarity Nodes at a CARB rooftop monitoring location in Sacramento, CA and recorded the results (3 of these devices were later removed during analysis based on outlying data). In addition to our Clarity Nodes, this collocation included one BAM, two Environmental BAMs (eBAM) — a more portable BAM intended to produce reliable estimates on a daily average basis rather than an hourly or minute-by-minute basis— and several Purple Air low-cost-sensor-based monitors. Collocation was concluded on July 5, 2019.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*EhA_wkiywXeV_nN7tdgwOw.png" /><figcaption>CARB’s T Street location rooftop monitoring location in Sacramento, CA — where the collocation was performed.</figcaption></figure><h4>What was this collocation designed to achieve?</h4><p>The original goal of this study was to collect air quality data using Clarity Nodes and determine how well they perform compared to the BAM. To our surprise, we were fortunate that the rooftop monitoring station happened to play host to an eBAM (Environmental Beta-Attenuation Monitors) and 6 PurpleAir monitors to which CARB allowed us data access. This allowed us to take part in a collocation analysis in which the readings of Clarity Nodes, BAMs, an eBAM, and several PurpleAir monitors could all be compared. eBAM monitors are marketed as a portable, scalable derivative of the BAM. PurpleAir is a low-cost sensor based air quality monitor that collects data using the same internal sensor hardware as the Clarity Nodes — for this reason, having a set of PurpleAir monitors allowed for a sanity check of our Clarity-BAM comparison data as well as an opportunity to see how our monitor compared to another widely popular low-cost sensor-based monitor. We did not expect much difference given that they use the same sensor. Conversely, the eBAM Monitors collect data using a beta-attenuation technique, which is expected to result in slightly different measurements of concentration than the Clarity Nodes and PurpleAir. In theory, this monitor operates similarly to the BAM, but is often observed as less precise, especially at lower pollution concentrations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*Swt8KhTwsN-tl1lzZhgtpw.png" /></figure><h4>Comparing Each Low-Cost Sensor to the BAM</h4><p>The Shair and Groundwork teams installed Clarity Nodes on the rooftop of CARB’s T Street location in Sacramento, CA and collected data from May 24th to July 5th, 2019. These Clarity Nodes took measurements approximately every 17 minutes. The 6 PurpleAir monitors to which we had data access collected data at a faster average sampling rate of once every 2 minutes, while the eBAM and the regulatory grade BAM provided readings representing an hourly average. (Note: For this analysis, we isolated the “primary” data stream from Channel A from each Purple Air air quality monitor)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/339/1*NEGUza9wEDla75NzSuHfLA.png" /><figcaption>Clarity Node-S monitors lined up along CARB’s rooftop railing as part of the collocation.</figcaption></figure><p>As per the original intention of our collocation, data collected by the Clarity Nodes were made into hourly average values and then compared to hourly data produced by the BAM. As an added bonus of the collocation study, we were able to compare hourly averaged data from the PurpleAir monitors and data from the eBAM to hourly BAM output. For the purpose of this blog post, we report preliminary results (subject to change) from a single month-long period during this collocation: June 5, 2019 through July 5, 2019 (UTC). The figure below shows the concentrations measured by all relevant Clarity Nodes and PurpleAir monitors as well as the eBAM and BAM over this month-long period.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*TAHaESZdCubMzk1xRMjtgg.png" /><figcaption>PM2.5 concentrations measured by the Clarity Nodes, PurpleAirs, eBAM, and BAM over a month during the collocation. Note the impressive spike in concentrations around July 4th (fireworks and grills!).</figcaption></figure><p>The Shair team determined minimum, mean, and maximum hourly average concentrations for the Clarity, PurpleAir, and eBAM monitors, normalized to the respective regulatory monitor with a simple multiplicative scaler. The PurpleAir monitor had the strongest correlation to the BAM, with an average r2 value of 0.65. Despite the use of the same internal sensor, the Clarity Node units demonstrated a considerably lower average r2 value of 0.52. The eBAM unit that we used in this analysis showed particularly poor hourly correlation with BAM output with an r2 of 0.09. (However, an analysis of daily correlation was, as expected, much higher at r2 = 0.88. Note: we would also expect the r2 values for the BAM comparisons of both the PurpleAir and the Clarity Node-S monitors to increase when compared at the daily average scale.)</p><p>As with any study, it is important to be aware of the major sources of error. Generally, air quality monitors are able to record measurements more accurately when ambient concentrations are relatively high (&gt; 10 ug/m3, for example) In this case, the ambient concentrations of particulate matter in the Sacramento air were relatively low at about 6 ug/m3 on average, which is ultimately good for the residents of Sacramento, but proved to be a source of trouble for our study. It appears all monitors were affected by this issue, especially the eBAM, which is known to produce particularly low-precision hourly estimates.</p><h4>An Unexpected Outcome</h4><p>The discrepancy observed between the correlation of PurpleAir monitors to BAM data (r2 = 0.65) and the correlation of Clarity Nodes to BAM data (r2 = 0.52), which use the same sensor hardware to determine particulate matter concentrations, is unintuitive. In general, PurpleAir performed better against the BAM than the Clarity Nodes. <em>Why?</em></p><p>With all the additional data we obtained from the rooftop in Sacramento, we decided to dig into the differences between the Clarity Nodes to the PurpleAir monitors. Addressing this discrepancy early in the process allowed us to apply our findings before the Richmond study was completely underway. One major difference that we observed between the two monitors is the sampling frequency. The number of samples a sensor collects every hour is a major contributor to the quality of the data recorded — more samples per hour will reduce error but also produce a more temporally-representative average of the concentration during that hour. The sensors in the PurpleAir sampled the air almost nine times more frequently than those in the Clarity Nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*9AHeFcSUun1fWnL_x-zwNA.png" /><figcaption>The distribution of correlations (r2 values) between hourly average estimates produced from individual monitors as compared to BAM output and delineated by type.</figcaption></figure><p>As expected, we found that higher sampling frequencies resulted in less error during hourly averaged assessments. Given their higher sampling frequencies, PurpleAir devices may produce more reliable average hourly concentrations than Clarity Nodes.</p><p>To confirm our suspicion that the sampling frequency was affecting the results, we ran a simulation exploring what would happen if the PurpleAir device sampled about as frequently as the Clarity Nodes. To do this, we created a new artificial hourly average, which included only one out of every nine PurpleAir samples. This set the PurpleAirs’ effective sample rates at about the same value as the Clarity Nodes’. We then ran the same correlation analysis and found that the correlation between the new “resampled” PurpleAir and the BAM was lower than it was previously. In fact, the average r2 of the PurpleAir correlation to the BAM decreased to 0.53 — nearly identical to the average r2 of the Clarity Node. Additionally, as shown in the below figure, the standard deviation of r2 across PurpleAir devices increased to more closely match that of the Clarity Nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*45rOCYTSJiLmjLuVYcyblw.png" /><figcaption>The distribution of correlations (r2 values) between hourly average estimates produced from individual monitors as compared to BAM output and delineated by type with the addition of a new “type” representative of PurpleAir monitors with their original data resampled to more closely match the sampling frequency of the Clarity Nodes.</figcaption></figure><p>We then took this one step further and simulated data from PurpleAir data by skipping between 1 (i.e., taking every second measurement) and 29 measurements per “sample.” The goal of this assessment was to explore whether a larger trend exists between sample frequency and correlation of hourly averages with BAM output. This result is shown in the figure below. As the sample frequency decreased (toward the right), the correlation with BAM output decreased (p &lt; 0.01 when linearly tested. It should be noted that a non-normality of the data is indicated by a Shapiro-Wilk test [p &lt; 0.001]; more analysis is needed to confirm the significance of this inverse trend).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r9uWk4HOdQzwsS0vLCWLNg.png" /><figcaption>The average correlation between PurpleAir hourly average concentrations and BAM output as sample frequency is reduced (i.e. as the number of skipped data points per sample increases). The first point (from the left) is the original PurpleAir average r2 . The relationship is inverse (p &lt; 0.01, r2 =0.24, RMSE= 0.03).</figcaption></figure><h4>Conclusion</h4><p>Our co-location at CARB’s T Street monitoring location was a great success.</p><p><strong>Monitor reliability</strong></p><p>First and foremost, our collocation analysis demonstrated a reasonable correlation between the Clarity Node-S monitors that we are using in Richmond and the tried, trusted, and far-more expensive regulatory grade Beta Attenuation Monitor (BAM; r2 = 0.52). Correlations spanned a range that included the level of correlation observed between two BAM units collocated at a regulatory monitoring site in Vallejo several years ago (r2 = 0.58),(1) and suggest reasonable device sensitivity at ambient pollution concentrations and the potential for calibration. While not explicitly discussed in this write-up, Clarity devices also showed high correlation with each other, suggesting low inter-device variability, indicating that individual Clarity monitors should respond similarly to varying levels of PM2.5. This has implications for various collocation strategies, especially one in which a realtime collocation-based correction factor is developed for a single Clarity monitor and applied to other Clarity monitors in the region.</p><p><strong>Same sensor, different results</strong></p><p>We also uncovered a major discrepancy between the “reliability” of the Clarity Node and the PurpleAir (as assessed via correlation between hourly averaged monitor data and BAM output) despite their use of the <em>same internal sensing hardware</em>. Clarity Node correlations with the BAM were, on average, 0.52 while PurpleAir correlations with the BAM were about 25% higher with an average correlation of 0.65. After some experimentation with the data, we discovered the discrepancy to be related in large part to the number of samples taken every hour — more samples in an hour typically resulted in a higher reliability. When the PurpleAir data were made to mimic the Clarity Node’s sampling frequency, their correlation with BAM data became much more similar to the Clarity Node, dropping to an average correlation of 0.54.</p><p><strong>Findings that improve the Richmond study and inform future study design</strong></p><p>The results of this down-sampling simulation suggest that the Clarity Node units may substantially increase their reliability (r2 vs. the BAM) by increasing their sample frequency and, as a result, may be made to better represent true pollution conditions. This finding proved to be a major milestone in the study. Consequently, we’ve been working with Clarity to increase the sampling frequency of the monitors located around Richmond to improve the quality of their results.</p><p><strong>Moving beyond critical limitations</strong></p><p>PM2.5 concentrations were particularly low during the collocation measurement period, averaging approximately 6 ug/m3 — a value just barely above the BAM hourly detection limit of approximately 4.8 ug/m3 (2) and likely at or below the reliable detection limits of the low-cost sensors used in the Clarity Node and Purple Air. We expect to see ambient PM2.5 concentrations that exceed these values, and so future collocation testing of the Clarity Node should be done under higher concentrations to better understand the reliability of these devices under conditions that maybe more typical in the Richmond area. One idea moving forward might be to establish long-term collocations of a handful of Clarity Nodes throughout the study area.</p><p><strong>A big win for AB617</strong></p><p>This project sets precedence for future AB617 sensor-based projects by demonstrating the utility that can come from a pre-study collocation trial. From quantifying the general reliability of two popular low-cost sensor-based monitors to shedding light on sample frequency best practices, this work specifically adds great value to the community-oriented work currently underway in Richmond. We are excited to apply what we’ve learned here to improve this and future AB617 work.</p><h4>Acknowledgements and References:</h4><p>Special thanks to the CARB Advanced Monitoring Techniques Section (AMTS) Monitoring and Laboratory Division for allowing us to utilize their space and data, to David Lu of the <a href="https://medium.com/u/9f1c03cd92f6">Clarity Team</a> for providing support during the deployment visit, and to <a href="https://medium.com/u/d38438d987d6">Justin Bandoro</a> of <a href="https://medium.com/u/d9728fa0f7b7">Ramboll Shair</a> for his assistance with quality assurance in the analysis. The authors are employees of Ramboll, which is subcontracted for technical expertise on the Groundwork Richmond Air Rangers grant.</p><p><strong>1.</strong> Holstius et al. 2014. Field calibrations of a low-cost aerosol sensor at a regulatory monitoring site in California. Atmospheric Measurement Technology. 7:1121–1131. doi:10.5194/amt-7–1121–2014. Available at &lt; <a href="https://www.atmos-meas-tech.net/7/1121/2014/">https://www.atmos-meas-tech.net/7/1121/2014/</a>&gt;.</p><p><strong>2.</strong> <a href="https://metone.com/wp-content/uploads/2019/04/bam-1020_detection_limit.pdf">https://metone.com/wp-content/uploads/2019/04/bam-1020_detection_limit.pdf</a></p><p><em>Update: an earlier version of this blog stated “</em> These devices are very expensive — upwards of $500,000 for a BAM when accounting for maintenance, according to a recent presentation given by a member of the Bay Area Air Quality Management District. It is not economically feasible to implement a BAM on every street corner. Low-cost sensors that perform well against regulatory grade equipment open the door for everyone, including those in disadvantaged communities, to access real-time, local air quality information.<em>” This has been updated for clarity and to more appropriately fit the article’s context.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e4d2ec3ca6a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ramboll-shair/gaining-clarity-into-low-cost-sensors-4e4d2ec3ca6a">Gaining clarity into low-cost sensors</a> was originally published in <a href="https://medium.com/ramboll-shair">Ramboll Shair</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How machine learning and open source data can dramatically improve the quality of low-cost sensor…]]></title>
            <link>https://medium.com/ramboll-shair/how-machine-learning-and-open-source-data-can-dramatically-improve-the-quality-of-low-cost-sensor-580293d9ea52?source=rss-c97cdb307306------2</link>
            <guid isPermaLink="false">https://medium.com/p/580293d9ea52</guid>
            <category><![CDATA[air-quality]]></category>
            <category><![CDATA[sensors]]></category>
            <category><![CDATA[environment]]></category>
            <category><![CDATA[applied-machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Drew Hill]]></dc:creator>
            <pubDate>Tue, 08 Oct 2019 18:16:30 GMT</pubDate>
            <atom:updated>2019-10-08T18:16:30.096Z</atom:updated>
            <content:encoded><![CDATA[<h3>How machine learning and open source data can dramatically improve the quality of low-cost sensor output.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KCPCyj1ONTSZlPyDh_MQqg.png" /></figure><h3>Low-cost sensors: a true “disruption”</h3><p>Understanding the quality of the air we breath once required specialized field skills, large semi-permanent sample staging, and a hefty budget (often $10,000+ per monitor!). Today, low-cost sensors are changing the way we think about air quality assessment. While <a href="https://www.mdpi.com/1424-8220/17/8/1879">academics </a>have been cleverly adapting inexpensive technologies <a href="https://www.aqmd.gov/docs/default-source/aq-spec/resources-page/the-ucb-pats-monitor-a-short-history-with-list-of-publications-from-around-the-world.pdf?sfvrsn=0">like smoke detectors</a> to estimate air quality on a budget since at least the early 2000’s, a new family of affordable ready-made sensors — like the <a href="https://www.purpleair.com/sensors">Purple Air</a>, <a href="https://foobot.io/">Foobot</a>, and <a href="https://clarity.io/solution">Clarity Node</a> — are rapidly emerging as a boon to consumers, governments, and environmental consultants. Today, it seems that all you need is a $200 handheld gadget and an internet connection to get real-time air quality estimates right on your smart phone. But, as someone’s gran-dad somewhere always said, “there’s no such thing as a free lunch.”</p><h3>No free lunch</h3><p>Low-cost sensors often measure proxies of pollutant concentrations — like the amount of light (measured in voltage) that is scattered (by particulate pollutants) when a beam is sent through a parcel of air — rather than the actual parameter of interest. They’re also able to keep their small form factor and low prices by using lower grade hardware that allows more noise to infiltrate the measurement signal. Perhaps most importantly, the signal of many low cost sensors can be heavily affected by constantly changing factors like temperature, humidity, and the complex characteristics of the pollution being measured. That scattering light beam, for example, will scatter less light from darker pollution than it will from the same concentration of lighter pollution because more of that light will be absorbed (think wearing a dark shirt on a bright sunny day). For the most part, manufacturers and academics are making great strides in accounting for signal noise and temperature effects, but accounting for real-time changes in pollution properties, especially for particulate matter (PM), has remained a substantial hurdle. A new approach using carefully selected publicly available information on relevant environmental changes and data analytics may help us clear this hurdle.</p><h3>We can (and have begun to) overcome this hurdle!</h3><p>Subject matter experts (environmental consultants, academics, government officials, etc.) can tell you what sort of <em>high-level</em> events are going to <em>generally</em> affect local air pollution characteristics. For example, up-ticks in traffic, nearby wildfires, major power outages, and weather will all impact the contributors to and the makeup of local pollution. They will also tell you that trends in air quality measured several miles upwind are likely to provide insight on nearby air quality. However, the relationships between these factors and the pollution characteristics that affect low-cost sensor output are highly complex.</p><p>In an analysis recently presented at the Air Sensors International Conference in Oakland, CA (<a href="https://asic.aqrc.ucdavis.edu/sites/g/files/dgvnsk3466/files/inline-files/Drew%20Hill_Upload_0.pdf">click here for the full presentation!</a>), we demonstrated that <a href="https://cran.r-project.org/web/packages/SuperLearner/index.html">ensemble </a>machine learning methods (random forests and support vector machines alongside GLM and GLM net) can be used to account for sensor-relevant changes in local pollution composition to substantially improve the quality — and actionability — of low-cost sensor data. We also showed that information on such high-level events can be captured with reasonable spatial and temporal resolution from freely available (mostly government) data repositories using open source products like <a href="https://www.r-project.org/">R</a> and <a href="https://ropensci.org/">R Open Sci</a>.</p><h3>SUCCESS!</h3><p><em>Uncalibrated</em> PM2.5 data from Plantower sensors inside five low-cost air quality monitors — Clarity Nodes, the manufacturer of which does <a href="https://clarity.io/solution">offer a calibration service</a>, which we did not use here — are plotted against co-located regulatory grade PM2.5 monitors in the image below. A perfectly representative low-cost sensor would produce dots that fall directly along the black line — this black line is where government monitor concentrations are equal to the low-cost sensor estimates. What we see, however, is a consistent over-estimation of pollution concentrations (the blue line, or central tendency of the low-cost sensor data, does not follow the black line, or the central tendency of the government monitor) and some noise (the amount of scatter around the blue line).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/0*L3d0A5EMqRPDqyvv" /></figure><p>Processing these data using a combination of ensemble machine learning methods and features produced from open source data produced pollution estimates with dramatically less bias (made the sensor output better match the government monitor output!) for every device. Other than a few outliers, our model allowed us to reproduce “true” concentrations (the government monitor’s measurements) with noticeably less bias than the original sensor estimates and potentially less error, but more analysis is needed. 10-fold cross-validated predictions are plotted against “true” concentrations in the colorful image below. (note: do let me know if you want more stats — I can draw some more-refined RMSE, etc. info if there is interest)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*48y_8j44Ba-9HvgM" /></figure><h3>What’s the big deal?</h3><p>We used predictions from a machine learning model to <em>reliably </em>estimate “true” air pollution concentrations at <em>many points in space and time</em> using low-cost sensor output, local weather data, satellite wildfire estimates (<a href="https://asic.aqrc.ucdavis.edu/sites/g/files/dgvnsk3466/files/inline-files/Drew%20Hill_Upload_0.pdf">see the presentation!</a>), and a few other open-data-based variables with a free ensemble machine learning statistical package … oh, and it was done on a laptop.</p><p>I believe that this targeted multi-source data analytics framework can be used to turn the glut of data collected by consumers around the world over the last several years and openly published by manufacturers (e.g., <a href="https://www.purpleair.com/gmap?&amp;zoom=3&amp;lat=39.51596757727815&amp;lng=-99.35539180755615&amp;clustersize=45&amp;orderby=L&amp;latr=137.1519509363168&amp;lngr=360">Purple Air</a>) into research-grade information on environmental air pollution and human exposures.</p><p>With the right finesse and collaboration among government agencies, some improvements to their data transfer processes [note: the <a href="https://data.commerce.gov/">US Dept. of Commerce</a> has already mode some good progress here]), and a bit of elbow grease, the multi-source data analytics framework might also be expanded to convert low cost sensor output into more-reliable air quality data in real-time without substantially expanding cost or resource outlay beyond the price of the low-cost sensor itself.</p><p>The possibilities are very exciting. I feel privileged to have the chance to work on these sorts of issues in my current job, and look forward to sharing more updates and insights on the state of the field!</p><h3>Acknowledgements</h3><p>Co-authors on <a href="https://asic.aqrc.ucdavis.edu/sites/g/files/dgvnsk3466/files/inline-files/Drew%20Hill_Upload_0.pdf">the presentation</a> were Ajay Pillarisetti and Kirk Smith of the University of California, Berkeley and Shari Libicki of Ramboll (formerly Environ). Several months of data from Clarity Node devices were generously provided by Clarity (while they do provide a sophisticated calibration service, we only used <em>uncalibrated</em> data). Clarity also provided co-located regulatory grade monitoring data from the San Joaquin Valley Air Pollution Control District and the Bay Area Air Quality Management District.</p><p>This article originally appeared on Nov 6, 2018 as a LinkedIn publication by <a href="https://www.linkedin.com/in/hilldrew">Drew Hill</a>, written as an employee of <a href="https://www.linkedin.com/company/shair-by-ramboll/">Shair </a>and <a href="https://www.linkedin.com/company/ramboll/">Ramboll</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=580293d9ea52" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ramboll-shair/how-machine-learning-and-open-source-data-can-dramatically-improve-the-quality-of-low-cost-sensor-580293d9ea52">How machine learning and open source data can dramatically improve the quality of low-cost sensor…</a> was originally published in <a href="https://medium.com/ramboll-shair">Ramboll Shair</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Know What You Are Breathing]]></title>
            <link>https://medium.com/ramboll-shair/know-what-you-are-breathing-41319dd776fc?source=rss-c97cdb307306------2</link>
            <guid isPermaLink="false">https://medium.com/p/41319dd776fc</guid>
            <category><![CDATA[environment]]></category>
            <category><![CDATA[sensors]]></category>
            <category><![CDATA[air-quality]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Drew Hill]]></dc:creator>
            <pubDate>Tue, 08 Oct 2019 14:50:49 GMT</pubDate>
            <atom:updated>2019-10-08T19:43:49.748Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HdM8PVrwHAFBKT1PhdPO9A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@elidefaria?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Eli DeFaria</a> on <a href="https://unsplash.com/s/photos/breathe?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>(This article originally appeared on Dec 3, 2018.)</p><p>On Friday, the New York Times published a provocative piece on the state of air pollution monitoring in the United States. The article, <a href="https://www.nytimes.com/2018/11/30/style/air-quality-pollution-monitors.html">“Do You Know What You’re Breathing” by Nellie Bowles</a>, is a powerful piece on the need for increasing the availability and accessibility of actionable, hyper-local air quality information. We wanted to take a few paragraphs to share why its message is near and dear to our hearts, and what Shair is doing to address it.</p><h3>“Why didn’t we know about this already?”</h3><p>One of the article’s interviewees, Jim Leach, a retired IT professional, lives in a valley in Layfayette, California. His valley neighbors? Eight lanes of highway. The pollutants produced in and pulled through this highway valley are unlikely reflected in the concentration readings of his nearest government monitor which, according to the article, is six miles away and in a residential area.</p><p>Mr. Leach eventually discovers a concerningly high trend in is neighborhood pollution with a low-cost sensor he purchased from an online organization, <a href="https://www.purpleair.com/">PurpleAir</a>. Mr. Leach is often asked “Why didn’t we know about this already?” The answer: a low density of government monitors placed throughout the US.</p><p>Let’s be clear: the US regulatory air monitoring network is one of the most advanced out there. The US Environmental Protection Agency has been a major force driving air quality monitoring, regulation, and improvement world-wide. Unfortunately, traditional, high-grade monitors and the stations required to house them are very expensive — often in the range $15,000 — $100,000 (USD). The resource limitations posed by the costs of siting, purchasing, operating, and maintaining these devices has produced a regulatory measurement paradigm characterized by monitors that are few and far between. According to the president of the American Lung Association, Janice E. Nolen, only about one-third of US counties even have air quality monitors* (*Presumably regulatory monitors).</p><p>Low-density networks of expensive (and operationally complicated!) monitors have been an important first step in understanding regional air pollution and developing a population-scale understanding of the health and property effects of air quality. But, as Mr. Leach discovered at his California valley home, trends in air quality are much more hyper-local than networks like these can capture.</p><h3>The role of low-cost sensors</h3><p>Consumers and local air quality agencies are beginning to demand action to address the gaps in traditional air quality monitoring networks. Companies like <a href="https://clarity.io/">Clarity</a>, PurpleAir, <a href="https://getawair.com/">Awair</a>, and <a href="https://plumelabs.com/en/">Plume Labs</a> — which produce air quality monitors from low-cost sensor hardware (“low-cost sensors”) and often price their devices in the $150 — $1,000 range — are springing up around the world to meet these needs. As a result, thousands of low-cost sensors have recently come online. According to the Times article, PurpleAir now has more than 3,000 monitors feeding data to their (<a href="https://www.purpleair.com/map#1/2.3/-30">publicly accessible</a>) cloud daily. This is an amazing development and the beginning of a major shift in the air quality assessment paradigm. But what do we do with these data once they’re in hand?</p><h3>“What is this stuff and how do I need to be protected”</h3><p>In the article, Mr. Leach poignantly notes that “[p]eople are starting to wake up and ask what is this stuff and how do I need to be protected.” In essence: we need to get a better handle on hyper-local pollution levels, but we also need to understand what that pollution is and where it comes from so that we can take action to mitigate those sources and more effectively protect ourselves.</p><p>The data these low-cost sensor networks collect are doing and will continue to do wonders for public awareness, hotspot detection, and a variety of other important areas. But more is needed. The data coming from these networks must be more than just a canary in a coalmine. We need to know what we’re breathing so that we can know what to fix.</p><h3>Our mission</h3><p>Shair is designed to complement existing and planned hyper-local sensor networks. By combining sensor data with sophisticated spatial and physical modeling approaches, Shair targets and quantifies the emissions sources contributing to harmful, hyper-local pollution exposures so that cities and citizens can take action. In short: Shair turns sensor data into more-actionable data.</p><p>If this sort of thing interests you, drop us a line! We love meeting and collaborating with like-minded folks, and would be thrilled to assist anyone interested in applying Shair to their data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41319dd776fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ramboll-shair/know-what-you-are-breathing-41319dd776fc">Know What You Are Breathing</a> was originally published in <a href="https://medium.com/ramboll-shair">Ramboll Shair</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why is hyper-local air quality information important?]]></title>
            <link>https://medium.com/ramboll-shair/why-is-hyper-local-air-quality-information-important-a3eb15ae4cd3?source=rss-c97cdb307306------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3eb15ae4cd3</guid>
            <category><![CDATA[wildfires]]></category>
            <category><![CDATA[air-quality]]></category>
            <category><![CDATA[climate-change]]></category>
            <category><![CDATA[sensors]]></category>
            <category><![CDATA[public-health]]></category>
            <dc:creator><![CDATA[Drew Hill]]></dc:creator>
            <pubDate>Mon, 30 Sep 2019 14:33:57 GMT</pubDate>
            <atom:updated>2019-09-30T14:33:57.085Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H_45JjxjEzqKO5_d" /><figcaption>Photo by <a href="https://unsplash.com/@jamesmhammond?utm_source=medium&amp;utm_medium=referral">James Hammond</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>One major reason: so that you can know when to take action. Simple steps like turning on the furnace fan (with a highly MERV-rated furnace filter from the hardware store) can improve indoor air quality during outdoor pollution events. A member of the Shair team did just that to protect their family from the wildfire smoke engulfing the SF Bay Area during the Camp Fire. Their indoor PurpleAir sensor captured the shifts as they turned the furnace fan on and off throughout the day.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/0*gEs26d7xIdu2_0VI.png" /></figure><p>After the weekend, our team member gathered measurements from a nearby outdoor PurpleAir (the top/higher data series) and plotted them alongside their indoor PurpleAir monitor. The furnace and its fan were turned off and on at various points, but not all transitions were logged or remembered, and some may be affected by recall bias :). Note: some indoor peaks include dinner time cooking, too. The time frame of the original image is the very beginning (leftmost side) of the indoor data series. While a clear relationship exists between the outdoor and indoor concentrations (a good sign for the reliability of the PurpleAir low-cost sensor!), a strong correlation between indoor concentrations and furnace timing is maintained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><p>Learn more about MERV-rated furnace filters and indoor air purifiers <a href="https://www.epa.gov/indoor-air-quality-iaq/air-cleaners-and-air-filters-home">on the EPA’s website</a>.</p><p>&lt;note: an ever-so-slightly different version of this post originally appeared on <a href="https://www.linkedin.com/feed/update/urn:li:activity:6467419691534082048">SHAIR’s LinkedIn page</a> during the Camp Fire Bay Area smoke event on approximately November 13, 2018&gt;</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3eb15ae4cd3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ramboll-shair/why-is-hyper-local-air-quality-information-important-a3eb15ae4cd3">Why is hyper-local air quality information important?</a> was originally published in <a href="https://medium.com/ramboll-shair">Ramboll Shair</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>