<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Brian Kemp on Medium]]></title>
        <description><![CDATA[Stories by Brian Kemp on Medium]]></description>
        <link>https://medium.com/@brian.l.kemp?source=rss-80b62109329e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*4fdIyJTSM3hWqGR4.jpg</url>
            <title>Stories by Brian Kemp on Medium</title>
            <link>https://medium.com/@brian.l.kemp?source=rss-80b62109329e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:31:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@brian.l.kemp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Is more always better? Lessons in Machine Learning from aggregated Census data]]></title>
            <link>https://medium.com/@brian.l.kemp/is-more-always-better-lessons-in-machine-learning-from-aggregated-census-data-83abf9644725?source=rss-80b62109329e------2</link>
            <guid isPermaLink="false">https://medium.com/p/83abf9644725</guid>
            <category><![CDATA[census]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Brian Kemp]]></dc:creator>
            <pubDate>Tue, 20 Oct 2020 23:34:34 GMT</pubDate>
            <atom:updated>2020-10-20T23:34:34.260Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/348/1*pPlY3hHkzUOLvODOVZEAcQ.png" /></figure><p>One of the powers of data science in recent years has been its ability to generate great insights into behaviors by using micro-level data in order to find relationships that are non-obvious or hard to pin down through more traditional analytical tools. One of the main points of Seth Stephens-Davidowitz’s 2017 book “Everybody Lies” on insights from data science is the power of “zooming-in” and making the most of some of the extremely fine-grained information now at our fingertips.</p><p>Reflecting on my <a href="https://medium.com/@brian.l.kemp/the-5-keys-to-a-great-solar-demographic-581d1b07a215">prior experience</a> in analyzing data from the Australian Bureau of Statistics, I was intrigued by the power of prediction available from looking at data at differing levels of geographic aggregation. When I stumbled across the extremely detailed <a href="https://datapacks.censusdata.abs.gov.au/datapacks/">DataPacks</a> available I thought this would be a great opportunity to explore how training models with different levels of geographic aggregation affect performance and other potential insights.</p><p>Out of curiosity, I set my target predictor as the participation rate of people who worked from home on census day. Using data aggregated at different Statistical Area (SA) levels, I set out to answer the below key questions:</p><ul><li>What is the impact on model accuracy (e.g. is more detailed data prone to overfitting)?</li><li>How do the various features compare between models in terms of feature importance? Are the same features consistent between models or do they differ substantially?</li></ul><p>For context, the different SA levels have the below characteristics:</p><ul><li>SA3 — has 358 areas, with an average population of ~32,000 per region</li><li>SA2 — has 2,311 areas, with an average population of ~5,000 per region</li><li>SA1 — has 57,523 areas, with an average population of ~200 per region</li></ul><p>This gives a large degree of variance in granularity to work with in analyzing the impacts of training models at different levels.</p><h4>Getting around the Census Data</h4><p>My first challenge was finding a way around the DataPacks files themselves. With &gt;15,000 columns worth of data with helpful titles such as “All_inc_NS_UnE_lfw_Tot” as well as being aggregated at multiple (often inconsistent) levels, there had to be some way of nicely consolidating the data, but it wasn’t going to be easy.</p><p>Luckily, the DataPacks files themselves had an Excel file for metadata on the features of the dataset, and the long description of each column header was mostly consistent in using capitalisation or numbers to indicate different levels of data within a single column. For example “Males_20_24_years_Never_Married” has 3 levels of data: sex (males), age (20-24 years), and marital status (never married).</p><p>By using regular expressions to split this data (and some manual intervention for poorly formatted exceptions) I was able to reduce the number of measures from ~15,500 to ~1,000 across 64 different categories. From this I built a new metadata file for reference in an Extract-Transform-Load pipeline for use in consolidating tables, refined to only include selected categories of interest, ready for use in training a random forest regression model.</p><p>This function and all the underlying analysis used in writing this article is available <a href="https://github.com/blkemp/Census-ASGS-Level-and-Model-Sensitivity">here</a>.</p><h4>Picking a metric for assessment</h4><p>It may be expected that because we are comparing model efficacy it would be worthwhile exploring RMSE as noted <a href="https://medium.com/usf-msds/choosing-the-right-metric-for-machine-learning-models-part-1-a99d7d7414e4">here</a>. Given the increasing variance in the predicted outcome at different SA levels, by failing to take the population variance into account RMSE will tend to inflate the error rate of the models trained on more disaggregated SA levels. Accordingly, I have focused on R2 score as a metric due to my desire to explore the explanatory power of models.</p><p>I considered using Adjusted R2, but the high dimensionality simply made the results irrelevant (Adjusted R2 is supposed to always be lower than R2, and this was not the case where n samples &lt; k parameters as often occurred at the SA3 level). Additionally, since the number of features is the same between models, the only driver of differences between Adjusted R2 and regular R2 is the sample size. Given the entire point of this analysis is to determine the power of having more training samples, using a metric which seeks to mitigate this as a factor seemed disingenuous.</p><h4>What is the impact of “zooming in” on model accuracy?</h4><p>This was the key question in the analysis; does disaggregating data into smaller geographic regions produce more accurate models because there are more points to train on, or does it tend to cause overfitting?</p><p>In short, more granular detail tends to increase model accuracy to a point, but can quickly become prone to overfitting and statistical noise.</p><p>Consider the below 3 charts, showing the relationship between the <em>actual</em> and <em>predicted</em> worked from home participation rate of a test sample of regions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*FtiKhXG41lwlmXiWaWc-Ww.png" /></figure><p>As the regions progress in levels of granularity the R² score first increases when moving from the SA3 level to SA2, from 0.82 up to an astonishingly good 0.87, but then quickly turns into a mess when modelling with SA1 level data. In particular the models at both the SA2 and SA1 levels seem to be fairly poor at predicting regions with zero people working from home, which compounds significantly when you get to the SA1 level which has a large number of such regions.</p><p>This is further confused by <a href="https://www.abs.gov.au/ausstats/abs@.nsf/Latestproducts/1160.0Main%20Features6Aug%202017?opendocument&amp;tabname=Summary&amp;prodno=1160.0&amp;issue=Aug%202017&amp;num=&amp;view=">the intentional addition of “noise” by the ABS</a> to its datasets in an effort to protect the privacy of citizens and sensitive information, which has a larger impact on individual samples as they become exponentially smaller when moving from SA3&gt;SA2&gt;SA1.</p><p>This additional noise makes handling outliers incredibly important, as without it SA2 model performance drops precipitously. Some things come up which are hardly representative of a reasonable prediction, such as the region of Molonglo which had a population of 3, all of which worked from home. Or even worse at the SA1 level where there were multiple regions that recorded more people who “Worked From Home” than the overall reference population!</p><h4>Reliability of model accuracy comparisons</h4><p>While the above analysis certainly gave some good insights into the drivers of differential performance at different levels of geographical aggregation, in the course of compiling this modelling it also became apparent that the test:train split could influence significant fluctuations in model performance. This could mean that the above difference between SA2 and SA3 R2 scores for SA3 and SA2 level data (0.83 vs 0.87) was within the realms of normal variation in model performance and there was no real benefit at all from even this first step toward disaggregation!</p><p>To assuage these concerns I did a comparison based on a power analysis between SA2 and SA3 trained models with 200 different test train splits to get a more comprehensive view on general model performance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*23MWaUeQnp38maLYa9fgjQ.png" /></figure><p>From this mass-analysis of model performance it definitely seemed like the first examples above were outliers in their very high R2 scores, and the overall gap between model performance halves. While this may seem to be bad news for SA2 level data, comparing the two datasets statistical performance gave a p-value of 0.000003 asserting that SA2 level data creates more reliable prediction models than SA3, which is just an astonishingly high level of confidence.</p><h4>Do the key features highlighted by the model change?</h4><p>As I ran through the analysis, I ensured I extracted the feature importances of the models to get a feel for their consistency and just to make sure the models made some kind of sense. From this “eyeball” view feature importances do change between models, but not significantly. At both SA2 and SA3 levels the top 2 features were the proportion of “55–64 year old managers” and “65–74 year old managers”, although they did swap in importance between datasets. The remaining features also remained fairly consistent, being a mix of indicators of age and “management/professional” occupations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*wkbp1fo-uByn4AkYjpxrfw.png" /><figcaption>Feature importances for the SA3 level model trained, note that while the features state “by Sex” that sex has been removed from this model and this line is actually a total of both Male &amp; Female</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*XebViFe2ydkXB5QvKi-nBw.png" /><figcaption>Feature importances for the SA2 level model trained</figcaption></figure><p>Given the high variance in the predictive power of each model as tested, I was quite surprised to see these themes remain consistent throughout the different levels. Especially important to me was that the “directionality” of feature impacts was consistent between models (e.g. an increase in the proportion of Managers aged 65–74 increased the likelihood of working from home in both SA3 and SA2 level models).</p><p>While this strong relationship between older managers and working from home is quite apparent when looking at correlations in the data at the SA3 level, with the proportion of 65–74 year old managers in a population having a Pearson’s correlation coefficient of 0.8 with the working from home participation rate at the SA3 level. I have found in past experiences that these correlations do not always hold when creating predictive models and didn’t expect it to hold when “going deeper” into the data.</p><p>For good measure I also ran the data removing age as a feature and adding in sex, which again had similar “theme consistency” across models, although the specifics themselves could switch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*k84kV-DvfNWE4m7Ui5qqUg.png" /><figcaption>Feature importances for the SA3 level model trained including sex but excluding age as a feature</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*_i10-HkGevLtwBitE4DHhA.png" /><figcaption>Feature importances for the SA2 level model trained including sex but excluding age as a feature</figcaption></figure><p>By training models at this level there were significantly fewer categories (there are fewer categories of “sex” captured in the data than “age”) and the normalised weights of feature importance had much lower variance between models trained at different levels. With minimal sacrifices to R2 this made me further suspect the perils of overfitting by simply throwing more data at the problem, although the consistency in themes did provide some level of comfort. As Gwern has extensively detailed: <a href="https://www.gwern.net/Everything">Everything is Correlated</a>, so it is probable that comfort is misplaced.</p><h4>Is there much to learn about the influences for working from home?</h4><p>In a year like 2020 where vast swathes of the global population have been forced to adopt work from home practices, were there any lessons to learn from the 2016 Census? In short, yes, and it was fascinating. However! This is not the target of this article, if you are interested in this I strongly suggest you check out the <a href="https://github.com/blkemp/Census-ASGS-Level-and-Model-Sensitivity">Git repository</a> “Working from home feature analysis” Jupyter file where I dove into the detail. Do note that I compiled this analysis in 2019, so there are no allusions to Covid-19 or reflections on 2020 in any of the workings. Additionally it was largely abandoned as my real job starting taking up more time in my life, it’s a great start and there is much to build upon but it is far from being able to draw any conclusions at this point.</p><h3>Conclusion</h3><p>In all, the analysis of this Census data is a good representation of both “the power of zooming in” and “the curse of dimensionality”. By zooming in from the equivalent of “county level” data to “suburb level”, predictive power increases substantially as training samples increase in size. But, when zooming in too far, to what could be typified as a “city block level”, the models can start jumping at shadows trying to fit in all the edge cases, resulting in much less effective predictions overall.</p><p>Taking these lessons and applying them to my analysis of the drivers of working from home was a huge help. While the temptation is to solve problems with “more data!”, often the real solution is more thoughtfully splitting and refining your data. This became especially apparent when I started analysing the drivers of working from home and started splitting datasets by sex rather than treating it as one homogeneous block, where the drivers of working from home were quite different between Male and Female workforce participants, but strong model performance (R2&gt;0.8) after disaggregating.</p><p>There is a balance to be found here, and it speaks to the “art” of Data Science.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83abf9644725" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 5 keys to a great solar demographic]]></title>
            <link>https://medium.com/@brian.l.kemp/the-5-keys-to-a-great-solar-demographic-581d1b07a215?source=rss-80b62109329e------2</link>
            <guid isPermaLink="false">https://medium.com/p/581d1b07a215</guid>
            <category><![CDATA[solar-energy]]></category>
            <dc:creator><![CDATA[Brian Kemp]]></dc:creator>
            <pubDate>Sun, 09 Jun 2019 22:54:04 GMT</pubDate>
            <atom:updated>2019-06-09T22:54:04.432Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Lessons from the Australian census about the best regions for solar sales</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WXszgLI4eq2BNmh4a8a2iQ.jpeg" /></figure><h3>Introduction</h3><p>There’s a lot going on in the world right now around clean energy technology. As power prices go up, there are more reasons for consumers to look into installing their own solar system: be it to save money, save the environment, or just to tap into some sweet <a href="https://www.solar.vic.gov.au/">government rebates</a>.</p><p>With all this bubbling demand, you may thinking to yourself “I should set up a solar business” and of course the first question that comes after that thought is “where?”</p><p>Underlying that fundamental question of “where?” lies a couple of others:</p><blockquote>· What are the key features that influence solar installations in an area?</blockquote><blockquote>· How do changes in these features effect the demand for solar?</blockquote><blockquote>· What are the growth prospects for the regions I can see based on these features?</blockquote><p>Luckily, the Australian Bureau of Statistics (ABS) has a wealth of data we can use to help answer these questions. If you’re Australian you likely best know the ABS from filling out your census forms every 5 years, where they ask everything from the number of people in your house, to what kind of house you live in, to how far you went to get to work and if you rode your bike there!</p><p>The ABS <a href="https://www.abs.gov.au/databyregion">publish all this census data</a> by geographic region, along with some other fields they capture on an annual (instead of 5 yearly) basis. Leveraging this dataset and combining it with “the new hotness” of machine learning, we can get some great insight into the best areas to target for customers with the highest propensity to invest in solar, and probably come up with some provocations for policymakers while we’re at it.</p><h3>Part 1: What makes a region invest in solar?</h3><p>One of the areas the ABS has been tracking since 2014 is number of “small scale solar panel system installations”: by stitching all of the data by region together into one file and then training a <a href="https://medium.com/@williamkoehrsen/random-forest-simple-explanation-377895a60d2d">random forest model</a> on this, we can get to a pretty robust outcome for predicting the number of installations per household based on all other factors.</p><p>With the model trained, looking at the model’s “feature importances” shows that the following 5 areas explain around 2/3 of the model’s final prediction:</p><p>1. Percent of working population employed in Agriculture, Forestry and Fishing (primary producers)</p><p>2. Population density (how many people per square kilometer)</p><p>3. Number of apartments</p><p>4. Latitude</p><p>5. Average commuting distance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*uitMNsNjoz-Vc1CCMUz0yA.png" /></figure><p>The green bars show the importance of the individual characteristic, while the blue bars show cumulatively how important the features are as they are factored in. From this you can tell the share of primary producers in a region is about twice as important as population density, which itself is about twice as important as the number of apartments or the region’s latitude.</p><p>“Okay,” you say “but <em>important</em> in what way?”</p><h3>Part 2: How do these features effect solar uptake?</h3><p>It’s all well and good to say “the number of farmers is important when estimating demand for solar”, but does this mean that more farmers will increase or decrease demand? Based on the media sentiment about Queensland during the latest election, maybe you’d think farmers don’t really care about clean energy so it would decrease. Or maybe you see a lot of farmers enjoying being “grid independent” so you think it would increase. Or it could even be some kind of funny curve, where it increases to a point, then decreases! Best to let the data do the talking.</p><p>The great thing about using a machine learning model is you can “fake” the data you feed into it, to come up with some pretty great ways of seeing the impacts of changes to any given feature on the final prediction it makes. By creating some “fake” data by changing the original values given by the ABS for each of the above features, we can see some interesting patterns emerge in how these can influence predicted demand from as much as +7% to nearly -20%!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i2MCLxCNJjanosh7g9tEMw.png" /></figure><p><strong>1. <em>Percent of working population employed in primary industry</em></strong> — There seems to be a real “tipping point” in this feature, where once you hit a given % employed in primary industries, you gain an immediate boost to solar demand, but it peaks quickly and never decreases.</p><p><strong>2. <em>Population density</em></strong> — The spread of effect on this one is a bit more broad, where looking at regions which have more “spread out” populations give a quick sharp boost to the predicted number of installations, but flattens out significantly after this point. Similarly, getting more urban and dense hurts demand for solar, but only so much.</p><p><strong>3. <em>Number of apartments</em></strong> — Increasing the number of apartments in a region <em>strongly</em> decreases demand; from the chart it looks like exponential losses! This one feels very intuitive, given apartments themselves typically are not very “solar friendly” in design, and when coupled with apartments typically being “rentals” it gives significantly less incentive for the occupier to invest in a solar installation.</p><p><strong><em>4. Latitude</em></strong> — This one is a bit unintuitive. After a certain point of increasing latitude (moving further south), there are strong boosts to solar demand. This is contrary to basic science indicating that solar has higher potential at lower latitudes (closer to the equator) as nicely demonstrated <a href="http://profhorn.meteor.wisc.edu/wxwise/radiation/sunangle.html">here</a>. Whether it is a proxy for government intervention (subsidies and rebates at a state level), an indicator of higher electricity prices in the south, reflective of a more temperate climate meaning the effective yield is higher (due to fewer cloudy days), are all open questions for further exploration.</p><p>5. <strong><em>Average commuting distance</em></strong> — As you decrease commuting distance, you decrease demand for solar. But increases in distance don’t seem to have an impact. This again seems to be a proxy for “urban-ness” of a region.</p><p>So as we can see there are a lot of things going against solar, most relating to how urban a given region is.</p><p>So with all that in mind, if you’re wanting to make sure you’re in the right region for a solar business it seems like less urban is better.</p><h3>Part 3: What are the growth prospects?</h3><p>Thank goodness we have that key feature of “urbanity” identified, as “Population Density” itself is a great overall proxy for how urban a region is, and population density happens to be a data point that the ABS tracks every year along with number of solar installations.</p><p>By grouping the regions into 4 segments based on their population density, we can get a pretty good view on the trends in installations over time and get a feel for how growth is looking.</p><p>The 4 segments are based on “quartiles”, but to make it easy to picture in your mind’s eye, I’ve given them representative labels:</p><p><strong>· Q1 — Rural</strong>. Country regions around small towns like Junee in outback NSW.</p><p>·<strong> Q2 — Regional</strong>. Regions around larger country towns like Wagga Wagga or Albury.</p><p><strong>· Q3 — Suburban</strong>. Suburbs (or groups of suburbs) with typically no agricultural land, like West Pennant Hills or Fairfield.</p><p><strong>· Q4 — Urban</strong>. Dense suburbs close to the heart of Australia’s major cities, like Potts Point or Bondi in Sydney.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*z4xfTciT2_llipqtSJDQoA.png" /></figure><p>Looking at these trends you can see the biggest markets are definitely around those “regional” areas, with installations accelerating as time progresses! The growth prospects are further supported by looking at the average growth rates from 2014–17:</p><p><strong>· Rural:</strong> 13.1%</p><p><strong>· Regional:</strong> 11.0%</p><p><strong>· Suburban:</strong> 9.5%</p><p><strong>· Urban:</strong> 9.2%</p><p>It definitely seems that “the trend is your friend” for solar installations almost everywhere.</p><h3>Final Thoughts</h3><p>Using machine learning models on the publicly available data from the ABS gave us some great insight into the factors that influence demand for solar in Australia.</p><p>It looks like the best market in Australia is outside the major cities, where population density is far lower and you have fewer apartments. These areas are showing the strongest trends in growth in installations, and already represent around 60% of the total installations in Australia. This means good business not just for those looking to install new systems, but also for those who wish to leverage future opportunities in maintenance.</p><p>So if you’re interested in the business of solar panels, get out of the city and say g’day to those Aussies going green in the bush.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=581d1b07a215" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>