<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kat Hempstalk on Medium]]></title>
        <description><![CDATA[Stories by Kat Hempstalk on Medium]]></description>
        <link>https://medium.com/@drkatnz?source=rss-673dc508e869------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*gdIPec0qvZ0VNQXk.jpg</url>
            <title>Stories by Kat Hempstalk on Medium</title>
            <link>https://medium.com/@drkatnz?source=rss-673dc508e869------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:06:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@drkatnz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[No, AI will not kill us all.]]></title>
            <link>https://drkatnz.medium.com/no-ai-will-not-kill-us-all-ccc84929289d?source=rss-673dc508e869------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccc84929289d</guid>
            <category><![CDATA[climate-change]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[responsible-ai]]></category>
            <dc:creator><![CDATA[Kat Hempstalk]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 03:56:51 GMT</pubDate>
            <atom:updated>2026-09-22T22:03:52.424Z</atom:updated>
            <content:encoded><![CDATA[<p>But AI could do a lot of damage to humanity, if we let it.</p><p>There have been a lot of high profile exits from frontier AI companies lately, with the ex-employees citing concerns about lack of oversight and that “AI could kill us all”. Anthropic’s CEO even released <a href="https://darioamodei.com/post/we-must-pace-the-frontier">his view</a> (later reinforced by competitors like OpenAI) that we need to slow down to pace ourselves to ensure AI’s safety. I’ve spent my career applying AI to solving problems — long before LLMs were popular — and I <em>almost </em>agree. Any technology has the potential to cause harm, and the risk is higher with new technologies that are not yet fully understood, so we should be wary of the harm it causes. However, where my view differs is that instead of slowing down, I think we should be more thoughtful about the problems we use AI to solve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vSrQBJyJpsESfDrtF8KamQ.jpeg" /><figcaption>AI, robots and humans can work together for the good of humanity and the planet.</figcaption></figure><p>Previously AI was built and managed by data scientists like myself. We typically trained machines to solve specific problems: sometimes as simple as auto-completing a form, but could be as advanced as a cohort of models working together to create a self-driving car. We approached it with a balance of scientific rigor and pragmatism. The models had to be good enough to solve a problem, but also work in the real world. Early in my career when working on AI camera systems I spent hours figuring out how to take images of cows, because you can’t ask them to look at a camera and say “cheese”. We had to put in controls and guardrails (and in the case of cows, <em>literal </em>rails) to ensure the AI was successful and it was part of a product people would want to use.</p><p>With the rise of LLMs, the barrier to entry has been lowered for access to AI and almost everyone using a computer has been exposed to AI whether they like it or not. LLMs are typically “general knowledge”, in the sense they don’t specialise on one problem or knowledge base. They can be used for anything provided the instructions can be written down and the output is digital. Frontier labs are racing to make these models smarter so they are more capable than even the greatest experts, across all categories of knowledge. It’s possible for a model to learn about an area and speak with authority on it, complete a task, or even tackle some of our hardest problems (like the recent possible solution to the <a href="https://openai.com/index/navier-stokes-solution/">Navier-Stokes Millenium prize</a>), without being forced into thinking about whether the model should do those things, or whether the right controls are in place.</p><p>At a personal level this can expose you to simple risks where the AI tool is incorrect, and you don’t know until something bad happens later. For example, producing a recipe that has an extra ingredient that could make the dish inedible or toxic. You might vibecode a website that is easily hacked. Or maybe you annoy a friend by sending them some AI slop when they were expecting something you made or said. These are all real risks, and some variation of them has probably happened to anyone using an AI tool for personal use. Maybe even you start to feel obsolete when AI is more capable at solving a task that you were previously an expert at. These are not the risks the researchers are worried about, but they can still cause harm.</p><p>Organizations can also be exposed to similar risks from LLMs, only the stakes can be higher since they have a bigger exposure. An incorrect LLM response could make a product unusable or even deadly. Websites, software and hardware might be buggy, or more vulnerable to hacks. Society might lose faith in your brand because they see you using AI for something they expect to be human. Your entire business might disappear overnight if AI is capable of replacing it. These are also not the risks the researchers are worried about, but they can do a lot of damage too.</p><p><strong>The things that are worrying to researchers are things like:</strong></p><ul><li>AI becoming sentient and/or smarter than all humans combined, and what does that change about our obligations towards it if does? Are there tasks we should no longer ask it to perform? How will that impact society? What could it do for or against us with the tools and knowledge it has, and how it feels about us? If its goals do not match human survival, will it attempt to make us extinct? Is Skynet our future?</li><li>Rogue people or organizations that leverage AI to cause more harm than they might be capable of without it. Could AI shut down the internet if asked to create a self-sustaining botnet (virus) that infiltrates every device? Could organizations create unique attacks on infrastructure or services that we all rely on? Could someone create a biological weapon that might be used against us? Could someone press the big red button to launch all missiles?</li><li>If the machines make us obsolete at certain tasks, how do we perform those tasks when the machine is unavailable? If we are no longer the experts, how would we know when it is right or wrong? How do we choose our path forward in the world? Has society peaked?</li></ul><p>These are big, scary, and potentially widespread impacts that could shape entire generations. Some of them are plausible , and some of them are more out of science fiction. These are not the risks that keep me up at night, mostly because these are risks that exist today in some form. Just replace the word “AI” in the sentences above with “powerful humans” and I’m sure you can think of some famous/infamous people throughout history who could easily fulfil AI’s role in each of those concerns. For most things in the world there are already good controls in place that prevent bad things from happening, and progress (and obsolescense) has been key for humanity’s survival thus far. As the world has changed, we have adapted (sometimes begrudgingly). Despite adversity, we are still here.</p><p><strong>There are two things that I worry about.</strong> The first is a variation on obsolescence. <strong>I worry that people are starting to outsource thinking and literacy to AI.</strong> I have seen people who I greatly respect author something using AI that is a fraction of their normal quality and/or inappropriate for the task “because they could” with AI. I’m sure they had good reasons for doing so such as needing to produce something fast, prove a point, or even just navigate a situation they were not expert in. But, it saddens me to see people that have previously prided themselves on their knowledge, critical thinking and experience just… outsource it. And outsource it to something that lacks their personal flair. My worry is that if our best and brightest lead by this example, we set our future generations up for a less creative, lower capability future. What motivation have the next generation to learn if there is a machine that can do things for them?</p><p>Humanity has advanced significantly over the last few thousand years due to increasingly levels of literacy, knowledge sharing and scientific progress. We have the potential to rapidly undo generations of progress if we stop thinking for ourselves. One might argue that if the machine can do it, why do we need to? Instead, I ask: <strong>what can the machine not do that we should be thinking about instead?</strong></p><p><strong>The other thing I worry about is whether the speed of which AI solves problems can outpace the speed at which AI makes them worse.</strong> We have already seen this recently with Anthropic’s Claude models: the “Mythos” model was held back, then made available only to limited people, over concerns that it could easily find bugs in software. The main worry was that by releasing the model to everyone there would be a period of time that software worldwide was vulnerable to model-generated attacks, before developers were able to patch the bugs that the model could find. I feel AI presents great opportuntities for solving big problems like curing disease, feeding our world’s population, and reversing climate change. But it also is at risk of creating more harm first: AI in its current form uses lots of electricity and water, releases lots of carbon emissions, and has few guardrails on its use. <strong>I think instead of pacing AI’s development we should focus it on the problems it brings first, and solve them concurrently.</strong> If we aim for a future where machines support and protect us, we must start working towards that future now.</p><p><strong>I strongly believe that AI has helped — and will continue to help — humanity solve problems. AI is not inherently good or evil, but it might choose a side based on how we shape it. <em>Let’s shape it responsibly.</em></strong></p><p><strong>Disclosure</strong></p><p>This article was 100% written by a human that happens to like punctuation, and highlighting for effect. The author currently works on applying AI for good, and this article solely reflects their views. The single image in this article was AI generated, and is only there because Medium insisted on having an image for the article. ¯\_(ツ)_/¯</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccc84929289d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Prefect Pipeline Code for Building a Basic Data Lake]]></title>
            <link>https://drkatnz.medium.com/the-prefect-pipeline-code-for-building-a-basic-data-lake-5ac3628ab6e3?source=rss-673dc508e869------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ac3628ab6e3</guid>
            <dc:creator><![CDATA[Kat Hempstalk]]></dc:creator>
            <pubDate>Wed, 18 Aug 2021 05:13:42 GMT</pubDate>
            <atom:updated>2021-08-18T05:13:42.130Z</atom:updated>
            <content:encoded><![CDATA[<p>Earlier this month I wrote <a href="https://medium.com/swlh/building-a-basic-data-lake-with-prefect-s3-glue-and-athena-676410fbbdc4">an article about building a basic data lake</a> with Prefect and AWS, and promised a follow- up post where I’d share some of the code from the pipeline. This is that post.</p><p>First, a quick review: for our org, we set up a basic data lake using Prefect to assist in copying the data from AWS Aurora (MySQL) to S3. Prefect is a workflow orchestration engine, meaning that it can be used to schedule and run tasks (i.e. pieces of code) and report on their success or failure. Since data pipelines can easily fail thanks to any number of things — bad permissions, changes in the underlying data, lack of memory, etc — using an orchestrator such as Prefect allows us to be notified easily if anything goes wrong. The pipeline can also be inspected easily, no need to ssh into a machine and trawl through log files to figure out what’s caused it to fail.</p><blockquote>In Prefect, each pipeline is a flow, and the flow performs multiple <em>tasks </em>— like extracting the data, transforming the data, or loading the data. The flow is a directed acyclic graph (DAG), meaning that tasks that are dependent on others won’t be run unless their predecessors complete successfully. Although Prefect has a cloud offering, and a nice UI if you choose to self-host the engine, it’s also possible to run the flow as a python script locally, rather than as part of an orchestrated workflow automation system.</blockquote><p>Prefect is run using a server — the orchestration engine — and agents, which work on the tasks. The agents are designed to ‘pull’ — they will poll the server for tasks, and pick one up when it becomes available. We chose to use Docker agents, which pull the flow from a Docker repository before running it on a Docker virtual machine. The advantage with doing so is that our code is all neatly packaged up in a docker image and we don’t have to worry about dependencies when the task runs. This did cause some challenges, however, with setting up the agent on AWS EC2.</p><h3>Building out the Prefect Flow</h3><p>In Prefect, you construct a series of <em>tasks </em>into a <em>flow</em>. Each task is really just a python function with a decorator to recognise it as a task and ensure Prefect monitors it correctly. The flow then strings together multiple tasks and runs it as a complete program.</p><p>Our flow to our data lake is simple: we need to get the connection details for the database, get the data from the database, transform it, and finally write the data out to AWS S3.</p><p>Our basic flow looks like this:</p><h4>Imports</h4><p>Here are the imports we used (our complete code is part of a proprietary system, so no GitHub repository link is available, sorry):</p><pre># imports needed for code examples<br>from prefect import task, Flow, Parameter<br>from prefect.client.secrets import Secret<br>import pandas as pd<br>import pymysql<br>import boto3</pre><p>The associated python requirements are:</p><pre>prefect<br>pandas<br>pymysql<br>boto3</pre><h4>DB Connection Task</h4><p>The first task is to get the secrets from the Prefect Secrets store ready for passing into the other tasks. It’s a typical pattern when dealing with cloud systems (e.g. Prefect, GitHub Actions) to get passwords from their secrets store instead of hard coding it or storing it in a risky way. When running Prefect locally, secrets were provided via the TOML format Prefect config file (~/.prefect/config.toml). In cloud, they are stored in a Hashicorp Vault and entered via the Prefect cloud UI.</p><p>When retrieving values we took the approach of getting all the secrets at once, then passing them along to the other tasks as parameters. However, Prefect runs Secret().get() as a task so trying to get them all at once was a little redundant. Our approach was to set up a task returning a dictionary of the secrets (don’t do this, just do Secret(‘secretname’).get() where you need it):</p><pre>@task<br>def get_db_vars():<br>    &quot;&quot;&quot;Get the secrets.&quot;&quot;&quot;<br>    secret_db = {<br>        &#39;db_host&#39;: Secret(&quot;DB_HOST&quot;).get(),<br>        &#39;db_name&#39;: Secret(&quot;DB_NAME&quot;).get(),<br>        &#39;db_user&#39;: Secret(&quot;DB_USER&quot;).get(),<br>        &#39;db_pass&#39;: Secret(&quot;DB_PASS&quot;).get()}<br>    return secret_db</pre><h4>MySQL Extract Task</h4><p>After getting the secrets, our first real pipeline task is a MySQL extraction: we log into our database, grab a day’s worth of data, and return it as a pandas dataframe. Note the only thing that is ‘Prefect’ syntax here is the decorator for the function denoting it as a ‘task’.</p><pre>@task<br>def get_data(db_host, db_name, db_user, db_pass, extract_date=None):<br>  # connect to db<br>  connection = pymysql.connect(<br>   host=db_host,<br>   user=db_user,<br>   password=db_pass,<br>   db=db_name<br>  )</pre><pre>if extract_date is None:<br>   # get all of yesterday’s data<br>   query = &#39;<br>   SELECT * FROM reports<br>   WHERE date_start = SUBDATE(CURRENT_DATE, 1)&#39;<br>  else:<br>   # get data for a given day<br>   query = &#39;SELECT * FROM reports<br>   WHERE<br>   date_start ={extract_date}&#39;.format(extract_date=extract_date)</pre><pre>#read all the data you want into a dataframe<br>  report_df = pd.read_sql(query, connection)<br>  return report_df</pre><h4>Transformation Task</h4><p>Ah, the bit where the magic happens. Again, this task is relatively straightforward: it takes in the pandas dataframe, deserialises the data and makes sure dates and values are in the right format, and then returns it back to the main flow. Again, the only thing prefect-like about this is the task decorator for the function.</p><pre>@task<br>def transform_data(report_df):<br>    &quot;&quot;&quot;Convert data to decoded format.&quot;&quot;&quot;<br>    if len(report_df) &gt; 0:<br>        # iterate through and decode<br>        report_df[&#39;decoded&#39;] = report_df[&#39;encoded&#39;].map(decode)<br>    # drop the encoded column<br>    report_df = report_df.drop([&#39;encoded&#39;], axis=1)<br>    # finally, lets turn the dates into proper dates<br>    report_df[&#39;date_start&#39;] \= <br>        report_df[&#39;date_start&#39;].map(convert_date)<br>    return report_df</pre><h4>Formatted Data Output Task</h4><p>The final step in our flow: outputting the transformed data frame to S3. The key things to note are:</p><ol><li>Our output format is parquet</li><li>We ‘partition’ our data in S3 by using a filename that includes a partition name and value. So, if we were writing today.parquet, we’d instead write it in the following file location: <br>s3://my-bucket/data/tablename/partition=value/today.parquet<br>e.g. s3://datalake/data/reports/date=20210101/today.parquet</li></ol><p>This is later used by Glue to catalog the data, and when it is pointed at the data folder it will create a table called ‘reports’ with a partition scheme of ‘date’. A partition is to data lakes what an index is to databases — it helps the system find the data you are after more efficiently.</p><pre>@task<br>def write_transformed_data(report_df,<br>                           bucket_name=&#39;mah-bucket&#39;):<br>    &quot;&quot;&quot;Write transformed data to prod bucket.&quot;&quot;&quot;<br>    # get today&#39;s date<br>    prefix_date = datetime.now()<br>    if len(report_df) &gt; 0:<br>        # use the first record for naming instead<br>        prefix_date = report_df[&#39;date_start&#39;].iloc[0]<br>    prefix_date = prefix_date.strftime(&#39;%Y%m%d&#39;)</pre><pre>    prefix = &#39;data/reports/date=&#39; + prefix_date<br>    now = datetime.now().strftime(&#39;%Y%m%d-%H%M%S&#39;)<br>    filename = prefix + &#39;/&#39; + now + &#39;_reports.parquet&#39;<br>    print(&#39;Writing filename&#39;, filename)<br>    # write to s3<br>    session = boto3.Session(aws_access_key_id=Secret(<br>        &quot;AWS_CREDENTIALS&quot;).get()[&#39;ACCESS_KEY&#39;],<br>        aws_secret_access_key=Secret(<br>        &quot;AWS_CREDENTIALS&quot;).get()[&#39;SECRET_ACCESS_KEY&#39;])<br>    s3_client = session.client(&#39;s3&#39;)<br>    dataframe_to_s3(s3_client, report_df,<br>                    bucket_name, filepath=filename,<br>                    format=&#39;parquet&#39;)</pre><h4>Final Flow</h4><p>These three tasks make up our final flow, which is basically a standard ETL (Extract, Transform, Load) data pipeline task.</p><pre>with Flow(&#39;Extract reports to S3&#39;) as flow:<br>    # parameters for backfill<br>    end_date = Parameter(&#39;end_date&#39;, <br>                         default=str(datetime.now().date()))</pre><pre>    # actual flow<br>    secrets = get_db_vars()<br>    data = get_data(db_host=secrets[&#39;db_host&#39;],<br>                    db_name=secrets[&#39;db_name&#39;],<br>                    db_user=secrets[&#39;db_user&#39;],<br>                    db_pass=secrets[&#39;db_pass&#39;],<br>                    extract_date=end_date)<br>    # write_raw_data(data)<br>    transformed = transform_data(data)<br>    write_transformed_data(transformed)</pre><pre>flow.run()</pre><p>And, that’s it. You could cut and paste the code out of here and it would almost work: I’ve omitted some of the functions such as convert_date which are arbitrary transformations we do in our pipeline.</p><p>Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ac3628ab6e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Case Against Self-Service Data and Analytics]]></title>
            <link>https://medium.com/swlh/the-case-against-self-service-data-and-analytics-39a2bdb9164c?source=rss-673dc508e869------2</link>
            <guid isPermaLink="false">https://medium.com/p/39a2bdb9164c</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[self-service]]></category>
            <category><![CDATA[reporting]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Kat Hempstalk]]></dc:creator>
            <pubDate>Fri, 12 Feb 2021 05:45:26 GMT</pubDate>
            <atom:updated>2021-02-12T22:20:24.815Z</atom:updated>
            <content:encoded><![CDATA[<p>Damn. You just discovered the latest sales report doesn’t include the new category of product and you know it’s going to take forever for the always-busy internal analytics to make the tiny wee change to add it. Why can’t they just let you just do it yourself? In fact, you know what you’re doing and have authority, so you have them to add you to the tooling and you do exactly that. Done. See, now that wasn’t so hard, was it?</p><p>Sound familiar?</p><p>It should. I am yet to meet an analyst — or someone who works with one — who hasn’t felt this pain at some point. The answer typically is to give the people who ask for access, access. Once upon a time, I would have agreed with this and I would have been the person asking for access too! But, I’ve since come to realise that going all-in on self-service is the wrong approach.</p><h4>Why data is walled off</h4><p>Let’s take a step back and discuss why the data and analytics was walled off in the first place. Reasons might include:</p><ul><li><strong>Security</strong>: privacy, security, and contractual requirements might mean access to data is limited. Best-practice security usually requires minimum access: only those who absolutely need it to do their jobs get access.</li><li><strong>Structure</strong>: the data and reporting simply isn’t organised well enough to make it easy or safe to share.</li><li><strong>Tooling</strong>: there aren’t the tools available to work with the data or reports. Or, even if the tooling is available it might be excessively expensive or challenging to work with which in turn limits the number of people who you give access to.</li><li><strong>Literacy</strong>: the data is organised well but there’s a lack of trust that you will treat it with due care, giving more people access would risk the ‘cleanliness’ or integrity of that structure. Also, there may be a lack of trust, or a genuine lack of competence, that you know what the data means or how to correctly analyse it. Giving unfettered access will therefore endanger decision making that may be based on the data or reports.</li><li><strong>Interest</strong>: no one cares about data enough to want to look at the data themselves, they would much prefer someone else did it for them.</li></ul><p>These reasons are all good reasons to limit access to data and analytics: they ensure your organisation is able to make sound data-driven decisions, without putting the data (or organisation) at risk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YWlvD35LysmfonwQsTTJbg.png" /><figcaption>A real fortress. Well, an old one that was easier to find than a data fortress.</figcaption></figure><h4>The impact of a fortress</h4><p>However, even though there might be great reasons to limit access to data and analytics, new problems are caused by doing so. The most common problem I’ve experienced is an analytics team who simply can’t keep up with demand: they have so many requests for reports they have lead times of weeks for tasks that may only take minutes to complete. In turn, this generates a behaviour amongst analysts of wanting to be the knight in shining armour that rides in and saves the day by turning around a task quickly, or by allowing their friends to jump the queue. It may also have the opposite effect: analysts get so tired of repeating the same tasks over and over they get bored and quit.</p><p>You might find you constantly need to keep adding analysts to the team, but even as the team grows the backlog never seems to shrink.</p><p>With a fortress built around data and reports it may also be difficult for those needing it for decision making to get information in time — so even though the data is eventually provided and is correct, it’s not used because it is provided too late. Decision makers may lose interest in working with the analytics team because they are viewed as ‘slow’ (even though they are always busy!). Instead, they begin to make decisions on ‘gut feel’, and may lose interest in the data entirely.</p><h4>Lean / agile may help…</h4><figure><img alt="Kanban board" src="https://cdn-images-1.medium.com/max/1024/0*NxitcLP0ytoU3fG6.jpg" /><figcaption>Kanban board. Image courtesy of <a href="https://commons.wikimedia.org/wiki/File:Simple-kanban-board-.jpg">Wikimedia Commons</a>.</figcaption></figure><p>… but prioritisation is not the only problem. We managed to speed up our central analytics team at one organisation by improving our planning process. We limited work in progress and used cost-of-delay to prioritise tasks, and worked off a Kanban board with daily standups, and weekly grooming and retrospectives. It had a great effect on analyst behaviours: because there was now a defined process for prioritization it was difficult for jobs to jump the queue, or analysts to pick easier/smaller tasks to look like heroes. It also caused us to improve some of our lead times — particularly for smaller and more urgent tasks — but ultimately the size of the backlog was still ridiculous and the problem of data being relatively inaccessible still remained.</p><h4><strong>Self-service to the rescue?</strong></h4><p>The answer at this point seems easy: to lessen the load on the analytics team, just open up access so people can sort themselves out. This means less tickets coming into the team so they can work on the ‘hard’ stuff, and the smaller jobs go away and take care of themselves. And they do.</p><p>But then again, they don’t. Short term you may have solved the speed which which you can make decisions from data, but long term you might find:</p><ul><li>It’s difficult to audit who has access to what (and whether they should), and whether they have accessed data they have permission to. For example, giving access to the server containing all of the data warehouses might mean you have a simple permissions model, but perhaps there’s a table with personal data in there and you’ve unwittingly given an employee a means to stalk an ex. (Ok, this is kind of worse case, but stranger things have happened!)</li><li>Your ‘clean’ data store suddenly has hundreds more tables/files/etc, no one knows what half of them are for but you’re too scared to delete them in case something breaks.</li><li>You’ve blown your budget with tooling: suddenly you’ve gone from 10 licenses to 1000 because everyone <em>needs</em> access.</li><li>Analytics ‘silos’ form: the manager in another part of the org who now has access realises they are wasting too much time trying to do it themselves, so they hire someone person to do it for them. That person at best might keep that manager happy, but at worst is a law unto themselves who leaves behind a trail of destruction (including incorrect reports, poorly written code, and their own data silos) and ultimately more work for the main analytics team.</li></ul><p>And, most likely: even if the underlying data/analytics is standardised, you end up with several versions of ‘the truth’. This makes analysts sad. One of the reasons data-driven decisions are powerful is because they are objective. You are able to make a decision based on fact, bringing in emotion by choice. When several versions of ‘the truth’ exist decision making becomes a more emotional decision about whose version to trust. And as most analysts will tell you: you can torture data to say whatever you want. So, the data-driven decision making in this scenario is actually the loudest-(or most highly paid)-person-in-the-room decision making.</p><h4>If not self-service then what?</h4><p><strong>It’s a balance</strong>. Self-service is great, but it needs guard rails: protections in place to ensure it achieves the outcomes intended without creating more problems along the way. There is value in structure and security: ensuring not just that the data is safe, but the decisions made from it are sound, too.</p><p>I have been fortunate to work in a number of great technology companies and each has been at different stages of data and analytics competency — as I too have! <strong>But, the answer for me has always been about relationship building.</strong></p><figure><img alt="Two people talking" src="https://cdn-images-1.medium.com/max/200/0*xqWU6_aj-vZYpNYX.png" /><figcaption>Image courtesy of <a href="https://thenounproject.com/term/relationship/1016570/">the Noun Project</a>.</figcaption></figure><p>As an analyst there was value in collaborating: in retrospect if I had spent more time working together and sharing with other analysts we would have found the commonalities so work wasn’t repeated. We could have easily standardised and automated common reports if only we’d bothered to talk to each other rather than independently working on tasks. This would have removed the repetition from our backlog, and generated a better culture amongst the team.</p><p>As an data and analytics manager it’s worth taking the time to build the relationships with not just peers — but more senior and junior people too — and understand what outcomes they hold most dear. Understanding their needs makes it easy to draw a line (with them!) between what goes data needs to stay in the fortress, and what can be released for general consumption. Once you have an agreed set of data / measures / graphs / reports / dashboards with agreed access levels and tooling, it frees up the data team the data team to work on more interesting tasks and create more value for the organisation. And when you have a relationship, you’ll both be more keen to help each other get to where you want to go.</p><p>Building relationships can be hard for analysts as many of us are introverts and other people are scary. But, start small with one person and see how you go. The alternative is probably more daunting!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39a2bdb9164c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/the-case-against-self-service-data-and-analytics-39a2bdb9164c">The Case Against Self-Service Data and Analytics</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Basic Data Lake With Prefect, S3, Glue, and Athena]]></title>
            <link>https://medium.com/swlh/building-a-basic-data-lake-with-prefect-s3-glue-and-athena-676410fbbdc4?source=rss-673dc508e869------2</link>
            <guid isPermaLink="false">https://medium.com/p/676410fbbdc4</guid>
            <category><![CDATA[prefect]]></category>
            <category><![CDATA[athena]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[s3]]></category>
            <dc:creator><![CDATA[Kat Hempstalk]]></dc:creator>
            <pubDate>Wed, 13 Jan 2021 00:29:58 GMT</pubDate>
            <atom:updated>2021-01-13T09:51:29.936Z</atom:updated>
            <content:encoded><![CDATA[<p>I’m currently working with an indoor sensor startup, <a href="https://smoothsensors.io/">Smooth Sensors</a>, and one of our challenges so far has been bespoke data analysis: when a customer has asked us to tell them something interesting from their data that is not part of the standard reporting we provide. The main reason this is a problem is that our sensor platform serialises the data in a highly compact format to minimise data transfer, and is stored in our database without deserialisation. In order to perform any analysis — including simple daily means and ranges of values — we go through the painful process of first extracting the data of interest locally, deserialising it, and finally analysing it.</p><p>Obviously this isn’t ideal, and as an interim measure to solving this without a massive system rewrite we decided to build a basic data lake. Being a startup the key factor here is cost: we simply don’t have the funds to use the technologies we’d like to because most of them incur substantial upfront costs. The advantage with a data lake is that storage is cheap, and compute only costs us when we need it.</p><h3>Building a basic data pipeline</h3><p>We quickly settled on S3 as a good storage medium: we can pack a years worth of our data into AWS S3 for the low cost of $2 USD per month — even though we record over five million data points each day! We also already use AWS for the core platform, so whilst we might get started faster with something like BigQuery, we’d prefer to keep everything in one place for now.</p><p>Our main database is hosted using AWS Aurora, so our first step is to extract the data from our database, and dump it somewhat unceremoniously into S3. AWS does provide AWS Data Pipelines, which allows you to schedule extracts from databases in various ways, but at the time of writing this the documentation was woefully inadequate with respect to setting it up with Aurora (oddly?). Also, although the costs are not excessive ($1USD per pipeline per month, not including resources used for running the pipeline) we weren’t sure how this would scale if we needed it to.</p><p>In the end, we scripted our pipeline using <a href="https://www.prefect.io/">Prefect</a>. Prefect is a python library that is specifically designed for dataflow automation: it allows for scheduling of tasks, and progress reporting on task success or failure. It has the added advantage of setting us up nicely if we want to scale our pipelines in future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VLkbmziw_5q5s3JFgPH7qQ.png" /><figcaption>Our pipeline, viewed as the tasks in a timeline</figcaption></figure><p>In Prefect, each pipeline is a <em>flow</em>, and the <em>flow</em> performs multiple <em>tasks </em>— like extracting the data, transforming the data, or loading the data. The <em>flow</em> is a directed acyclic graph (DAG), meaning that tasks that are dependent on others won’t be run unless their predecessors complete successfully. Although Prefect has a cloud offering, and a nice UI if you choose to self-host the engine, it’s also possible to run the flow as a python script locally, rather than as part of an orchestrated workflow automation system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HteV-_oSKGyBnacRQmd5aw.png" /><figcaption>Our extract workflow as a DAG in Prefect. All those little tasks are just to get the secrets (e.g. database host, username, password etc).</figcaption></figure><p>For those of your reading this and wondering “why not Airflow?”, well, good question. Amazon does have a managed version of AirFlow, too. And although we’ve got nothing against Airflow, Prefect is pretty swish and was simple to get started so we went with that. And we used the cloud variant: the free tier covered our first use case of extracting the data, and we felt it would scale easily and without excessive cost as we might (hopefully!) need.</p><p>Our first Prefect pipeline is straightforward: it has one task to get the data out of our MySQL database, a second task to transform that data (deserialise it), and a third task to finally write it out to S3. We also ended up including some table copies in our final flow. Some example code for doing this will be provided in a follow-up post.</p><p>One key thing we did here to minimise cost was to use CloudWatch events and AWS Lambda functions to turn on/off our Prefect agents. We only need to be running our pipeline for a small time each day, so having an instance on and idling doesn’t make much financial sense. <a href="https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/">The guide on how to turn on/off instances on a schedule is well-documented (click here to go to it) </a>and we were able to set this up in less than half an hour.</p><h3>Building a basic data store structure</h3><p>AWS S3 is basically a bottomless file system, but one advantage of using this store (aside from it being extremely cheap!) is that if you add a little bit of structure to your filenames you can turn it into a partition-ready big data store.</p><p>The trick is this, your filenames should be in the format (we are using parquet as the data format, cos it’s awesome):</p><pre>s3://bucketname/databasename/tablename/partitionname=partitionvalue/data.parquet</pre><p>Replacing all the names with the appropriate names for your system. For example, our structure looks something like this:</p><pre>s3://my-data-bucket/sensor-data/raw-sensor-data/date=20210101/data.parquet</pre><p>Okay, okay, it just looks like a nice way to structure files, but I promise you being pedantic about this file structure is worth it. Parquet is also worth considering if your data has lots of rows and fixed columns: it’s an efficient file format for that kind of data and in our case what is 250MB of uncompressed CSV files (30MB compressed as gzip) turns into 20MB of parquet data with no data loss.</p><h3>Supercharging the data store with Glue and Athena</h3><p>By now we’ve got our data piped out to S3, and neatly stashed away in a tidy filesystem — but that doesn’t actually help us with doing anything useful with it. In order to operate on it and be able to do SQL queries over it like its a database, we need to utilise Glue and Athena.</p><p><a href="https://aws.amazon.com/glue/">AWS Glue</a> is a cataloguer: it runs over the S3 bucket and makes a record of what data it finds, and the schema it is in. Conveniently because we used parquet the schema is encoded in the data files, so we don’t need to use any custom data classifiers. Also, because we structured the filesystem in the way Glue expects, we can just point Glue at the S3 bucket, and set it to run just after our Prefect pipeline, and hey presto, it finds the fresh data moments after it is written to the store and enters it into its catalog.</p><p>Speaking of Presto, our final data lake step is <a href="https://aws.amazon.com/athena/">AWS Athena</a>, which is an AWS version of the popular <a href="https://prestodb.io/">Presto distributed query engine</a>. It uses the Glue Catalog to know where to find the data we’ve tucked away in S3, and by default anything that we’ve catalogued using Glue will appear in the Athena interface. This means we can run our SQL query over all of our deserialised data, and summarise it in a matter of seconds… for only a few cents.</p><p>The reason why it is so quick and cheap is thanks to our partitions: if we use date=datehere in our query WHERE clause, the query engine only needs to look at those folders — quickly narrowing the data that is scanned. This reduces cost (as well as making it fast) because Athena charges you based on the data that is scanned.</p><p>In fact, our basic data lake costs us less than $5 USD to maintain — a few cents per month for every sensor we have deployed in the field. We don’t expect to maintain that cost (i.e. it’ll go up) as we add more analysis, aggregations and data points going forward — but it’s a great start to unlocking our data for new customer use cases.</p><h3>Architecture Diagram</h3><p>Our final basic setup looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*wDuQCmVGUZbspY3jnWtzaA.png" /><figcaption>Final Basic Data Lake Setup</figcaption></figure><h3>In Summary</h3><p>Our basic data lake is really nothing momentous — yet. At present it’s just an easy to manage deserialised version of our main database. Some tables are copied, one is transformed via Prefect and S3, and then exposed using serverless services such as Glue and Athena. However, although it might be basic now — it’s also extremely cheap to run, and easily scales as our data and use cases grow. Our turnaround time on bespoke analysis has rapidly reduced, without a huge cost overhead.</p><p>The investment in time to build this wasn’t huge either: it took one developer three work days to complete the basic pipeline described here — even though they have never touched any of the technologies mentioned before. As a startup this small time and money investment has set us up extremely well for the future, and already paid for itself in reduced time spent performing data extraction and transformation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=676410fbbdc4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/building-a-basic-data-lake-with-prefect-s3-glue-and-athena-676410fbbdc4">Building a Basic Data Lake With Prefect, S3, Glue, and Athena</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[WTF Is a Data Lake?]]></title>
            <link>https://medium.com/swlh/wtf-is-a-data-lake-51c5a9ed2e0f?source=rss-673dc508e869------2</link>
            <guid isPermaLink="false">https://medium.com/p/51c5a9ed2e0f</guid>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[athena]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Kat Hempstalk]]></dc:creator>
            <pubDate>Mon, 04 Jan 2021 05:19:38 GMT</pubDate>
            <atom:updated>2021-01-08T07:29:55.975Z</atom:updated>
            <content:encoded><![CDATA[<p>A few days ago I was chatting to a programmer friend about being able to quickly build a basic data lake for some sensor data (more on that another time). After I had finished, they politely asked “what’s a data lake?” — which came as a shock to me because I was certain their organisation had one but clearly no one there had ever bothered to explain to them what a data lake actually is. Whilst I filled them in on the spot, I’m just going to leave this here in case it helps someone else too.</p><figure><img alt="Loch Ness — not a data lake." src="https://cdn-images-1.medium.com/max/1024/0*B6oo7DvXi-FrFawD.JPG" /><figcaption>Not a data lake. Photo source: <a href="https://commons.wikimedia.org/wiki/File:Loch_Ness.JPG">Wikimedia CC-BY-SA 3.0</a></figcaption></figure><h3>WTF is a database?</h3><p>Let’s start with the stuff that I hope most programmers know: databases. In general, most computer programs require some sort of data store — sometimes to store the state of the program, but usually to store information recorded by the program itself. In it’s simplest form it can be thought of like an Excel spreadsheet: a single table (or many tables if you use sheets) of columns and rows of data that have been added as they happen. In fact, some people use Excel as a database (please don’t), even though it’s not designed to handle the kind of operations you might expect from a database especially when dealing with large volumes of data (it just happens to be easy to use).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*-rqlN-ayCFZghrL-_jRlxg.png" /><figcaption>Simple purchases table</figcaption></figure><p>Common database software includes Microsoft SQL Server, PostgreSQL, and MySQL. Cloud platforms such as AWS, GCP and Azure often have their own cloud-based flavours of these (e.g. <a href="https://aws.amazon.com/rds/aurora/">AWS Aurora</a>, <a href="https://cloud.google.com/sql">GCP Cloud SQL</a> and <a href="https://azure.microsoft.com/en-us/services/sql-database/">Azure SQL Database</a>). SQL stands for Sequential Query Language and is the type of code used to interact with data inside these databases.</p><p>Databases for production software are often known as <em>transactional</em> databases, partly because they record individual <em>transactions </em>such as a purchase, a booking, a search on a website or a click on a webpage. But, in technical terms, it’s actually because they support the safe rolling-back of such transactions should there been an error.</p><p>They can be and also often are <em>relational</em> databases, when different tables are related to each other in some way: for example, you might have a table of users and other tables such as clicks, searches and bookings where you record the transaction the user has performed. In this example the tables are related to each other by users.</p><p>The key thing about databases (particularly at scale) is that they often need to be fast to store and retrieve data, so that the user gets a good experience interacting with the software. This means the database is often expensive to run, as high-end hardware is required to keep it fast even when its full of data and always busy processing it.</p><h3><strong>WTF is a data warehouse?</strong></h3><p>Although databases are great at transactional data, they can get slow when you want to do any significant number crunching — such as summarising how many customers have made a booking today — because these transactions typically require a lot of compute power to calculate and this will take valuable processing time away from the machine being able to store and retrieve normal transactional data. To maintain the speed of production software, often programmers (and database administrators) will choose to separate analytics and reporting from the main database by using a data warehouse.</p><p>A data warehouse is simply a database that is (typically) separated from your main production database and records <em>aggregated</em> data or sometimes <em>archival</em> data. Often it is initially created with a read-only snapshot of your production database, with various stored procedures run over it to produce new additional tables. These new tables might do things like take an original where each row is an individual purchase and create a new table where each row is the total purchases for a given customer on a given day (or summarise it to an even higher level such as a row being the total purchases of all customers on a given day).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*CPR_rmFzjzrpFUvUF5jU6w.png" /><figcaption>Simple data warehouse table for purchases — note this is an aggregated form of the database table shown earlier.</figcaption></figure><p>Data warehouses are also often used to bring [copies of] many different production databases into one place where they can be easily queried.</p><p>Typical software to run a data warehouse is the same as for databases (they are just a special kind of database, after all!), but there is also software available that is optimised for this purpose. A common tool is an OLAP (OnLine Analytics Processing) database, where tables are set up in <em>Facts </em>and <em>Dimensions</em>, to make it easier to create on-the-fly reports. More modern tools such as <a href="https://www.snowflake.com/">Snowflake</a> separate storage and compute even further and can easily summarise production transactional data on-the-fly.</p><p>The servers that run a data warehouse aren’t usually as beefy as those for a transactional database; most of the time the analytics they perform are not needed immediately and users can afford to wait for their reports. Data warehouses may not have the same resilience for the same reason: they are not customer facing and not having the same uptime or backup strategy as core production databases isn’t going to affect the customer experience. Of course, this doesn’t stop people from using them in production, even when cautioned not to!</p><p>Sometimes (sadly) data warehouses can also become a bit of a free-for-all, where many users get access to create tables and reports, and without appropriate curation you can end up with many versions of ‘the truth’. This can be resolved by ensuring there is someone responsible for the governance of data within the warehouse, not just the production database(s).</p><p>Anyway, the key thing about data warehouses is that they are aggregated databases used for reporting purposes, typically on separate machines from their production database counterparts.</p><h3>Ok, ok, get to the point already — WTF is a data lake?</h3><p>In a world that’s hot on Big Data and Artificial Intelligence, there’s now a need to mix the responsiveness of production databases and all the data they hold, with the power of bespoke summarisation from data warehouses: and this is the sweet spot of a data lake. They were initially heralded as a dumping ground for all your data, kind of a “just throw it in as you have it and let the software do the work” — but the software to do this hasn’t fully matured yet.</p><p>Instead, data lakes are a place where storage is cheap and data can be stored in an unstructured way (like an AWS S3 bucket), but are often semi-structured to make it easier to both enforce privacy and to perform analytics. This structure can be as simple as file/folder naming and location conventions, whereby naming your files with the right format and placing them in the right folder tells the query interface how it should be operated on.</p><p>Compute is usually separate from storage for a data lake: any data processing typically happens on-demand by temporary compute resource, rather than an always-on server that is waiting for instructions. This means that data lakes — unlike their database and data warehouse counterparts — are not limited in the operations they can perform since they usually don’t have fixed hardware.</p><p>However, data lakes are not curated like data warehouses (although they can be): data is usually stored raw and may only ever be aggregated or filtered on-the-fly, rather than by a stored procedure that neatly stores it in a table. Also, data lakes do not typically contain data that is as up-to-date as production databases (but it’s often not far off it): a source of the data in the lake may be the transactions in the database, so data can only flow through to the lake once it has been successfully processed elsewhere. Data can also be rawer than in the production database, recording events as they happen rather than the end result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*1_ixoQAUqoZ9-IPyxHqmDg.png" /><figcaption>Data Lake table example — click data leading to purchase that would normally be thrown away</figcaption></figure><p>One common analogy is that a data lake is a like a real-world lake: an unfiltered and massive body of water. In comparison, a data warehouse is a bottle of water: filtered, clean and safe to drink.</p><p>Although they are a form of database (a bit like Excel being a database), and you could make an (expensive and slow) data lake using database software, the software is a different beast and is usually made up of a storage component (e.g. AWS S3) and one or more compute component(s) (e.g. AWS EMR or AWS Athena). Software for data lakes at present are typically in cloud-hosted environments: AWS, Azure, and GCP. Because of the storage/compute separate they can easily be set up in a bespoke way using multiple cloud(s) and also with on-premise solutions (e.g. a storage cluster running Hadoop and Spark).</p><h4>So, why use one?</h4><p>Data lakes can store more than both production databases and their respective data warehouses combined: storage is cheap so often all data is thrown into the lake, regardless of whether it is useful or not. A good example is click data: a production database might only store a day worth of data for efficiency reasons (i.e. let’s not fill up our hard drive and bring our database down), a data warehouse might summarise that day into a single row in its store (this is probably what we’d report on), whereas a data lake might store all click events for all time in the unsummarised format.</p><p>The advantage of having a data lake (in addition to the other stores) is that you can start to utilise some of the data you might have otherwise thrown away. You can now report with greater granularity, or run power-hungry tasks such as training machine learning models directly from your lake (rather than on, say, extracts from production). And best of all, data scientists won’t bring down production data stores when they try to build their own SkyNet!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51c5a9ed2e0f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/wtf-is-a-data-lake-51c5a9ed2e0f">WTF Is a Data Lake?</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[It is done.]]></title>
            <link>https://drkatnz.medium.com/it-is-done-af0ca35835ff?source=rss-673dc508e869------2</link>
            <guid isPermaLink="false">https://medium.com/p/af0ca35835ff</guid>
            <dc:creator><![CDATA[Kat Hempstalk]]></dc:creator>
            <pubDate>Tue, 05 Sep 2017 07:26:54 GMT</pubDate>
            <atom:updated>2017-09-05T07:26:54.800Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NcxaIAmY2U4VSoAQPczy3g.jpeg" /><figcaption>The final guitar (so much awesome the camera couldn’t focus)</figcaption></figure><p>6 months ago I decided to build my own guitar, today… I finished that guitar. So, so, happy.</p><p>Last time I posted about the guitar I was busy with the clear coat — or more specifically, dealing with the drips down the back. Turns out that sanded out really easily, so it was just a case of sanding where I needed to, and having another go. I think I probably put half a dozen “coats” on in the end. I gave it more than the rest because it wasn’t nearly as thick as the colour coat.</p><p>And then I sanded. The Internet led me to believe that I could sand this clear coat down, then apply car polish to bring it back up to a shine. The Internet was wrong. My gloss coat ended up being a satin finish. I thought about putting a final clear coat on after my last sand, but then decided that I probably shouldn’t do that after putting car polish on the coat. And so a satin finish it was. I was quite upset about this until I got the guitar together and decided it looked pretty awesome in satin and probably would have looked pretty naff in gloss. That’s what I’m going to keep telling myself anyway.</p><p>The last part of the build was actually the least difficult — it was just a matter of screwing everything into the right places (and using the holes I’d drilled earlier). Also, because I’d made a tiny shim for the neck it only took a little bit of setup (read: bridge adjustment) to get the strings to a height where they didn’t buzz and the whole thing is playable. In tune. Just… perfect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b1wD5jGbEsP12Wl2Jz61KQ.jpeg" /><figcaption>Body after the last sand.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cb2MHrAKYXp-EuQSokzwCQ.jpeg" /><figcaption>Tremolo claw installed. The tremolo came with 4 springs but there was 5 holes and symmetry seemed important for sanity.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*md8Jx9Z6NlzaO2Q-sYR2_g.jpeg" /><figcaption>Finished body with everything screwed in</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j8YlikB1Fy4HUylQXBZBHA.jpeg" /><figcaption>The back side with the cover plate installed.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TcRmgyuxrL1eIuUdmk_ysg.jpeg" /><figcaption>Neck on (neck plate is gold, photo makes it look black).</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AVA2uef8XBeLzhEruHPZ_g.jpeg" /><figcaption>So pretty.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af0ca35835ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Home automation with a Broadlink RM Mini 3 and Google Home]]></title>
            <link>https://drkatnz.medium.com/home-automation-with-a-broadlink-rm-mini-3-and-google-home-712faa074254?source=rss-673dc508e869------2</link>
            <guid isPermaLink="false">https://medium.com/p/712faa074254</guid>
            <category><![CDATA[ifttt]]></category>
            <category><![CDATA[google-home]]></category>
            <category><![CDATA[universal-remote]]></category>
            <category><![CDATA[broadlink-rm-mini-3]]></category>
            <category><![CDATA[stringify]]></category>
            <dc:creator><![CDATA[Kat Hempstalk]]></dc:creator>
            <pubDate>Sat, 02 Sep 2017 00:23:30 GMT</pubDate>
            <atom:updated>2018-06-23T09:24:27.169Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>NB: There is now a better way of using the RM Mini 3 with Google Home — simply install the Broadlink ihc app, hook up the RM Mini into the app, then go into the Google Home app and link the device. If you still want to try hard mode, feel free to keep reading.</strong></p><p>I recently purchased a Google Home device and have been since been looking at ways of using it to automate more things around my house. I came across <a href="https://www.reddit.com/r/googlehome/comments/5bmhqf/is_there_an_ir_blaster_i_can_get_that_works_with/">this reddit thread</a> which suggested using the <a href="http://www.ibroadlink.com/rmMini3/">Broadlink RM Mini 3</a> (aka Blackbean) device as a universal remote (IR blaster). At $25NZ delivered (at the time of purchase) I figured it wouldn’t hurt to try it out.</p><p>I bought this one off AliExpress: <a href="http://s.aliexpress.com/bEZfaYJb">http://s.aliexpress.com/bEZfaYJb</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*EE9aM3pPf6Q6Oy7KApRPvg.png" /></figure><p>Took about a week to get to New Zealand. The reddit thread suggests a complicated way of setting up the device with about half a dozen different Android Apps in the mix. After much trial and error I found a simpler way.</p><p>I used (all free):<br>- If This Then That (<a href="https://ifttt.com/">https://ifttt.com/</a>)<br>- Stringify (<a href="https://www.stringify.com/">https://www.stringify.com/</a>) <br>- RM Bridge (<a href="http://rm-bridge.fun2code.de/">http://rm-bridge.fun2code.de/</a>)</p><p>Interestingly, you may only need one of the first two (IFTTT/Stringify) depending on whether you are ok with prefixing your command to Google Home as “Ok Google, tell Stringify …” (then just use Stringify), or whether you are just performing a single action (IFTTT) or multiple (Stringify). You’ll definitely need the RM Bridge app though.</p><h3>Setting up RM Bridge</h3><p><a href="http://rm-bridge.fun2code.de/">RM Bridge</a> is the app that talks to the actual Broadlink RM Mini 3. I did the initial device setup using the <a href="http://www.ibroadlink.com/app/">Broadlink App</a> — which just hooks up the device onto the Wifi network. I ended up deleting the Broadlink app as it had some weird permission requests and kept trying to install stuff on my phone. In short: I didn’t trust it. The RM Bridge app doesn’t rely on the base Broadlink App, so after you’ve done the device setup you can just remove it.</p><p>RM Bridge is a small HTTP server that runs on your phone that can talk to the Mini device. There isn’t default support for the Mini 3 but there’s a workaround here: <a href="http://fun2code-blog.blogspot.co.nz/2016/07/rm3-mini-workaround.html">http://fun2code-blog.blogspot.co.nz/2016/07/rm3-mini-workaround.html</a></p><p>In order to use the workaround you’ll need to know the MAC address of the RM Mini 3 on the network. I found this by looking at the active devices on my router.</p><p>The RM Bridge app gives you an IP:port to talk to the app on. On <a href="http://rm-bridge.fun2code.de/rm_manage/index.html">the code learning page</a> you can add codes — you type a name for the code, hit “learn code” and then punch the button on the remote. If successful it will return you a URL that you can pop in a browser on your local network and have it fire the code.</p><p>This gave me a code that looked like this:<br>http://myphoneiphere:port/code/TVonoff</p><p>Unfortunately this will only work on the local network. IFTTT/Stringify require a publicly accessible URL, so it meant I had to look up my IP (http://whatsmyip.org) and also port forward the port on my router to the RM bridge port on my phone. I enabled the auth on the RM Bridge app too, because if I’m making this accessible to the outside world I’d like to make sure it’s not easy for malicious people to get to it.</p><p>Finally, my URL was:<br>http://mypublicip:port/code/TVonoff</p><h3>Setting up Stringify</h3><p><a href="https://www.stringify.com/">Stringify</a> is an app that lets you bind <strong>multiple actions</strong> to a condition, a little bit like IFTTT (which only allows you to bind a <strong>single </strong>action to a condition). I wanted to be able to turn on my TV and sound system together, so I had multiple actions to do — one RM bridge call for each. Stringify actually lets you hook up to the Google Assistant directly, so I didn’t strictly need IFTTT in the loop, but you have to start with “Ok Google, tell Stringify &lt;x&gt;” and I wanted to be able to say “Ok Google, turn the TV on” not “Ok Google, tell Stringify to turn the TV on”. Less is more.</p><p>I created a new flow, adding 4 items: first an “IFTTT” as my “when condition”, followed by a “Maker” call (web URL with GET call using RM bridge links), a two second timer, and a second “Maker” call. The idea was that when IFTTT triggers, the TV turns on (using the public URL), and two seconds later the sound system turns on too. The reason I put a delay in there is that I didn’t want the RM Mini to try and execute two codes at once.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*O24I07P6kFT9eFUqqdsaig.png" /><figcaption>My flow for the TV</figcaption></figure><h3>Setting up IFTTT</h3><p><a href="https://ifttt.com/">If This Then That</a> (IFTTT) is a popular tool for executing a single action when a condition occurs. The advantage with using this tool is that I don’t have to prefix my command to Google Home with “tell Stringify…”. If you are OK with that, you can swap the IFTTT step in Stringify for a Google Assistant condition and you’ll have a working system. You will need to link your Stringify account with the Google Assistant if you do this, it will guide you through this when you add the Assistant to the flow for the first time.</p><p>In IFTTT I set up a simple action so that when I said “Turn the TV on” to the Google assistant it would trigger the Stringify flow I had set up earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*mFKYkbK04JDIqBRe2oMXMA.png" /><figcaption>Stringify flow</figcaption></figure><p>If you have just one action to perform (i.e. one Maker URL call) you can just use the IFTTT app — no need for Stringify . One thing to note is that the Maker/Web URL call is called “Webhooks” in IFTTT now.</p><p>Now when I say “Turn the TV on” — after a few seconds delay both the TV and the sound system comes on. Good times.</p><p><strong>Edit</strong>: one small catch: the RM bridge app likes eating the battery on my phone — not surprising as it’s running a web server I guess. If you have an always-on server on your network, there’s <a href="https://github.com/jor3l/broadlinkrm-ifttt">this webhook server</a> that might help. Also <a href="https://github.com/davorf/BlackBeanControl">this python library</a>, and <a href="https://home-assistant.io/">the home assistant app</a> that you could try.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=712faa074254" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The scary bits of my guitar]]></title>
            <link>https://drkatnz.medium.com/the-scary-bits-of-my-guitar-5ef22c42a453?source=rss-673dc508e869------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ef22c42a453</guid>
            <category><![CDATA[diy]]></category>
            <dc:creator><![CDATA[Kat Hempstalk]]></dc:creator>
            <pubDate>Tue, 29 Aug 2017 09:07:52 GMT</pubDate>
            <atom:updated>2017-08-29T09:07:52.127Z</atom:updated>
            <content:encoded><![CDATA[<p>After what seemed like an eternity, all the bits and pieces finally arrived. Unfortunately the company I ordered from forgot to pack one of the key bits, but it could be sourced locally so only had to wait a couple of days for that last bit (the nut).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sTKTW4EJ0K42saswNiB4mw.jpeg" /><figcaption>My guitar’s neck</figcaption></figure><p>One of the interesting things about the neck is that the frets had been fitted before the final coats of paint had been put on, so the frets have paint… some of which is chipping off. I was told this was often fixed with steel wool, but after trying that decided that I was also probably filing the frets down too. So I cleaned off the worst of it and just left it as is. I’m pretty sure it will come off whilst being played, anyway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pn5kADlRYIE4OtiA9OM7Yg.jpeg" /><figcaption>Tuners, bridge, strings, and some other bits and pieces</figcaption></figure><p>I had been waiting for all the bits before doing final painting, mostly because I needed to screw everything in and I figured drilling a finished body would be far more stressful than one that I could easily touch up. I decided it would be a bad idea to drill in the pickguard (with pickups) until I had the neck, tuners and bridge screwed in, since otherwise the strings might not align under the pickups (this is important, misplaced strings = bad sound).</p><p>The neck I had was mostly finished: I needed to fit the tuners, fit the nut, and drill some holes so it fitted into the body. This is not as straight forward as it sounds. The nut was too wide for the slot in the neck, so I had to sand it down in all directions until it fitted into the slot at the top. I was going to glue it down with proper glue, but decided hot glue might be a safer bet if it needed to be removed later.</p><p>The tuners were… harder. The holes that had been predrilled in the neck weren’t quite large enough to fit the bushings for the tuners, so I had to file them out. I had ordered a reamer from TradeMe to do this, but it was taking forever to arrive, so ended up buying a bastard file from Mitre 10. Patience is not my strong suit when it comes to toys. Each hole had to be carefully filed out, without being made oval, or chipping the finish. This meant LOTS of masking tape, and I still managed to chip the finish on at least two of the holes. Fortunately, I had filed from the front and chipped on the back, so the stuff-ups were conveniently covered with the tuners later on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QNVftUOUdJH5kKWNAmvnmg.jpeg" /><figcaption>Bushing on the left. This is the general concept — file in hole.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*crp2uhNDR-WQuwh6tWkgiA.jpeg" /><figcaption>In the sun with the file and the neck, more tape was added before any filing was done.</figcaption></figure><p>Once the holes were filed out I had to squish a bushing into them. The bushings were supposed to be tight — so I only filed out the holes enough that I could easily get them in about halfway. Pliers were used to put them completely in. I wrapped the head in a towel so the pliers wouldn’t crunch and ruin the finish: this turned out to be a bad plan because the towel fluff caught in the bushings as I pushed them in. There’s probably still some fluff in there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KCSJhszR_2N-BE3KxuRy_A.jpeg" /><figcaption>Bushings in.</figcaption></figure><p>After the bushings were fitted, the tuners are dropped through from the other side, screw holes marked, tuners removed, screw holes drilled, tuners replaced and screwed in. The hard part is getting the tuners straight. Even though the pegs fit through the tuners and can’t move, they can rotate in the hole. So, I held a ruler hard against the tuners as I marked the holes to ensure they were in the right position. Also, the holes weren’t marked with a sharpie: I used a drill bit to start the holes, just gently rotating it by hand.</p><p>Drilling the holes was stressful too: I wanted to make sure I wasn’t going to go through to the other side of the neck. The Internet showed me that I could put some tape on the drill bit to match the depth of the screw, and to not drill past the tape. This worked like magic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_rye3_Ec4u7CIj8rpHLxjw.jpeg" /><figcaption>Drilling with tape. Note the chip from filing out the second hole. Not cutey cutey.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mHi-m1OjMHcKyq5bGhGCGw.jpeg" /><figcaption>Completed drilling.</figcaption></figure><p>Once the holes were drilled, I screwed the tuners in — again holding a ruler to the side to make sure everything lined up. It did. I am so happy with the result. It’s the bit I’m most proud of so far.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZP3uJj9VsLdmBTurrfJzyA.jpeg" /><figcaption>Tuners from front (nut in too).</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UrVUN145_kHutN5oTXeP3A.jpeg" /><figcaption>Tuners from back. Just… gorgeous.</figcaption></figure><h3>And now, a non-pictorial update…</h3><p>Because the parts took so long, I haven’t been able to paint the guitar with it’s final colour coat, and clear coats… because I was waiting to line everything up. Now I didn’t actually take photos of this bit, but after getting the tuners in, I screwed the bridge in gently, clamped the neck in gently, and put the two outside (E) strings on the guitar. I didn’t put the strings under tension — which turned out to be a mistake because the alignment could have been better — but lined up the pickguard (with pickups) under the strings and then marked and drilled out the holes for that. Plus all the other holes for the backplate, strap locks and so on.</p><p>Then… the scary bit — drilling the holes in the neck to bolt it to the body. It was scary because it turned out the alignment of the pocket wasn’t very good, and so had I bolted it in without the strings in alignment the low E string would have been off the side of the neck! The thing would have been completely unplayable. Fortunately I put the strings under tension, realised what was happening, and tilted the neck into the correct alignment before tightening the clamp. I did mark the back of the neck slightly with the clamp, but I’m ok with this because it will be covered with a neck plate. I’m not convinced I got the holes quite right, but the strings look to line up pretty well so let’s hope they stay that way after the painting is finished.</p><p>Speaking of which… the final colour coat is on (yay!). Not 100% perfect but 99.95% so pretty darn close (I found a little dot of primer when I put the first clear coat on).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1cVcptdXxzp6v_D1eKPwDg.jpeg" /><figcaption>Final colour coat, post sanding.</figcaption></figure><p>I have also put on a layer of clear coat, but I didn’t realise the clear coat wasn’t nearly as thick (read: it’s runny) and so after spraying the front and turning the guitar around, discovered paint running down the back. it doesn’t mark as badly as colour coat does, but still, those runs will have to be sanded out. I also need to be far more patient with the coats — lighter coats with gaps in between to make sure it’s tacky rather than runny for the later coats.</p><p>I also changed the painting rig for the last colour coat — using some zip tied/duct taped coat hangers (and a broom) as a holder. The nice thing is I can move the guitar safely inside when I’m done, without risking dropping it or picking up dust on the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ix9ng3PNk_alG_CgpfYYSw.jpeg" /></figure><p>Smart, eh?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ef22c42a453" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Soldering up a storm]]></title>
            <link>https://drkatnz.medium.com/soldering-up-a-storm-ff7176101aae?source=rss-673dc508e869------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff7176101aae</guid>
            <category><![CDATA[hardware]]></category>
            <dc:creator><![CDATA[Kat Hempstalk]]></dc:creator>
            <pubDate>Sun, 06 Aug 2017 08:22:10 GMT</pubDate>
            <atom:updated>2017-08-06T08:22:10.049Z</atom:updated>
            <content:encoded><![CDATA[<p>Whilst most of the parts for my electric guitar are taking the slow boat out of the US to NZ, some of the smaller parts arrived earlier than I expected. This allowed me to take a break from sanding and painting, and instead solder up the pickguard with all the electronics.</p><p>A standard stratocaster has all the electrical components mounted on the pickguard, which is then screwed directly onto the body. I don’t have a lot of electronics experience, but as the instructions for the pickups pointed out: this isn’t rocket science. Here’s what the wiring diagram looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*60jeOVgC_F5GNUdqEASj7A.png" /><figcaption>Strat SSS wiring diagram — © Seymour Duncan</figcaption></figure><p>The pickguard I went for is cream, and I also bought some “creme” buttons/knobs to match. They weren’t a great match. The pickguard was spot-on the colour I wanted, but the knobs were a bit too dark. Thankfully something that is easily and cheaply replaced later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8xkspNpQ6iPnp60yjJvs1g.jpeg" /></figure><p>My plan is for the guitar to have a crimson body, cream plastics, and gold metal bridge/tuners/screws. So the remaining parts were bought with that in mind. I also wanted it to sound awesome, and settled on the <a href="http://www.seymourduncan.com/pickup/california-50s-set">Seymour Duncan California 50’s set</a> of pickups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Hk29WF6MSQveid0f8mg_A.jpeg" /></figure><p>Given that I don’t do a lot of electronics, my workbench is my dining room table, cunningly disguised with a towel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0r8ZqaR0-4jJq7qMevKgyw.jpeg" /><figcaption>Workbench</figcaption></figure><p>And now for more pictures…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DEkjLRIu96jA6409--hJ8g.jpeg" /><figcaption>The pots and 5 way switch in place.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jLhrCcUlCfCwGInLHRCFlQ.jpeg" /><figcaption>Pickups screwed in.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3pOj6AXqrH4bjQ-ntQBKgg.jpeg" /><figcaption>Front with everything screwed in.</figcaption></figure><p>And finally…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OCVbmIqY-Is3m8D7Zigwzw.jpeg" /><figcaption>All the wiring done!</figcaption></figure><p>It took just over an hour to get all the soldering done. I took my sweet time with it because I wanted to make sure it was right. The wires that came with the wiring kit (pots and switch) was plenty for the whole project but I wish they had been a little more generous since ‘lil ol’ me might have liked a little bit more room for error.</p><p>I haven’t yet plugged it in and tested it goes, but I happy with my soldering and it matches the diagram nicely. I’m pretty sure it’ll be fine.</p><p>I also got some screw terminal blocks (chocolate blocks) for the jack — so I don’t need to solder the jack to the pickguard, I can connect via the blocks. This makes getting the pickguard in and out a bit easier.</p><p>When I placed the pickguard onto the guitar I realised that there was quite a bit of play in the positioning. The body is new, so it’s never had a pickguard on it before and there are no guide holes for the pickguard. This presents a small problem as the pickups should be nicely aligned under the strings in order to get the best sound: without the neck/bridge it’s impossible to be sure that the pickups will align. I now need to wait for the remainder of the parts, fit the neck, bridge, and tuners, and a string or two. Then, the pickups can be aligned and the holes marked and drilled for the pickguard.</p><p>The catch here is that I don’t have the neck/bridge yet. Also, if the guitar has the clear coat on top when the holes are drilled there’s a chance it will crack the clear coat (bad!). So I can’t finish the body until the rest of the parts arrive.</p><h3>Speaking of which…</h3><p>The body is now red! I have started to put the colour coat on. Here’s coat one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nk43QJQOiXfJ5Idv8mRbfg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AoDdwUmZYsHpdF8Tz4DcCQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4zLZWGQ9OCWXvjZ3ZjwSMA.jpeg" /></figure><p>Coat one post sanding (yes, I realise I sanded down to the primer but coat one wasn’t very thick):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jXEQKaU-O9mvKWRq4e7dFg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fFfTzS-S2_AUXF_eTjyovQ.jpeg" /></figure><p>And then I rejigged the way I was hanging the guitar to make it easier to move inside to cure. I had been hanging it with fishing line but found that I’d have to lay the guitar down inside. This risked damaging the paint before it had fully set. I wanted to move to a system where I could lift it inside without needing to put it down. I started with a homemade hanger cut from cardboard because I was too lazy to go out and buy some cheap coat hangers (my nice ones I didn’t want to risk getting paint on). This worked quite well but the hanger part failed slightly, however fortunately the guitar was inside, dry and didn’t have far to fall.</p><p>Coat two…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wdL1k-A4k-vuII_w8fx6Ig.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*moRLkZV_pUAOZqATKTHbdQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tUpbgD88eceNDuLI0nC2FA.jpeg" /><figcaption>Pre-sanding coat two.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eCNlK1da8yTLS4Ff0Wbi5A.jpeg" /><figcaption>Preparing to wet sand.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XTiwczXkYmxjZfo8RZOFFg.jpeg" /><figcaption>Post sanding coat two.</figcaption></figure><p>And how it looks with the pickguard…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kygBweMzNRi8ylOI82EKCA.jpeg" /><figcaption>With the pickguard resting on top, coat one post sanding.</figcaption></figure><p>Shit is starting to get real!</p><p>Now I am waiting for the rest of the parts so I can drill the holes for the pickguard. Then it will be a final colour coat, sand, and clear coat on the body. Then final assembly, and hopefully — hopefully… awesomeness.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff7176101aae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A neck for my guitar.]]></title>
            <link>https://drkatnz.medium.com/a-neck-for-my-guitar-4efe15d4de36?source=rss-673dc508e869------2</link>
            <guid isPermaLink="false">https://medium.com/p/4efe15d4de36</guid>
            <category><![CDATA[guitar]]></category>
            <dc:creator><![CDATA[Kat Hempstalk]]></dc:creator>
            <pubDate>Wed, 26 Jul 2017 10:17:06 GMT</pubDate>
            <atom:updated>2017-07-26T10:17:06.334Z</atom:updated>
            <content:encoded><![CDATA[<p>The body of a guitar is only half the equation: you need a neck too. The guitar I’m building is a stratocaster style, so has a bolt-on neck. The alternative is a glue-on neck, which is on some types of guitar like the Les Paul. The type of neck boils down to the <a href="http://www.dummies.com/art-center/music/guitar/glued-in-versus-bolt-on-guitar-necks/">guitar style</a>, being either Fender-style (bolt-on, strat &amp; tele) or Gibson-style (glue-on, Les Paul).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*87x788azVAJkrvTy.JPG" /><figcaption>A bolt on style neck.</figcaption></figure><p>There’s lot of arguing about which is better, but it’s irrelevant here: the guitar I’m building has a bolt-on style neck.</p><p>I decided I wanted to do as little woodwork as possible, so buying a finished neck was the option I wanted to go for. Unfortunately, doing this is harder than it sounds.</p><h3>CITES protected wood</h3><p>Guitar necks for electric guitars are usually two parts: a fretted fingerboard, and a neck “back”. The backs are usually maple, but the fingerboard wood varies. The back wood must be straight and strong, maintaining it’s shape despite the tension of the strings. Electric guitars also have a metal truss rod that runs down the length of the neck and essentially controls how it bends. The truss rod can be adjusted to ensure the notes can be cleanly played, and in tune. Classical/acoustic instruments do not have a truss rod; they rely on the neck being thick/strong enough to withstand the tension of the strings.</p><p>On the fingerboard the most common sort of wood is rosewood, which unfortunately for us guitarists is an endangered species of wood. CITES — Convention of International Trade of Endangered Species — recently declared all species of rosewood protected. Brazilian rosewood has been protected for some time, but Indian and African species are now protected too. This “protected status” means that rosewood cannot be shipped over borders (but within is fine) without export licenses. It does not matter whether it is raw wood, or part of an instrument… it will need a licence either way.</p><p>Existing instruments need documentation that proves they were purchased before the regulations came into effect. New instruments must be sold with accompanying licenses.</p><p>Part of the problem is that the part of the wood that is used on instruments is in the middle: a large tree only delivers a small amount of wood for use. The process of obtaining the wood for use on an instrument (or high-end furniture) is therefore incredibly wasteful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qf55IiDZMbwleWEE.jpg" /><figcaption>Rosewood</figcaption></figure><p>Although I am not disputing this protected status, it has unfortunate ramifications for my guitar build. Necks are already difficult to purchase by themselves, because they are often on an instrument for life (if treated well) so there is little need for replacements. Also, NZ is small enough that no one holds stock of necks, you either buy a fully made instrument, or you import the neck in from overseas.</p><p>I would feel terrible ripping a perfectly good instrument apart for its neck, and the new CITES regulations and their export orders means that it is extremely difficult to get a rosewood neck into the country. It’s quite the dilemma.</p><p>The solution, of course, is to import a neck that doesn’t have a rosewood fingerboard. This leaves you with two choices: ebony (a black wood) and maple (a light coloured wood). Necks with ebony fingerboards are about as rare as hens teeth and twice as expensive, so maple is the sensible option. I personally don’t like the feel of maple, but most neck backs (not fingerboards) are made from maple anyway, so it’ll at least look nice.</p><p>So in the end I ordered <a href="http://www.wdmusic.com/strat_neck_vintage_maple_soft_v.html">this</a>: a single piece construction (i.e. one bit of wood made the fingerboard AND back) maple neck. Now I just have to wait for it to arrive!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*uz3X_tCkecOscsdk.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4efe15d4de36" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>