<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Unruly Engineering - Medium]]></title>
        <description><![CDATA[Find out more about Unruly&#39;s technology stack, open source contributions and product development team. - Medium]]></description>
        <link>https://medium.com/unruly-engineering?source=rss----d1d70e201bfd---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Unruly Engineering - Medium</title>
            <link>https://medium.com/unruly-engineering?source=rss----d1d70e201bfd---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:27:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/unruly-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The many sides of pair programming and the importance of deliberate practice of it]]></title>
            <link>https://medium.com/unruly-engineering/the-many-sides-of-pair-programming-and-the-importance-of-deliberate-practice-of-it-78ffd26293bb?source=rss----d1d70e201bfd---4</link>
            <guid isPermaLink="false">https://medium.com/p/78ffd26293bb</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[pair-programming]]></category>
            <dc:creator><![CDATA[Paul Cox]]></dc:creator>
            <pubDate>Tue, 24 Sep 2019 14:20:03 GMT</pubDate>
            <atom:updated>2019-09-25T11:43:12.578Z</atom:updated>
            <content:encoded><![CDATA[<p>Pairing can be a very triggering topic, so I’ll start by stating my bias: I am an advocate of pair programming; I generally enjoy it but more importantly, I think it results in better software delivery. I should also state that I’ve actually only ever worked as a developer in teams that have done the majority of work paired (5 teams across 2 companies)… I appreciate this puts me in a minority position in the industry. I’ll try and provide scientific backing throughout the post, so that it’s not just my limited personal anecdotal experience.</p><p>I’ll start by giving a brief intro to pair programming and then relating a shifting pairing experience I had recently, before going on to talk about some criticisms of pairing and my responses to these, provide a checklist we often use for pairing and then some musings. This is quite a long read, feel free to skip to the end for some key takeaways (but then come back and read the rest 🙂 ).</p><p>Pairing is two people working on the same bit of work at the same time, and in its most common form involves two developers sat at the same machine working on the same code. One person is the “driver” and they control the machine itself; writing out the code and being concerned with the “small picture” details e.g. syntax. The other person is the “navigator” and they tend to navigate through the code design details; they take a “big picture” view of things. These tend not to be rigid roles, and are very context dependent. There are other ways to split the work e.g. “ping pong-ing”: one person writes a failing test, the other then writes the simplest code to make it pass and then writes the next failing test; repeat. The general idea being that you have a very short feedback loop between decisions on code design. Within a team you will tend to “rotate” pairs every so often e.g. each day, this tends to involve one person staying on the piece of work and the other switching out for someone else in the team to join in their place.</p><p>Recently I was pairing with a colleague, Nat, more junior than me, who I’ve paired with often before and who I have a good relationship with. The first day Nat started leading and I made a conscious decision to give her more of the reigns for the day and to lean the focus more towards learning/ mentoring; this consisted of me being in control less, and providing hints more than solutions. The next day we shifted back to a more traditional pairing dynamics, with a more even split between driving duties &amp; discussions on solutions. I think we did test/ implementation ping-pong: one person writing a test, the next person making it pass in the simplest way and then writing the next test; repeat. This was a good example of pairing not being one thing, it can shift depending on what the aim is.</p><p>It would have been better to have been more explicit about what we wanted to achieve on each of the days. We shifted between the two pairing styles quite naturally during the two days, this might be down to us having paired a fair amount already (one of the many benefits of constant pairing in a stable team, building up the psychological safety) but I was aware on the first day that it was more tiring for her and that I didn’t know whether she was in the mindset for it. Being explicit at the beginning of the day has multiple benefits: you can focus on it more/ have buy-in if you both know what you’re planning for the day, if a learning/ mentoring day then one person doesn’t feel extra pressure because things are going slower, you can make sure that both people are in the right mindset for the pairing dynamic. Thankfully Nat fed back in our team feedback session that she appreciated the day and the extra time we took to focus on learning. The decision to work this way was made easier because there was less “pressure” on delivery at this point, but this shouldn’t be needed to take the time to focus on learning.</p><p>Particularly on the “learning focused” day, the uneven workload when pairing like this can be particularly tiring for the person driving, so be conscious of taking breaks regularly rather than just when you feel you personally need to. It can be good to set a regular interval for the breaks so that you don’t forget and so the other person doesn’t feel they have to request one. For similar reasons, this is dependent on each person’s energy levels for the day. Another way we could have split the work would be for me to drive/ type but just to implement what Nat says — different dynamic &amp; learning experience, takes away the pressure of the person learning having to type but means they don’t build up muscle memory by doing it themselves.</p><h3>Common criticisms of pairing</h3><h4>It’s stressful/ tiring</h4><p>Correct… I’ll be open, even now I can still find pairing stressful, especially with new people… imposter syndrome is very much a thing and getting used to someone takes time. This can be improved by increasing psychological safety within the team (which has huge benefits for the team <a href="https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html">generally</a>) but it takes constant effort; it takes time to build but can be destroyed in a fraction of that time. Also, adding/ removing just 1 person from a team effectively makes it a new team and so the safety starts at 0 again, and although a strong previous base can make it faster to build again, be extra aware as soon as team structure changes to not lose momentum.</p><p>All that being said, I don’t think it means you shouldn’t pair, I believe the benefits outweigh this downside, you just need to be conscious/ appreciative of this. There are also a number of things you can do to mitigate some of this:</p><ul><li>Regular breaks — this may seem like wasted time, but pairing is intense, you’ll be more productive overall</li><li>Other team tasks/ activities that don’t require pairing — research, spikes (afterwards throw away spike and pair on actual solution), support tasks, learning, blogging</li><li>It gets easier after practice and building psychological safety in the team (this has other benefits to the team as well); although it’ll likely always still be tiring</li><li>Following on from previous point, being open with pair around mental energy fluctuations during the day &amp; on different days</li></ul><h4>Opportunities to fail</h4><p>Another common criticism is that you don’t have a chance to fail in a friendly way, or you feel pressure to not appear stupid in front of your pair. In my opinion the “chance to fail on your own/ it’s good to hit your head against a brick wall sometimes” criticism is actually one of the most valid ones, but even there I think there are other ways of achieving this without giving up the benefits of pairing on the vast majority (if not all) production code e.g. spikes/ giving time for personal projects/ share leading on particular strands of work.</p><h4>Differing levels of experience/ context</h4><p>There are varying criticisms that come from this but some are: someone gets bored, or someone loses track; Junior/ senior pair is a waste of the senior person’s time; Two junior people together is dangerous; Two senior people together is a waste</p><p>I think some of this relies on the assumption that learning has a one-way benefit, whereas in my experience, both people get a huge amount out of it; having your assumptions questioned can be really beneficial in making you question why you actually do things in a particular way and if that’s actually the best way. Teaching someone also forces you to really understand a topic, which helps your own development. Also, being able to explain something to different types of people helps you to communicate with other roles involved e.g PMs. “Experience” is also multifaceted and context dependent e.g. different technologies, domains, someone may have more experience at a technology but be newer to the domain. There are also different outcomes you can focus on from different pairing combinations — this is good, particularly if done consciously. Dan North talks about this a little in his Dreyfus squared model <a href="https://www.youtube.com/watch?v=lvs7VEsQzKY">https://www.youtube.com/watch?v=lvs7VEsQzKY</a>.</p><h4>Different learning/ processing speeds</h4><p>People are different and diversity of thought has been shown to be important in a multitude of studies, different processing speeds have different positives/ negatives and mixing these can result in some tension but handled with empathy can result in a healthy tension that results in better solutions. Something that can help is that you don’t have to constantly be sat together… go away and research independently/ spike/ think and then come back together. It’s important to build up awareness &amp; empathy of different styles over repeated pairings.</p><h4>The misconception that pairing is for knowledge transfer only</h4><p>Often the benefit that gets talked about with pairing is that it’s there for silo breakdown and learning; both “bus factor” and domain/ technical learning for people less familiar. Although I do think these are useful, I actually view knowledge transfer as a surprisingly small part of the benefit to pairing.</p><p>Some other benefits of pairing:</p><ul><li>Fast, instantaneous, feedback on code (compared to code review). A higher number of people buy in to the fast feedback benefit of TDD &amp; releasing frequently, pairing is another side of this</li><li>Fewer bugs as you have two eyes on the code and thinking about edge cases</li><li>Faster delivery of story/ feature (traded off against possibly slightly fewer features, although I’m not sure that’s even the case) as less likely to get stuck</li><li>Better code design as you can chat about different ideas and their tradeoffs on the spot, with the most context</li><li>More consistent code in team</li><li>More understandable code by other people/ future you; having to articulate reasons means you are less likely bake your current pre-existing biases/ context in to the code</li><li>Team connection</li><li>More consistent delivery speed; less likely to get stuck for extended periods of time, fluctuations in energy levels get balanced out</li><li>Forces individual biases to the surface, which can increase the benefits of a diverse team, as long as you provide opportunities for the team to address them.</li><li>Team code ownership: when one person writes some code, there is a bias to feel like they own it, which can result in blaming someone for a bug in it, make criticisms more personal, or a reluctance for others to change it. Having multiple people write the code helps alleviate some of this “preciousness”.</li></ul><p>It could be argued that you can get a number of these benefits from code reviews, but in my limited experience and from chatting to others, you get them to a much lesser extent and at an extended feedback loop. It’s easy to fall in to the trap of skim reading code reviews as proper ones take a large amount of time, but this is a dangerous trap. Proper code reviews take time to understand the context and thought processes that went in to decisions and as they are often done at the end of a story, there is a large amount of code to review. The back and forth of the extended feedback loop adds an extra overhead of constant context switching. Waiting for someone to review code also encourages larger change sets, less frequent deploys and reliance on branching… in my opinion these all add to the chance of larger bugs that are harder to track down, more time and annoyance spent merging. Most code consists of trade-offs and the nuance of these can be lost during a review, particularly if scheduling “face-to-face” reviews is hard and the default is text comments on a pull request. Text comments on a pull request also lose the emotional nuance of conversation, this can cause comments to be perceived incorrectly and it’s easy to fall into the trap of them sounding passive aggressive.</p><blockquote>“They found that for a development-time cost of about 15%, pair programming improves design quality, reduces defects, reduces staffing risk, enhances technical skills, improves team communications and is considered more enjoyable at statistically significant levels.” — The Costs and Benefits of Pair Programming; Alistair Cockburn, Laurie Williams <a href="https://collaboration.csc.ncsu.edu/laurie/Papers/XPSardinia.PDF">https://collaboration.csc.ncsu.edu/laurie/Papers/XPSardinia.PDF</a></blockquote><p>Tuple have put together a nice summary of some of the research in to pairing <a href="https://tuple.app/pair-programming-guide/scientific-research-into-pair-programming">https://tuple.app/pair-programming-guide/scientific-research-into-pair-programming</a> (disclaimer: Tuple offer a remote pairing product and so likely to have the same, if not greater, bias to positive research than myself)</p><h4>Less flexible working day e.g. having to work similar hours/ no remote work</h4><p>This is one of the trickier ones. Tools for remote pairing are getting much better (screen sharing through Slack/ Zoom, VSCode, Tuple), but fundamentally they still rely on two people being sat at their computers at the same time, this can be a blocker for teams across multiple time zones. I have no experience with this, but I wonder whether regular overlapping blocks of time to pair would enable at least some of the benefits, without waiting for a code review on pull request when they feature is done. Please let me know if you have experience of being in this situation and what you’ve found.</p><h3>Checklist</h3><p>We came up with the below checklist as a way to code our agreed working practices, although we could do better at explicitly using them at the start of the pairing session both as a reminder to be more explicit with some of the decisions that vary day to day, but also as a ritual to get in to a pairing mindset. (Dave Snowden talks about the power of rituals in cognitive activation: <a href="https://youtu.be/r8T7wlJ8DgM?t=1525">https://youtu.be/r8T7wlJ8DgM?t=1525</a>)</p><figure><img alt="Pairing checklist attached to screen: e.g. Set expectations of each other and the task for the day" src="https://cdn-images-1.medium.com/max/1024/1*X-gO7dJtsQ7PudV666u35Q.jpeg" /><figcaption>Pairing checklist attached to monitor</figcaption></figure><h3>Musings</h3><p>My gut feel is some of the reluctance to pairing is previous negative experiences where it hasn’t been treated as a skill that takes practice.</p><p>Pairing is a skill, only practicing it on hard things isn’t the best time to practice, you want to build up the skill in easy situations so that it is more habitual when you really need it. It needs to be deliberate practice (provide link/ context), just sitting on the same machine doesn’t count; you won’t get better, &amp; you’ll miss out on many of the benefits. Although some of the latest literature on deliberate practice indicates a reduction in its influence on elite performance compared to some of the original studies, this is relative to other factors that result in the differences in performance between performers e.g. genetics, not the importance of deliberate practice itself in improving your own skill level. The benefits of deliberate practice over “basic practice” is the more relevant factor for individuals/ teams improving their pair programming as a skill e.g. if you accept that pair programming is beneficial, then getting better at pair programming has performance increases for team delivering products, and so deliberate practice is vital.</p><p>It also feels like some of this comes from a tension between what’s best for the product/ company and what’s best for the individual… these don’t have to be completely mutually exclusive but if one of the benefits to pairing less is the flexibility it provides (and the multiple benefits this gives, including hiring/ retaining people), then this is traded off against the belief that pairing results in better code/ products and benefits the company more directly.</p><p>I’m coming round to the idea of rotating pairs less frequently; switching pairs every day is great for knowledge sharing on a story, but as I’ve mentioned through the post, knowledge sharing is actually one of the smaller benefits. I actually think the other benefits mentioned earlier in the article benefit from slightly stickier pairs. You get time to build up closer working relationship with your pair, which benefits team relationships; you maintain more context on story, which means you spend less time context shifting/ getting up to speed; you can get in to deeper discussions about the feature and code design because there feels like less of a time pressure. The question of when to rotate does then become a little trickier as stories will start/ stop at different times, so when do you rotate? This seems to work out fairly organically around other things going on e.g. research days/ meetings/ time off, certainly once you get used to pairing as a team, it may be trickier at the beginning. I do still think that you probably want to switch pairs, or add someone to create a <a href="https://en.wikipedia.org/wiki/Mob_programming">mob</a>, on a story if it takes more than a few days; in my opinion, the benefits of having more points of view on things that are more complex outweighs the downsides.</p><h3>Takeaways</h3><ul><li>There is more to pairing than knowledge transfer</li><li>It’s a skill, practice deliberately</li><li>Tension between what’s best for individual (e.g. flexible working) vs what’s best for the team/ product</li><li>Yes, pairing is tiring, particularly at the beginning. This doesn’t mean you shouldn’t do it, it means you should take that into account with breaks &amp; emotional awareness (easier said than done, see above point)</li></ul><p>Please leave comments (both on pair programming &amp; the writing generally). Although I am an advocate of pair programming, I’m also a firm believer in not having anything cargo culted and that we should be constantly trying to challenge/ question and improve our working practices, as long as this is happening on a level playing field of understanding the reasons behind things, rather than a straw-manned/ misunderstood argument based on anecdotal experiences.</p><h4>Further reading</h4><p>For more detail about the reasons behind pairing, and other XP practices, read Extreme Programming Explained by Kent Beck</p><p>Studies on deliberate practice:</p><ul><li><a href="https://www.ncbi.nlm.nih.gov/pubmed/18778378">Deliberate practice and acquisition of expert performance: a general overview by Ericsson KA</a></li><li><a href="https://www.frontiersin.org/articles/10.3389/fpsyg.2014.00751/full">Facing facts about deliberate practice by Hambrick et al</a></li><li><a href="https://pdfs.semanticscholar.org/8177/d5d8897d192225b1c58b6842cdfec54bd81b.pdf">The Relationship Between Deliberate Practice and Performance in Sports: A Meta-Analysis by Macnamara et al</a></li></ul><p>Badass: Making user awesome is a great book on general skill acquisition (<a href="http://shop.oreilly.com/product/0636920036593.do">http://shop.oreilly.com/product/0636920036593.do</a>) and conference talk by her (<a href="https://www.youtube.com/watch?v=r4dNaflEgP4">https://www.youtube.com/watch?v=r4dNaflEgP4</a>)</p><p>Project Aristotle @ Google <a href="https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html">https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html</a></p><p><a href="https://docs.google.com/document/d/1Re7m-BpgbDDUIwL9fvVK5t6dTA5WBNlEcRN1cjqvuAM/edit?usp=sharing">The many sides of pair programming and the importance of deliberate practice</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78ffd26293bb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unruly-engineering/the-many-sides-of-pair-programming-and-the-importance-of-deliberate-practice-of-it-78ffd26293bb">The many sides of pair programming and the importance of deliberate practice of it</a> was originally published in <a href="https://medium.com/unruly-engineering">Unruly Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond Tech 2019 conf review]]></title>
            <link>https://medium.com/unruly-engineering/beyond-tech-2019-conf-review-363178a775f6?source=rss----d1d70e201bfd---4</link>
            <guid isPermaLink="false">https://medium.com/p/363178a775f6</guid>
            <category><![CDATA[inclusivity]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[conference]]></category>
            <category><![CDATA[diversity]]></category>
            <dc:creator><![CDATA[Paul Cox]]></dc:creator>
            <pubDate>Thu, 08 Aug 2019 07:24:26 GMT</pubDate>
            <atom:updated>2019-08-09T11:11:47.603Z</atom:updated>
            <content:encoded><![CDATA[<h3>Beyond Tech 2019 conference summary</h3><p>This was probably my favourite conference I’ve been to, and certainly the most important. The conference itself was very well run (attendee badge had areas for stickers for the gender pronouns you want to be referred to, coloured sticker to indicate comfortableness with being approached to talk to through the conference — although on the hidden bias side, what happens if you’re colour blind…), and adapted quickly to feedback e.g. round table sessions being moved because of noise around them. The atmosphere was great, and really conducive to important discussions around what we can do better as an industry. It was full of really brilliant talks, some of which I’ll go in to below, and all of which you can find <a href="https://skillsmatter.com/conferences/11735-beyondtech-2019#skillscasts">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tebRmdGZsj2nXyJEWKAHlg.jpeg" /><figcaption>Attendee badge with gender pronouns and coloured sticker to indicate how open you felt to interaction</figcaption></figure><p>One of my main takeaways is how foundational the area of diverse teams/ companies is; even with the best intentions in the world, we will miss things if teams aren’t diverse, the data models are trained on will be biased, vast business areas won’t be realised. This is even if the intentions of those involved are good… once you get in to neutral or bad intentions (e.g. actively maintaining the status quo because they are in a position of privilege), things get even worse. A quote from someone at the conference which I think sums up some of it is, and I’m probably paraphrasing, “to people in a privileged position, equality will seem like inequality”, we need to recognise this on both sides. It often comes up that a diverse company/ team is great for the bottom line, which is true, but Sara on one of the panels made a good point in that this shouldn’t be the only reason that we focus on this… what happens when an inclusivity initiative doesn’t benefit the bottom line, does that mean it’s not worth doing? Or should we still do it because it’s the “right” thing to do? I shared the video of a talk with my Mum, &amp; then realised that I didn’t know if it had subtitles &amp; so whether she’d be able to comfortably watch it.</p><p>It’s also great to see a larger number of ‘tech for good’ companies starting to form, or b-corp companies who don’t view profit as their only metric for success. It may just be my very narrow (and overly naive) view, but I get the impression that a lot of newer companies have a smaller focus on huge growth/ profits/ power, but instead focus on creating a sustainable business (in many senses of the word) that are truly for the benefit of people or the world; these should be encouraged, and consumers really do have power to make a difference in this. Every purchase is a decision on what you want society to look like. I don’t mean to preach, it would be incredibly hypocritical of me, I’m certainly far from perfect in this, but I think it’s important to have reminders and to constantly make small steps e.g. buying local can reduce environmental impact. Do you really need the product the next day, or can you buy from a local company that might take slightly longer but is more sustainable/ from a more ethical company? Are we okay selling ethics for convenience? Being in a position to make these decisions is also a position of privilege, e.g. paying a little more for a more sustainable/ ethical product, and should be treated as such and with a certain level of responsibility when we can, this is one of the prices we pay for “playing the game of life on the easy setting”. Not only does it help companies that are doing good, but it forces other companies to react and also move in that direction, which helps other consumers not in a position to make these decisions. I should add that privilege isn’t a constant, and that sometimes the extra energy it takes is too much at times, this is okay, your mental health is important and everyone has a finite amount of mental energy… if it is being used up in other important areas (depression, illness, family responsibility etc.) then it’s okay to give yourself some space, but just remember that this extra energy is what people from disadvantaged/ minority groups often have to expend on top of everything else. (See the <a href="https://en.wikipedia.org/wiki/Spoon_theory">spoon</a> <a href="https://butyoudontlooksick.com/articles/written-by-christine/the-spoon-theory/">theory</a> for a good way of thinking about this). Privilege is also relative and context specific. It’s not our fault, but it is a fact &amp; we do benefit from it. <strong>These conversations should make us feel uncomfortable.</strong></p><p>As a developer/ highly sought after profession, I (we) have a disproportionate amount of power to hold companies to account for their actions, as recent walkouts by Google employees has demonstrated. I (we) should make use of this power, especially if in a particularly privileged position (white, male, straight, cis, middle class with family support network), to hold companies we work for to account and push back on things we deem unethical.</p><p>Another action I took away from this conference, and I’d encourage others to do, was to add my gender pronouns to my Twitter &amp; Instagram profiles; this is a very minor change on my part but the hope is that it’s the sort of thing that can help normalise being vocal with how you want to be referred to and can make it easier for people in a marginalised group to feel comfortable to do the same without feeling different.</p><p>It would be nice if the most diverse conference I’ve been to, wasn’t the one where diversity was one of the main strands, but one where it’s not actually the focus.</p><p>If this has interested you, I’d highly recommend going along next year (I already have my ticket), early bird tickets are already available at a discounted rate <a href="https://skillsmatter.com/conferences/12381-beyondtech-2020">here</a>.</p><p>Below are some notes from some of the talks I went to, the more +s, the more highly recommended to go and <a href="https://skillsmatter.com/conferences/11735-beyondtech-2019#skillscasts">watch</a>.</p><p><strong>Day 1</strong></p><p>Data for the public good by Reema Patel</p><ul><li>Beneficial ethical right vs technical ownership e.g. property: right if pay towards mortgage vs name on deed</li><li>Data and AI reinforce power relationships</li><li>Facebook: benefit comes from collective data, not individual bits of data, who profits from this? Different to analogy of oil</li><li>Possibly have to look beyond laws &amp; regulations</li><li>Welcome trust: Fair NHS data sharing, make sure public services are not getting “ripped off” for their data</li></ul><p>Round Table Sessions: Responsible AI by Sam Mercer</p><ul><li>Someone brought up the point around different moral bases and how they affect moral AI decisions (this is covered in Righteous Mind)</li><li>Best intentions aren’t always good enough. Someone used the example of the coloured stickers on attendee badges, and how they might not work for people with colour blindness.</li></ul><p>Psychological Safety: Overprotection or Not? by Gitte Klitgaard ++</p><ul><li>Diversity vs inclusion — “when you’re in an unsafe environment, you feel like it’s your fault”</li><li>Needed so that you can step outside your comfort zone</li><li>Departments that are doing the best, <strong>admit</strong> the most errors</li><li>If you are a “full stack developer”, you’re probably overloaded</li><li>Doesn’t mean you can’t make a joke, but that the space is safe for someone to come and say your joke offended them — humans make mistakes</li></ul><p>Designing a Feminist Alexa: An Experiment in Feminist Conversation Design by Dr. Charlotte Webb +</p><ul><li>Feminist internet</li><li>f-xa.co</li><li>Don’t use “I” anywhere, to reinforce the point that it’s not a living entity</li></ul><p>Keynote: Closing the Skills Gap: Russ Shaw in Conversation with Sarah Luxford by Russ Shaw and Sarah Luxford</p><ul><li>Tech London advocates</li><li>Life long learning. “Learning bank” to have a pot of money to dip in to re-learn later/ throughout life</li><li>Ada college in Tottenham</li><li>John Lewis got a mention about doing well pushing retail tech</li><li>Recruitment process — do we ever ask any ethical questions as part of recruitment process; why not? Students resenting ethics modules because they feel it doesn’t actually help them get roles because it’s not “tested” during the recruitment process</li></ul><p>Keynote: Digital Activism — Saving the World with Open Data by Jaya Chakrabarti MBE ++</p><ul><li>b-corp e.g. patagonia</li><li>Tiscreport — supply chain transparency</li></ul><p><strong>Day 2</strong></p><p>Keynote: Can Democracy Survive the Internet? by Jamie Bartlett ++</p><ul><li>How can we run an election that we can all trust? The outcome of “fake news” will actually be that no-one will trust anything</li><li>Radicals by Jamie Bartlett — book</li><li>Radicals tend to be early adopters of tech</li><li>Selective one sided true news is more of a problem than “fake news”</li><li><a href="http://editions-hache.com/essais/pdf/kaczynski2.pdf">Unabomber manifesto about modern industrial-technological society</a></li></ul><p>Why Ethics Matter And What we Forget About Our Past by Stella Windsor +++</p><ul><li>“White women are like a band-aid for diversity”</li><li>“This is not London” — Yes it is! Change to “This is not who we want to be”</li><li>“Audit your damn meeting rooms” and go beyond this, make sure you’re listening to everyone</li><li>Windrush/ other marginalised groups, why are we not putting them first in the story instead of us</li><li>“skin coloured” band-aids, ballet shoes etc. only just starting to take in to account people with different skin colours</li><li>Conversely, the responsibility for equality shouldn’t fall on oppressed/ minority groups, they have enough to deal with already with the extra energy required to just be themselves in our society</li></ul><p>Social Problems have Social Solutions: A Case Study with Homelessness by Alex Stephany +</p><ul><li>Beam is the charity Alex runs</li></ul><p>Knowing Me Knowing you…. Ahaaaaa! by Jenny Martin +</p><ul><li>OCEAN</li><li>Focus on the strengths of people and grow those — well rounded team vs well rounded individuals</li><li>Strengths finder 2.0 from Gallup</li></ul><p>Panel Discussion on Diversity and Inclusivity in Tech hosted by Nancy Evbuomwan by Nancy Evbuomwan</p><p>Lightning Talk: Putting Top Coding Challenge Solutions to Work by Angela Bates</p><ul><li>Call for Code — developers coming together to build solutions to help solve problems caused by natural disasters</li></ul><p>Lightning Talk by Sara Shahvisiby Sara Shahvisi</p><ul><li>Fearless futures</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=363178a775f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unruly-engineering/beyond-tech-2019-conf-review-363178a775f6">Beyond Tech 2019 conf review</a> was originally published in <a href="https://medium.com/unruly-engineering">Unruly Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Personal Ethics Clash With Industry Norms]]></title>
            <link>https://medium.com/unruly-engineering/when-personal-ethics-clash-with-industry-norms-3cc41d0bec7a?source=rss----d1d70e201bfd---4</link>
            <guid isPermaLink="false">https://medium.com/p/3cc41d0bec7a</guid>
            <category><![CDATA[hiring]]></category>
            <category><![CDATA[recruitment]]></category>
            <category><![CDATA[ethics]]></category>
            <category><![CDATA[google-project-maven]]></category>
            <category><![CDATA[ethics-in-tech]]></category>
            <dc:creator><![CDATA[Gidi Meir Morris]]></dc:creator>
            <pubDate>Tue, 24 Jul 2018 09:40:44 GMT</pubDate>
            <atom:updated>2018-07-24T09:40:43.867Z</atom:updated>
            <content:encoded><![CDATA[<p>Last week I had the profound pleasure of attending the first ever <a href="https://www.coedethics.org/">COED:ETHICS</a> conference in London.</p><p>A first of its kind, the organisers of this conference posed a question which has long been on my mind: <strong>Can developers become agents for change which might inspire, inform and enforce ethical standards within the tech industry?</strong></p><h3>Motivation</h3><p>I found the conference truly unique and inspiring, but mostly, it really struck a personal chord with me and I thought it was worth taking the time to clarify why I felt it is important to take part in such an event.</p><p>Throughout my career I have been heavily involved in the recruitment process into the tech industry. I am also extremely passionate about improving the diversity within the industry and I am actively working to improve the inclusivity of the community of developers I am a part of.</p><p>Recently I found myself confronted with an ethical question which required an immediate answer: <strong>Are my personal ethics at odds with those of the industry I work in?</strong> And, more importantly, how much leeway do I have in following my own ethical guidelines when participating in this industry?</p><h3>Ethics As Part Of The Recruitment Process</h3><p>At my current company all developers take part in the interview process of candidates, which means that as part of my job (though I am in no way a member of our <em>People Team</em>) I assist in screening the CVs of candidates.</p><p>Several months ago a CV came across my desk from a developer whose previous employer had been named by newspaper headlines on a semi-regular basis. For the sake of keeping this post on topic and to avoid triggering an unrelated debate, I won’t name the company explicitly.</p><p>Instead lets use a pseudonym.</p><p>I shall hence force refer to this employer as <em>Oxford Statistica</em>; a totally made up company name. Totally.</p><p>This candidate was likely choosing to distance themselves from the reputation this company was amassing, which is a perfectly legitimate thing to do, but I found myself asking a serious question:</p><blockquote>How relevant is the nature of this company’s business practices to the question of this person’s future employment?</blockquote><p>As I contemplated this question, both in the context of current events and in the context of my personal experience in the industry, I came to two important conclusions.</p><h4><strong>The Industry Doesn’t Deem This Relevant</strong></h4><p>Thinking back over my time in the tech industry, I have come to the conclusion that at this moment in time, the general question of the relationship between the ethical aspects of a company’s business practices and their bearing on the individuals working at the company, are limited to what a legal system might define as <em>culpability</em>.</p><p>What I mean by this is that the only place where it is considered legitimate to hold a candidate to account for the ethical practices of their previous employer is when these practices were found to be illegal and that the law had found this individual responsible in some form or another for this wrongdoing.</p><p>In fact, I would go so far as to say that, as far as I can tell, the industry in general would find the dismissal of candidate on ethical grounds without proven <em>culpability</em> as a form of <em>discrimination</em>, just as dismissing a candidate for being “A Tory” would be discrimination.</p><h4><strong>I Am Biased</strong></h4><p>The second, and perhaps most important one, was that I am in fact biased against this candidate. As an often overly vocal voice in the <em>D&amp;I</em> efforts of my employers, I talk often about the concept of the <em>biases</em> that interviewers are often tripped-up by when interviewing candidates.</p><p>In the case of this candidate, I posed, I am no longer capable of avoiding <em>Confirmation Bias</em>, specifically I am more likely to support the thesis that this candidate has taken part in unethical practices, than to be convinced otherwise.</p><p>This is the very definition of bias, and as I had already come to the conclusion that my concerns are not deemed relevant by the industry, I chose to withdraw myself from this candidate’s interview process.</p><p>This means that another developer stepped in and conducted the interview in my place. This meant the candidate got a fair and balanced interview, instead of having to face me, an interviewer who was already biased against them.</p><p>I felt comfortable with my decision in this case, but found myself severely torn up inside, as I realised this question was one I am likely to encounter more and more as we uncover more misdoings by companies and individuals throughout our industry.</p><h3>Asking The Question</h3><p>In the wake of this interview I came across the mention of the <strong>COED:ETHICS</strong> conference and noticed this blurb on the conference site:</p><blockquote>After the Facebook, Uber and Volkswagen scandals, where are we as an industry? How did we get here and will things get better or worse? <strong>80% of developers don’t feel it’s their responsibility if unethical products go live. Are they right?</strong></blockquote><p>No single more relevant question could have been asked of me at that very moment, and so I quickly grabbed myself a ticket, booked the day off and prepared mentally to seek out other individuals who might be as frustrated as myself in the quest to answer it.</p><h4>Have I Found An Answer?</h4><p>Though I discussed the topic with several people at the conference, I neither got a clear answer, nor was I expecting to.</p><p>The participant at such a conference, are by the vary nature of such a grassroots event, individuals who are not entirely comfortable with the status quo.</p><p>This means that they would likely lean towards change, not retaining things as they are, and as the status quo seems to be that of stifling debate on ethics in favour of company profits, I knew that any answer I got would likely be biased towards one edge of the spectrum.</p><p>That said, I have come up with my own position on this question and I thought this would be a nice opportunity to voice it.</p><h4>Why the Ethics and Practices of Former Employers Are Relevant</h4><p>My personal position is that we <em>should</em> be asking candidates about their position on the ethics of the business practices of their former employers.</p><p>There are several reasons for my position.</p><h4>They Are Often Applicable To The Work</h4><p>In the case of the candidate from <em>Oxford Statistica</em>, the company has been accused of dancing on the edge of criminality in its use of personal data, avoiding legal action by the skin of its teeth thanks to the archaic nature of our legal system.</p><p>In the case of my current employer, we could potentially use similar methods to boost our revenues, but we explicitly choose to prioritise privacy over shady business practices. We rely heavily on the ethics of our employees to act as safeguards, especially those of our developers, data scientists, and analysts as they could easily violate these guidelines without the rest of the business ever knowing.</p><p>In fact, we employ a <strong>Privacy By Design</strong> practice by which every single Story, Developer Task and Operation is assessed for privacy issues by the developers as part of the design process, to ensure they keep privacy concerns at the forefront of their minds.</p><p>If a candidate is coming from an environment where the demanded ethics around privacy are divergent from our own, then how can we possibly ignore that? Surely, the only way to ensure that our practices are retained, is to ensure candidates are willing to adhere to them.</p><h4>Questioning a candidate on such questions is the opposite of exhibiting bias</h4><p>As I explained above, in the case of the <em>Oxford Statistica</em> candidate, I did not feel empowered by the industry to ask this question. Nor did I feel that my company could take a divergent stance on this issue on its own, as it would end up being labeled a discriminator. With that in mind I decided I could not question the candidate on such issues, but without debating these topics I was afraid I would end up unintentionally biasing against the candidate, which is why I chose to withdraw.</p><p>It is my belief however that had I been able to ask the candidate about their position on such question it would actually allow me to avoid these unintentional biases.</p><p>To understand my position it is first crucial to understand that the intention behind such questions <strong>is not to hold the candidate to account for the actions of their former employer</strong>.</p><p><em>Firstly</em>, such a demand would be unfair as many employees might have been completely oblivious to the methods employed by their employers.</p><p><em>Secondly</em>, even if a candidate was aware, or even took part, it is important to give them a chance to explain themselves, as it might be that they did in fact speak up, did try to change things and most of all, as human beings, they too can make mistakes, learn from them and improve. All we’re doing is making the question of ethics part of the conversation, and with the right empathy we can make these questions as valid as technical exercises.</p><p><em>Thirdly</em>, on a personal level, it would be completely <strong>hypocritical</strong> on my part to hold such actions against the candidate, as I spent the first 7 years of my career working for the Defence industry. I would not feel it fair for a potential employer to reject me on that basis without first giving me the opportunity to discuss the issue, as ethics are often subjective and are open to interpretation.</p><p><em>Fourth</em>, equally important, by discussing these ethical questions with the candidate we, as potential employers, are also giving the candidate the opportunity to asses <strong>our ethics</strong>. As a candidate I would want to know about the ethical position of a company before I chose to work there, and I would definitely appreciate the opportunity to ask a potential employer tough questions on how <em>my work</em> might be used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KXxo-7cbLr3F-ip44AADXg.png" /><figcaption><a href="https://www.theguardian.com/commentisfree/2018/may/16/google-business-war-project-maven">Google Project Maven</a></figcaption></figure><p>As we saw when Google’s bid for a Pentagon contract which would have offered its AI technology to isolate targets for attack by scanning drone feeds, employees care a great deal about how their work is used. At the time, an open letter was signed by over 3,000 Google employees forcing the company executives to pull the plug on the project. Had these employees had the opportunity to discuss these topics with the company as part of their interview process, they could have avoided this situation all together and could have avoided offering their expertise to a company that would use the product of their work in a manner which might be contradictory to their personal ethics.</p><p>One might argue that had they not been employed by Google they would not have had the power to stop the project, but my argument is that by forcing the ethics into the conversation we can separate the “ethical” companies from the “unethical” ones and by doing so we can hold the unethical companies to account by using the power of the community as a whole.</p><h3>Where From Here</h3><p>Nothing much has changed, but I feel that after the COED:ETHICS conference I am slightly more empowered to express my point of view.</p><p>By holding the conference our community is beginning to challenge the status quo, and even though it is still far from affecting change, it is sowing the seeds.</p><p>I, for one, will continue to keep my ethical questions to myself when interviewing candidates. Despite the importance I feel they hold, I don’t feel the industry is ready to accept this change, but I will continue to discuss the validity of such questions as I make my way through the industry.</p><p>Perhaps change isn’t here yet, but if it’s up to me, change will come long before I retire.</p><p>Read more on Gidi’s personal blog at <a href="http://www.gidi.io/">gidi.io</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cc41d0bec7a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unruly-engineering/when-personal-ethics-clash-with-industry-norms-3cc41d0bec7a">When Personal Ethics Clash With Industry Norms</a> was originally published in <a href="https://medium.com/unruly-engineering">Unruly Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data engineering tech at Unruly]]></title>
            <link>https://medium.com/unruly-engineering/data-engineering-tech-at-unruly-bb8b4afa2758?source=rss----d1d70e201bfd---4</link>
            <guid isPermaLink="false">https://medium.com/p/bb8b4afa2758</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Vijay]]></dc:creator>
            <pubDate>Mon, 23 Jul 2018 10:58:26 GMT</pubDate>
            <atom:updated>2018-07-26T14:51:37.590Z</atom:updated>
            <content:encoded><![CDATA[<p>Data engineering began at Unruly with a product experiment as with many of the products we build. As an ad-tech organisation, we wanted to demonstrate the commercial value of predicting if a user will complete watching a digital video ad. This provides value by meeting objectives (KPIs) of advertisers, saving money, providing the end user more relevant ads and helping Unruly to optimise ad serving.</p><p>Additionally, this experiment allowed the technical team to define a base architecture and technology stack for data pipelines and machine learning at Unruly. This post will share with you our technical architecture, taking a look at how it has evolved and the choices made. We assume a degree of familiarity with data engineering and machine learning technologies.</p><h3><strong>Some jargon</strong></h3><p><strong>Events</strong> are domain events from any other systems. The user events are a subset of these events that capture user interaction with a digital ad. These are useful for some of our machine learning models (for example video complete prediction) as they<strong> </strong>represent<strong> </strong>user context and behaviour.</p><p><strong>Transactions</strong> are a related set of events e.g. events for a single user interacting with an instance of a digital ad. Think of this as a ‘squashing’ of events into a single row.</p><h3><strong>Data scale</strong></h3><p>Anywhere from 0.5B to 1.5B events per day are received; this can be between <strong>50 to 100GB of data </strong>(compressed, batched per hour) daily. Whilst this is not ‘big data’, our supply volumes are ever increasing and we have considered this factor in our technology choices.</p><h3>A picture of things at the beginning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mXxb-ipxHHvax1dE" /></figure><h3>A picture of things now</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kRlVvPHNEdkcGrHp" /></figure><p>This is a continuously evolving architecture and we expect it to change further as new and exciting challenges are faced — watch this space!</p><h4><strong>BigQuery for data warehousing</strong></h4><p>There are <a href="https://www.google.co.uk/search?q=google+bigquery+vs+redshift&amp;oq=google+bigquery+vs+redshift&amp;aqs=chrome..69i57j0j69i64.5052j0j7&amp;sourceid=chrome&amp;ie=UTF-8">many comparisons </a>of <a href="https://aws.amazon.com/redshift/">Amazon’s Redshift</a> and <a href="https://cloud.google.com/bigquery/">Google’s BigQuery</a>, and it is a close call as their features are very similar.</p><p>The team chose BigQuery due to its relative simplicity and low barrier to entry. We liked the fully managed database-as-a-service approach as it supports our need for a <a href="https://panoply.io/data-warehouse-guide/bigquery-architecture/">scalable, highly-performant data store</a> without having to concern ourselves with running our own infrastructure. Even Redshift as a cloud service needs a certain degree of infrastructure configuration and optimisation that BigQuery does not require.</p><p>Events are partitioned by date — this keeps our costs low by restricting the partitions queried and query performance high by parallelising queries across partitions.</p><h4>Dimensions as immutable snapshots</h4><p>Dimensions imported as daily batches are stored as immutable snapshots into timestamped partitions. This is useful for reprocessing data with <a href="https://medium.com/@maximebeauchemin/functional-data-engineering-a-modern-paradigm-for-batch-data-processing-2327ec32c42a">consistent, reproducible results</a>.</p><h4>Batch data processing</h4><p>The team’s data pipelines are batch oriented rather than streaming— this is conceptually and operationally a simpler model. Right now, these are single daily batches. As we discover new business use cases or face challenges of growing latencies and larger data sets, we are prepared to consider a mini-batch or streaming oriented processing architecture.</p><h4><strong>Apache Airflow for pipelines</strong></h4><p><a href="https://airflow.apache.org/">Apache Airflow</a> is a data processing framework (for ETL and other workflows) with similar purpose to <a href="https://github.com/spotify/luigi">Luigi</a>, Oozie or Azkaban.</p><p>Using Airflow helped to solve many observability problems. We had challenges with knowing which stages in the pipeline had failed as the original bash script was writing everything to a single log file. It was also difficult to monitor the pipeline, debug which parts had failed and alert on failures — all things that Airflow helped to solve out of the box.</p><p>Airflow was also a strategic choice as it gave a balance of simple first setup (using a Local executor) with scope to scale in the future (e.g. using a Celery or Mesos executor).</p><p>It has provided many other useful features:</p><ul><li><strong>DAGs (Directed acyclic graph)</strong> — the power of dependency graphs helps reduced completion times by utilising concurrent tasks. It also stops downstream tasks from executing when there are upstream failures and manages recoverability.</li><li><strong>Backfill</strong> — useful for upstream failures or requirement for historic data — Airflow makes backfill super easy via the web UI or CLI.</li><li><strong>Pipelines as code </strong>— not just any code but Python code (not XML declarations!).</li><li><strong>Abstractions </strong>— putting reusable workflow components into subdags means future pipelines are easier to create. Pipelines can also be generated dynamically at run time e.g. an import pipeline per available source file, or a processing pipeline per partition of data.</li><li><strong>Airflow operators </strong>— we incrementally moved bash scripts over to Airflow, and converted parts to utilise the in-built operators, reducing the amount of code to maintain and allowing the team to be more productive.</li></ul><h4>Vowpal Wabbit for machine learning</h4><p><a href="http://hunch.net/~vw/">Vowpal Wabbit</a> is an extremely fast machine learning engine since it using <a href="https://en.wikipedia.org/wiki/Feature_hashing">hashing trick</a>. It also handles categorical and multi-label features very well.</p><p>This is vital for performant training on huge data sets and low latency model predictions. Vowpal also makes discovery of feature importance relatively easy.</p><h4>Standardised on EC2 infrastructure</h4><p>The pipeline was migrated to run on <a href="https://aws.amazon.com/ec2/">EC2 infrastructure</a> from Google Compute, and all new services going forward are hosted on EC2.</p><p>This is in line with the infrastructure used by other teams, and allowed us to utilise the common Terraform and Puppet code base for easily rebuilding hosts.</p><h4><strong>Python vs Bash</strong></h4><p>Our first pipeline was written with a combination of Bash and Python. There were a few problems with this:</p><ol><li>Testing was more difficult</li><li>Unnecessary knowledge of two languages</li><li>Achieving reusability &amp; consistency</li></ol><p>Much of the data science community is using Python — so it made sense to standardise on this language.</p><p>The team have been steadily replacing Bash scripts with Airflow operators (which are coded in Python) — this has reduced the amount of code to maintain.</p><p>Custom Python code is easily unit and integration tested— this has helped to build quality into the code. Structuring and packaging of custom Python components is easier — we distribute versioned, reusable libraries via Artifactory that are pip installable.</p><h4><strong>Data analysis infrastructure</strong></h4><p>During the experiment, the definition of the model and exploratory data analysis work was done locally on our Data Scientist’s machine. Subsequently, we have built infrastructure for <a href="https://jupyter.org/">Jupyter Notebook</a> with popular data libraries like <a href="https://pandas.pydata.org/">Pandas</a> and <a href="http://scikit-learn.org/stable/">Scikit-learn</a>. All notebooks are committed to Git repositories, retaining a traceable version history like the rest of our codebase.</p><p>This repository and central service for data analysis and model building facilitates reproducibility, knowledge sharing and increased transparency.</p><h4><strong>Model serving infrastructure</strong></h4><p>The first approach of model deployment during the experiment was to deploy manually onto the consumer hosts. This proposed some key challenges, primarily due to the dependency on other teams:</p><ol><li>Regular deployments are difficult</li><li>Monitoring was challenging — especially real-time monitoring of scoring</li><li>Experimentation (e.g. A/B testing, new features) was difficult</li></ol><p>A centralised service for model scoring owned by the Data Engineering team helps solve these problems, however we also needed to consider the requirement of low latency by the consumers. A locally applied model incurred zero network latency, however a centralised service would. To solve this model serving infrastructure is deployed in the same availability zones as consumer instances to keep network latency at a bare minimum (&lt;5ms).</p><h4><strong>What more could be done</strong></h4><p><strong>Parallel batches or streaming in the pipeline</strong> — Pipelines currently import data in sequential batches (e.g. a batch for each source file). The main pipeline takes around 3.5 hours and since models are trained daily this is sufficient for our current needs. As our data scales or there is a need to have more regularly trained models, a quick win is to import batches in parallel. Beyond this, another option is to stream the data e.g. via <a href="https://kafka.apache.org/">Kafka</a>.</p><p><strong>Athena for data warehousing — </strong>with the increasing maturity of <a href="https://avro.apache.org/">Avro</a> and <a href="https://parquet.apache.org/">Parquet</a>, there is opportunity to store data in one of these formats rather than CSV, and utilise <a href="https://aws.amazon.com/athena/">AWS Athena</a> for querying data directly from S3. Redundant infrastructure (BigQuery) could be removed, saving on costs, whilst achieving similar levels of performance and ease of use (via a SQL-like interface) as BigQuery. Here is a good <a href="https://dzone.com/articles/how-to-be-a-hero-with-powerful-parquet-google-and">overview of Parquet</a>.</p><h4><strong>In summary</strong></h4><p>Unruly’s data engineering team has made many choices and tradeoffs for our machine learning pipelines, and we continue to adapt to change and evolve our technical stack and architectural patterns along the way.</p><ul><li>Big Query is a highly scalable and performant data store that has provided the simplicity of use and low barrier to entry (a fully managed scalable service with a SQL interface); however there is opportunity to simplify by using AWS Athena.</li><li>Vowpal Wabbit is an extremely fast machine learning engine that uses the hashing trick and can handle large amounts of data such as real-time events.</li><li>Airflow provides observability, pipelines as Python code and other key features like backfill, recoverability and dependency management.</li><li>Python as the primary language makes sense as a large part of the data science ecosystem is Python oriented, and it has simplified automated testing and distribution of reusable libraries.</li><li>The use of Jupyter notebooks with pandas/scikit-learn/matplotlib has made data analysis and knowledge sharing super easy.</li><li>Building a model serving infrastructure has enabled regular model deployment, real-time monitoring of models in production and effective A/B experiments.</li></ul><p>If you have a passion for machine learning and are excited about working collaboratively to develop data architecture, technologies and processes, check out our <a href="https://unruly.co/about-us/were-hiring/?/development#hiring-list">open roles</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb8b4afa2758" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unruly-engineering/data-engineering-tech-at-unruly-bb8b4afa2758">Data engineering tech at Unruly</a> was originally published in <a href="https://medium.com/unruly-engineering">Unruly Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started in machine learning]]></title>
            <link>https://medium.com/unruly-engineering/getting-started-in-machine-learning-b51b3b1917ad?source=rss----d1d70e201bfd---4</link>
            <guid isPermaLink="false">https://medium.com/p/b51b3b1917ad</guid>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Vijay]]></dc:creator>
            <pubDate>Fri, 20 Jul 2018 18:03:39 GMT</pubDate>
            <atom:updated>2018-08-03T09:48:53.303Z</atom:updated>
            <content:encoded><![CDATA[<p>Someone at work recently asked me about getting started in the field of machine learning. Embarrassing realisation — I should have noted this stuff down and share more widely. I am by no means finished with my journey, but there are some things that have helped me already.</p><p>These are some things that worked for me — please go ahead and adapt to your own context and learning style. Most importantly experiment and find what works for you, and let me know if I can help in any way!</p><p>To be clear, this is not a detailed blog about how to do machine learning (e.g. linear regression, neural networks etc.) — there are other people who can do this better than me. This is a post about what tools, resources and strategies can be used to learn those things.</p><p>The only assumption made is that you have some software development experience — at minimum a familiarity with an object-oriented programming language. Moving from software engineering into the world of data science was a huge challenge and required an accelerated learning curve, though having basic computer science, programming and SQL skills helped a lot. Where possible, I will try to include resources for those without that base.</p><h4><strong>Timeline of learning</strong></h4><p>My journey sort of looked like the diagram below. Truth be told, this is the ideal timeline if I had the foresight to do things in the right order — instead I jumped into things too early, stopped and revisited strategies when most appropriate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LSCdIogigXNRUNd3NQ0SSg.png" /></figure><h4><strong>Python essentials — Development environment</strong></h4><p>Setup a basic environment upfront for experimenting with Python and machine learning. Use Conda to isolate python environments — it makes experimenting with different setups much easier e.g. a different version of Python or pip libraries.</p><p>Checklist:</p><ol><li>Install Conda — <a href="https://conda.io/docs/user-guide/install/index.html">https://conda.io/docs/user-guide/install/index.html</a></li><li>Create a new conda environment</li></ol><pre>conda create -n py36 python=3.6</pre><pre>conda activate py36</pre><p>3. Install some dependencies via pip</p><pre>pip install pandas scikit-learn</pre><p>4. Install Jupyter notebook- <a href="https://jupyter.readthedocs.io/en/latest/install.html">https://jupyter.readthedocs.io/en/latest/install.html</a></p><pre>pip install jupyter</pre><pre>jupyter notebook</pre><h4><strong>Python essentials — Jupyter</strong></h4><p><a href="https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/what_is_jupyter.html">Jupyter Notebooks</a> is a web-based IDE for developing Python applications that combines code with rich text elements (paragraph, equations, figures, links, etc….). It allows easy experimentation of Python programs and let’s you start hacking away with minimal setup (see previous section).</p><p>It’s particularly useful for data analysis and machine learning since Python code can be combined with visuals and comments about the data, and therefore documenting data analysis and machine learning outcomes becomes easy. Please pay it forward and share your own Jupyter notebooks on Github!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*04r1LGfrOfl9NHCwT6T2qQ.png" /><figcaption><a href="https://jupyter.org/">https://jupyter.org/</a></figcaption></figure><h4><strong>Python essentials — Language</strong></h4><p>Python seems to be used so widely in data science and has great libraries and frameworks. It’s more popular than R in my opinion — Python is not only easier to learn but there are so many clear, well written example projects on the web that are implemented in Python. Everything else rests on knowing Python, so definitely the place to start.</p><p>If you have used other languages extensively, then check out these resources to get into Python quickly:</p><ul><li><a href="https://docs.python.org/3/tutorial/index.html">The Python Tutorial</a></li><li><a href="https://www.learnpython.org/">Learn Python</a></li></ul><p>If you’re an absolute beginner to programming, then try:</p><ul><li><a href="https://pythonbasics.org/">Python Basics</a></li><li><a href="http://thepythonguru.com/">The Python Guru</a></li><li><a href="https://www.amazon.co.uk/Programming-Python-Complete-Introduction-Developers/dp/0321680561">Programming in Python 3: A Complete Introduction to the Python Language (book)</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*vs5H5vxWGHEQIrrToZbBCg.png" /><figcaption><a href="https://www.python.org/">https://www.python.org/</a></figcaption></figure><h4><strong>Data analysis</strong></h4><p>Jumping into machine learning without learning some statistics and data analysis techniques is a mistake in my opinion. Understanding the statistical quality and distribution of the data helps you decide if machine learning is even possible and which techniques are most appropriate.</p><p>Start with learning the basics of statistics. At this stage, don’t worry about mastering this area; but instead focus on the basics — if you are refreshing your knowledge then this could help:</p><ul><li><a href="https://www.mv.helsinki.fi/home/jmisotal/BoS.pdf">https://www.mv.helsinki.fi/home/jmisotal/BoS.pdf</a></li></ul><p>If you are new to statistics and want to cover more thoroughly, then the following will help you form a solid understanding of the basics. Be aware that a course will add a few months to your learning timeline:</p><ul><li><a href="https://www.amazon.co.uk/Head-First-Statistics-Dawn-Griffiths/dp/0596527586">Head First Statistics</a></li><li><a href="https://www.amazon.co.uk/Statistics-without-Tears-Introduction-Non-Mathematicians/dp/0141987499">Statistics without Tears: An Introduction for Non-Mathematicians</a></li><li><a href="https://www.amazon.co.uk/Statistics-Introduction-Teach-Yourself-Learn/dp/1473652006">Statistics: An Introduction: Teach Yourself: The Easy Way to Learn Stats</a></li><li><a href="https://www.udemy.com/statshelp/">Workshop in Probability and Statistics</a></li><li><a href="https://eu.udacity.com/course/intro-to-statistics--st101">Intro to Statistics</a></li><li><a href="https://www.coursera.org/learn/basic-statistics">Basic Statistics</a></li><li><a href="https://www.khanacademy.org/math/statistics-probability">Statistics and Probability</a></li></ul><p>Get familiar with Pandas — it is a Python library for data analysis and really quite easy to get into once you have a foundation of statistics and Python. It makes learning data analysis fun — you get a sense of progress, it is easy to use and abstracts the details away (statistical functions, plotting).</p><p>This book will help you learn Pandas:</p><ul><li><a href="https://www.amazon.co.uk/Python-Data-Analysis-Wes-Mckinney/dp/1491957662">Python for Data Analysis, 2e</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*d8GSjlwKZ4u9PH0VvVgzTg.jpeg" /><figcaption><a href="https://pandas.pydata.org/">https://pandas.pydata.org/</a></figcaption></figure><h4>Machine learning — introductory articles</h4><p>If you need a birds-eye view of machine learning before jumping into the details, here are a few short blog posts to dip your feet into:</p><ul><li><a href="https://medium.com/@ageitgey/machine-learning-is-fun-80ea3ec3c471">https://medium.com/@ageitgey/machine-learning-is-fun-80ea3ec3c471</a></li><li><a href="https://towardsdatascience.com/machine-learning-from-scratch-part-1-76603dececa6">https://towardsdatascience.com/machine-learning-from-scratch-part-1-76603dececa6</a></li><li><a href="https://medium.com/ml-research-lab/chapter-1-machine-learning-introduction-6e29f8df43e9">https://medium.com/ml-research-lab/chapter-1-machine-learning-introduction-6e29f8df43e9</a></li><li><a href="https://www.digitalocean.com/community/tutorials/an-introduction-to-machine-learning">https://www.digitalocean.com/community/tutorials/an-introduction-to-machine-learning</a></li><li><a href="http://www.r2d3.us/visual-intro-to-machine-learning-part-1/">http://www.r2d3.us/visual-intro-to-machine-learning-part-1/</a></li></ul><p>Of course, there are loads of these type of introductory articles on Medium, so go explore!</p><h4><strong>Machine learning — specialist course</strong></h4><p>The mathematical aspects of machine learning were a scary thought. To be fair, there is no way to get away from the mathematics entirely; though if you are willing to accept the math works and just apply it, this will definitely help.</p><p>Try <a href="https://www.coursera.org/learn/machine-learning">Coursera’s Machine Learning by Andrew Ng</a> — this is a in-depth course, quite academic in style and will take several weeks to complete. It requires you to learn Octave/Matlab which can be a distraction — but Andrew encourages you to trust that the math works (yet provides you enough intuition about why it works) and moves you on swiftly to applying the algorithms to practical examples. The exercises are invaluable so do not skip them — they become so addictive! This course is the de-facto standard for anyone entering into the world of machine learning.</p><p>Also check out <a href="https://developers.google.com/machine-learning/crash-course/">Google’s Machine Learning Crash Course</a> — it’s lighter than the Coursera Machine Learning course and could be completed with a week of solid effort. There is a prerequisite of basic machine learning knowledge for this course (it skips on some details), whereas the Coursera one does start right from the beginning and covers everything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*LrVmQhw4g_Src-Utw_F-bg.png" /><figcaption><a href="https://www.coursera.org/learn/machine-learning">https://www.coursera.org/learn/machine-learning</a></figcaption></figure><h4><strong>Open data sets</strong></h4><p>There are many open data sets on the web — my favourite place is Kaggle which hosts data science competitions. If you are not ready to jump into the competitions, you could simply explore the data sets and practising machine learning on them.</p><p>Two simple data sets to start out with:</p><ul><li><a href="https://www.kaggle.com/c/wine-quality">https://www.kaggle.com/c/wine-quality</a></li><li><a href="https://www.kaggle.com/c/boston-housing">https://www.kaggle.com/c/boston-housing</a></li></ul><p>To really understand some machine learning algorithms under the covers, you could try to implement them from scratch instead of using libraries (e.g. scikit-learn). You can compare the results of your algorithm implementation with results from scikit-learn to get a sense of confidence that your own implementation works.</p><p>Learn from past work by exploring the many open-source Jupyter notebooks (e.g. on Github) for various Kaggle competitions. Look out for the ones with data analysis, exploration of multiple machine learning approaches, and plentiful comments and conclusions throughout.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*TanbLjUI4RbkrF2YecBDlg.png" /><figcaption><a href="https://www.kaggle.com/">https://www.kaggle.com/</a></figcaption></figure><h4><strong>Build a reference library</strong></h4><p>Reading these books cover-to-cover will be overwhelming — instead use them for reference when solving real problems e.g. open data sets.</p><ul><li><a href="https://www.amazon.co.uk/Introduction-Statistical-Learning-Applications-Statistics/dp/1461471370">An Introduction to Statistical Learning</a> (there is a <a href="http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf">free copy</a> on the web)</li><li><a href="https://www.amazon.co.uk/Data-Science-Scratch-Principles-Python/dp/149190142X">Data Science from Scratch: First Principles with Python</a></li><li><a href="https://www.amazon.co.uk/Python-Data-Analysis-Wes-Mckinney/dp/1491957662">Python for Data Analysis, 2e</a></li><li><a href="https://www.amazon.co.uk/Programming-Collective-Intelligence-Building-Applications/dp/0596529325">Programming Collective Intelligence: Building Smart Web 2.0 Applications</a></li><li><a href="https://www.amazon.co.uk/Python-Machine-Learning-scikit-learn-TensorFlow/dp/1787125939">Python Machine Learning: Machine Learning and Deep Learning with Python, scikit-learn, and TensorFlow</a></li><li><a href="https://www.amazon.co.uk/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1491962291">Hands-On Machine Learning with Scikit-Learn and TensorFlow</a></li></ul><p>Form a list of high quality books by checking reviews, asking other people in the community (Stack Overflow, Quora, at work) or read a few chapters before you buy. Some books are really not well written — I find this to be quite common in the data science community where everyone seems to be writing books!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ptdrYK0gwz99rugJRvsHsQ.jpeg" /><figcaption>Lovely books</figcaption></figure><h4><strong>Blogs &amp; papers</strong></h4><p>A few blog posts were mentioned as part of machine learning introduction section. This is about reading much more widely and deeply. Interesting topics could include:</p><ul><li>Machine learning in relation to data science as a whole</li><li>New or less well known machine learning algorithms</li><li>Machine learning in specific domains e.g. finance, weather forecasting, medicine, advertising, image/video classification</li><li>Performance analysis of machine learning techniques</li><li>Implementing algorithms from scratch in Python or other languages</li><li>Patterns/technologies/frameworks/libraries for machine learning</li></ul><p>There are many blogs out there : my favourite is <a href="https://towardsdatascience.com/">Towards Data Science</a>. This author has captured some good ones too:</p><ul><li><a href="https://medium.com/@exastax/top-20-data-science-blogs-and-websites-for-data-scientists-d88b7d99740">https://medium.com/@exastax/top-20-data-science-blogs-and-websites-for-data-scientists-d88b7d99740</a></li></ul><p>Most papers come out of the research departments of top academic institutes. These are too numerous to list — but search around for papers from Stanford, Berkeley, MIT etc. They usually have repositories somewhere. Some other sources of research papers:</p><ul><li><a href="https://www.researchgate.net/">ResearchGate | Find and share research</a></li><li><a href="https://ai.google/research/pubs/">Publications - Google AI</a></li><li><a href="https://www.springeropen.com/">SpringerOpen</a></li></ul><h4><strong>Meetups</strong></h4><p>Meetups can be hit and miss — a few can be unclear on level of proficiency required or tend towards extreme sales pitches. Meetups on machine learning are much easier once you have some foundational knowledge — before that they can be super confusing and a waste of time from a learning perspective.</p><p>A few groups I recommend in London:</p><ul><li><a href="https://www.meetup.com/PyData-London-Meetup/">PyData London</a></li><li><a href="https://www.meetup.com/London-ODSC/">London Data Science ODSC</a></li><li><a href="https://www.meetup.com/Data-Engineering-London/">Data Engineering London</a></li></ul><p>Do your research and check if the talk has something interesting to you and is within your current range of understanding. The other benefit to learning is meeting people in the community and getting their recommendations.</p><h4><strong>Conferences</strong></h4><p>I cannot comment much on this as I haven’t yet been to a data science conference (though I plan to attend ODSC in 2018 — yay!). A bit of research and a few recommendations have helped form this list so far:</p><ul><li><a href="https://conferences.oreilly.com/strata">Strata</a></li><li><a href="https://odsc.com/">ODSC</a></li><li><a href="http://www.kdd.org/conferences">KDD</a></li><li><a href="https://mlconf.com/">MLConf</a></li><li><a href="http://www.dataengconf.com/">DataEngConf</a></li></ul><p>As with meetups, most conferences assume a degree of proficiency — might not be wise to go to one right at the start of your journey, but could be a great way to valid your learning much later in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*5umAMeS4JhX-PqGma0tYAg.png" /><figcaption><a href="https://odsc.com/">https://odsc.com/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*OGsuLr8ESQ2g5tMvwIsOAA.jpeg" /><figcaption><a href="https://conferences.oreilly.com/strata">https://conferences.oreilly.com/strata</a></figcaption></figure><h4><strong>Find a mentor</strong></h4><p>After several months there were a million things jumbling around in my head. After playing with a few data sets I had learnt a lot, failed some more and was left confused about what was next and what is missing in my learning.</p><p>With help from my mentor, I formed a <a href="https://recurse.henrystanley.com/post/better/">mind map my known-unknowns</a> and this provided structure around the details I needed to learn.</p><p>A mentor in data science can also direct you to learn the most important things first and help you apply the <a href="https://en.wikipedia.org/wiki/Pareto_principle">80/20 rule</a> effectively.</p><p>An effective way to structure learning is to take real problems (e.g. from Kaggle) and work with a mentor to explore the different ways to solve the problem and tick things off your list as you go along.</p><h4><strong>Join a data team</strong></h4><p>If you can, get onto a data science team in an organisation that is solving real data problems with machine learning. Not only do you get a local community to engage with on a daily basis, there is no better end-to-end immersive learning experience than working on a machine learning data product alongside talented people.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*JIaBxjDFhoWO5Qhjy4xG5g.jpeg" /><figcaption><a href="https://www.flickr.com/photos/toomuchdew/8119694946">https://www.flickr.com/photos/toomuchdew/8119694946</a></figcaption></figure><h4><strong>Summary</strong></h4><p>This was a broad level overview of how you could get going in your learning of machine learning and data science. There is so much that is not captured in this post, simply because I have not got there yet! I welcome your suggestions for other resources that you have found useful.</p><p>Be resourceful in your approach to accelerated learning — some things will not be applicable in your context or may not work for your learning style. Be willing to try things and throw them away if they do not work for you — form your own learning pathway.</p><p>Thank you to <a href="https://unruly.co/">Unruly</a> (my current workplace) for supporting me in learning about data science! Being part of the data team, a learning budget (20% time, conferences and books) and access to a community of people with similar interests — without these things my rate of learning would be much slower!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b51b3b1917ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unruly-engineering/getting-started-in-machine-learning-b51b3b1917ad">Getting started in machine learning</a> was originally published in <a href="https://medium.com/unruly-engineering">Unruly Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Experiments in making my role redundant: from Team Test Lead to Team Member]]></title>
            <link>https://medium.com/unruly-engineering/experiments-in-making-my-role-redundant-from-team-test-lead-to-team-member-c45e102b9759?source=rss----d1d70e201bfd---4</link>
            <guid isPermaLink="false">https://medium.com/p/c45e102b9759</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[coaching]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[apps]]></category>
            <dc:creator><![CDATA[Esther was a Tester]]></dc:creator>
            <pubDate>Thu, 31 May 2018 12:21:53 GMT</pubDate>
            <atom:updated>2018-05-31T12:21:53.135Z</atom:updated>
            <content:encoded><![CDATA[<p>There are few times in your career that you truly get a chance to reflect on the job you’ve done. Even fewer I would imagine that you leave a company with your head held high, proudly explaining that you achieved your experiment in making your role redundant. Classic questions such as “who will replace you?” and “how’s the handover?” that one would normally expect are instead replaced with requests to communicate how others can continue the approach at scale.</p><p>Therefore before my sieve-like memory forgets my pre-Unruly life, this blog recounts adventures in quality with mobile apps teams at my previous company and the journey on how I went from Test Team Lead to Team Member. Ultimately how I left my team to autonomously own their quality practices and to socialise their work, leading the way forward for the wider company and other teams.</p><h3>Do we need dedicated testers for teams to deliver quality at speed?</h3><h4>The Experiment</h4><blockquote>If we remove [QA]/[Tester]/[Test Engineer]/[Test Lead]/[DevOp] as a siloed role and responsibility then the team will share the burden and together work to improve the quality of the app.</blockquote><p>Let me set the scene... 7 teams, roughly 2:3 tester to developer ratio, 6 different apps across 2 platforms (iOS/Android) for 4 brands. The then Test Team lead had their role split to handle the workload — enter me. Surprisingly knowing very little about mobile app development turned into my biggest advantage, as I didn’t have any bias influencing my approach. First step was to help identify the teams challenges in quality, and what might contribute to their lack of confidence. Best thing about starting a new job: asking tonnes of questions. Always results in the most informative and honest answers.</p><h4>The Challenge</h4><p>After scribbling my way through 5 notebooks (I’ve since invested in a higher page per book ratio), the teams appeared to have a few key difficulties in the name of quality:</p><ul><li><strong>Blame culture</strong>. App clients are bound to services, and it’s easy to blame another team for their service than it is to work together to understand the impact. With a fundamental breakdown in communication between teams, lack of architecture education, and no service contracts meant the teams pointed the finger at others.</li><li><strong>Role silos</strong>. It seemed easier to blame someone else for not doing their job, than it is to work together to ensure the job is done right. Having distinct roles in the team meant more finger pointing.</li><li><strong>Legacy code</strong>. Inheriting outsourced apps meant technical debt, severe lack of unit tests, lack of understanding, and ultimately a fear of owning the code. Developers had little confidence in making changes, and therefore had an over reliance on testers and E2E tests to catch issues. Easier to blame legacy code than it is to take ownership.</li></ul><p>What did this all result in? Our customers having a bad quality experience.</p><h4>The Vision</h4><blockquote>“If you are really good at it, they won’t even know it’s you.” — Joakim Sundem</blockquote><p>The first steps to helping solve these issues was to curate a shared approach in which we could remind ourselves what we value in quality. Helping to support why should the team invest in quality. We collaborated and defined a Quality Manifesto for the team.</p><p><em>We are uncovering better ways of </em><strong><em>testing apps</em></strong><em> and delivering value to our </em><strong><em>customers</em></strong><em>, by doing it and helping others do it. Through this work we have come to value:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qM4N0JJ11ETON77_bkKGlQ.png" /><figcaption>Quality Manifesto as prioritised by the teams</figcaption></figure><p><em>That is, while there is value in the items on the right, we value the items on the left more.</em></p><h4>The Process</h4><p>The curation of the manifesto enabled us to have lots of constructive conversations as teams. It was a little bit of a game changer. Everyone started thinking outside the box. It never was, and never intended to be, rules to develop by. You could argue that that this manifesto actually could be interpreted as: <em>we as a team will work together to ensure we develop what we understand to be the best implementation at the time; automate this understanding.</em></p><p>Through facilitated sessions with the teams, we drilled down into the definitions of each of these values. The intention that we iterate on these definitions as we progress. Understanding impact<strong> </strong>really summarises the ideal state whereby the team don’t make assumptions on what they are developing on behalf of users, stakeholders, customers or anyone with a stake in that feature. Understanding what risks, benefits, how to measure, and ensuring that the right thing is actually being implemented helps to drive a quality outcome. It also promotes communication outside of the team, a step closer to addressing blame culture habits.</p><p>The second and third values on not leaving all the testing to just before shipping, and automating repetitive tasks (perhaps naively) self explanatory and should be well understood across software development in 2018. But it’s good to reinforce all the same. Also questions what automation means. Is it just automating manual tests? How can we automate more infrastructure, deployment processes, Slack bots for user feedback, wrapping legacy code in tests for understanding…?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*cCiN3UiLKUzGyt17T6F3uA.png" /><figcaption>Adapted and inspired by Derek Huethers <a href="https://www.leadingagile.com/2017/02/e-shaped-staff/">Break Organisational Dependencies with an “E-Shaped” Staff</a></figcaption></figure><p>Reflecting on the last two values invokes the memories of when I was applying for Unruly. I shared an old Quality Vision presentation I had done with an ex-colleague of mine. He asked a very simple question:</p><blockquote>“I see you mentioned T-Shaped. Have you considered working towards E-Shaped?”</blockquote><p>While I don’t believe individuals should be striving to be defined a letter of the alphabet (I-shaped, T-shaped…), the term ‘E-shaped’ has been <a href="https://culturecartography.wordpress.com/2012/07/26/business-trend-e-shaped-people-not-t-shaped/">referenced since 2010</a>, and the fundamentals of this concept really do help cement forth and fifth values on team responsibility and collaboration. The concept involves teams removing dependencies that could block them, and instead building a team of next-generation specialists. In theory, this can result in teams with limitless potential.</p><h4>The Methods</h4><p>One of most surprising contributors towards the teams embodying E-shaped traits was the changes in team structure. The absence of a person in a role would once be considered a breaking change, instead it became a gap that could be learnt and filled by the teams.</p><p>Over the course of a year we lost Android developers, iOS developers, PMs, BAs, Scrum Masters, designers, and UX. Not to mention other personal changes throughout the business. We mourned primarily for the loss of the people and their contributions, not for the role they were hired in. Within a year of starting, the ratio of testers to developers reduced from 2:3 to 1:4.</p><p>So how did we fill the gaps? We experimented with many different methodologies, to varying degrees off success. One such experiment was rotating titles (<em>Batman — saving the team from technical distractions such as CI, infrastructure, flakey tests, server side changes. Demo-gorgan — demoing value to internal stakeholders every sprint. Release Captain — taking care of the manual side of app releases</em>) ensuring that everyone upskilled in difference aspects of the day to day.</p><p>The teams started bringing more under their control, and naturally started progressing towards becoming E-shaped. They learnt how to use data to drive decisions. Financial impact, user impact and risk analysis was all learnt. They joined architectural discussions to ensure all clients were understood. They gathered business requirements, engaged in user research and UX sessions, wrote their own copy, and discovered new means of gather translations. They even started designing their own assets.</p><h4>The Methods in Testing</h4><p>As part of this experimental journey, I helped to work with the team to redefine the traditional testing pyramid. <em>For more in depth technical implementation, Hesham Salaman wrote a fantastic </em><a href="https://medium.com/@hesham.salman/the-ios-testing-manifesto-e1bc821cc4c3"><em>Testing Manifesto</em></a><em> that complements the pyramid extremely well for iOS development.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DyBJ_PqyYmNiiOIp-bZweg.gif" /></figure><p><strong>“Team of Testers” base layer — Exploratory Testing<br></strong>- Test pre, during, and post writing code. From API creation, UI implementation, to A/B experimentation.<br>- Always test on a physical device. Customers don’t use emulators.<br>- Gather feedback from colleagues, stakeholders, customers, your family, your dog…<br>- … and understand their needs.<br>- If you are repeating it, automate it.</p><p><strong>Network independent foundation layer — Unit/Snapshot/UI Tests<br></strong>-<strong> </strong>Offline, offline, offline. <br>- TDD.<br>- All tests run against mocks, stubs, fakes, spies, or dummies.<br>- Regularly update offline stubs against dependent services.<br>- Test the stubbed data, don’t assume.<br>- <a href="https://github.com/xcbuddy/microfeatures-guidelines">Micro-features</a> can be independently tested without needing to build the entire app.<br>- Feature integration should be covered by UI Tests that are network independent.</p><p><strong>Integration layer — Contract Tests<br>- </strong>Consumer driven contracts work to document in code how the clients are consuming services.<br>- Tests run against all supported environments<em><br>- </em>Using a collaborative, in-house, open framework to ensure full transparency</p><p><strong>“The Icing on the Cake” layer — Red Route Tests<br></strong>- Name stems from city planning road markings, see <a href="https://tfl.gov.uk/modes/driving/red-routes/rules-of-red-routes">Red Routes</a>.<br>- Identify the minimum number of positive journeys through the app that gives the team confidence in releasing. These form the definition of a Red Route.<br>- These could be driven by User Story Mapping.<br>- Run against <strong>Staging</strong> and <strong>Production</strong>.<br>- Ignore feature flags until they are released into production.<br>- Remove tight coupling of UI driven acceptance tests.</p><p><strong>Taking the emotion out layer — Data Driven Testing<br></strong>- The apps leveraged tools such as <a href="https://firebase.google.com/">Firebase</a>, <a href="https://www.appdynamics.com/">AppDynamics</a>, and <a href="https://appfigures.com/">Appfigures</a> to promote fast customer feedback.<br>- Phased releases.<br>- A/B testing bug fixes. Utilising feature flags.<br>- Always run and monitored in production.<br>- This can be assisted by introducing Error Budgets.</p><h4>The Results</h4><p>There are 3 key results: quality, speed and number of dedicated testers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*ZdLuK74k9fPSIE4OWRRgpA.png" /><figcaption>Quality: iOS Crash Free Users Jan 2017-Jan 2018, objective set at 99%</figcaption></figure><p>Ask most people who work in software and they too will express how hard it is to measure quality. Especially when technology and tools change constantly. I determined crash free users as a solid metric that could measure customer impact. Both apps set an objective of 99% crash free users, they both met/surpassed it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*Cp28bIxleQ7Nsj0sRixrZw.png" /><figcaption>Speed: Cycle time in teams Jan 2017-Jan 2018</figcaption></figure><p><a href="https://confluence.atlassian.com/agile/glossary/cycle-time">Cycle time</a> decreased for both teams. What does this mean? Teams are delivering against their definition of done faster. What this graph doesn’t show, is that post Christmas these times reduced further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*V7P2s3t7yLA-pdoomdkiAw.png" /><figcaption>Dedicated Testers: Ratio of QA/Tester to Engineers Jan 2017-Jan 2018</figcaption></figure><p>Finally, the number of dedicated testers. As of January 2018 there were still those whom identified as testers or QA. Myself included. Shortly thereafter, the ratio of testers to developers shifted to 0:1, with team members identifying themselves as engineers. One tester measured his progress by taking the interview for a developer role. He got the job.</p><h4>The Conclusion</h4><p>Due to successfully making my role redundant, it allowed me to pursue new adventures with Unruly, working to contribute to the success of the developments teams and the company as an <a href="https://medium.com/unruly-engineering/a-coaching-structure-iteration-1-4dd1c0369a62">Agile Test Coach</a>. Thankfully this new chapter begins with teams owning their own quality! Now I get to indulge in working with teams to continuously improve their current practices.</p><p>This experiment has cemented my belief that a <strong>great</strong> <strong>teams</strong> will <em>always</em> be better than rockstars in defined roles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c45e102b9759" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unruly-engineering/experiments-in-making-my-role-redundant-from-team-test-lead-to-team-member-c45e102b9759">Experiments in making my role redundant: from Team Test Lead to Team Member</a> was originally published in <a href="https://medium.com/unruly-engineering">Unruly Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hierarchical Java Configuration]]></title>
            <link>https://medium.com/unruly-engineering/hierarchical-java-configuration-c20a421ee272?source=rss----d1d70e201bfd---4</link>
            <guid isPermaLink="false">https://medium.com/p/c20a421ee272</guid>
            <category><![CDATA[extreme-programming]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Alex Wilson]]></dc:creator>
            <pubDate>Wed, 18 Apr 2018 12:51:45 GMT</pubDate>
            <atom:updated>2018-04-18T12:51:44.702Z</atom:updated>
            <content:encoded><![CDATA[<p>Hard-coding secrets in application code is not ideal.</p><p>It makes rolling them hard by coupling them to an artifact, and is not particularly secure, if the artifact itself is compromised.</p><p>It was, once-upon-a-time, the easiest way to get the job done. Saddling us with technical debt, which needed to be paid back.</p><p>How can migrate these secrets into configuration files, environment variables, or a third-party service, in a safe and incremental way?</p><p><a href="https://github.com/unruly/java-config-fallback">unruly/java-config-fallback</a></p><p>We started building java-config-fallback as a way of facilitating this process, and decided to <a href="https://gds.blog.gov.uk/2017/09/04/the-benefits-of-coding-in-the-open/">build it in the open</a> from Day 1.</p><p><strong>How it works</strong></p><p>This library exposes an entry-point with Configuration.of(...) and accepts any number of ConfigurationSource parameters.</p><pre>Configuration config = Configuration.of(<br>    environment(),<br>    properties(&quot;/etc/myapp/config.properties&quot;)<br>);</pre><p>Values can be retrieved with .get(String key), returning an Optional&lt;String&gt; which represents a potentially missing value (for example, if the key is not present in any of the sources)</p><pre>config.get(&quot;some-key&quot;); // Optional&lt;String&gt;</pre><p>The input sources are composed together such that querying for a key returns the <strong>first</strong> corresponding value it finds (in the order passed to the .of(...) method), falling back to successive sources if not found.</p><p><strong>Design Decisions</strong></p><p>There are already libraries (such as <a href="https://github.com/ufoscout/properlty">https://github.com/ufoscout/properlty</a>) that do something very similar — why build our own?</p><ul><li><strong>Focus on cleanliness and simplicity</strong>: at time of writing, the core library is less than 100 lines of code.</li><li><strong>Architecting for extensibility</strong>: the ConfigurationSource entry point is used to retrieve a specific value for a key, not to load all possible values into memory. This opens the door to providers for external services that operate on a request-by-request basis.</li><li><strong>Use modern Java features</strong>: we designed ConfigurationSource as a <a href="https://www.oreilly.com/learning/java-8-functional-interfaces">functional interface</a>, so custom providers can be defined in-line as lambda expressions:</li></ul><pre>Configuration config = Configuration.of(<br>  (key) -&gt; &quot;always-this-value&quot;<br>);</pre><ul><li><strong>Short-circuit querying</strong>: Incorporating .orElseGet(() -&gt; ...) into the composition of the sources stops the query being evaluated if a value is found “higher up”. For services where the call is non-trivial or expensive, this is much more efficient.</li><li><strong>Do one thing well</strong>: The library doesn’t coerce types (opting for String key, String value) or assume behaviour if a source cannot be queried (missing file or slow network call). This is up to the user to decide, and simplifies a lot of our other design choices.</li><li><strong>Eat our own dog-food</strong>: This library is in use at Unruly, helping us transition our app configuration and advance our architecture. We don’t want to release anything we wouldn’t use ourselves.</li></ul><p><strong>What’s coming later</strong></p><p>Adding some new ConfigurationSources is something we want to achieve before the big 1.0.0 release, such as:</p><ul><li>New file types — XML, JSON, …</li><li>3rd party services — Vault, Consul, AWS Parameter Store</li><li>Local services — data stores, <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-dynamic-data-retrieval">AWS instance metadata</a></li></ul><p>Tracking where a given property came from, to confirm that properties are being loaded from the specified place (<a href="https://github.com/unruly/java-config-fallback/issues/1">https://github.com/unruly/java-config-fallback/issues/1</a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c20a421ee272" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unruly-engineering/hierarchical-java-configuration-c20a421ee272">Hierarchical Java Configuration</a> was originally published in <a href="https://medium.com/unruly-engineering">Unruly Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lights are out down memory lane (part 2)]]></title>
            <link>https://medium.com/unruly-engineering/lights-are-out-down-memory-lane-part-2-3e3dd8dd5596?source=rss----d1d70e201bfd---4</link>
            <guid isPermaLink="false">https://medium.com/p/3e3dd8dd5596</guid>
            <category><![CDATA[episodic-memory]]></category>
            <category><![CDATA[diversity]]></category>
            <category><![CDATA[life]]></category>
            <category><![CDATA[memories]]></category>
            <dc:creator><![CDATA[Paul Cox]]></dc:creator>
            <pubDate>Mon, 19 Mar 2018 15:25:52 GMT</pubDate>
            <atom:updated>2018-03-19T15:25:58.324Z</atom:updated>
            <content:encoded><![CDATA[<p>In my first <a href="https://medium.com/p/c753ec3eca77">post</a> on my stumbling down memory lane I gave an introduction to how I believe my memory works and how it affects my life &amp; interactions, below I go in to some more specific situations.</p><h3>Social situations e.g. many are based on people retelling memories</h3><p>It’s amazing how when you start realising your brain works in a different way you suddenly start noticing more and more situations where social norms assume a particular way of thinking without most people being aware. Take for instance meeting someone new, you tend to start chatting about things you’ve done right? You bond over shared experiences. You relate stories of things that happened to you. People laugh/ cry/ cheer along with the story. You probably take this for granted. What happens when you don’t have any stories to share? It’s not that you’ve never done anything “interesting”, but you have no detailed memory to fall back on to put the story across in a way that would be engaging. How interesting would hearing about someone’s experience be if they were only able to relate fairly surface level details of an event. How many debates entail relating memories of things? This applies to social situations in both professional &amp; personal lives.</p><p>It’s also very baked in to entertainment; books, films, TV quite often have a narrative around someone reliving past events, or how often do they talk about memories making a person who they are? The sci-fi genre often talks about downloading people’s memories in to other bodies/ machines.</p><h3>Jobs</h3><p>Not only can memories be used as an implicit ice breaker, but quite often they are used explicitly so: have you ever done a team building exercise, or joined a new company, where they ask you to describe an experience from you past? This can make things more stressful for someone that can’t relate these. It gets even worse when you are interviewing; how many job applications/ interview processes ask you to describe a particular situation when you did X? I may be able to come up with something but it could be quite a shallow description, which adds to the pressure of an interview… are they thinking I’m making it up as I can’t really describe it, or describe how I felt in the situation?</p><h3>Past, present, &amp; future relationships*</h3><p>As I mentioned previously, memories are often used as an ice breaker in new social situations, or people you don’t know very deeply, but it’s similar in long lasting relationships. How many relationships do you maintain because of past shared experiences? “Remember that time we…” etc. Now try to imagine long relationships without this glue, it’s probably hard to even imagine. I can tell you from experience, it makes it much harder to maintain things. Again, this might at first seem like only a negative. I disagree, a positive side of this is bad or toxic relationships are less likely to be prolonged because of shared good memories, grudges aren’t held as long/ as deeply (granted this also opens you up to being taken advantage of). It’s almost an inbuilt protection against the <a href="https://en.wikipedia.org/wiki/Sunk_cost">sunk cost fallacy</a> in some situations. Relationships are only maintained with activity from both sides, or the interaction as it currently is, there is less reliance, for good and bad, on credit in a social bank.</p><p>Another side to this is that people I click with, or even just more generally meet for a short time, are almost instantly the same in my head as people I’ve known for years. I can build up fairly deep relationships with people relatively quickly. To me this is generally a positive, but again, when viewed through someone’s different lens things can be misconstrued. I know I can come across as being too familiar with people I’ve just met, messaging people I haven’t known very long too much as only older friends(?) would (also related to my reduced timeline in my head, see other post). Past relationships might atrophy, but then if it’s someone I click with then it’s very likely to return very quickly once the relationship starts again, not because I can bring past memories with them to support it, but because I click with them in the moment.</p><p>*relationships in the broad sense, including friendships, colleagues etc.</p><h3>Live in the moment</h3><p>I tend to live in the moment, and I think a large amount of this is due to my lack of episodic memory. I can’t dine off of past experiences by reliving them, even by going back and viewing pictures or videos, I have to constantly be doing things (gigs, travelling etc.) to keep experiencing things/ keep things interesting. I don’t really have nostalgia for things, which means I’m more likely to do something e.g. listen to music, that I enjoy in the here and now, rather than doing something because it reminds me of a past memory/ time in my life.</p><h3>No timeline in my head</h3><p>A side effect of how my memory works is that I have much less of a timeline in my head; things that have happened within the last year/ few months are pretty equivalent to things that happened years ago. The ordering of events can also be off.</p><h3>Moving on from events, both happy &amp; sad, quickly</h3><p>I move on from events very quickly, both happy and sad. I think on the sad side it can be split in to two types:</p><ul><li>standalone sad/ painful events</li><li>sad events with a past attached to them e.g. a breakup</li></ul><p>In the first type, I very quickly lose the attachment to the emotions that I felt in the situation, even if I “remember” the event itself.</p><p>In the second situation, and this may sound quite sad to some, but I effectively have a much lower emotional bank balance with someone; I don’t really have the past experiences to remind me of them to make me feel sad (the flip side being I also don’t have the happy memories of experiences with someone)</p><p>I hope these posts have helped explain me a little to some, and still been informative to others who don’t know me. Please do reach out with any comments and questions, or to counter that I don’t behave in the ways I’ve described! I also don’t want to use any of this an excuse for behaving in a crappy way, so call me out on it if I do, it probably wasn’t intentional…</p><p>My parting request to readers: don’t assume everyone is perceiving life in the same way that you are, take the time to step back and see if their different lens is leading to a misunderstanding, or serving to exclude them. An extension to this, which isn’t really directly related to my memory, but will serve you well <a href="https://www.stephencovey.com/7habits/7habits-habit5.php">“Seek first to understand, then to be understood”/ “Listen to understand, not to form a response/ retort”</a></p><p>If you want to come work for an awesome company that gives you time &amp; flexibility for personal development, we’re <a href="https://unruly.co/about-us/were-hiring/">hiring</a>.</p><p>Further reading:</p><p>As far as I can tell, it’s <a href="https://en.wikipedia.org/wiki/Episodic_memory">episodic memory</a> that I lack.</p><p><a href="https://www.wired.com/2016/04/susie-mckinnon-autobiographical-memory-sdam/">Wired article</a> that resonated with me and prompted me to start thinking about it more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e3dd8dd5596" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unruly-engineering/lights-are-out-down-memory-lane-part-2-3e3dd8dd5596">Lights are out down memory lane (part 2)</a> was originally published in <a href="https://medium.com/unruly-engineering">Unruly Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lights are out down memory lane (part 1)]]></title>
            <link>https://medium.com/unruly-engineering/lights-are-out-on-memory-lane-part-1-c753ec3eca77?source=rss----d1d70e201bfd---4</link>
            <guid isPermaLink="false">https://medium.com/p/c753ec3eca77</guid>
            <category><![CDATA[psychology]]></category>
            <category><![CDATA[episodic-memory]]></category>
            <category><![CDATA[memories]]></category>
            <category><![CDATA[diversity]]></category>
            <dc:creator><![CDATA[Paul Cox]]></dc:creator>
            <pubDate>Mon, 19 Mar 2018 15:25:11 GMT</pubDate>
            <atom:updated>2020-01-25T09:37:48.395Z</atom:updated>
            <content:encoded><![CDATA[<p>Take a moment to think back to your childhood… picture a memory… describe it in as much detail as you can… How vivid is it? Can you feel how you felt then? Did it feel like you were actually there? Do you now feel the emotion? Maybe childhood is a little far back… how about a memory from another time in your past… try again.</p><p>That song that pops up that sends you back to your teenage years maybe?</p><p>First album you bought?</p><p>First concert you went to?</p><p>A breakup?</p><p>First kiss?</p><p>I’ll give you a moment more to do this…</p><p>So, a large number of you probably found this relatively easy and are wondering where this is going… another portion of you might have struggled a little more, particularly with your childhood, but could start to describe some memory, and place yourself in it, feeling the emotions that you felt at the time… I have none of that… I have very few memories of my past at all. I don’t mean of the distant past, I mean any of my past. The few “memories” I do have I almost wouldn’t describe as memories. I can talk about some factual aspects of experiences I’ve had, or things I’ve been told/ learnt/ read about, but I can’t actually re-live a particular memory, experience the emotions of it.</p><p>I’m certainly generally restrained in expressing emotion but I do experience a range of them during everyday life, so it’s not that I don’t have any to recall in the first place. Having said that, I’m not sure whether the “problem” is not being able to fully store the memories, or an inability to retrieve them. However, this is probably more of a philosophical question and not so relevant for this discussion.</p><blockquote>I remember things, I have very few memories.</blockquote><p>I am very good at remembering faces, but not names, which might be related to more of a pattern matching based memory, and may also explain my strength at maths.</p><p>I use to just think I had a bad memory, it took me until a few years ago to realise it’s much more nuanced than this. I’ve always been quite self-reflective and so I’ve gradually been reflecting on how this affects many aspects of my life; from relationships in general, my perception of time, living day to day and the appreciation of “now”, moving on from past events (both happy &amp; sad), and social situations.</p><p>I should add that I still see myself as what society thinks of as a highly functioning human… I did well at school/ uni, have a good job as a software developer for a great company (I’m currently using gold cards &amp; blog days at the company to blog about this), was a regional level swimmer and county level tennis player, have good interactions and relationships with others, have travelled etc. Granted, I also appreciate that I’ve been very privileged in my upbringing and the journey I’ve taken up until this point: a white heterosexual male from a middle class family with parents that gave me the freedom to pursue my interests and then offer a healthy amount of pushing to improve. Similar neural quirks may have a more detrimental affect for someone in less of a privileged position, an appreciation of which will hopefully become more obvious as I go in to details in later posts.</p><p>This rings more and more true the more I’ve come to understand how many social norms are based on the most prevalent lens through which the world is viewed, or the lens that is most prevalently thought of as the “correct” lens (subtle difference between the two), and this has a disproportionate affect on anyone that is already viewed as a minority in a particular situation. Which is why an appreciation of these different lenses is so important for a genuinely open view to the many aspects of diversity; this could be, and may well end up being, a whole other post.</p><p>This first post is meant more as an introduction and then in the next one I’ll go in to a little more detail for some of the areas of life I’ve found it to affect.</p><p>A few aims of these posts in rough order of intention:</p><ul><li>To help myself think about and understand how my memory works/ affects aspects of my life; help to minimise any downsides, make use of any upsides</li><li>Give people I know a better understanding of how it affects things I do and how I react to situations; particularly things that could be misinterpreted</li><li>Raise an awareness for anyone reading this who thinks “Ah, that’s how I perceive things as well”</li><li>Make people pause and think of others in their lives whose memory might work in a similar way, or the other end of the spectrum and how this might affect things</li><li>More broadly, that most traits for how people think/ experience things are a spectrum and that people you interact with are likely to live somewhere different on this spectrum to you</li></ul><p>Following on from last point, it isn’t just that their personality trait might be different, they may experience the same event/ interaction in a completely different way, and it may be functionally impossible for them to fundamentally change this. Viewing their actions through your lens of how you experience life, instead of theirs, can result in a large amount of misunderstanding and unnecessary friction. On the flip side, it’s important to understand how your own actions might be perceived by others when they view it through a different lens. This can be extended to other innate traits and cultural/ nurture factors, that influence this lens. This isn’t just an empathy/ it’s not nice to offend people’s point of view, it’s also useful from a selfish point of view; we interact with ever increasing numbers of people, in both our professional and personal lives, and if you want someone to do something for you, they are much more likely if you frame it in a way that makes sense through their lens.</p><p>As I’ve gradually become more and more aware of how the way that my memory operates affects things, I’ve mentioned it to more and more people. It’s been an interesting mix of reactions: fascination that lack of episodic memory is a thing and how it affects how I experience life (both from sympathetic and an almost jealous point of view!), and sudden light bulb moments of them realising others in their lives might experience things in a similar way. I also only ever really viewed this as a negative trait, but I’ve come to appreciate that as with most things in life, it’s just different; it has its positives and negatives. I only have access to my perception of things so I’m unsure how unusual it is, or how much I’m talking about is related to my memory vs other personality traits of mine vs more generally how everyone operates. I imagine, as with most things, it’s a mixture of all of these. So, feel free to ask questions (I’m pretty open about it all), or relate similar experiences.</p><p>In part 2 I go in to more specific situations I’ve found it to affect, you can read it <a href="https://medium.com/p/3e3dd8dd5596">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c753ec3eca77" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unruly-engineering/lights-are-out-on-memory-lane-part-1-c753ec3eca77">Lights are out down memory lane (part 1)</a> was originally published in <a href="https://medium.com/unruly-engineering">Unruly Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detecting when you’re in IntelliJ]]></title>
            <link>https://medium.com/unruly-engineering/detecting-when-youre-in-intellij-d4ea0fb86972?source=rss----d1d70e201bfd---4</link>
            <guid isPermaLink="false">https://medium.com/p/d4ea0fb86972</guid>
            <category><![CDATA[intellij]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Tom Johnson]]></dc:creator>
            <pubDate>Thu, 15 Mar 2018 12:43:36 GMT</pubDate>
            <atom:updated>2018-03-15T12:42:40.626Z</atom:updated>
            <content:encoded><![CDATA[<p>How do you make your tests aware if you’re running them from your IDE or not? That’s what I’ve been wrestling with, and there were some surprising results, so I thought it would be worth sharing.</p><p>But first of all, why on earth would you ever care?</p><p>When I joined the team at Unruly, we had a suite of acceptance tests for our<br>exchange, each of which called main() at the beginning of the test so we had something to test against. That meant we could just invoke the test from within IntelliJ, debug the exchange from a test, etc. Everything was nice and convenient.</p><p>Some things, however, were too convenient. As we were starting the exchange<br>programmatically, we could do things like inject test doubles, replace statics,<br>and generally fool around with the guts of the exchange. We weren’t <em>just</em> invoking main(), we were constructing a test exchange and starting that.<br>This meant that our acceptance tests weren’t really acceptance tests — and that isn’t just a semantic argument.</p><p>On more than one occasion, we broke application startup. Because our tests didn’t actually <em>start</em> the exchange, not in the way we did in production, we only discovered that when a botched deploy left us without a service in production.</p><h4>Test Against What You’re Going To Deploy</h4><p>So we changed things. We removed the various injections and dependencies, stubbing out behaviour at the edges rather than shoving mocks into internals, and got to the point where we could run the exchange and its tests in different processes. That way, we were actually testing against the exchange we were deploying, instead of a jumble of exchange components.</p><p>Then we codified that, by removing the ability of tests to start the exchange.<br>The only way to run the acceptance tests was to already have an exchange started and ready to run against. Doing this with a single instance of the exchange for the entire AT run (as opposed to a fresh one for each test class) also forced us to clean up some accidental dependencies on initial state.</p><p>Not only was this more principled, giving us better, more meaningful coverage, it was also faster. The relatively expensive startup was being done once, instead of dozens of times (once per acceptance test class). The gain wasn’t quite as large as I’d hoped for, but it was noticeable.</p><p>Sounds like a pure win, right? Well, except for the experience in the IDE.</p><h4>Frictionless Feedback is Important</h4><p>In order to run tests in the IDE, you had to remember to start the exchange first. If you changed code, you had to remember to restart it. In order to debug, you had to juggle in your head which process to restart (the tests? or the exchange?), and in order to build and deploy, you also had to remember to stop the exchange.</p><p>Even when you remembered, it was just <em>fiddlier</em>. A few extra clicks, a requirement to use the mouse instead of the keyboard.</p><h4>Robust or Frictionless? Why not both?</h4><p>The fast-feedback loop of everyday development was worse. It was easy to under-estimate the impact that had. Also, the additional robustness and performance of the build script meant we didn’t necessarily need it in the IDE.</p><p>So what if we ran the tests against an exchange in a different process in the<br>full deploy script, but when we’re in the IDE we start up the exchange in the<br>same process?</p><p>That requires two things: making the tests toggle-able as to whether they start<br>the exchange or not, and passing the toggle in. Surprisingly, the hard part was<br>the latter.</p><h4>-DstartExchange=true</h4><p>The first idea was to pass in a system property when running ATs from Maven.<br>If the property is set, the tests are being invoked from a Maven build, so don’t<br>start the exchange. Otherwise, do.</p><p>I was very surprised when that didn’t work: the property was being set even when running tests in IntelliJ. It turns out that IntelliJ looks at the Maven config, and applies any command-line arguments to test targets when running tests.</p><p><em>Thanks, IntelliJ!</em> That was some help I didn’t need or want.</p><p>As a result, I can’t distinguish from where the tests are being run via arguments provided in the Maven script.</p><h4>Run configurations</h4><p>The second idea would be to pass in a system property when running the ATs from <strong>IntelliJ</strong>, with custom run configurations. It’s easy to edit configurations to pass in an argument, and Maven is unaware of them. It’s also easy to share and check them in, so it’s not something you need to think about when setting up a dev environment.</p><p>Of course, we don’t just use a pre-existing run configuration: sometimes we’ll run “all tests”, but then sometimes we’ll run “only this case which just failed”, or “only this case I just wrote”. That’s fine — you can edit a default configuration, and then any new JUnit configuration will inherit that.</p><p><strong>But you can’t share and check in a default configuration.</strong></p><p>Which means that you have ATs working in the IDE, <em>until they don’t</em>, and it’s totally unclear as to what the difference is between this case running by itself<br>and as part of a suite.</p><h4>Something totally horrible</h4><p>The third idea was a terrible, horrible, no-good, very bad idea. What if I just<br>looked in the system properties to see if, y’know, <em>maybe</em> there’s something there that tells me this is being run from IntelliJ?</p><p>And there was. The sun.java.command property is set to com.intellij.rt.execution.junit.JUnitStarter &lt;…various settings…&gt; when run from IntelliJ. So that gives me my function:</p><pre><br>private static boolean isRunningInIntelliJ() {<br> return System.getProperty(“sun.java.command”,“”)<br>              .contains(“intellij”);<br>}<br></pre><p>Let me repeat: I know this is horrible. It’s fragile — I have no reassurance that this will continue to work in future versions of IntelliJ, and I’m certain it won’t work from Eclipse.</p><p>But sometimes, in the absence of a principled way of solving a problem, all you have are hacks. After all: it works, and if the alternative is either a loss of<br>robustness or frictionlessness, then you’re better off swallowing your pride and embracing the hack.</p><p>But if anyone knows a nice, principled way of doing this, please: let me know.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4ea0fb86972" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unruly-engineering/detecting-when-youre-in-intellij-d4ea0fb86972">Detecting when you’re in IntelliJ</a> was originally published in <a href="https://medium.com/unruly-engineering">Unruly Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>