<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ray Berger on Medium]]></title>
        <description><![CDATA[Stories by Ray Berger on Medium]]></description>
        <link>https://medium.com/@RayBB?source=rss-7ddb752ac03a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7Ala-UxXsqe50loYsqge8A.jpeg</url>
            <title>Stories by Ray Berger on Medium</title>
            <link>https://medium.com/@RayBB?source=rss-7ddb752ac03a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 14 Nov 2026 09:33:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@RayBB/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What is Zhong Tai? Front End, Back End, …Middle End?]]></title>
            <link>https://medium.com/@RayBB/what-is-zhong-tai-front-end-back-end-middle-end-e6456bbe0fef?source=rss-7ddb752ac03a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6456bbe0fef</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Ray Berger]]></dc:creator>
            <pubDate>Thu, 19 Mar 2020 14:01:00 GMT</pubDate>
            <atom:updated>2020-03-19T14:01:00.958Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is Zhong Tai? Front End, Back End, …Middle End?</h3><h4>Spoiler Alert: It’s just another way to say platform</h4><p>The 2020 O’Reilly Software Architecture Conference training I attended, which I reviewed <a href="https://medium.com/@RayBB/oreilly-software-architecture-conference-2020-a-young-engineer-s-review-7236380e8f2d?sk=bd839e6300470989b5fae5b23b7514c2">here</a>, suggested that we read Thoughtworks’ <em>Radar</em>. In the <a href="https://assets.thoughtworks.com/assets/technology-radar-vol-21-en.pdf">latest volume</a>, I came across a strange phrase: Zhong Tai. The definition in <em>Radar</em>, ”<em>an approach to delivering encapsulated business models,”</em> wasn’t exactly elucidating. Googling Zhong Tai revealed a dearth of knowledge about the subject on the Western Internet.</p><p>A friend, who is fluent in Mandarin, helped me learn more. She determined that the Chinese characters for this are 中台 and linked me to a few Chinese sites discussing it. I plugged the sites into Google Translate and started digging.</p><p>There seems to be two primary ways of thinking about Zhong Tai:</p><ol><li>From a business perspective, thinking of capabilities</li><li>From a technical perspective, thinking of architecture</li></ol><h3>A Business Perspective</h3><p>This perspective is mostly based on listening to the Thoughtworks <a href="https://thoughtworks.libsyn.com/zhong-tai-innovation-in-enterprise-platforms-from-china">podcast</a> on the topic. When looking at it from a business perspective, Thoughtworks makes Zhong Tai seem like something between SaaS solutions and whitelabel solutions. The core idea is that you can transfer business capabilities to another organization, internal or external. Internally, this means breaking down barriers between departments to encourage sharing services and resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*hYmedN5gHESXIuRSn3wCZQ.png" /><figcaption>Photo by <a href="https://unsplash.com/@birderee?utm_source=medium&amp;utm_medium=referral">KT</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>One example of this: if you’re a food delivery company, you’ve already built out a full infrastructure around driver tracking, routing, queueing, and much more. If someone wants to start a makeup delivery company, they could reuse your whole infrastructure around delivery. The Thoughtworks podcast didn’t mention how this “reuse” of business capabilities would work in terms of licensing or who actually runs the servers. It seems like Zhong Tai would be most well suited for a company that owns multiple smaller companies that are similar to one another</p><h3>A Technical Perspective</h3><p>When digging into the Chinese web I quickly found that the idea of Zhong Tai is attributed to Alibaba, a Chinese e-commerce giant. They wrote an entire <a href="https://book.douban.com/subject/27039508//">book</a> on Zhong Tai with a title that roughly translates to: <em>The way of enterprise IT architecture transformation: Alibaba’s middle-end strategic thinking and actual combat</em> (ISBN: 9787111564805). I was able to find a copy of the book online and ran it through Google Translate. I read the first chapter and learned about the brief history of Zhong Tai.</p><ul><li>In 2003, Taobao, a C2C website, was founded as part of the company Alibaba.</li><li>In 2010, Taobao launched Tmall, a B2C website which became wildly successful.</li><li>These two websites, although both owned by Alibaba, each had their own e-commerce system. Payment processing and orders were completely different on each platform.</li><li>They realized they were wasting massive resources maintaining them independently so they created a shared business division so they reuse parts of the e-commerce system.*</li></ul><p>*Like most of this article, most of my understanding is based on Google Translate. Technical words/concepts don’t translate so easily so I’m making my best guess for some of this.</p><p>When looking at this from a technical perspective, Zhong Tai seems to be about doing something that’s already quite common in Western tech companies: creating shared platforms. One company might make a payment platform that could be used by multiple departments or organizations. If that payment platform was made available to the public you might call it <a href="https://stripe.com/">Stripe</a>.</p><p>Why is it called middle end? A core problem Zhong Tai is trying to solve is that the frontend changes too fast for the backend to keep pace. The idea is that you can create a middle layer (or end) that depends on your platform (or backend). That middle layer can be changed very quickly to keep up with the changes on the frontend. I didn’t find any great examples in reading. However, one example might be: if many teams in a company rely on machine learning (ML), the company could create a platform for training, maintaining, and accessing ML models. A middle end would be built on that platform, and could house business logic and expose data to a frontend.</p><p>One recent <a href="https://36kr.com/p/5286443">article</a> harshly criticized Zhong Tai. The author said that Zhong Tai is very expensive and while it may help big companies it will squash small companies. The author claimed that there isn’t any common definition of Zhong Tai, other than that it involves reusing technology. They also felt that it set expectations far too high and didn’t appreciate the complexity and uniqueness of businesses.</p><h3>Etymology</h3><p>中台 are the Chinese characters for Zhong Tai. <a href="https://en.wiktionary.org/wiki/%E4%B8%AD">中</a> is pronounced zhōng and means middle. <a href="https://en.wiktionary.org/wiki/%E5%8F%B0">台</a> is pronounced tāi and means, among many things, stage, but in our case it is translated to “end.” Note: 中台 is also the phrase for <a href="https://en.wikipedia.org/wiki/Cross-Strait_relations">China-Taiwan relations</a> so that may cause confusion when translating or searching this phrase.</p><h3>Conclusion</h3><p>Zhong Tai, as best I can tell, is the concept of creating platforms that can be shared across your organization, or possibly outside of your organization. When written like that, it isn’t anything new or shiny. Maybe that’s why Zhong Tai hasn’t been well documented on the Western Internet. Maybe I am grossly oversimplifying a complex concept based on poor translations. If that’s the case let me know and I’ll be happy to update the article.</p><h3>Sources</h3><ul><li><a href="https://www.zhihu.com/question/57717433/answers/updated">Contextualizing Zhong Tai</a></li><li><a href="https://36kr.com/p/5286443">Criticizing Zhong Tai</a></li><li><a href="https://thoughtworks.libsyn.com/zhong-tai-innovation-in-enterprise-platforms-from-china">Thoughtworks Podcast</a></li><li><a href="https://book.douban.com/subject/27039508//">Alibaba book on Zhong Tai</a></li><li><a href="https://assets.thoughtworks.com/assets/technology-radar-vol-21-en.pdf">Thoughtworks Radar</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6456bbe0fef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[O’Reilly Software Architecture Conference 2020: A Young Engineer’s Review]]></title>
            <link>https://medium.com/@RayBB/oreilly-software-architecture-conference-2020-a-young-engineer-s-review-7236380e8f2d?source=rss-7ddb752ac03a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7236380e8f2d</guid>
            <category><![CDATA[conference]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Ray Berger]]></dc:creator>
            <pubDate>Tue, 03 Mar 2020 14:09:56 GMT</pubDate>
            <atom:updated>2020-03-03T14:09:56.825Z</atom:updated>
            <content:encoded><![CDATA[<h4>Katas, Microservices, and Reactive Streams: a Buzzword Oasis</h4><p>In 2015 O’Reilly started their Software Architecture Conference. Five years later, I attended the 2020 O’Reilly <a href="https://conferences.oreilly.com/software-architecture/sa-ny">Software Architecture Conference</a> in New York City. What follows is a summary of the two day training, a summary of my favorite talks, and miscellaneous thoughts. This was my first endeavor into formal software architecture so I approached this with a fresh mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rp1Nof3-SaEOgZ9uvtRf6Q.jpeg" /><figcaption>A tightly cropped photo of my conference badge</figcaption></figure><h3>Training</h3><p>I signed up for the Fundamentals of Software Architecture training by <a href="https://twitter.com/markrichardssa">Mark Richards</a>. It seemed like the best fit for my interests since I wasn’t looking for tooling-oriented training. The fundamentals were a good fit for a young engineer like myself. The training was based on Mark’s new book, <a href="https://en.wikipedia.org/wiki/GRPC">Fundamentals of Software Architecture</a>, so you can read it for yourself if you want to learn more.</p><p>When the training started I was a little leery. It seemed like this might be a training session full of fluff and a smooth talking consultant. That was not the case. Mark quickly dove into the content and started asking questions that helped break down popular assumptions. One such belief is that architects set rules for everyone to follow and lay out some grand plan for the organization. However, an architect is in a more collaborative role. They might set some rules but they also create guidelines and work with engineers and business folks to discover and prioritize system attributes, or <a href="https://en.wikipedia.org/wiki/ilities">ilities</a> as they may be called.</p><p>A key theme of the training is that <em>everything</em> is a tradeoff. People are quick to espouse the benefits of a new technology but there is always a tradeoff to using that new technology. The tradeoff is not always easy to find. One way to help find them is through the lens of <a href="https://en.wikipedia.org/wiki/Non-functional_requirement">quality attributes</a> (aka nonfunctional requirements). These attributes should always be determined in collaboration with the business stakeholders.</p><p>The training suggested these steps to discover attributes with stakeholders:</p><ol><li>Show stakeholders a list of attributes</li><li>Ask them to pick the the ones they’d like to see in their system</li><li>Ask them to pick seven that are important</li><li>Ask them to pick three of those seven that should be the highest priority*</li></ol><p>*Asking them to rank them would be a fool’s errand. So don’t.</p><p>The difference between these attributes may be unclear at first. For example, is elasticity or scalability more important for a live sports streaming service? Like everything, it depends. But understanding the difference between scalability (the ability to easily add more capacity as user base grows) and elasticity (the ability to handle very spiky traffic) is important. As such, it’s necessary to examine these attributes through the lens of architecture.</p><p>Throughout the training we engaged in <a href="https://archkatas.herokuapp.com/rules.html#">architecture katas</a>. The katas brought the room to life and gave everyone a chance to practice what we learned. They were difficult but a lot of fun and gave us a chance to get to know each other.</p><p>Occasionally, we discussed tooling specific to architects. <a href="https://github.com/TNG/ArchUnit">Archunit</a> was the most novel to me. It is a testing library to enforce architecture decisions at a code level. You could create a test to check if the presentation layer is directly calling the persistence layer and, in combination with other CI/CD tools, prevent merging when the test fails.</p><p>As an architect, you should focus on expanding your breadth of knowledge or, <a href="https://www.edagoodman.com/the-value-of-knowing-what-you-dont-know-you-dont-know/">knowing what you don’t know</a>. The training suggested that it’s more important to know about a lot of technologies and their general overview than to know a lot about a few technologies. If you know one technology really well, you’ll see it as the solution to many problems. If you know many technologies at a high level you’ll know what tools to evaluate to help make the right decision.</p><p>Free resources to help expand your breadth of tech knowledge:</p><ul><li><a href="https://www.infoq.com/">https://www.infoq.com</a> — Covers many tech trends and even has an architecture specific section</li><li><a href="https://dzone.com/refcardz">https://dzone.com/refcardz</a> — Refcardz provides concise summaries of technology topics</li><li><a href="https://www.thoughtworks.com/radar">https://www.thoughtworks.com/radar</a> — Published twice a year. Predicts and discusses upcoming trends</li></ul><p>Expanding your knowledge base is critical for an architect. If you think you don’t have enough time for reading these then try reframing if by replacing every “I don’t have time” with “it’s not a priority.” If you decide it is a priority, you could put that in place by spending 20 minutes each morning to see what’s happening in the tech world before you dig into your email.</p><p>We also covered architecture certifications. Even if paying for the certificate isn’t worth it to you, filling out the certification <a href="https://certification.opengroup.org/docs/OpenCA_Conformance_Requirements_v1.1.pdf">application</a> will show you where there are gaps in your skills and experience. The most widely accepted certifications in the field are from <a href="https://www.opengroup.org/certifications/certified-architect-open-ca">The Open Group</a> and the <a href="https://www.eacoe.org/">Enterprise Architecture Center of Excellence</a>.</p><p>The second half of the training focused on the nuts and bolts of architecture patterns. We covered the following architectures: <a href="https://en.wikipedia.org/wiki/Multitier_architecture">layered monolith</a>, <a href="https://learning.oreilly.com/library/view/software-architecture-patterns/9781491971437/ch03.html">microkernel</a>, <a href="https://en.wikipedia.org/wiki/Microservices">microservices</a>, <a href="https://en.wikipedia.org/wiki/Service-oriented_architecture">service-oriented</a>, <a href="https://en.wikipedia.org/wiki/Event-driven_architecture">event-driven</a> (including mediator and broker <a href="http://radar.oreilly.com/2015/02/variations-in-event-driven-architecture.html">topologies</a>), and <a href="https://learning.oreilly.com/library/view/software-architecture-patterns/9781491971437/ch05.html">space-based</a>. We spent time examining the <a href="https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing">fallacies of distributed computing</a>.</p><p>The final topic was communicating decisions. They recommend using <a href="https://github.com/joelparkerhenderson/architecture_decision_record">Architecture Decision Records</a>, which are kind of like changelogs with reasoning but for high level implementation details.</p><h3>Talks</h3><p>The two days of training were followed by two days of talks. Some were just okay and others expanded my understanding of architecture. The not-so-good ones were either too generic or not closely related to architecture. They would be a better fit for a more general software conference.</p><p>My favorite talks:</p><h4>Technical Debt: A Master Class</h4><p>r0ml Lefkowitz | <a href="https://s3.amazonaws.com/learning-programmer.com/OReillySACon-TechDebt4.pdf">slides</a> | <a href="https://learning.oreilly.com/videos/oreilly-software-architecture/9781492050506/9781492050506-video323975">description</a></p><p>He gave an overview of the history of technical debt, explained why it’s hard to grasp (hint: it’s not something you pay back), and argued that technical debt primarily comes from dependencies.</p><h4>Maturing an Agile organization with intentional architecture</h4><p>Nimisha Asthagiri (edX) | <a href="https://web.archive.org/web/20200228185135/https://docs.google.com/presentation/u/1/d/1S425SpVsXdVRarteRNW8cWvH3mTSFd7IRDdKdPNpq7M/export/pdf?id=1S425SpVsXdVRarteRNW8cWvH3mTSFd7IRDdKdPNpq7M&amp;pageid=g7e400b553d_2_4">slides</a> | <a href="https://conferences.oreilly.com/software-architecture/sa-ny/public/schedule/detail/79997">description</a></p><p>She primarily talked about how edX has tried to get architects and developers to collaborate. On the surface it isn’t an interesting topic but she gave very concrete examples such as hosting architecture lunches with and without topics and different ways of organizing their “architecture guild” that failed and are still in trial.</p><h4>Where do great architectures come from?</h4><p>Mary Poppendieck (Lean Software Development Series) | <a href="https://cdn.oreillystatic.com/en/assets/1/event/307/Where%20do%20great%20architectures%20come%20from_%20Presentation.pdf">slides</a> | <a href="https://conferences.oreilly.com/software-architecture/sa-ny/public/schedule/detail/81599">description</a></p><p>Mary started working at Bell Labs over half a century ago. She started off working on a phone system with redundancy and localization. Years later Google faced similar problems with redundancy at a hardware level. Today we’re going through the cycle again with AI and smart homes.</p><h4>I don’t understand micro-frontends</h4><p>Luca Mezzalira (DAZN) | <a href="https://cdn.oreillystatic.com/en/assets/1/event/307/I%20don_t%20understand%20micro-frontends%20Presentation.pdf">slides</a> | <a href="https://conferences.oreilly.com/software-architecture/sa-ny/public/schedule/detail/79739">description</a></p><p>Micro-frontends are new to me and this was a good overview. The idea is to adapt Domain Driven Design to the frontend. Instead of having a huge front-end and forcing all the teams to use the same tools, you can create independent frontends. This enables faster deployment of changes and more flexibility for teams. How do you share headers/footers when following this pattern? Luca’s team did analysis and found that those components only changed a few times in the past several years so they decided to copy the code into each (of about 5) frontends instead of complicating matters with a shared code library.</p><h4>Challenges to internationalize the largest digital bank in the world</h4><p>Luiz Hespanha (Nubank), Lucas Cavalcanti (Nubank) | <a href="https://cdn.oreillystatic.com/en/assets/1/event/307/Challenges%20to%20internationalize%20the%20largest%20digital%20bank%20in%20the%20world%20Presentation.pdf">slides</a> | <a href="https://conferences.oreilly.com/software-architecture/sa-ny/public/schedule/detail/79885">description</a></p><p>This was one of the most concrete talks of the conference. They told the story of the Nubank architecture from providing a frontend for an outsourced credit card processor to deploying customized Kubernetes clusters across the world.</p><h3>Miscellaneous Thoughts</h3><ul><li>The organizers talked a lot about networking at this conference and how valuable it is. Do architects collaborate across organizations? I’m curious how this value manifests to architects.</li><li>In the middle of the keynote they stopped for two minutes and asked everyone to talk to their neighbors. I loved this.</li><li>I met great people and talked about mainframes, security tooling, and zhong tai. But it was not easy to choose who to strike up a conversation with.</li><li>For a company that is supposed to showcase the latest and greatest in tech, O’Reilly’s <a href="https://www.oreilly.com/conferences/mobile-app/">events app</a> is pretty rough 😬</li><li>They sent a lot of emails reminding people to charge their devices before the event. It must have worked because there weren’t many chargers around but the ones I saw were usually available.</li><li>Some tables at lunch displayed table topics such as what’s next?, what do you want to change in your organization?, and <a href="https://en.wikipedia.org/wiki/GRPC">gRPC</a>. The topic cards were a nice indicator of who people wanted to talk at lunch.</li><li>There was also a talk that mentioned <a href="https://www.reactive-streams.org/">reactive streams</a>, which is basically a way to give your processor the ability to control the amount of data coming from the source. This was a new idea for me.</li></ul><p>All in all, I really enjoyed the conference and got a lot out of it. For the price, I don’t think I could recommend it to anyone who was footing the bill themselves. If your company is willing to pay for it (thanks NYT) then you should go for it. Thanks for reading and best of luck with your software architecture journey.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7236380e8f2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Drone.io, pipenv, Docker, and the Quest for CI/CD]]></title>
            <link>https://medium.com/@RayBB/drone-io-pipenv-docker-and-the-quest-for-ci-cd-22797f630049?source=rss-7ddb752ac03a------2</link>
            <guid isPermaLink="false">https://medium.com/p/22797f630049</guid>
            <dc:creator><![CDATA[Ray Berger]]></dc:creator>
            <pubDate>Fri, 31 Jan 2020 20:20:09 GMT</pubDate>
            <atom:updated>2020-01-31T20:20:09.454Z</atom:updated>
            <content:encoded><![CDATA[<p>I wanted to use Drone.io to run tests for an app that uses pipenv for dependency management. Turns out it wasn’t as simple as I thought. The problem isn’t specific to Drone but more of a challenge with Docker and pipenv. I couldn’t find a good solution when browsing help forms so I’m just posting the solution I eventually came to.</p><p>Goal: Fast Drone tests with pipenv</p><p>Problem: pipenv installs requests in a random(ish) directory and install is relatively slow</p><blockquote>“<em>The virtualenv’s name is the project’s root directory name, plus the hash of the full path to the project root.</em>” <a href="https://github.com/pypa/pipenv/issues/796#issuecomment-333372544">source</a></blockquote><h3>Solution</h3><h4>In Docker:</h4><ul><li>Use Pipenv docker <a href="https://hub.docker.com/r/kennethreitz/pipenv/">image</a> as base so pipenv is correctly</li><li>In your image, set working dir to where the src will eventually be clone to</li><li>You need to set this as the working dir because of the way virtualenv names directories (see above). If you create it elsewhere and then try to copy it to the src directory it won’t work. More info <a href="https://stackoverflow.com/a/58772116/620699">here</a>. You could simply reinstall every time but that slows things down considerably.</li><li>Set PIPENV_VENV_IN_PROJECT=1 so virtualenv is created in project dir <a href="https://pipenv.kennethreitz.org/en/latest/install/#virtualenv-mapping-caveat">source</a></li><li>Run pip install</li><li>Move the .venv to a different folder so it’s not deleted when git clone happens</li></ul><h4>In Drone:</h4><ul><li>Use our new image as base</li><li>The default starting directory is root of my git repo</li><li>`mv /app/.venv .` — this moves our pipenv dependencies into cloned folder</li><li>`pipenv install — dev — deploy` install any dependencies that weren’t in our image already</li><li>— deploy makes it so we get an error if the Pipfile.lock is out of date</li><li>— dev just makes us install dev dependencies along with regular ones</li><li>`pipenv run pytest` works great now</li><li>To deploy to GAE:</li><li>Create a requirements.txt from Pipfile <a href="https://pipenv.kennethreitz.org/en/latest/advanced/#generating-a-requirements-txt">source</a></li><li>Deploy to GAE with those requirements</li></ul><p>Thanks:</p><p><a href="https://pythonspeed.com/articles/pipenv-docker/">https://pythonspeed.com/articles/pipenv-docker/</a> — for some inspiration</p><p>Example Dockerfile.buildbox:</p><pre><strong>FROM </strong>kennethreitz<strong>/</strong>pipenv<br><strong>WORKDIR /</strong>drone<strong>/</strong>src<strong>/</strong>github.com<strong>/</strong>my-org<strong>/</strong>my-repo<br><strong>COPY </strong>Pipfile<strong>* </strong>.<strong>/<br>ENV <em>PIPENV_VENV_IN_PROJECT</em></strong>=1<br><strong>RUN </strong>pipenv install <strong>--</strong>dev <strong>--</strong>deploy<br><strong>RUN </strong>mv .venv <strong>/</strong>app</pre><p>Example .drone.yml</p><pre><strong>pipeline</strong>:<br><br>  <strong>run_core_tests</strong>:<br>    <em># the &amp;buildbox_image of the line below creates a variable that we can call elsewhere with *buildbox_image<br>    </em><strong>image</strong>: &amp;buildbox_image us.gcr.io/my-gcp-project/private-custom-build-env-image:somehash<br>    <strong>pull</strong>: &amp;pull_enabled false <em>#This is important so we don&#39;t accidentally get stuck with an old build image during dev<br>    </em><strong>commands</strong>: <em># These should be cleaned up into a build step and a test step<br>      </em>- mv /app/.venv .<br>      <em># Dev makes it so we install dev dependencies<br>      # Deploy makes it so we will throw an error if Pipfile.lock is out of date (aka out of sync with Pipfile)<br>      # https://pipenv.readthedocs.io/en/latest/cli/#cmdoption-pipenv-install-d<br>      # Most dependencies should already be installed from the docker image, however, if we added some recently this ensures they&#39;ll be installed<br>      </em>- pipenv install --dev --deploy<br>      <em># Verbose makes output a bit more readable<br>      # Capture=no allows us to see print statements<br>      # pythonwarnings allows us to hide the python warnings<br>      </em>- pipenv run pytest --verbose --capture=no --pythonwarning=ignore<br>      <em># Now that the tests passed, lets make the requirements.txt in case we want to deploy to GAE<br>      # This needs to happen in the same block as pipenv install in case we have any new dependencies<br>      </em>- ./tools/create_requirements.sh<br>    <strong>when</strong>:<br>      <strong>event</strong>: [push, tag]</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22797f630049" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using xdotool and Websockets to Automate the OnePlus Clicker Game]]></title>
            <link>https://medium.com/@RayBB/using-xdotool-and-websockets-to-automate-the-oneplus-clicker-game-ce1d2cb3f459?source=rss-7ddb752ac03a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce1d2cb3f459</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[xdotool]]></category>
            <dc:creator><![CDATA[Ray Berger]]></dc:creator>
            <pubDate>Mon, 05 Nov 2018 02:34:15 GMT</pubDate>
            <atom:updated>2018-11-05T23:59:45.523Z</atom:updated>
            <content:encoded><![CDATA[<p>Last month OnePlus unveiled a promotion for their then-upcoming OnePlus 6T. The premise was simple: tap or click the screen to unlock mystery prizes. The more you clicked, the better the prize would be. After clicking to the first prize, just 60 clicks, I decided try automating the process.</p><p>If you go the game’s website from your desktop, you’ll get a message asking you to open it on mobile. Luckily, it’s easy enough to <a href="https://developers.google.com/web/tools/chrome-devtools/device-mode/">simulate a mobile</a> device using the Google Chrome developer tools.</p><h4>xdotool</h4><p>Clicking a few thousand times using a tool like AutoHotkey would be easy. Unfortunately AutoHotKey is not available on linux, but there is an alternative: <a href="https://manpages.ubuntu.com/manpages/cosmic/man1/xdotool.1.html">xdotool</a>. With a few quick Google searches I was able to setup <a href="https://gist.github.com/RayBB/3b713a4be9d8cd4a9e8fc0c0f9147c05">this</a> bash script to start clicking for me. Within a few hours I was up to 60,000 clicks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*qHUeQyB2VrZ3wSVXHDE87Q.png" /></figure><p>The next milestone was 300,000 clicks. I wanted to give it a shot to see what prizes laid behind such a massive number of clicks. Unfortunately, there were several downsides to the xdotool method:</p><ul><li>I could no longer use my computer</li><li>It was relatively slow</li><li>The browser would sometimes crash (likely due to the devtools storing a lot of info)</li></ul><h4>Websockets</h4><p>When checking the network requests I found that the game wasn’t using regular HTTP requests but rather sending all the info through <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API">websockets</a>. I had experience simulating HTTP requests but wasn’t sure how to go about simulating websocket frames.</p><p>First, I tried simulating a click on the screen using plain old javascript. After taking the code from <a href="https://stackoverflow.com/questions/2705583/how-to-simulate-a-click-with-javascript">this</a> stackoverflow question, I was able to get the click event to dispatch but for some reason it wasn’t increasing the counter. Puzzled, I turned on the JS debugger just before firing the click event. Somewhere in the jumble of minified JS the script was checking if the click event was legit using the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Event/isTrusted">isTrusted property</a>. Essentially, isTrusted is a security feature in browsers to see if the click was generated by the user or javascript. I didn’t find any simple way around it.</p><p>When digging through the JavaScript I got the idea to bypass the event handler and execute the code directly that was sending the message to the websocket. Between the minified code and my lack of familiarity with Websockets I couldn’t manage find the exact code that was sending the frame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d20eAOPJFIdCtEmF" /><figcaption>Example frames in Google Chrome</figcaption></figure><p>Next, I tried to find the Websocket object in memory to send frames myself using that. I had no luck finding the object in memory. What if I just make a new connection?</p><p>From what I could see, there were really only two kinds of frames being sent: one for authentication and a frame for each click event.</p><p>Authentication frame*:</p><pre>420[“authenticate user”,{“username”:”raybb”,”token”:”ff45fb994e3124a5e0ce7bf7f27745dcbe62e71275532f982498b798&quot;,”squad”:””,”locale”:”en-gb”,”count”:62301,”csrf”:”0xsn5yq9SAb65OvTXDl3OjzRoVPgr1zVvDtzoVR4ZhjdptbHClRSVaSJDXKwXy8b”}]</pre><p>Examples of the frames (1st, 2nd, and 10th)*:</p><pre>421[“tap_logs”,{“username”:”raybb”,”timestamp”:1540163213460,”touchX”:19,”touchY”:78}]</pre><pre>422[“tap_logs”,{“username”:”raybb”,”timestamp”:1540163297196,”touchX”:389,”touchY”:737}]</pre><pre>4200[“tap_logs”,{“username”:”raybb”,”timestamp”:1540163347125,”touchX”:306,”touchY”:442}]</pre><p>*these frames have modified values for privacy reasons.</p><p>From experimenting, I found:</p><ul><li>token and csrf: didn’t change between refreshes</li><li>count: the number of taps I had when page loaded (changing it had no impact)</li><li>The auth frame always started with 420 (which we shall call the frame counter)</li><li>The frame counter <em>mostly</em> increased by 1</li><li>The leftmost digits of the frame counter never reached 43. For example: [420, 421, 422, ..., 429, 4200, 4201, ..., 4299, 42000, ...]</li></ul><p>With this in mind, I made my first attempt to simulate a click with a websocket:</p><pre>const socket = new WebSocket(&#39;wss://socket-unlock.oneplus.com/socket.io/?EIO=3&amp;transport=websocket&#39;);</pre><pre>socket.addEventListener(&#39;open&#39;, function(event {<br>   console.log(&quot;opened&quot;);<br>   socket.send(authFrame);<br>   socket.send(firstTap);<br>  });</pre><p>It would connect successfully but the first tap wouldn’t send 🤔</p><p>It turns out that the server only accepts taps after it has sent a response confirming authentication. The confirmation frame always starts with 430 and contains your username. So all I had to do was wait for that frame to come back and then send my taps.</p><pre>const socket = new WebSocket(&#39;wss://socket-unlock.oneplus.com/socket.io/?EIO=3&amp;transport=websocket&#39;);</pre><pre>socket.addEventListener(&#39;open&#39;, function(event) {<br> console.log(&quot;opened&quot;);<br> socket.send(authFrame);<br>});</pre><pre>socket.addEventListener(&#39;message&#39;, function(event) {<br> if (event.data.indexOf(&quot;raybb&quot;) &gt; -1) {<br>  if (event.data.indexOf(&quot;430&quot;) &gt; -1) {<br>   socket.send(firstTap);<br>  }<br> }<br>});</pre><p>After figuring that out, it was pretty easy to write the rest of the code. You can see the full script <a href="https://gist.github.com/RayBB/2aef69a5f6d33e68fd4bf6e1627430c7">here</a>.</p><p>The only issue with this method was that sending taps would eventually slow down. After sending a few thousand taps the script would get noticeably slower, and stay at that speed. I suspect it was due to the recursive nature of the script, which makes many layers of scoped variables to build. I changed the code to send 100 taps in each level of recursion. That seemed to help a bit but the slowing was still noticeable. Then I blocked all scripts from loading on the OnePlus website (so it was essentially just an empty page). That also seemed to help a bit.</p><p>I didn’t end up finding a solution to the slowing script. The contest was ending before too long. It wasn’t much of an issue and before I knew it I was on my way to 1 million taps. I may not have won any prizes from this but I did learn about click events, Websockets, xdotool, and a little more about poking through sites with the debugger.</p><p>Thanks for reading 🤓👏</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*lmVwoGsyIXIlQHjjsuy8QA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce1d2cb3f459" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to unsubscribe from all the pesky GitHub emails when added to a new organization]]></title>
            <link>https://medium.com/@RayBB/how-to-unsubscribe-from-all-the-pesky-github-emails-when-added-to-a-new-organization-1e92cf5e27e9?source=rss-7ddb752ac03a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e92cf5e27e9</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[email]]></category>
            <category><![CDATA[life-hacking]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Ray Berger]]></dc:creator>
            <pubDate>Tue, 23 Oct 2018 02:42:01 GMT</pubDate>
            <atom:updated>2018-10-23T02:42:01.353Z</atom:updated>
            <content:encoded><![CDATA[<p>On the first day of my internship at BuzzFeed last summer I was added to their GitHub organization. Along with read access to several repos, I was also automatically subscribed to watch each repo.</p><p>In the first few hours I received hundreds of email notifications for PRs, comments, issues, and more. When I got home to find all these emails a quick Google Search showed that going to <a href="https://github.com/watching">https://github.com/watching</a> would allow you to see all the repos you were watching and unwatch them.</p><p>Many of my colleagues clicked the handy “Unwatch all” button on the top of the page. That wouldn’t work for me since I was watching several repos that I wanted the notifications from.</p><p>As such, I whipped up <a href="https://gist.github.com/RayBB/27086092a16b69f56bdcf081f0c373cc">this</a> handy little script to unwatch all the repos containing the organization name. All you have to do is paste it in the browser console and watch it do it’s magic! Fast forward to today, as I start working at the New York Times, and I found this script to be handy again. As such, I decided share it on here!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*K7M8XZfNRW-df_C7.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e92cf5e27e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web Bluetooth as a force for good with hackNY and Simprints]]></title>
            <link>https://medium.com/hackny/bluetooth-in-the-browser-a-magic-trick-by-tech-for-good-with-hackny-44c139196835?source=rss-7ddb752ac03a------2</link>
            <guid isPermaLink="false">https://medium.com/p/44c139196835</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[social-enterprise]]></category>
            <category><![CDATA[bluetooth]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[social-good]]></category>
            <dc:creator><![CDATA[Ray Berger]]></dc:creator>
            <pubDate>Tue, 02 Oct 2018 16:47:27 GMT</pubDate>
            <atom:updated>2018-10-21T01:48:53.286Z</atom:updated>
            <content:encoded><![CDATA[<p>By Raymond Berger and Clifford Lezark</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ewSK8l9zNLaCrA_Bo059lw.jpeg" /><figcaption>Team presenting at demo night</figcaption></figure><p>This summer <a href="http://hackny.org/">hackNY</a> expanded its commitment to the Summer of Good initiative by organizing several social good projects for fellows to contribute to outside of their internships. The goal of this initiative is to support nonprofit companies who have technologically-ambitious missions but lack the capacity to achieve them internally. It’s a perfect match between the Fellows and non-profits, since college interns are some of the only programmers on Earth willing to donate their late nights and weekends for months and these organizations are the only institutions willing to put college student volunteers in charge of entire mission-critical projects. While most of the organizations hackNY paired with were local to NYC, our team worked with<a href="http://simprints.com/"> Simprints</a>, a startup based in Cambridge, UK. This is largely because Tristram, the CTO of Simprints, graduated from Eckerd College just a few years before me and I met him through a professor.</p><p>Simprints is a social enterprise that works with NGOs (non-governmental organizations) to provide identification services for recipients of aid. The problem these NGOs often face is that the communities they work with lack access to formal identification. This is due to a number of reasons including physical seclusion, inadequate government infrastructure, and political disenfranchisement. Over one billion people worldwide are ineligible to receive food, medicine, financial support, and other forms of aid because they can’t provide proof of their identity the way we can. Having grown up without many resources, the recipients often have no formal identification and only a rough idea of when they were born. To overcome this issue, Simprints has engineered a durable and low-cost fingerprint scanner alongside a simple, secure, open-source identification solution that can be integrated into existing digital services. Together, these two innovations offer an alternative to formal identification that can be deployed and managed locally.</p><p><a href="https://www.linkedin.com/in/tristramnorman/">Tristram Norman</a>, co-founder and CTO, was our main point of contact with Simprints. <a href="https://www.linkedin.com/in/samadarshad/">Samad Arshad</a>, a firmware developer at Simprint was also a huge help. Our team on the hackNY side consisted of <a href="https://www.linkedin.com/in/amberrawson/">Amber Rawson</a>, <a href="https://www.linkedin.com/in/bentyang/">Ben Yang</a>, <a href="https://www.linkedin.com/in/clifford-lezark-702b17ba/">Clifford Lezark</a>, and <a href="https://www.linkedin.com/in/rlberger/">Raymond Berger</a>.</p><h4>Problem</h4><p>At the time of writing, Simprints fingerprint scanners can only pair with Android phones. This has worked well for some organizations but other NGOs don’t have workflows that are compatible with android phones and, for various reasons, need to use a traditional computer instead. Our task was to come up with a solution for Simprints that would allow their scanner to connect to Bluetooth-enabled computers without the use of device-specific drivers.</p><p><strong>Our Solution</strong></p><p>We built a solution using Web Bluetooth, a wildly-underused API introduced by Google in 2015 that leveraged the modern browser’s flexibility to communicate with bluetooth devices. Using<a href="https://webbluetoothcg.github.io/web-bluetooth/"> Web Bluetooth</a> would be much easier than writing bluetooth drivers and having to support them across multiple operations systems and hardware configurations.</p><p>We quickly realized that this would be possible, but the application would only work on macOS and Chrome OS. Windows and Linux implementations of Web Bluetooth were much less developed and the API hadn’t been adopted by any other browsers. In theory we just needed to figure out the Web Bluetooth API and interface with the hardware, which had already been mailed to us. We knew that Web Bluetooth only supported BLE (Bluetooth Low Energy), and we knew the scanner had the hardware to support BLE but we just couldn’t make them connect. After a couple of hours spinning our wheels and talking with Tristram, we realized there was no BLE firmware developed for the scanner. We had to make our own.</p><p><strong>Firmware</strong></p><p>The challenge came from the thousand cuts we endured porting firmware written for a fifteen-year old standard to an eight-year old standard. Embedded programming can be complicated in part because it wrestles with hardware constraints. BLE adds more complication because it enforces further constraints in the interest of battery life and security. Vero’s firmware wasn’t built with those constraints in mind, so many aspects of it’s functionality had to be rewritten from the ground up. Rewriting the code for Vero wouldn’t have been too difficult if we had access to the source code for its proprietary Bluetooth stack at the outset of the project. A bluetooth stack, which in this case was graciously implemented for and licensed to Simprints free of charge, defines how firmware is written for a given device using a given protocol. However, no infrastructure for a BLE implementation of Vero’s firmware was provided so we needed to build it ourselves — boilerplate templates and all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/0*dBSwhjeIA4-quKY0" /><figcaption>The Simprints scanner — Vero</figcaption></figure><p>A month into the project we received an example of a BLE-ready project using the same Bluetooth stack. It allowed us to build a firmware with read/write functionality to ensure that the device could at least connect to our application, and enabled us to prototype the full firmware, but more problems developed. The stack’s compiler wouldn’t compile our code or tell us what problems it was facing, so we rebuilt the compiler. The compiled code needed to be built into firmware using equally confusing tools, so we had to learn how to use those tools. The amount of compatibility testing necessary before we could even load our prototype firmware onto the scanner threatened to take weeks to complete. Though we made a lot of progress, finishing the firmware would still take more time than we had. At the outset, we understood that embedded programming could be complicated and were prepared to work within the bounds of the hardware and software. What we weren’t prepared for was the complications imposed by the nature of the projects origin.</p><p>A redeeming takeaway is that all of our struggles were for a good reason — maintaining security for Simprints’ clients and the people they help. A defining feature of the Web Bluetooth API is the amount of attention afforded to security measures. With IoT security being such a pressing issue, the architects of Web Bluetooth chose to take precautions that are inherently difficult to work with but completely worthwhile. With sensitive biometric data passing over the air, ensuring its safe and secure movement across devices is worth the challenges.</p><p><strong>Chrome Extension</strong></p><p>While work with firmware development was ongoing we also worked on building a Google Chrome extension for Simprints. We decided that a chrome extension would be the best solution for a few reasons.</p><ol><li>It would be easy to install and automatically update.</li><li>It is always visible in the browser, so it’s easy to access.</li><li>We could potentially inject the unique identifier into a clients website if needed.</li></ol><p>From a technical standpoint the extension is rather simple. Essentially all we did was copy the Android app workflow and add a pretty coat of paint on top. To do this we used Vue.js and Bootstrap. Why Vue and Bootstrap? For a relatively small application they did the job well and are well documented. At this point, the extension doesn’t actually connect to the scanner due to the lack of firmware. However, it is a great way to show clients a possible new way to interface with the scanner. If they show enough interest, further resources will be put into developing the firmware and extension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/0*0Z0tEQCQsPBHYZr-" /><figcaption>Chrome extension demo</figcaption></figure><p><strong>Thoughts on Volunteering</strong></p><p>Generally speaking, non-profit organizations don’t come to mind often when most programmers think of innovative, life-changing careers. But as we discovered from working with SimPrints, some of the neatest work we had ever heard of is being done in this space. Non-profits are using Bluetooth in the browser to hook up biometric scanners in Bangladesh, creating tools that enable journalists to harness big data for their reporting, retooling government services to improve their workflow, and so much more. The opportunities available today in the non-profit space for tech workers are just as cutting-edge as anything being done in the private sector, and they also offer the chance to reap good karma and a good paycheck.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44c139196835" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackny/bluetooth-in-the-browser-a-magic-trick-by-tech-for-good-with-hackny-44c139196835">Web Bluetooth as a force for good with hackNY and Simprints</a> was originally published in <a href="https://medium.com/hackny">hackNY</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>