<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alex Scammon on Medium]]></title>
        <description><![CDATA[Stories by Alex Scammon on Medium]]></description>
        <link>https://medium.com/@alexscammon?source=rss-22814b3fd0ff------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*GTUUbHAJpiEdBFVtIPic_Q.png</url>
            <title>Stories by Alex Scammon on Medium</title>
            <link>https://medium.com/@alexscammon?source=rss-22814b3fd0ff------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:33:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alexscammon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[G-Research Creator Talk: Wes McKinney — Apache Parquet & Arrow]]></title>
            <link>https://alexscammon.medium.com/g-research-creator-talk-wes-mckinney-apache-parquet-arrow-cf0d5b0ada16?source=rss-22814b3fd0ff------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf0d5b0ada16</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[apache-parquet]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[apache-arrow]]></category>
            <dc:creator><![CDATA[Alex Scammon]]></dc:creator>
            <pubDate>Mon, 03 Oct 2022 20:18:19 GMT</pubDate>
            <atom:updated>2022-10-03T20:18:19.043Z</atom:updated>
            <content:encoded><![CDATA[<h3>G-Research Creator Talk: Wes McKinney — Apache Parquet &amp; Arrow</h3><p>When you’re known for a hit, the fans always want to hear you play it. It’s been some 57 years since The Stones released “<a href="https://en.wikipedia.org/wiki/(I_Can&#39;t_Get_No)_Satisfaction">Satisfaction</a>,” and not even the death of its drummer or a global health crisis could keep the band from touring and playing it to sold-out stadiums packed with people who demand to hear the song for themselves.</p><p>When Wes McKinney came to G-Research Headquarters, everyone expected to hear him talk about <a href="https://pandas.pydata.org/">Pandas</a>, since it’s what he’s widely known for. Amongst our data-minded crew, the open-source library of data analysis tools for Python he created is roughly equivalent to the opening chords of Keith Richards guitar. It accelerates our pulses, and we want to hear him expound on that. That’s the hit we know and love.</p><p>Pandas is what he is known for now, and it’s definitely a<em> Beast of Burden</em> for G-Research that <a href="https://www.gresearch.co.uk/article/faster-string-processing-in-pandas/">we spend a lot of time with</a>, but McKinney has recently developed something that may come to play an even larger role in the daily lives of our team. Something potentially more powerful and more useful to what we do with big data, especially as we do more and more with columnar data from different and disparate data environments.</p><p>McKinney talked to us about his work spearheading the <a href="https://parquet.apache.org/">Apache Parquet</a>, and now the <a href="https://arrow.apache.org/">Apache Arrow</a> project, which aims to make it easy for data-heavy engineers to work with columnar data in memory.</p><p>He told us that Parquet was initially created to overcome networking limitations. The pipes weren’t wide enough to stuff such big data files down, he explained. But as file sizes have exploded, e.g. big data got even bigger, the CPUs started to become the problem. McKinney imagines that we may soon see a rise in specialized silicon explicitly fabricated to deal with big data application processing.</p><p>Apache Arrow makes it easy to move data across different environments without having to reformat it. McKinney talked about taking data from a variety of databases and data formats— <a href="https://spark.apache.org/">Spark</a>, <a href="https://drill.apache.org/">Drill</a>, <a href="https://impala.apache.org/">Impala</a>, <a href="https://hbase.apache.org/">HBase</a>, <a href="https://kudu.apache.org/">Kudu</a>, <a href="https://cassandra.apache.org/_/index.html">Cassandra</a> and Parquet — and instead of copying and converting it each time you needed to move it, transferring it in the Arrow data format. It would certainly seem to cut down on the number of read-write operations, thereby upping both processing speed and improving data efficiency.</p><p>One of the greatest advantages of Arrow, McKinney points out, is that it allows for chunking, so that big data can be broken into component parts and processed in smaller batches. That allows for greater flexibility for application development.</p><p>The Rolling Stones released “Satisfaction” in the summer of 1965, and it was an immediate hit, holding the number one spot on the charts for four weeks in August and remaining a perennial favorite. But if the Rolling Stones had stopped there, we never would have gotten “<a href="https://en.wikipedia.org/wiki/Gimme_Shelter">Gimme Shelter</a>” three years later, which is, if not musically superior, then at least its equivalent.</p><p>Wes McKinney may be known for Pandas, it was a big hit for him, after all. But like The Rolling Stones, we may be now witnessing an even bigger development in the future of data storage and processing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf0d5b0ada16" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We, the Merry Maintainers]]></title>
            <link>https://medium.com/aerospike-developer-blog/we-the-merry-maintainers-baaf7cc58226?source=rss-22814b3fd0ff------2</link>
            <guid isPermaLink="false">https://medium.com/p/baaf7cc58226</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[philanthropy]]></category>
            <category><![CDATA[hashicorp-vault]]></category>
            <category><![CDATA[hashicorp]]></category>
            <category><![CDATA[aerospike]]></category>
            <dc:creator><![CDATA[Alex Scammon]]></dc:creator>
            <pubDate>Mon, 23 Aug 2021 18:24:36 GMT</pubDate>
            <atom:updated>2021-08-26T19:50:05.350Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*IAcy3idu4iU9KLeBnhni4w.jpeg" /></figure><h3>We, the Merry Maintainers — Gifting the Aerospike Vault Plugin</h3><h3>Gifting the Aerospike Vault Plugin</h3><p>Being a modern, responsible, software-defined citizen can mean making the lives of those around you easier and better. It can mean making subtle improvements that make things run more smoothly, consume fewer compute resources or take less time. Improvements can be subtle or profound, but they move the ball forward for anyone using what you’ve created. It’s good to make things better.</p><p>Sometimes, though, those most resistant to change systems and processes are the same people who created the systems themselves. Nowhere is that more evident than in the world of software development and design. Developers can be haunted by the ghosts of their efforts, desperate to hold fast to what they once deemed to be a critical feature, or necessary function — blind to how their software is actually used in the wild, by actual users. They can grasp tight to how they initially imagine “users” might interact with their product or service, despite all evidence to the contrary.</p><p>Open source development helps make transparent design choices and, though it’s by no means a perfect way of building software, it shines light into the minds of software-makers. The easiest way to create an open source project is to start it right from the beginning in the open, as I discussed in <a href="https://www.gresearch.co.uk/article/armada-the-decision-to-open-source/">my previous writing</a>. And if the users don’t end up using it in the way the developers intended, well, they can go right into the code and tinker with the functionality themselves until they get it just the way they want.</p><p>It may seem natural that we ‘give away’ projects all the time in the course of the open source work we do at <a href="https://www.gresearch.co.uk/">G-Research</a>. In many senses, open source work is all about giving code and ideas and time and energy to the common good. However, code can feel very personal, just like all other human creations. When it actually comes time to give code away, it can get very emotional and it isn’t always easy for maintainers to let go.</p><h3>Plate-spinning maintainers</h3><p>It’s easy to imagine the work of a maintainer as stolid, stationary — boring, even. Not so! Better to think of the work of a maintainer as being most similar to that of a professional <a href="https://en.wikipedia.org/wiki/Plate_spinning">plate spinner</a>, using the momentum and gyroscopic effect of software development to carry it forward into whatever new demands the future holds. It requires a deep knowledge, careful balance and a gentle touch.</p><p>Few do it better than <a href="https://github.com/jgiannuzzi">Jonathan Giannuzzi</a> on our open source team, who managed to carry the <a href="https://github.com/aerospike-community/vault-plugin-database-aerospike">Aerospike-Vault plugin</a> beyond what would suit the needs of our organization alone, into something the wider community could find value in. It didn’t make sense merely to create a Vault plugin for Aerospike in-house and keep it all to ourselves: we knew that members of the Aerospike community would want this eventually. Heeding the call, Jonathan took up the charge, built the plugin, wrote tests, set up some CI, wrote the documentation, verified that everything worked in our production environment, and then dutifully rolled out the repository into public view. He kept the plates spinning too — giving his teams what they needed — while building something for everyone else.</p><p>As Jonathan did all of this, we couldn’t help feeling that the plugin needed a better venue. Why, we asked ourselves, would anyone come looking for an Aerospike plugin in the Github org of a quantitative research firm? Isn’t this plugin so generically useful to enough people that it should have a more prominent place in the Aerospike or Vault community pages?</p><p>We did all the basic marketing: adding ourselves to the community-contributions pages for Aerospike and Vault. This was a good start and we even had some early positive feedback from our efforts. It’s not exactly wrong for a third party company to author and maintain useful glue like this — it happens all the time, and just as often from individual contributors.</p><p>But it still didn’t feel right to us.</p><p>Fortunately, our internal teams had already established a good relationship with the people at Aerospike, so we took the opportunity to reach out. When we did, we discovered that they knew about the plugin and were hoping to piggy-back on the groundwork that Jonathan had laid down. We floated the idea that we move the repository into their organization so that it would get more visibility and to nobody’s surprise but ours, they thought it was a great idea.</p><p>Furthermore, they expressed an idea that we were thinking ourselves: this plugin should eventually live in a Hashicorp organization repository itself. We couldn’t agree more. But we all also agreed that it might take a few steps to get there and that moving the plugin to the Aerospike community was a positive step towards that end goal.</p><p>The opportunity was just too much to ignore, so we moved the git project from the G-Research repository to the Aerospike community repository:</p><ul><li><a href="https://github.com/aerospike-community/vault-plugin-database-aerospike">https://github.com/aerospike-community/vault-plugin-database-aerospike</a></li></ul><p>A few short months later, we’re even more pleased to see that this work is now recognized as an official part of the Vault project itself:</p><ul><li><a href="https://www.vaultproject.io/docs/configuration/storage/aerospike">https://www.vaultproject.io/docs/configuration/storage/aerospike</a></li></ul><p>While this movement of code from one location to another may seem trivial, the real effort was for us to overcome our emotions around “letting go” of “our” project. Of course, we recognized this was an opportunity for our work to prosper and grow into something more than we could achieve by ourselves, but moving the project over felt like handing the keys to your red Porsche 911 to your teenager. They’ll wreck it! They’ll destroy the transmission! The tires! Ack!</p><p>It was important for us to remember that this software isn’t static, or carved in stone, or even “ours” once it’s in the world as an open source project. It’s a living thing that grows with its community and it needs increased collaboration and contribution to strengthen its roots and flourish. Opening it up to outsiders actually gave the project the space it needed to breathe and brought others in by virtue of its utility. And when others used it, they were able to see opportunities for improvement that we never would have. With this plugin, we had enough rationalizations, realizations and observations of this truth that we were able to overcome our emotional attachment to the project.</p><p>Doing open source isn’t just simple philanthropy. Who we are and what we think of ourselves is all wrapped up in what we create and it can be a real struggle to let go of our creations. But it is very much worth it. We are really pleased that the Aerospike-Vault plugin has moved on beyond us, and we’re thrilled that the Aerospike team can derive immediate benefit from it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=baaf7cc58226" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aerospike-developer-blog/we-the-merry-maintainers-baaf7cc58226">We, the Merry Maintainers</a> was originally published in <a href="https://medium.com/aerospike-developer-blog">Aerospike Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Armada: our Decision to Open-Source]]></title>
            <link>https://alexscammon.medium.com/armada-our-decision-to-open-source-14488e318d35?source=rss-22814b3fd0ff------2</link>
            <guid isPermaLink="false">https://medium.com/p/14488e318d35</guid>
            <category><![CDATA[batch-processing]]></category>
            <category><![CDATA[multi-cluster]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[hpc]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Alex Scammon]]></dc:creator>
            <pubDate>Mon, 10 May 2021 15:06:50 GMT</pubDate>
            <atom:updated>2021-05-13T21:51:17.380Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xdwo6sDcCOQ1d4utwa_wkA.jpeg" /></figure><p>We made the decision to open-source the <a href="https://armadaproject.io/">Armada Project</a> before we wrote a single line of code.</p><p>This was a completely new choice for G-Research. We have made a few projects public and open-source over the years (<a href="https://github.com/G-Research/ParquetSharp">ParquetSharp</a>, and <a href="https://github.com/G-Research/geras">geras</a>, for example), but we had always started the project internally and moved it into the open when we felt it was ready. Yet others we inherited when their original maintainers no longer had the time or passion to contribute to their original project (C<a href="https://github.com/G-Research/consuldotnet">onsul.NET</a>, <a href="https://github.com/jupyter-incubator/sparkmagic">SparkMagic</a>).</p><p>We had never, however, started a project as open-source from its inception. With Armada, we made a conscious decision to start completely open from the outset. I’d like to share our rationale for this decision and some of our learnings from along the way.</p><h3>Two Approaches to an Open-Source Project</h3><p>The main alternative to starting a project in the open is to work on it internally, bring it as close to perfection as possible, and open-source it later. There are pros and cons to each approach.</p><h3>Open-Sourcing Later</h3><h4>Pros</h4><ul><li>You get to confirm that your project works in production before trying to sell the public on it — everyone wants to be part of something that works and is good.</li><li>You don’t have to share all your early mistakes and missteps with the world — everyone wants to look good, seemingly right from the start.</li></ul><h4>Cons</h4><ul><li>You receive input from the community later in the process, possibly leading to a project that doesn’t apply to the needs of a more general population — nobody wants what you’ve built because it only solves your specific problem.</li><li>You may inadvertently tie your project to some internal process or architecture, thereby requiring more work to release the project publicly — nobody wants what you’ve built because it only solves a problem internal to your systems.</li><li>You may never get around to it. So many things, so little time — nobody wants what you’ve built because it doesn’t scratch the one specific itch that they have.</li></ul><h3>Open-Source from the Start</h3><h4>Pros</h4><ul><li>By sharing the work publicly, you are encouraged to follow best practices from the start — everyone likes to be seen doing the right thing.</li><li>It allows others to join your project, potentially providing valuable input into the design of the project or actual code contributions that accelerate the project’s progress — everyone likes to contribute in a meaningful way.</li><li>The design of the project is (hopefully) free of the constraints or dependencies of your company’s internal development process or infrastructure from the get-go — everyone who needs to be involved in the decision making process is.</li></ul><h4>Cons</h4><ul><li>Sharing the project too early may actually turn some potential collaborators off from the project — nobody wants to get behind a half-baked project.</li><li>G-Research is particularly sensitive to security concerns, so there is extra development time involved in developing a project publicly. We have an elaborate system of levers and pulleys to synchronize code from inside to outside and vice versa. The system is brittle and the sync can be broken if you make a simple mistake. Your own company may not have quite as onerous a process, but there is often some additional amount of operational overhead associated with contributing to open-source projects — nobody wants to over-share things that they shouldn’t.</li></ul><p>There isn’t exactly a right or wrong choice here. In some situations, we continue to prefer the “Open-Source Later” approach, particularly in situations where we feel that it is valuable to verify that our approach works in our production environment first. For example, we recently released and moved the <a href="https://github.com/aerospike-community/vault-plugin-database-aerospike">Aerospike plugin for Vault</a> to the Aerospike-community repository, but only after we felt confident that it was indeed working correctly for our own internal users.</p><h3>The Rationale for Armada</h3><p>The rationale for starting Armada in the open was based in the aims of the project itself.</p><p>We thought long and hard about creating yet another batch processing tool. We worried that it was an increasingly crowded space, and that someone else would eventually have the same requirements that we had. Namely, we needed something to coordinate the fair-use scheduling of batch-processing jobs across Kubernetes clusters.</p><p>For us, the number of machines that we need to run our quants’ insatiable desire for processing exceeds the number of machines that is recommended in a single Kubernetes cluster — perhaps by an order of magnitude. The official guidance on large clusters is 5,000 nodes per cluster (<a href="https://kubernetes.io/docs/setup/best-practices/cluster-large/">https://kubernetes.io/docs/setup/best-practices/cluster-large/</a>), although our experience suggested a lower number was a more practical threshold. We could push the limit if we tweaked a lot of little settings, but it seemed like pushing the envelope in this dimension would ultimately lead to unexpected issues.</p><p>Another consideration was that we wanted the ability to perform cluster-wide upgrades without taking down the entire cluster. This, again, suggested multiple clusters running side-by-side which would allow us to selectively take a cluster offline for maintenance without affecting the overall running of the business.</p><p>While our rationale was directing us towards a multi-cluster setup, we also wanted to keep the interface to the system simple. We didn’t want to burden our users by requiring an intimate knowledge of the infrastructure their jobs will run on. Armada sits above all of our clusters and distributes batch-processing jobs to those clusters intelligently. Users submit their job to Armada and the complexity of where the job actually runs is abstracted away. Our hope is that this makes our users’ lives easier by providing a simple way to submit a batch job to the massive amount of infrastructure required to run all of G-Research’s jobs.</p><h3>Open From the Start</h3><p>With all this in mind, we still could have developed the code in-house first and then open-sourced it later. Our decision to start as an open-source project from the beginning was really driven by three points:</p><ol><li>We hoped a coalition of willing contributors would be able to help design and build a better tool than if we built it by ourselves. We knew other organizations suffered from the same problems we did and were working on their own ways of addressing the same concerns.</li><li>Often when a useful tool is built and kept in-house, someone in the open-source world will eventually have the same brilliant idea. As that open-source tool starts to eclipse our internal efforts, we are forced to decide between a migration effort to the open-source option or valiantly maintain our private tool forever. If, instead, we put our initial efforts towards an open-source tool, we set up the possibility of not having to pay that migration cost.</li><li>Given the core position that this technology would take in our overall infrastructure, the temptation to build something bespoke would be too great if we started in-house. To force ourselves to generalize the architecture, we needed to start in the open to avoid falling into this trap.</li></ol><h3>Collaboration from the Get-Go</h3><p>We benefit by having more input into the product. We believe that a diversity of ideas, viewpoints and methodologies is one of the strengths of open-source software and will make our own software stronger if we adhere to the same principles.</p><p>Before the project even started, we spent a lot of time talking to our peers and confirming that this could be a useful project. We reached out to people who had built other versions of batch-scheduling capabilities into Kubernetes, including <a href="https://github.com/kubernetes-sigs/kube-batch">k8s-batch</a> and <a href="https://github.com/volcano-sh/volcano">Volcano</a>. We eventually all sat down and collaborated on a design document outlining the overarching goals for all of us to bring batch-processing to Kubernetes natively:</p><ul><li><a href="https://docs.google.com/document/d/1jgXiLNZUh7Voz-MHeggrC227tQogEVA4wbtlPIKzZPI/edit#heading=h.4p9vyufcbgne">Bring Batch Capability into Kubernetes Natively</a></li></ul><p>Among the group were representatives from research institutions such as CERN and the University of Michigan; corporate representatives from Google, Amazon, Huawei, VMWare and JD.com; and various CNCF core committers and some Apache PMC’s. We also presented our work to the CNCF End-User group to see if our issues were relevant for anyone in that audience.</p><p>By soliciting all of this feedback, we grew increasingly assured that what we were trying to do — batch processing across multiple Kubernetes clusters — was a concern for other parties. Others shared our need, and this gave us the encouragement to try and fill it.</p><p>If others suffered the same problems, there might be folks willing to help build the solution with us. From our initial conversations with this group, there was clearly a willingness to give input and advice and encouragement. All of this helped convince us that we should undertake the project and initiate it in the open.</p><h3>Avoid Building and Switching</h3><p>We hoped that we could avoid building a proprietary system in-house and then migrating to an open-source alternative later. We thought that if we managed to build something that was useful to us and others, it had a chance of being the thing we would use for years to come.</p><p>From another angle, if our project was truly useful, it would almost certainly be developed at some point in time in the open-source world. Because the Kubernetes ecosystem develops so fast, if the open-source alternative caught on it would quickly eclipse whatever we had built internally. Eventually, we would be forced to decide between maintaining an in-house tool with fewer features than the open-source alternative or migrate away from what we had built. That was a decision we didn’t want to have to make.</p><p>These are still early days for the <a href="https://armadaproject.io/">Armada Project</a>. We hope our solution will catch on. We encourage this to happen by ensuring that others are engaged with our production process and understand the key design decisions at every step by working out in the open — in full view of a community of collaborators and contributors.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14488e318d35" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>