<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by rmoff on Medium]]></title>
        <description><![CDATA[Stories by rmoff on Medium]]></description>
        <link>https://medium.com/@rmoff?source=rss-e6a1d2163309------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xfIBj3LCTOsLyWz6ahkdtw.jpeg</url>
            <title>Stories by rmoff on Medium</title>
            <link>https://medium.com/@rmoff?source=rss-e6a1d2163309------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:56:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rmoff/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Try To Keep Up With The Data Tech World (A List of Data Blogs)]]></title>
            <link>https://medium.com/@rmoff/how-i-try-to-keep-up-with-the-data-tech-world-a-list-of-data-blogs-e16e11645ba8?source=rss-e6a1d2163309------2</link>
            <guid isPermaLink="false">https://medium.com/p/e16e11645ba8</guid>
            <category><![CDATA[rss]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[rmoff]]></dc:creator>
            <pubDate>Wed, 22 May 2024 13:19:49 GMT</pubDate>
            <atom:updated>2024-05-22T15:16:44.442Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fvbb7G0BUZS0ikbvbf8FEw.jpeg" /><figcaption>Nothing to do with data, but a nice photo of the Boston skyline at sunrise 🌅</figcaption></figure><p>I do my best to try and keep, if not abreast of, then at least aware of what’s going on in the world of data. That includes RDBMS, Event streaming, stream processing, open source data projects, data engineering, object storage, and more. If you’re interested in the same, then you might find this blog useful, because I’m sharing my sources :)</p><h3>Sidenote: allow me some nostalgia</h3><p>In the <a href="https://rnm1978.wordpress.com/2009/03/"><em>really</em> old days</a> not that many people blogged. It was kinda weird. There’d be a handful of folk in your circle of interest who also blogged, and you’d maybe have a <em>Blogroll</em> linking out to their blogs and/or feeds.</p><p>In the old days, a bit after that, more people started blogging, and one had to be organised about following what was written. The fantastic technology that is RSS came about. I used Google Reader (RIP) and followed a ton of blogs. Then Google Reader was infamously killed off, and I switched to Feedly. Then I…dunno. I moved jobs, I didn’t have time, I made excuses, whatever. I stopped following blogs so closely, and instead got my current tech news from Twitter and from colleagues. This was all about seven years ago, and I even <a href="https://rmoff.net/2017/03/11/keeping-up-with-the-deluge/">wrote about it</a>.</p><p>In that time, probably around the late 2010’s, RSS started to fade. &lt;RANT&gt; <em>This is to my eternal horror, and an illustration of the enshittification of the internet. Instead of having an easy way to share material openly using an standard and open format that people could consume with whatever platform or tool they wanted, a bunch of smart people wanting to make money off other people’s content decided it was far more sensible to put up walls (yes, the irony of x-posting this to Medium </em>😬)<em>. Instead of being able to pull a set of RSS feeds into one place, the reader would instead traipse around numerous platforms to read the material there. Eyeballs, amirite? That, plus a lively tech twitter scene and some clever algorithms meant that folk got their fill of tech news directly. It might have been from blogs, but often direct linked instead of from an RSS feed being followed.</em> &lt;/RANT&gt;</p><p>Fast-forward to today, and I’m back with a centrally managed RSS feed — and it’s working great 😃</p><h3>RSS is dead…long live RSS! (All Hail Inoreader)</h3><p>I’m using an excellent RSS reader called <a href="https://www.inoreader.com/">Inoreader</a>. I spent a bunch of time last year looking at options and settled on this one. It’s got some really nice features on top of simply collecting RSS feeds and organising them into folders including:</p><ul><li>Web scraping for blogs that don’t offer RSS (grrrr) — usually does a half decent job of telling you when there’s a new article to read</li><li>Newsletter subscriptions (get a bespoke email; newsletters sent to that address show up in your feed)</li><li>Share articles to lots of services plus custom integrations</li><li>Search across feeds</li><li>Feed filters and rules</li><li>Create custom RSS feeds from your folders</li><li>Load full content (even if the payload isn’t in the RSS feed itself)</li></ul><p>There’s a free plan — I pay for the pro one because it’s worth it and I like supporting good projects.</p><h3>So…where are these blogs then?</h3><p><strong>Here’s the OPML for </strong><a href="https://rmoff.net/code/rmoff-opml-2024-05-22.xml"><strong>all the blogs that I follow at the moment</strong></a>.</p><p>My RSS feeds are skewed towards my interest in the data industry — streaming, analytics, data engineering, RDBMS. And just to make it fun it’s got some general tech feeds (HackerNews, Lobste.rs), some podcasts, and some running blogs too. Cos, it’s mine and I curate it :-P</p><p>The feeds are split into folders of roughly logical grouping. I’ve shared each one below as individual OPML — but if you want to follow along dynamically as I add (or remove) feeds from these folders, you can use the RSS feed of each.</p><p>There is also a web page view (HTML) of each folder too.</p><h3>Aggregators</h3><p>A bunch of tags of interest on Medium, Lobste.rs, plus some InfoQ feeds</p><p><a href="https://www.inoreader.com/stream/user/1005111311/tag/Aggregators">RSS</a> | <a href="https://www.inoreader.com/stream/user/1005111311/tag/Aggregators/view/html?cs=m">HTML</a> | <a href="https://www.inoreader.com/reader/subscriptions/export/user/1005111311/label/Aggregators">OPML</a></p><h3>Companies using data</h3><p>Everyone who is building stuff with data and writing about it</p><p><a href="https://www.inoreader.com/stream/user/1005111311/tag/Companies%20using%20data">RSS</a> | <a href="https://www.inoreader.com/stream/user/1005111311/tag/Companies%20using%20data/view/html?cs=m">HTML</a> | <a href="https://www.inoreader.com/reader/subscriptions/export/user/1005111311/label/Companies%20using%20data">OPML</a></p><h3>Data projects &amp; vendors</h3><p>Everyone who is building products for doing stuff with data. Includes both vendors and non-profit projects.</p><p><a href="https://www.inoreader.com/stream/user/1005111311/tag/Data%20Projects%20and%20Vendors">RSS</a> | <a href="https://www.inoreader.com/stream/user/1005111311/tag/Data%20Projects%20and%20Vendors/view/html?cs=m">HTML</a> | <a href="https://www.inoreader.com/reader/subscriptions/export/user/1005111311/label/Data%20Projects%20and%20Vendors">OPML</a></p><h3>Software</h3><p>Handful of projects that I follow closely and want to see when there are new releases</p><p><a href="https://www.inoreader.com/stream/user/1005111311/tag/Software">RSS</a> | <a href="https://www.inoreader.com/stream/user/1005111311/tag/Software/view/html?cs=m">HTML</a> | <a href="https://www.inoreader.com/reader/subscriptions/export/user/1005111311/label/Software">OPML</a></p><h3>Individuals &amp; Consultancies</h3><p>What it says on the tin. Bit of a broader scope on this one, sometimes just interesting people in the broader field of tech.</p><p><a href="https://www.inoreader.com/stream/user/1005111311/tag/Individuals%20%26%20Consultancies">RSS</a> | <a href="https://www.inoreader.com/stream/user/1005111311/tag/Individuals%20%26%20Consultancies/view/html?cs=m">HTML</a> | <a href="https://www.inoreader.com/reader/subscriptions/export/user/1005111311/label/Individuals%20%26%20Consultancies">OPML</a></p><h3>Forums</h3><p>Apache Kafka and Apache Flink Reddit subs, plus interesting tags on StackOverflow</p><p><a href="https://www.inoreader.com/stream/user/1005111311/tag/Forums%20etc">RSS</a> | <a href="https://www.inoreader.com/stream/user/1005111311/tag/Forums%20etc/view/html?cs=m">HTML</a> | <a href="https://www.inoreader.com/reader/subscriptions/export/user/1005111311/label/Forums%20etc">OPML</a></p><h3>Marketing &amp; DevRel</h3><p><em>How DARE you put them in the same subheading?</em> Well, cos there’s <strong>lots</strong> of overlap.</p><p><a href="https://www.inoreader.com/stream/user/1005111311/tag/Marketing">RSS</a> | <a href="https://www.inoreader.com/stream/user/1005111311/tag/Marketing/view/html?cs=m">HTML</a> | <a href="https://www.inoreader.com/reader/subscriptions/export/user/1005111311/label/Marketing">OPML</a></p><h3>Podcasts</h3><p>A mishmash of podcasts, covering tech, UK politics, current affairs, history, and more. Many of these are dormant, but I keep the subscription in the hope that one they they’ll come back :)</p><p><a href="https://www.inoreader.com/stream/user/1005111311/tag/Podcasts">RSS</a> | <a href="https://www.inoreader.com/stream/user/1005111311/tag/Podcasts/view/html?cs=m">HTML</a> | <a href="https://www.inoreader.com/reader/subscriptions/export/user/1005111311/label/Podcasts">OPML</a></p><h3>Running</h3><p>Yep, no tech here. Move on, or join in 🏃‍♂️</p><p><a href="https://www.inoreader.com/stream/user/1005111311/tag/Running">RSS</a> | <a href="https://www.inoreader.com/stream/user/1005111311/tag/Running/view/html?cs=m">HTML</a> | <a href="https://www.inoreader.com/reader/subscriptions/export/user/1005111311/label/Running">OPML</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e16e11645ba8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Apache Flink: What is Flink?]]></title>
            <link>https://medium.com/@rmoff/learning-apache-flink-what-is-flink-2dc049bea3fe?source=rss-e6a1d2163309------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dc049bea3fe</guid>
            <category><![CDATA[apache-flink]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[flink]]></category>
            <category><![CDATA[stream-processing]]></category>
            <dc:creator><![CDATA[rmoff]]></dc:creator>
            <pubDate>Mon, 02 Oct 2023 15:13:11 GMT</pubDate>
            <atom:updated>2024-05-21T09:47:03.338Z</atom:updated>
            <content:encoded><![CDATA[<p>My <a href="https://rmoff.net/2023/09/29/learning-apache-flink-s01e01-where-do-i-start/">journey</a> with <a href="https://flink.apache.org/">Apache Flink</a> begins with an overview of <em>what Flink actually is</em>.</p><p>What better place to start than the <a href="https://nightlies.apache.org/flink/flink-docs-release-1.17/#apache-flink-documentation">Apache Flink website itself</a>:</p><blockquote><strong><em>Apache Flink</em></strong><em> is a framework and distributed processing engine for stateful computations over </em>unbounded<em> and </em>bounded<em> data streams. Flink has been designed to run in </em>all common cluster environments<em>, perform computations at </em>in-memory<em> speed and at </em>any scale<em>.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9djp-PfUkFcDzFiYOGypuA.jpeg" /></figure><p>In this post, I’m going to summarise my current-possibly naïve-understanding of:</p><ul><li><a href="https://rmoff.net/2023/10/02/learning-apache-flink-s01e02-what-is-flink/#so-what-is-flink">What Flink is</a></li><li><a href="https://rmoff.net/2023/10/02/learning-apache-flink-s01e02-what-is-flink/#use-cases">What Flink is used for</a></li><li><a href="https://rmoff.net/2023/10/02/learning-apache-flink-s01e02-what-is-flink/#users-of-flink">Who uses Flink</a></li><li><a href="https://rmoff.net/2023/10/02/learning-apache-flink-s01e02-what-is-flink/#how-do-you-run-flink">How do you run Flink</a></li><li><a href="https://rmoff.net/2023/10/02/learning-apache-flink-s01e02-what-is-flink/#the-flink-community">Where to find the Flink community</a></li></ul><h3>So: What is Flink?</h3><p>I found a couple of <a href="https://www.dataengineeringpodcast.com/apache-flink-with-fabian-hueske-episode-57">excellent</a> <a href="https://overcast.fm/+BAj84H3884">podcasts</a> from <a href="https://www.linkedin.com/in/fhueske/">Fabian Heuske</a> and my <a href="https://decodable.co/">Decodable</a> colleague <a href="https://www.linkedin.com/in/metzgerrobert/?originalSubdomain=de">Robert Metzger</a> respectively that gave some really useful background on the project:</p><ul><li>Flink started life as a research project in 2011, called <a href="https://scholar.google.com/citations?view_op=view_citation&amp;hl=en&amp;user=Q1LJyvQAAAAJ&amp;citation_for_view=Q1LJyvQAAAAJ:_FxGoFyzp5QC"><em>Stratosphere</em></a>.</li><li>It was donated to <a href="https://www.apache.org/">Apache Software Foundation</a> in 2014.</li><li>Version 1.0 released 2016, latest version is <a href="https://flink.apache.org/downloads/#apache-flink-1171">1.17</a> .</li><li>Whilst it was originally designed for batch, it always used streaming principles, making its move into stream processing a logical one</li><li>Hadoop revolutionised the distributed processing of data at scale, but was “dumb”. Flink aimed to use some of the principles whilst bringing in important learnings from the RDBMS world that had been missed in Hadoop. Flink includes a bunch of things that you’d have to build for yourself in Hadoop, such as pipelined execution (e.g. all stages run concurrently and stream data), native join operators, and it re-use of data properties such as the data being sorted or partitioned already in a certain way.</li><li>JVM-based. <a href="https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/overview/">SQL</a> and <a href="https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/overview/">PyFlink</a> added in recent years.</li><li>Flink is a Distributed system. <a href="https://overcast.fm/+H1YOnxO3I/05:50">Each</a> worker stores state.</li><li>It <a href="https://overcast.fm/+H1YOnxO3I/23:29">supports</a> exactly once state guarantee with checkpointing across workers that stores the processing state (such as aggregations), as well as the metadata of input sources (e.g. Kafka topics offsets) all on a distributed filesystem (e.g. S3)</li><li>Event time processing. <a href="https://overcast.fm/+H1YOnxO3I/21:42">Uses</a> watermarks (same as Google data flow), which enable you to trade off between completeness and latency.</li><li>🤯 Everything is a stream; it’s just some streams are bounded, whilst others are unbounded.</li><li><strong><em>-&gt; Wait, What? Everything is a Stream?</em></strong></li><li>-&gt; From my background with Apache Kafka and the <a href="https://www.michael-noll.com/blog/2018/04/05/of-stream-and-tables-in-kafka-and-stream-processing-part1/">stream-table duality</a>, this source-agnostic framing of events is different, and I can’t wait to explore it further. I’m interested to see if it’s just a matter of semantics, or if there is something fundamentally different in how Flink reasons about streams of events vs state for given keys.</li></ul><h3>Flink Use Cases</h3><p>The documentation for Flink lays out <a href="https://flink.apache.org/use-cases/">three distinct use cases</a> for Flink. Under each are linked several examples, mostly from the Flink Forward conference.</p><p><a href="https://flink.apache.org/use-cases/#event-driven-applications-a-nameeventdrivenappsa"><strong>Event-driven Applications</strong></a>, e.g.</p><ul><li><a href="https://www.youtube.com/watch?v=Do7C4UJyWCM/">Fraud detection</a></li><li><a href="https://www.youtube.com/watch?v=rJNH5WhWAj4/">Anomaly detection</a></li><li><a href="https://www.youtube.com/watch?v=_yHds9SvMfE/">Rule-based alerting</a></li><li><a href="https://jobs.zalando.com/tech/blog/complex-event-generation-for-business-process-monitoring-using-apache-flink/">Business process monitoring</a></li><li><a href="https://www.youtube.com/watch?v=0cJ565r2FVI/">Web application (social network)</a></li></ul><p><a href="https://flink.apache.org/use-cases/#data-analytics-applicationsa-nameanalyticsa"><strong>Data Analytics Applications</strong></a>, e.g.</p><ul><li><a href="https://www.youtube.com/watch?v=izYsMQWeUbE/">Quality monitoring of Telco networks</a></li><li><a href="https://www.youtube.com/watch?v=17tUR4TsvpM/">Analysis of product updates &amp; experiment evaluation</a></li><li><a href="https://eng.uber.com/athenax/">Ad-hoc analysis of live data</a></li><li>Large-scale graph analysis</li></ul><p><a href="https://flink.apache.org/use-cases/#data-pipeline-applications-a-namepipelinesa"><strong>Data Pipeline Applications</strong></a>, e.g.</p><ul><li><a href="https://ververica.com/blog/blink-flink-alibaba-search">Real-time search index building</a></li><li><a href="https://jobs.zalando.com/tech/blog/apache-showdown-flink-vs.-spark/">Continuous ETL</a> (a.k.a. “Streaming ETL”)</li></ul><p>One thing that’s interesting about the linked examples is that they are all from 6–7 years ago. One can look at this two ways. Put positively, it demonstrates what a long history and proof of success Flink has when it comes to experience in stream processing. Being snarky, one would cast it in the light that Flink is a technology of the past, on its way out with the Hadoops of this world.</p><p>I’d strongly reject the latter view. You may say that is obvious given that I now work for a company offering <a href="https://decodable.co/">a managed Flink service</a> 😉. But this in itself is a point to counter the snark. There are multiple companies <em>launching</em> Flink as a service-including companies which already had stream processing offerings based on other technology. Flink is a well-established technology with a strong <a href="https://flink.apache.org/roadmap/">roadmap</a> and a <a href="https://flink.apache.org/roadmap/#scenarios-we-focus-on">modern and cloud-native vision</a> for its future direction.</p><h3>Users of Flink</h3><p>Whilst the rise of managed Flink services is one proof-point demonstrating Flink’s popularity, the other irrefutable one is its <em>continued use</em> in a wide range of companies and use cases (and not just those from 6–7 years ago). A quick look through the back issues of <a href="https://www.dataengineeringweekly.com/">Data Engineering Weekly</a> and past sessions of <a href="https://www.flink-forward.org/events">Flink Forward</a> and other conferences demonstrates this.</p><p>Users with <strong>recent</strong> (in the last ~two years) published use cases include:</p><ul><li><a href="https://alibaba-cloud.medium.com/four-billion-records-per-second-f8eeabce934d">Alibaba Cloud</a></li><li><a href="https://www.flink-forward.org/sf-2022/conference-program#alexa--be-quiet---end-to-end-near-real-time-model-building-and-evaluation-in-amazon-alexa">Amazon</a></li><li><a href="https://www.infoq.com/presentations/apache-iceberg-streaming/">Apple</a></li><li><a href="https://www.youtube.com/watch?v=-wSbb4JSuZU">Booking.com</a></li><li><a href="https://medium.com/capital-one-tech/exploring-apache-flink-aws-kda-realtime-data-streaming-7201ed4ed197">Capital One</a></li><li><a href="https://doordash.engineering/2021/07/14/open-source-search-indexing/">DoorDash</a></li><li><a href="https://www.flink-forward.org/seattle-2023/agenda#model-inference-in-flink-sql-using-a-custom-http-connector">ING Bank</a></li><li><a href="https://tech.instacart.com/building-a-flink-self-serve-platform-on-kubernetes-at-scale-c11ef19aef10">Instacart</a></li><li><a href="https://www.flink-forward.org/seattle-2023/agenda#quality-scale-with-flink">JP MorganChase</a></li><li><a href="https://www.slideshare.net/FlinkForward/building-a-fully-managed-stream-processing-platform-on-flink-at-scale-for-linkedin-252866883">LinkedIn</a></li><li><a href="https://eng.lyft.com/wheres-my-data-a-unique-encounter-with-flink-streaming-s-kinesis-connector-6da3b11b164a">Lyft</a></li><li><a href="https://netflixtechblog.com/data-mesh-a-data-movement-and-processing-platform-netflix-1288bcab2873">Netflix</a> (<a href="https://netflixtechblog.com/keystone-real-time-stream-processing-platform-a3ee651812a">ad</a> <a href="https://www.google.com/search?q=site:netflixtechblog.com+flink">infinitum</a>)</li><li><a href="https://medium.com/pinterest-engineering/lessons-from-debugging-a-tricky-direct-memory-leak-f638c722d9f2">Pintrest</a> (this <a href="https://medium.com/pinterest-engineering/unified-flink-source-at-pinterest-streaming-data-processing-c9d4e89f2ed6">older article</a> lists more of their use cases)</li><li><a href="https://www.flink-forward.org/seattle-2023/agenda#protecting-reddit-users-at-scale-with-flink-stateful-functions">Reddit</a></li><li><a href="https://shopify.engineering/optimizing-apache-flink-applications-tips">Shopify</a></li><li><a href="https://speakerdeck.com/jeffchao/flink-forward-2022-squirreling-away-640-dollars-billion-how-stripe-leverages-flink-for-change-data-capture">Stripe</a></li><li><a href="https://www.flink-forward.org/seattle-2023/agenda#self-service-data-ingestion-platform-at-tiktok--powering--foryoupage-for--b--users">TikTok</a></li><li><a href="https://www.uber.com/en-GB/blog/real-time-exactly-once-ad-event-processing/">Uber</a></li><li><a href="https://vinted.engineering/2023/09/25/search-indexing-pipeline/">Vinted</a></li></ul><p>See also the Powered By Flink <a href="https://flink.apache.org/powered-by/">highlights</a> and <a href="https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink">complete list</a>.</p><h3>“What Can Flink Do” vs “Where Will You Find Flink”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x-mk3O268xY2wyG0YiLSqw.png" /></figure><p>I like this because it spells out <em>what</em> Flink does (e.g. Windowing, Streaming Joins, Filtering, etc), and separately, <em>where</em> it gets used for that.</p><h3>How do you run Flink?</h3><h4>Self-Managed</h4><p>Flink is a distributed system, which means that you don’t just buy one great big box and scale it up and up for capacity. Instead, you deploy it across multiple instances for both scalability and fault-tolerance.</p><p>The Flink documentation has a clear set of instructions for running Flink using the <a href="https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/resource-providers/standalone/overview/">binaries directly</a>, under <a href="https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/resource-providers/standalone/docker/">Docker</a>, and with <a href="https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/resource-providers/native_kubernetes/">Kubernetes</a>. Betraying its Big Data history, is also still supports <a href="https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/resource-providers/yarn/">YARN</a>.</p><h4>Managed Service</h4><p>Did I mention yet that <a href="https://decodable.co/">Decodable</a> offers a fully-managed Apache Flink service? :-D</p><p>You can find a list of other vendors that offer Apache Flink as a managed service in <a href="https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/overview/#vendor-solutions">the Flink documentation</a>.</p><h3>The Flink Community</h3><p>Just like any healthy open-source project, there is a good community around Flink. Per the Apache motto:</p><blockquote><em>If it didn’t happen on a mailing list, it didn’t happen.</em></blockquote><p>The <a href="https://flink.apache.org/community/">community</a> page on the Flink site lists numerous mailing lists. news@ and community@ are both pretty stagnant, but <a href="https://lists.apache.org/list.html?user@flink.apache.org">users@</a> is well-used with half a dozen posts per day. If you&#39;re contributing to Flink (rather than just using it) you&#39;ll want the dev@ list too.</p><p>Alongside the mailing lists, there is a <a href="https://join.slack.com/t/apache-flink/shared_invite/zt-22mklt3r5-89MjX41gqHsBk81ZoTDqXg">Slack group</a> with 3k members. It has a good layout of channels, and a handful of messages per day</p><p>You’ll also find a steady stream of Flink questions and answers on <a href="https://stackoverflow.com/questions/tagged/apache-flink">StackOverflow</a>.</p><h3>What’s next for Flink?</h3><p>There’s a <a href="https://flink.apache.org/roadmap/">comprehensive and well-maintained roadmap</a> for Flink. Changes are made through FLIPs ().</p><p>As well as what’s coming there’s also a clear list of <a href="https://flink.apache.org/roadmap/#feature-radar">features that are being phased out</a>. This level of insight into the project is really useful for a newbie-given how long Flink has been around (aeons, in internet years) there is going to be a lot of material published that is out of date and this chart will hopefully be a quick way to navigate that.</p><p>The roadmap page is notable for not only a list of planned features but also the <a href="https://flink.apache.org/roadmap/#scenarios-we-focus-on">general strategy</a> (which should help inform users as to whether their use cases are within sensible bounds) and even something close to my own heart: <a href="https://flink.apache.org/roadmap/#developer-experience">developer experience</a>. One particularly interesting bit that caught my eye is the idea of built-in dynamic table storage, described in <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-188%3A+Introduce+Built-in+Dynamic+Table+Storage">FLIP-188</a> and-if I understand correctly-spun out into its own Apache Incubator project, <a href="https://paimon.apache.org/">Apache Paimon</a>. Paimon describes itself as a “ <em>Streaming data lake platform</em>” and is definitely on my list to go and check out particularly after <a href="https://rmoff.net/2022/09/14/data-engineering-in-2022-storage-and-access/">my work last year on mapping out the data engineering landscape</a> as at first glance I’m not sure where it fits.</p><h3>Flink Resources</h3><p>The <a href="https://flink.apache.org/">Apache Flink project website</a> itself is an excellent resource. Especially when compared to some other Apache projects ( <em>cough</em>), it’s extremely well laid out, thoughtfully organised, and easy to use.</p><p>Some other good places for Flink information include:</p><ul><li>The <a href="https://www.flink-forward.org/">Flink Forward</a> conference (<a href="https://www.flink-forward.org/events">previous events</a>)</li><li><a href="https://cse.google.com/cse?cx=010150859881542981030:hqhxyxpwtc4&amp;ie=UTF-8&amp;q=Apache+Flink+&amp;sa=Search">Apache Flink presentations on SpeakerDeck</a></li></ul><p>There are also some good podcast episodes:</p><ul><li><a href="https://overcast.fm/itunes1193040557/data-engineering-podcast">Stateful, Distributed Stream Processing on Flink with Fabian Hueske</a></li><li><a href="https://overcast.fm/+BAj84H3884">Inside Apache Flink: A Conversation with Robert Metzger</a></li><li><a href="https://overcast.fm/+BAj87Wiuo4">Diving Deep into Apache Flink with Robert Metzger</a></li></ul><p><em>Originally published at </em><a href="https://rmoff.net/2023/10/02/learning-apache-flink-s01e02-what-is-flink/"><em>https://rmoff.net</em></a><em> on October 2, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dc049bea3fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Does This DevEx Engineer Do?]]></title>
            <link>https://medium.com/@rmoff/what-does-this-devex-engineer-do-eeb392cd70ee?source=rss-e6a1d2163309------2</link>
            <guid isPermaLink="false">https://medium.com/p/eeb392cd70ee</guid>
            <category><![CDATA[devex]]></category>
            <category><![CDATA[devrel]]></category>
            <dc:creator><![CDATA[rmoff]]></dc:creator>
            <pubDate>Tue, 23 May 2023 23:57:10 GMT</pubDate>
            <atom:updated>2024-05-21T09:33:25.508Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pILUYWI0LM9dRvVm.jpeg" /></figure><p><em>This was originally titled more broadly “What Does A DevEx Engineer Do”, but that made it into a far too tedious and long-winding etymological exploration of the discipline. Instead, I’m going to tell you what this particular instantiation of the entity does 😄</em></p><h3>Let’s Define DevEx first…</h3><p>Developer Experience (a.k.a. DevEx, DevX) is fundamentally about making the experience for developers a good, nay, positively joyful, one.</p><p>It’s about removing the “ <em>if you just read the first five chapters of the manual then you can use this … calculator</em> “ and instead making software behave as one would instinctively expect it to.</p><p>It’s pre-populating fields that can be pre-populated, opening APIs that can be opened, removing barriers and lifting developers up.</p><p>It’s about shipping that MVP but with the sharp edges rounded off and plotting a happy path for users so that the cringe-worthy mission statement of companies that “we build software that people love” can actually seem credible.</p><p>In a crowded market-place where alternative software is a click away making your software not just functional but also damned easy and enjoyable to use is vital. Developers are no longer chained to their workstations, doomed to use whatever piece of crap the IT director bought on the golf course from some salesman looking to hit their quarterly target.</p><p>Developers are empowered to use the software that not only works but which gets the job done efficiently, painlessly, and is easy to use. Don’t confuse <em>easy</em> with <em>simple</em> -complicated software can have a great developer experience — and simple software can unfortuntely also have a bloody awful developer experience.</p><h3>But Hang On, I thought DevEx Was… $DifferentThing</h3><p>Developer Experience (DevEx, also DevX) has taken on a similar-but-dual meaning in the last year or two. Both relate to developers and their experience, but one’s audience is developers <em>within</em> a company, and the other’s <em>outside</em> the company.</p><p>My familiarity with DevEx is based on the latter — working to improve the experience of developers external to the company, and specifically, those to whom the company is building and hopefully selling a product. Just like the broader discipline of Developer Relations as a whole-of which this version of DevEx is a part-there is no selling to be done here. DevRel, DevEx, and whatever other term you come up with, do not sell. Period. They educate, inform, and entertain the developer base, with a view to the developer then having such a great time with the software that they subsequently decide to open their cheque books and buy the software of their own volition.</p><p>The other DevEx which I’ve noticed more and more of recently is that serving the developers within a company. A good example of this angle is the wonderfully-named <a href="https://www.container-solutions.com/wtf-is-devex">WTF is DevEx</a> from Container Solutions. Microsoft have <a href="https://microsoft.github.io/code-with-engineering-playbook/developer-experience/">a good page on DevEx</a>, and the <a href="https://www.developerexperience.us/">DevX Community</a> has sprung up to provide a place for DevX practitioners to support each other.</p><h3>OK, so we’re talking about the DevRel-y DevEx</h3><p>Both kinds of DevX are important, and there is <em>huge</em> overlap between them — but there are some significant differences. DevX done under the DevRel banner will often include:</p><ul><li>Content creation</li><li>Documentation</li><li>Community interaction</li></ul><p>In a nutshell, it’s activities that enhance the <em>experience</em> of a <em>developer</em> not only when they’re just using software, but on their entire journey with it — from becoming aware of it, trying it, learning it, and so on.</p><h3>So… What Does This DevEx Engineer Do? <a href="https://rmoff.net/2023/05/23/what-does-this-devex-engineer-do/#so-what-does-this-devex-engineer-do">🔗</a></h3><ul><li>Write <a href="https://rmoff.net//2018/08/02/kafka-listeners-explained/">deeply technical</a> and <a href="https://www.confluent.io/blog/streaming-etl-and-analytics-for-real-time-location-tracking/">engaging blog posts</a>, along with more <a href="https://rmoff.net/2022/09/14/stretching-my-legs-in-the-data-engineering-ecosystem-in-2022/">analytical ones</a>.</li><li>Build best-of-breed <a href="https://docs.lakefs.io/quickstart">Quickstarts</a> and <a href="https://rmoff.net/2020/03/11/streaming-wi-fi-trace-data-from-raspberry-pi-to-apache-kafka-with-confluent-cloud/">fun</a> <a href="https://github.com/confluentinc/demo-scene/tree/master/mqtt-tracker">demos</a></li><li><a href="https://github.com/treeverse/lakeFS/issues?q=is%3Aissue+author%3Armoff+DX">Advocate for</a> and <a href="https://github.com/treeverse/lakeFS/pull/5313">Fix</a> DX in the product</li><li>Record and produce <a href="https://rmoff.net/2021/02/17/ksqldb-howto-a-mini-video-series/">video explainers</a></li><li>Teaching <a href="https://rmoff.net/2020/06/25/learning-golang-some-rough-notes-s01e00/">by learning</a></li><li>Engage with the online community wherever Developers are found</li><li><a href="https://twitter.com/rmoff/status/1587382202781913089">Shitposting</a>, <a href="https://www.linkedin.com/posts/robinmoffatt_choosing-open-wisely-snowflake-blog-activity-6973309528628973568-gjOJ?utm_source=share&amp;utm_medium=member_desktop">Shitposting</a>, and <a href="https://twitter.com/rmoff/status/1448230290657251338">Memes</a> (and more <a href="https://twitter.com/rmoff/status/1641113617662722064">shitposting</a>)</li><li>Manage and develop <a href="https://forum.confluent.io/t/why-yacp-yet-another-community-platform/54">community platforms</a></li><li>Author <a href="https://github.com/treeverse/lakeFS/pulls?q=is%3Apr+author%3Armoff+label%3Adocs">docs</a></li><li>Maintain and improve <a href="https://rmoff.net/2023/04/20/building-better-docs-automating-jekyll-builds-and-link-checking-for-prs/">Docs platform</a></li></ul><h3>Isn’t this the same as Developer Advocacy?</h3><p>To an extent, yes. I’m not interested in a discussion about job titles really, short to say that most people assume a Developer Advocate travels and does conference talks with other activities fitting around it. For me a DevX Engineer is more focussed on the building and writing, than speaking.</p><p><em>Originally published at </em><a href="https://rmoff.net/2023/05/23/what-does-this-devex-engineer-do/"><em>https://rmoff.net</em></a><em> on May 23, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eeb392cd70ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>