<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Graham Lannigan on Medium]]></title>
        <description><![CDATA[Stories by Graham Lannigan on Medium]]></description>
        <link>https://medium.com/@gshenanigan?source=rss-560e9b929369------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*z1iMhp7ZpuE0hOMq.jpg</url>
            <title>Stories by Graham Lannigan on Medium</title>
            <link>https://medium.com/@gshenanigan?source=rss-560e9b929369------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:31:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gshenanigan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[You’re not a real Data Engineer if you use no or low code tools]]></title>
            <link>https://gshenanigan.medium.com/youre-not-a-real-data-engineer-if-you-use-no-or-low-code-tools-b71cbf76862f?source=rss-560e9b929369------2</link>
            <guid isPermaLink="false">https://medium.com/p/b71cbf76862f</guid>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Graham Lannigan]]></dc:creator>
            <pubDate>Wed, 19 Jun 2024 17:35:33 GMT</pubDate>
            <atom:updated>2024-06-19T17:35:33.517Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rT_SqDochmGwfpE4HNvw5w.png" /><figcaption>Matillion ETL is a low/no-code ELT solution used to load and transform data in cloud data platforms.</figcaption></figure><p>I tend to read a lot of posts online to keep up with developments in the data industry. This usually consists of keeping an eye on the <a href="https://www.reddit.com/r/dataengineering/">data engineering subreddit</a>, people I follow on Twitter (no, I’m not calling it “X”, that’s stupid), my LinkedIn network and <a href="https://medium.com/tag/data-engineering">data topics on Medium</a>. I find this gives me a good blend of opinions and content to keep up to date with things (even if my reading list on Medium is getting wildly out of hand). One thing I find interesting is there seems to be a sense of elitism from some in the community who are quite vocal about what they perceive a Data Engineer to be (or not to be).</p><h3>Data Engineer vs Analytics Engineer</h3><p>Maybe this should be re-titled “what does a Data Engineer actually do?”. A <a href="https://www.reddit.com/r/dataengineering/s/WTGXSTwSVm">recent post on reddit</a> shared a table highlighting responsibilities of Data Engineers vs Analytics Engineers vs Data Analysts. A few highlights of this were:</p><ul><li>Build custom data ingestion integrations. (Data Engineer)</li><li>Develop and deploy machine learning endpoints. (Data Engineer)</li><li>Build and maintain the data platform. (Data Engineer)</li><li>Data warehouse performance optimization. (Data Engineer)</li><li>Provide clean, transformed data ready for analysts. (Analytics Engineer)</li><li>Train business users on how to use a data platform data visualization tools. (Analytics Engineer)</li><li>Work with business users to understand data requirements. (Data Analyst)</li></ul><p>The problem I see is that for some organisations this table might be absolutely correct. For others it might not. Personally, I’ve done all of these things and have never been titled Data Analyst, Analytics Engineer, or even Data Engineer, for that matter. For some, you’re only a Data Engineer if you’re dealing with the E&amp;L (Extract &amp; Load) parts of data processing. For others, if you only do the T part (Transform) then you’re an Analytics Engineer (I think we can thank <a href="https://getdbt.com/">dbt</a> for that one). The confusion created by splitting out these job titles was brilliantly summed up by one commenter on the DE subreddit who asked “WTF is an Analytics Engineer?”</p><p>It’s possible some of this is down to the maturity of the data industry. I think a lot of it is down to the size of the organisation in question. A lot of smaller shops will have people in multi-disciplinary roles, maybe even dealing with the whole data lifecycle. In larger orgs, it might be more like the definitions above. There are no hard and fast rules, and IMO there shouldn’t be. Even compared with IT or dev as a whole, there are a number of newer concepts which are yet to be widely understood. In saying that, many of the tasks above have been done decades ago by people titled “SQL Developer”, “BI Developer”, “DBA”, or any number of job titles that don’t fully encapsulate what somebody does.</p><h3>Tools of the trade</h3><p>I was interested to read a <a href="https://monimiller.com/blog/should-my-data-engineering-title-be-revoked-if-i-dont-know-python">post by Data Engineer Monica Miller</a> about her self-doubt around the fact she’s a Data Engineer who works almost exclusively in SQL and doesn’t know Scala or work heavily with python. You know what? I don’t know much python either. I can figure it out if I need to, and I have the experience to know when something is better handled in python, or SQL, or something else. I’ve spent plenty of time working with no/low code ETL and ELT tools, from my SQL Server days working with SSIS, to adopting Matillion as one of the few available ELT tools with native Redshift compatibility back in 2016.</p><p>As I’ve gone through my career I’ve come to agree with many smart people who have long argued that code is a liability, rather than an asset.</p><blockquote><em>Code is a liability, not an asset. So goal of software engineer is delivering the maximum amount of desired functionality at the cost of the least amount of code complexity, even as desired functionality evolves over time.</em></blockquote><blockquote><em>- Greg Brockman (@gdb) </em><a href="https://twitter.com/gdb/status/1606485739679932416?ref_src=twsrc%5Etfw"><em>December 24, 2022</em></a></blockquote><p>That’s not to say one shouldn’t write code, but do so for the right reasons. Write code that adds specific value for your use case, or because you don’t have the budget to buy the tool(s) that do what you need. Don’t spend time writing code that re-solves problems that are already solved. All you’re doing is adding technical debt that somebody (possibly you) will need to maintain in future. You wouldn’t write your own UDF to perform a SUBSTRING operation, would you?</p><p>Use the tool that gets you up and running and delivering value to your business as quickly as possible. There are very few truly “wrong” decisions in this industry. You can change your approach later. Sure it might cost time and money, but who’s to say your new CTO/CDO won’t bring in a new vendor anyway? There are <a href="https://mattturck.com/landscape/mad2024.pdf">so many tools out there</a> it’s impossible to define a specific stack that is the only way to do DE. Things change, and the key part of architecture is making sure you assess the landscape and make the most informed decision you can while being aware of as many risks as you can. Nothing is risk free, it’s about having reasonable mitigations in place should things go wrong.</p><p>For some teams that’ll be doing all their ELT via python code running via Airflow, for others it’ll be using Fivetran to land data and dbt to Transform it. For others it’ll be using Matillion or SSIS or other no/low-code tools to visually build pipelines, sprinkling in custom functions where the out of the box stuff doesn’t do what you need. And do you know what? They can all call themselves Data Engineers, because they are delivering data to their business by doing some or all of the ELT/ETL process.</p><h3>Am I a Data Engineer?</h3><p>All the elitism on social media does is gatekeep an industry that is massively growing and in need of more talented people who can understand the value in delivering clean, timely data to their business. With the exception of SQL, languages and tools tend to come and go. Fundamental understanding of data concepts such as data modelling, data pipelines, data quality, and so on will always be valuable, regardless of the tools used to deliver.</p><p>Do you think you’re a Data Engineer? Does your company call you a Data Engineer? Are you involved in the ELT process? Maybe you’re doing “old-school” ETL on so-called “legacy” on-prem solutions. Want to call yourself a Data Engineer? Go for it.</p><p><em>Originally published at </em><a href="https://picnicerror.net/data/youre-not-a-data-engineer-if-you-use-no-or-low-code-tools-2024-06-19/"><em>https://picnicerror.net</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b71cbf76862f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is ChatGPT coming for your job?]]></title>
            <link>https://gshenanigan.medium.com/is-chatgpt-coming-for-your-job-9c7c2c36e84e?source=rss-560e9b929369------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c7c2c36e84e</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Graham Lannigan]]></dc:creator>
            <pubDate>Thu, 02 Mar 2023 10:43:33 GMT</pubDate>
            <atom:updated>2023-03-02T10:43:33.414Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cJBAaf_afxtXaN0G.jpg" /></figure><p>I need your clothes, your boots and your employee ID. OpenAI’s Large Language Model (LLM) chatbot exploded onto the scene last November. In less than a week <a href="https://en.wikipedia.org/wiki/ChatGPT">ChatGPT had over 4 million users</a> and the Internet was awash with examples of prompts used and output from the chatbot. Some of the responses were so good that it didn’t take long for a wave of people claiming ChatGPT would make programmers, writers, support, and a whole host of other roles obsolete. A few months in and the doomsayers are still predicting widespread replacement of humans.</p><p>So how accurate are these claims of this particular AI offering taking our jobs?</p><h3>Should we all head for the Job Centre?</h3><p>So, will ChatGPT replace us all? There are certainly plenty speculating that it will be the death knell for careers as varied as software development to copywriting, even those saying it’s the beginning of the end for Google Search. Ignacio de Gregorio, speculated whether ChatGPT would kill Google in his <a href="https://medium.com/@ignacio.de.gregorio.noblejas/can-chatgpt-kill-google-6d59742ee635">Medium article: Can ChatGPT kill Google?</a>.</p><p>François Chollet, who is a key member of Google’s own deep learning work, called out the “AI effect” where people were prematurely jumping on the bandwagon of LLMs replacing us all, much like how we’re supposed to be all riding around in self-driving cars right now.</p><blockquote><em>The best minds in tech are falling prey to the “AI effect” — watching an AI system perform well on a number of special cases and assuming it will generalize broadly, like a human would.</em></blockquote><blockquote><em>Last time it happened on this scale was for self-driving cars in 2015–2016. </em><a href="https://t.co/HvMKhzRFbB"><em>https://t.co/HvMKhzRFbB</em></a></blockquote><blockquote><em>- François Chollet (@fchollet) </em><a href="https://twitter.com/fchollet/status/1623355738558169088?ref_src=twsrc%5Etfw"><em>February 8, 2023</em></a></blockquote><p>This certainly seems like a fair comment. We’re often so quick to jump on new tech, as amazing as it may be, and think it’s going to have an immediate, widespread impact. But new tech often takes time for mainstream adoption. Apple certainly didn’t invent smartphones and weren’t the first to market by a long shot, but they timed their entry well and built a buzz around the iPhone that prompted a major shift in public perception. ChatGPT is certainly causing a huge buzz but the fact we haven’t seen commmercialised entries from the major players (Google, Amazon, Meta) suggests there’s plenty more to come in this space.</p><h3>Language Models vs General purpose AI</h3><p>Large Language Models (LLM) are specifically trained and optimised for language-based use cases, such as recognising and interpreting text. Because they are able to interpret meaning from an input (or prompt), they can often produce output that, on the surface at least, appears highly plausible and accurate for the specific subject of the prompt. However, dig a little deeper and there are plenty of examples where these fail on the detail. Consider the below Tweet example, where ChatGPT is repeatedly asked to do a task and repeatedly fails, despite “knowing” the correct answer:</p><blockquote><em>With all the hype with ChatGPT, it’s interesting to see where it can fail miserably, even when taught the right things, even when it can verbalize a correct line of reasoning or definition. Oh, and it lies, too.</em><a href="https://twitter.com/hashtag/ChatGPT?src=hash&amp;ref_src=twsrc%5Etfw"><em>#ChatGPT</em></a><em> </em><a href="https://twitter.com/hashtag/ChatGPTFail?src=hash&amp;ref_src=twsrc%5Etfw"><em>#ChatGPTFail</em></a><em> </em><a href="https://t.co/8W6JSqM3kc"><em>pic.twitter.com/8W6JSqM3kc</em></a></blockquote><blockquote><em>- Daniel Rivera 🎮 (@danielrivera) </em><a href="https://twitter.com/danielrivera/status/1630983388562587653?ref_src=twsrc%5Etfw"><em>March 1, 2023</em></a></blockquote><p>General Purpose AI is what we think of when we picture a super-intelligent artificial life form in science fiction. They are able to learn, adapt and grow, using the sum of their experiences to handle any situation, much like humans. This is a much harder problem to solve than language. In any given situation there may be millions of variables which our brains will filter through our memories and experience, even our current emotional state, which is all considered in milliseconds and results in our decision. This is almost impossible to even conceive, never mind codify and build into an artificial brain.</p><p>The way this seems to be approached currently (to my very basic understanding) is to build specialised models for different scenarios and build other systems to combine the output and filter them through rules, much like one would go about building a multi-model recommendation system.</p><p>This doesn’t mean ChatGPT is bad. There are plenty of cases where LLMs are hugely useful, they just aren’t anywhere near the levels of general intelligence (most) humans can offer.</p><h3>Use cases for ChatGPT now</h3><p>Indeed, Microsoft are betting big on the utility of AI and LLMs, having been major investors in OpenAI for a number of years, and recently launching a GPT-3 integrated version of their own search engine, Bing.</p><p>ChatGPT as a tool to optimise workflows and increase productivity. Areas such as SEO where there can be a lot of text-based grunt-work, optimising descriptions and meta content, can be standardised and optimised. The human part becomes one of review and correction rather than creation.</p><blockquote><em>Using GPT3 to rebuild 4,000 product descriptions. Mesmerising to watch.</em></blockquote><blockquote><em>Background: 6 year old site, which has been on two platforms, lots of mojibake, redundant html, random line breaks / &lt;p&gt; tags — multiple char encoding etc.</em></blockquote><blockquote><em>Beautiful to watch it get cleaned &amp; standardised. </em><a href="https://t.co/s5TZJk3Jqh"><em>pic.twitter.com/s5TZJk3Jqh</em></a></blockquote><blockquote><em>- Lee Foot 🐍📈 (@LeeFootSEO) </em><a href="https://twitter.com/LeeFootSEO/status/1599888323081748480?ref_src=twsrc%5Etfw"><em>December 5, 2022</em></a></blockquote><p>We’ve already seen this in action with Github Copilot. Code has stricter syntax than spoken or written language and much less variety which means it is arguably easier to codify. Copilot has been well received (despite some quality concerns) and yet there are still plenty of software engineering jobs out there. Things like drafting documents, creating product descriptions, generating realistic filler text for websites are just a few simple examples of applications of generative LLMs that can be time-savers for professionals.</p><p>Simon Wardley, creator of Wardley Maps has been very successful at analysing developments in technology and their adoption.</p><blockquote><em>X : Does ChatGPT mean we’re close to building thinking machines?<br>Me : A long way into the future. The whole Chinese room argument. Think of ChatGPT as more stochastic parrots rather than a thinking machine. It’s “humanity” comes from our observation of the text.</em></blockquote><blockquote><em>- Simon Wardley (@swardley) </em><a href="https://twitter.com/swardley/status/1622570975043649536?ref_src=twsrc%5Etfw"><em>February 6, 2023</em></a></blockquote><p>He was adamant that cloud computing would become the norm, then serverless and predicts the next major leap is in conversational programming, the art of speaking to machines in natural language to create software. Far from removing the human from the equation, this democratises the ability to program. This in itself may be a threat to some in the industry, but much like the rise of WYSIWYG editors and no-code app creation tools haven’t made every web designer or software engineer redundant, conversational programming won’t do this either. Subject matter expertise, experience, and knowing your business will still be hugely important, much as they are now. Knowing how to code isn’t valuable in itself, it’s knowing how to use that skill to solve the right problems, something people can tend to forget when there is something new and shiny on the scene.</p><p>The more we reduce friction from the input experience, the more people can focus on identifying and solving the important problems and create value.</p><h3>The future</h3><p>So when will we be looking at viable General Purpose AI that can challenge the human capability to adapt and apply expertise to new and different disciplines?</p><p>I think we’re decades away. Beyond the technical challenges of creating the necessary neural capabilities, there are a whole host of secondary challenges to solve before we’ll see widespread adoption. The ethical concerns around AI bias are well documented, not to mention questions around transparency, data protection, security and commercial sensitivity. While these shouldn’t hold back innovation to a point, they are real blockers to adoption in a lot of organisations. Remember even now when cloud has become widely accepted, there are still organisations succeeding on what we’d call legacy tech due to compliance, commercial or infosec concerns.</p><blockquote><em>I think you massively overestimate rate of adoption. Many companies who won’t use public cloud or proprietary tech, nvm hurdles around quality control, data protection, IP, etc. No doubt it’ll be mind-blowing but I’d be amazed if we’re not still years from mainstream adoption.</em></blockquote><blockquote><em>- Graham 🏴󠁧󠁢󠁳󠁣󠁴󠁿 (@gshenanigan) </em><a href="https://twitter.com/gshenanigan/status/1607398751483887616?ref_src=twsrc%5Etfw"><em>December 26, 2022</em></a></blockquote><p>It’s an exciting time, certainly. OpenAI have taken a huge step towards bringing real-world examples of AI into the mass consciousness and the proactive and open-minded out there have been given a super-charged tool to add to their utility belt. I don’t think it’s yet a differentiator, but can certainly optimise a number of use cases today.</p><p>Are you using ChatGPT or any of OpenAI’s other models for any use cases today? If so, what? What are your impressions of it and what impact has it had?</p><p><em>Originally published at </em><a href="https://picnicerror.net/data/is-chatgpt-coming-for-your-job-2023-02-28/"><em>https://picnicerror.net</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c7c2c36e84e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Information Overload and the Second Brain]]></title>
            <link>https://gshenanigan.medium.com/information-overload-and-the-second-brain-20e4e909ecb7?source=rss-560e9b929369------2</link>
            <guid isPermaLink="false">https://medium.com/p/20e4e909ecb7</guid>
            <category><![CDATA[information-overload]]></category>
            <category><![CDATA[roam-research]]></category>
            <dc:creator><![CDATA[Graham Lannigan]]></dc:creator>
            <pubDate>Mon, 19 Apr 2021 21:20:26 GMT</pubDate>
            <atom:updated>2021-04-19T21:20:26.575Z</atom:updated>
            <content:encoded><![CDATA[<p>I joined my first Twitter Spaces uh...hangout…chat…thing…space(?) the other night, hosted by <a href="https://twitter.com/natterstefan">Stefan Natter</a>, talking about <a href="https://roamresearch.com/">Roam Research</a> and how powerful a tool it is for unlocking all sorts of potential, and helping focus and creativity. One comment made by <a href="https://twitter.com/GeorgeSilverman">George Silverman of MindSkills</a> echoed some thoughts of my own lately, about the sheer volume of information to which we’re exposed on a daily basis, and the mental capacity of our brains to process this.</p><p>Since the public emergence of the Internet only thirty-something years ago, the amount of information we process daily has increased exponentially, and shows no signs of slowing down. When you consider the pace of human evolution, taking hundreds and thousands of years to respond to environmental stimuli, are we currently in the midst of some Darwin-esque process of pushing the boundaries of our brains to see which ones survive and which break down? We have had only decades to adapt to this new information age, and it should perhaps be little wonder that there is such a rise in mental ill health, given all the extra processing our brains are doing, and all the additional stimuli for which we are simply not yet evolved to handle.</p><p>There’s a common suggestion (urban legend?) that we only use a small percentage of our brains, and there more capacity locked away in there. Whether this is true or not (I’m no neuroscientist), it has made me wonder whether at some point we will see the next evolutionary leap to allow humans to disseminate and process all this additional data? What I do know though, is that I often feel like I literally cannot process any more information after a day at work. I’m usually on a lot of calls, making lots of decisions and processing lots of info. By the end of it I feel like I can barely take in anything else.</p><p>There’s a well publicised story about Barack Obama and Mark Zuckerberg (not to mention many others) wearing the same outfit each day to combat decision fatigue. All of a sudden, this doesn’t sound like such a crazy idea. For me, I don’t own a wardrobe full of black (and slightly darker black) turtlenecks, so what do I do to try and help myself deal with this information overload?</p><h3>All roads lead to Roam</h3><p>My answer is <a href="https://roamresearch.com/">Roam Research</a>. There has been a rise over the past year or so of connected “second brain” applications. These are often characterised as note-taking or productivity apps, but these descriptions do them little justice. Roam and its ilk are a little like having your own personal Internet of You, into which you can sink all your notes, thoughts, ideas, and more, and link these together for recall exactly when you need them. I’ve done a little work with Graph Databases and understand the concept, so I very quickly had an inkling of the power this model could offer if there was a way to effectively define the nodes and their relationships.</p><p>This is one of the things that led me to <a href="https://roamresearch.com/">Roam Research</a> in the first place. I stumbled across it at a time when I was falling out of love with <a href="https://keep.google.com/">Google Keep</a>. While I continue to love the simplicity of post-its for input, it was getting too hard to use for recall and I was having to constantly edit and recombine my notes. I was considering having a go of either <a href="https://www.notion.so/">Notion</a> or <a href="https://miro.com/">Miro</a> when I stumbled across <a href="https://www.nateliason.com/blog/roam">Nat Eliason’s post</a> on Roam and decided to give it a try.</p><p>Being familiar with use cases for graph databases, I understood the concept straight away and had an inkling it could work extremely well. I wasn’t quite prepared for just how well it would work. Since January 2020 I have sunk everything into it. Meeting notes, thoughts, ideas, feelings, inspirational quotes, and everything else. Yes, my graph is a bit of a mess, and I need to spend some time refining some proper process, but it’s amazing how even in some of that chaos (especially in the early days) I can look into something and all of a sudden see a reference from 9 months ago that sparks an idea.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EyrGGugr_cLutNNo.png" /></figure><p>There is a handy Roam help graph that is available to all users for any help with onboarding.</p><p>I’ve nowhere near perfected my workflow, but I can genuinely say <a href="https://roamresearch.com/">Roam Research</a> has been a game changer for me. I still retain a lot of the information I process, but I’m slowly adapting to worry less about it and rely on my Roam graph instead.</p><h3>Roam wasn’t built in a day</h3><p>It has been, I think, about a year since Roam turned on payments, meaning most users will need to pay $15/month beyond their 30 day trial. This may seem steep in comparison to some other systems, but there’s a strong value proposition behind that, and Roam’s team, led by the vision of <a href="https://twitter.com/conaw">Conor White-Sullivan</a> have a feature-rich plan (I hesitate to say “roadmap” as they’re keen to stay open ended) and a fervent belief in not just Roam as a product, but the need for a whole new breed of tool that can act as a communal second brain and shake the foundations of the very Internet itself.</p><p>There are a few major omissions currently which the Roam team are working hard to not just release, but to make sure they absolutely nail the implementation:</p><ul><li>Mobile I/O is rather clunky and there’s no native app, relying on Progressive Web Apps (PWAs) running in the device browser as the closest thing currently avaialble. On the Twitter space I atteneded, <a href="https://twitter.com/GeorgeSilverman">George Silverman</a> was talking about waking up in the middle of the night and sinking a few ideas into Roam. This is exactly the type of thing it should be great for, but it’s a long way away in its current guise from scrawling a few words on a notepad and going back to sleep.</li><li>There’s also no API (yet). Despite this, there is a dedicated community already there featuring the likes of <a href="https://twitter.com/dvargas92495">roamjs by David Vargas</a> and <a href="https://twitter.com/roamhacker">RoamHacker</a>. Roam has a clever extension system that allows any JavaScript code to be executed within Roam, so it’s possible to build pretty much anything you can think of. Naturally there are associated risks with this, so it’s best used within caution (a fact which is very well communicated before you even get to the point of executing any code). When the official API releases I think Roam is going to get huge, fast.</li><li>Speaking of security, the lack of full end-to-end-encryption (E2EE) is a concern for some at present. Roam does allow creation of encrypted blocks, which provides some protection, but there is work to be done there. This is a mainly technical challenge though, rather than a contractual one. As <a href="https://twitter.com/conaw">Conor</a> has highlighted on multiple occasions, with Roam’s long term goals to be more collaborative and multi-user focused, E2EE is not an easy problem to solve (multiple editors requiring access to specific blocks and sharing keys). The 5-year $500 believer plan allows users to create offline graphs which again, may help some of the more security conscious, but that also comes with its own risks.</li><li>The Graph Overview, which provides the full visualisation of your graph is great early on. You can see individual nodes clearly and explore relationships. As this grows however, it becomes a big, zoomed out rectangle with very limited utility. One of the strengths of any graph visualisation is to be able to explore connections visually and jump from node to node, discovering unexpected connections. This is an are Roam could really improve, or perhaps this will be solved by plugins once the official API makes an appearance. I’d imagine there are plenty of people out there who have skills in this area.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hHrlNXW5F0oEPgJs.png" /></figure><p>The Graph Overview isn’t as useful as it could be currently, but it gives a great idea of the scale of a well-used Roam graph.</p><p>Despite the lack of these what some would call “key features”, Roam has had an incredibly strong start, and the leadership certainly has a vision towards which they are working.</p><h3>When in Roam…</h3><p>It’s odd to see a tool with such potential for so many use cases that a lot of people just don’t “get”. It’s not a question of intelligence or “vision” — Roam is bare bones. You get out what you put in, and it’s not immediately apparent “what” to put in. It’s a sea of emptiness with only your Daily Notes page waiting for you to start typing. The free form nature of it could drive some people insane, like staring into the void. As soon as you start typing, and adding ideas, and linking blocks and pages, you can see it taking shape. There’s definitely a “Eureka!” moment when you first discover a previously unconnected connection, and I think this is the part Roam could do more to help new users discover. It’s just in need of a little more work with the onboarding to help its power become more apparent.</p><p>In terms of combating information overload and helping organise my thoughts, Roam has been an absolute revelation and it’s not hard to see how a new class of tool could be emerging here. It’s not always the input of information that’s the issue, it’s the organisation and retrieval that is problematic. Hierarchies take maintenance. Search requires accuracy. The second brain? It just needs to connect two dots.</p><p><em>Originally published at </em><a href="https://picnicerror.net/web-technology/information-overload-and-the-second-brain-2021-04-16/"><em>https://picnicerror.net</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20e4e909ecb7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>