<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Harrington on Medium]]></title>
        <description><![CDATA[Stories by Daniel Harrington on Medium]]></description>
        <link>https://medium.com/@dharrington1995?source=rss-468ea54eadf8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*iiqGvpFDdNzP9aI-7cLskw.jpeg</url>
            <title>Stories by Daniel Harrington on Medium</title>
            <link>https://medium.com/@dharrington1995?source=rss-468ea54eadf8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:52:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dharrington1995/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Docker’s GenAI Stack: Pioneering the New Frontier of AI Development — Part I]]></title>
            <link>https://blog.mesh-ai.com/dockers-genai-stack-pioneering-the-new-frontier-of-ai-development-part-i-a2b52445c909?source=rss-468ea54eadf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2b52445c909</guid>
            <category><![CDATA[gen-ai-tools]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Daniel Harrington]]></dc:creator>
            <pubDate>Tue, 30 Apr 2024 15:08:47 GMT</pubDate>
            <atom:updated>2024-04-30T15:08:47.946Z</atom:updated>
            <content:encoded><![CDATA[<h3>Docker’s GenAI Stack: Pioneering the New Frontier of AI Development — Part I</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VtYZNAiHFpCO6mudfPafsw.png" /></figure><h3>Introduction</h3><p>The pace of development in the AI space can sometimes feel overwhelming with new papers, models and ideas being released and discussed on a seemingly daily basis. As this development continues, AI applications have moved beyond simple chatbot designs to more imaginative implementations which leverage these models beyond their base capability. One example of this is the GenAI Stack from Docker announced at DockerCon2023. This stack brings together some titans of the AI world to imagine a new way to develop and deploy AI applications, bringing together the key technologies of Docker, Langchain, Ollama and Neo4j.</p><p>In this article, I will be discussing this new development and why it is such an interesting step forward in this space.</p><h3>The Evolution of AI Development</h3><p>In a short space of time, the state of AI in the industry has changed massively. In the 18 months since ChatGPT’s release, general hype has slowly given way to genuine developments in the technology which have massively increased the efficacy of the technology beyond simple chatbot interfaces.</p><p>One paradigm shift that has taken the AI world by storm is the concept of Retrieval Augmented Generation (RAG). This broad design pattern comes in many forms but at a high level, it represents a move away from traditional Large Language Model (LLM) chatbots in which the user interacts directly with a base model (shown in Figure 1)…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9SoSZj18ZBLCtH_eaKbL3g.png" /><figcaption><em>Figure 1 — Interaction with a Base Foundational Model has its limits</em></figcaption></figure><p>…to a system in which some external information can be retrieved from outside of the model and added as context for the model (shown in Figure 2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UuUAXn7qUuRYJ3fv" /><figcaption><em>Figure 2 — RAG allows the LLM to access information outside of its training data</em></figcaption></figure><p>The key benefits of this pattern can be summarised as follows:</p><ul><li><strong>Reduced Hallucinations:</strong> by presenting factual information from a trusted source into the context window, the chances of hallucination are reduced as the model is ‘grounded’. Without RAG, LLMs are at the mercy of their training data which may contain little or no relevant information for the task you are trying to perform.</li><li><strong>More cost-effective:</strong> training and fine-tuning models on your own data is extremely costly and difficult to do. Conversely, even small very simple models can outperform larger models when integrated with RAG as even the largest models are restricted by their training data.</li><li><strong>More accurate and referenceable responses:</strong> with RAG, responses can be supplemented with direct references to the context which produced the response, thus enabling the user to ‘check the applications homework’.</li></ul><p>RAG has led to a new generation of LLM-based applications which can leverage novel and existing technologies to solve more complex and nuanced problems across a wide array of domains, from natural language understanding and conversation to content creation, information retrieval, and decision-making processes. One of these is the GenAI stack from Docker which combines some fantastic technologies to unlock some new use cases for AI.</p><h3>Core Technologies</h3><p>The GenAI stack incorporates a suite of technologies, each playing a pivotal role in simplifying and enhancing AI application development.</p><h4>Docker</h4><p>At the heart of this stack is Docker, a platform that drastically simplifies the deployment process of AI applications. By containerising applications, Docker ensures consistency across various environments, significantly reducing the “it works on my machine” syndrome that developers often face. For the GenAI stack, this means that applications can be developed consistently and made extremely portable, which is especially important in early-stage development where PoCs can be readily moved and tested by multiple parties without any barriers.</p><h4>Langchain</h4><p>Langchain is a development framework which simplifies the development of LLM-based applications. Its OpenSource Python library provides a convenient set of abstractions to enable LLM applications to be developed in a context-aware manner. It also provides a basic runtime for combining these components into chains and agents.</p><h4>Ollama</h4><p>Ollama is a lightweight tool for running open-source LLMs locally. Ollama’s release has been a huge step forward for LLM application development, as large expensive models are no longer needed to run in a cloud environment. Whilst these models do not have the same performance as large models like GPT-4, they allow for applications to be developed quickly and run in a local environment to prove out use cases inexpensively.</p><h4>Neo4j</h4><p>Neo4j’s technology has become the industry standard for graph databases. It excels in scenarios where relationships and connections between data points are as crucial as the data itself. Neo4j enables developers to model, store, and query data in a way that reflects real-world complexities, making it an ideal choice for AI applications that require a deep understanding of relational context, such as recommendation systems or knowledge graphs.</p><h3>Why Docker&#39;s GenAI Stack Matters</h3><p>This is an important release for the industry as this combination of technologies opens the door for novel use cases and efficient development of LLM-based applications. By reducing the barrier to entry for development, more developers can get acquainted with the technology and experiment quickly with new patterns to push the industry forward.</p><p>Some of the key benefits of the GenAI stack, and why they are so important are listed below:</p><p><strong>Pre-configured LLMs:</strong> By providing preconfigured LLMs developers can quickly get up to speed developing with these models. Further to this, being able to run LLMs locally drastically reduces the cost associated with development.</p><p><strong>Neo4j backend:</strong> Neo4j serves as the default database, offering graph and native vector search capabilities. This assists in revealing patterns and connections within data, thereby significantly improving the efficiency and precision of AI/ML models. Additionally, Neo4j functions as a persistent memory store for these models, which can serve as a knowledge graph to ground the LLMs for more precise outcomes.</p><p><strong>LangChain Management:</strong> LangChain enhances interactions among the LLM, your application, and the database, supported by a strong vector index. It provides a foundation for building applications driven by LLMs, featuring LangSmith, a novel approach for debugging, testing, evaluating, and overseeing your LLM-based applications.</p><p>By applying these key features, a plethora of different uses can be implemented. In the next section, I will explore the design pattern and some potential real-world use cases.</p><h3>A Simple Yet Powerful Design Pattern</h3><p>Whilst the components of the stack can be combined in different ways, a simplistic view of how these elements can be combined is shown in Figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pougF_i_T_xzMRJ8" /><figcaption>Figure 3 — Basic Design Pattern</figcaption></figure><p>This application could be run in the following way:</p><ul><li>An Agent Application serves as an interface where a user can input a natural language request via a chat interface.</li><li>This request is passed through an embedding model to create a vector representation of the request. Vectors are lists of numbers that can represent an object that has a direction and length. Comparing vector embeddings and determining their similarity is an essential part of semantic search, recommendation systems, anomaly detection, and much more.</li><li>A vector similarity search can be performed against the vector index of Neo4j. This means that for the data stored in your graph database, you could select one or more fields to be a vector index.</li><li>From the similarity search, you could get implicit results from the graph, for example:</li></ul><blockquote>Given a user description, return movies from the graph database with similar descriptions.</blockquote><blockquote>Given a user question, find a similar question from a graph containing forum questions.</blockquote><blockquote>Given a user paragraph, find similar style writing from a graph containing blog articles.</blockquote><ul><li>Once implicit results are obtained using the vector index, explicit results can be returned from the graph containing meta-data or enriched data from the knowledge graph.</li><li>These results can be combined in an LLM application with any other prompt engineering techniques to create a prompt to be sent to the LLM.</li><li>The LLM will produce natural language results which can be returned to the user leveraging domain-specific context obtained from the knowledge graph, drastically enhancing the quality of the response.</li></ul><p><strong>N.B</strong> A visualisation of vector similarity is shown in Figure 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*XLnYMBkUg0U0hbBRi539eg.png" /><figcaption>Figure 4 — Visual Representation of Vector Similarity</figcaption></figure><p>A similarity score between vectors is normally some function of the Euclidean distance or angle between the vectors in multi-dimensional space. Vectors that are ‘closer’ together have a higher score and will occur in similar semantic contexts i.e. in this case ‘car’ and ‘engine’ would be similar in high-dimension vectors but ‘sandwich’ would not be similar to either.</p><h3>Theory to Reality: Real-World Applications</h3><p>Having discussed how the elements of the stack can be applied, it is now possible to see how this design pattern can be applied to many different types of problems. In this section, I will briefly discuss some potential ways this pattern could be used.</p><h4>StackOverflow Augmented Chatbot</h4><p>When first developing on the GenAI stack, an out-of-the-box example is provided ready for immediate use. In this example, the application allows the user to import data from StackOverflow into Neo4j to create a knowledge graph of questions and answers with a given tag. The user can then use the application to ask programming questions.</p><p>Following the above design pattern, the user&#39;s question is then converted to a vector and a similarity search is performed to find a subset of similar questions to the users in StackOverflow. These are then returned as part of a RAG pattern to the LLM to enable it to produce a high-quality response with context taken directly from StackOverflow.</p><h4>Personalised Educational Content Generator</h4><p>This application would utilise the GenAI stack to create a dynamic, personalised learning experience for users. By integrating data from educational resources into Neo4j to form a knowledge graph, the application could provide customised learning materials based on the user&#39;s proficiency, interests, and learning goals.</p><p>A user inputs their learning objective or asks a specific question through an Agent Application interface. The request is processed into a vector using an embedding model, representing the user&#39;s query in a form that can be compared for similarity. Vector similarity search in Neo4j identifies related educational content, topics, or concepts stored in the knowledge graph. Based on the search results, a curated list of topics, reading materials, and practice exercises are generated. This list is tailored to the user’s current understanding and learning trajectory.</p><p>The LLM application could augment these materials by adding explanations, and summaries, or generating quiz questions, making the learning process more interactive and personalised. The user could also receive a comprehensive, custom learning plan that adapts over time as their knowledge and goals evolve.</p><h4>The Future of Data Discovery: The Revolutionary Data Catalogue Assistant…</h4><p>Unfortunately, you’re going to have to come back for Part II of this series to learn more about this one, where I will be taking you through a practical example of how an LLM-based Data Catalogue assistant can be created to reduce the barrier for non-technical users to understand their data estate… <strong>watch this space!</strong></p><h3>Navigating the Challenges: From Concept to Production</h3><p>While the GenAI stack offers a groundbreaking approach to AI development with its emphasis on privacy, minimal development requirements, and low operational costs, it presents certain challenges that need consideration.</p><p>Primarily designed for proof of concept (PoC) stages, the stack&#39;s reliance on local LLMs raises questions about performance compromises. While these local LLMs ensure privacy and contribute to significantly lower running costs, they cannot handle the demands of production workloads. Furthermore, the integration of Neo4j Knowledge Graphs, although powerful for data organisation and retrieval, introduces additional complexities in scaling from PoC to full-scale production environments. Addressing these challenges is crucial for organisations looking to harness the full potential of the GenAI stack in their operational framework.</p><h3>Conclusion: Embracing the Next Wave of AI Innovation</h3><p>In summary, the GenAI stack marks a significant development in the world of RAG applications. The layering of different technologies in the space has created a novel template which will surely be the basis for many enterprise applications in the years to come. Whilst, it is currently limited to a containerised solution, the building blocks are there for migration to production-grade cloud infrastructure. This will enable solutions to scale to larger data volumes and user bases.</p><p>In the next part of the series, I will be looking at how we can design safe LLM applications and apply the pattern discussed in a technical walk-through of a Data Catalogue assistant which will bring a lot of this theory to life.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2b52445c909" width="1" height="1" alt=""><hr><p><a href="https://blog.mesh-ai.com/dockers-genai-stack-pioneering-the-new-frontier-of-ai-development-part-i-a2b52445c909">Docker’s GenAI Stack: Pioneering the New Frontier of AI Development — Part I</a> was originally published in <a href="https://blog.mesh-ai.com/">Mesh-AI Technology &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Rapid Data Quality Assessment with ChatGPT’s Code Interpreter]]></title>
            <link>https://blog.mesh-ai.com/unlocking-rapid-data-quality-assessment-with-chatgpts-code-interpreter-520a1350c76e?source=rss-468ea54eadf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/520a1350c76e</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Daniel Harrington]]></dc:creator>
            <pubDate>Tue, 12 Sep 2023 08:49:21 GMT</pubDate>
            <atom:updated>2023-09-12T08:49:21.346Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0X7uQ3zO1Vl5tak8Auq0uw.png" /></figure><h3>Introduction</h3><p>In 2023 it seems that you cannot talk seriously about modern business, without mentioning the word ‘data’. It has become undeniably associated with any modern approach to driving value and insight into every corner of business regardless of the industry in which it is applied.</p><p>Accordingly, the demand for ‘data’ has increased exponentially across the board with company-wide data strategies becoming commonplace. This, combined with the accessibility and scalability of cloud computing, has meant that we are producing more data than ever with a predicted 120 Zettabytes to be produced in 2023 alone.</p><p>However, as anyone who works in the industry knows, quantity definitely does not mean quality. Collecting data is the easy bit, but generating meaningful insights and products that use this data borders on the impossible for teams.</p><p>In this article, I will be exploring one of the main blockers that prevent teams from getting from Raw data to the utopia of deployed ML models giving accurate and timely insights: <strong>Data Quality</strong>.<strong> </strong>Further to this, I will be demonstrating how one of the most revolutionary technologies of our generation can massively reduce the headache surrounding this divisive subject and accelerate your path from data discovery to business value.</p><h3>What Do We Mean By Data Quality?</h3><p>Data quality can mean different things to different people and when we talk about how ‘good’ or ‘bad’ data is, it can vary a lot based on context. It is important to always be objective and there are various measures that can be used to describe the quality of the data such as summary statistics, number of missing values and outliers to name but a few. Nevertheless, metrics alone do not give a concrete answer to whether data is good, so fundamentally <strong><em>the gap between the expectations and reality </em></strong>is critical for determining data quality.</p><p>Let’s say you are an e-commerce company looking to use data to make more informed decisions about marketing campaigns and inventory. You record all online transactions and you expect that each transaction contains information like transaction ID, price and transaction date. Upon inspection of recent data you find that less than 0.01% of transactions have missing data, the transaction amount statistics match the financial reports and less than 0.001% of transactions have an anomalous purchase amount. In this scenario, the <strong>reality matches the expectations </strong>of the data and therefore could be confidently used to make informed business decisions. As the expectation gap is small, the data is of good quality and will serve as the foundation for a successful data project.</p><p>Conversely, you may be a health organisation that has received anecdotal feedback about low satisfaction in your clinics, so you look to collect comprehensive feedback from patients about their overall satisfaction via a survey. Upon inspection you find that 20% of the surveys returned incomplete answers, the average wait time was only 2 minutes and a large proportion of results gave contradictory answers. In this scenario the <strong>reality does not match the expectation</strong> so provides little confidence in the efficacy of the data. Here the expectation gap is large, indicating that the data is of bad quality data and would likely lead to an ineffective project if used.</p><p>When beginning any data project it is extremely important to scope out the data available and assess its quality to determine what can realistically be achieved. However, far too frequently, the reality of the situation is not properly considered which leads to problems and delays later in the project long after expectations have already been set. This is where a modern approach to Data Quality assessment can help identify issues earlier and lead to a more informed project scope.</p><h3>The Issue With Data Quality Tools</h3><p>The data space has become inundated with tools to carry out a plethora of different tasks. There are more ETL tools and SaaS platforms than you can shake a stick at, and a lot of them promise the world in terms of solving data quality issues. However, as part of a discovery piece, it is unlikely that you would buy a tooling licence, connect the datasets, and run an ingestion to determine the quality of your data due to the time and effort implications — you would prefer something much quicker and more accessible.</p><p>There are of course Open Source options as well such as Great Expectations. This is a high-quality zero-cost option for many situations as it offers great flexibility, customisation and integrations. Nevertheless, the open-source version does require technical expertise to set up and a fair amount of investigative analysis will be required alongside this to determine specific data quality issues.</p><p>In a space where time is precious, and the cost of hiring expensive technical experts could impact the value of the project, it is important that data quality can be assessed quickly and effectively fed back to the project owners in a digestible manner. Enter ChatGPT.</p><h3>Rapid Data Quality Assessment Case Study</h3><p>Whilst it seems that the hype of ChatGPT has died down somewhat in recent months after the initial shock factor of its release, its power still does not cease to amaze me. One feature that has massively changed the game for GPT is the code interpreter feature. The Code interpreter is an official ChatGPT plugin that pushes the boundaries of what’s possible with AI by enabling data analytics, image conversions, code editing, and much more. Whilst this feature certainly brings the safety of LLMs into question, when used in the correct manner it has the potential to drastically change how we perform tasks like data quality assessment — prepare to be amazed!</p><h4>Warning! Hazardous Materials</h4><p>Whilst this case study will demonstrate the significant power of ChatGPT for use cases I must issue a warning and some caveats:</p><ul><li>The data used here is entirely fictional and bears no relation to any real-world datasets.</li><li>Significant care must be taken when carrying out activities such as uploading data to a 3rd party such as ChatGPT. OpenAi’s approach to GDPR and other data privacy regulations is a source of debate. Their <a href="https://openai.com/api-data-privacy">API Privacy Policy</a> should be consulted before uploading any data.</li><li>It is recommended that any data uploaded is highly obfuscated and any Personally Identifiable information is removed. You want to retain the data structure without risking uploading anything that you would not be happy with being exposed to malicious actors. There are various options for this but a good open-source tool is the Faker library for Python — the documentation for which can be found <a href="https://faker.readthedocs.io/en/master/">here</a>.</li><li>LLMs (Large Language Models) such as GPT can generate hallucinations and provide false information. There should always be a human in the loop to verify findings using these tools.</li></ul><p>There are of course alternatives to ChatGPT as a tool with many companies springing up to provide their take on LLM applications with different safety and privacy policies. These include companies like <a href="https://cohere.com/">Cohere</a>, <a href="https://inflection.ai/">Inflection</a> and <a href="https://www.anthropic.com/">Anthropic</a> which all provide similar offerings and can potentially help you manage your data more securely. For enterprise workloads, there is now also a Microsoft Azure OpenAI Service which allows you to leverage all of the powerful benefits of large pre-trained models (such as GPT-3, 3.5 &amp; 4) with the security of Azure. This service also includes features such as customisation of the models as well as content filtering to moderate how your applications are used.</p><p>Anyway, enough of the boring compliance stuff (kidding), let’s get into the case study.</p><h4>Setting the Scene</h4><p>To demonstrate the power of this feature, I would like to carry out a case study to highlight how this feature could be used to perform rapid data quality assessment and expedite your project in a low-cost manner. Imagine you are part of a local government looking to assess the Air Quality (see what I did there) in a small city — you want to set up an array of sensors around the city and perform predictive analytics to ascertain what times of the day/ year air quality is poor, to help identify potential causes and mitigations.</p><p>When beginning this project you want to ensure you are using high-quality sensors to derive maximum benefit from the project. As such you have acquired some sensors from two different manufacturers in order to compare their performance and make an informed business decision on which to use based on data quality factors.</p><h4>The Data</h4><p>You decide on a setup for this project which uses a simple architecture for the data quality assessment. The two sensor arrays are placed in similar locations around the city, and they connect via cellular network to a cloud database which collects the data from the arrays.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gId9ArThSZDxjn_k" /><figcaption>Figure 1 — Data Collection Architecture</figcaption></figure><p>The data recorded by the sensors goes to a table called Sensor Data, the schema of which can be seen in Table 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bLE0ViLDPyoHW5-0Nj6dCg.png" /><figcaption><em>Table 1 — Sensor Data Schema</em></figcaption></figure><h4>Rapid Data Quality Assessment</h4><p>As a local government, time and funds would likely be limited, therefore after a month, you would ideally like to analyse that data and begin to gain some insights from it. Ordinarily, this would be a long and arduous process for an analyst to perform but with our new virtual assistant, the process can be done in a matter of minutes.</p><p>To get started with rapid data quality assessment simply start a new chat in ChatGPT, select GPT- 4 as the model and select the code interpreter from the drop-down to begin the analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*v8ffP26jHLeQCJ8Q3GAgsA.png" /><figcaption>Figure 2 — Let’s get started</figcaption></figure><p><strong>Providing the Hazardous Materials section has been consulted</strong>, the data you have collected can be easily uploaded to the code interpreter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uGoD-OQ-3WAN7T1B" /><figcaption><em>Figure 3 — A simple request to the model</em></figcaption></figure><p>A simple request is provided, but with an intuitive system such as GPT, this is really all you need to obtain some incredible results. I will break down the analysis below.</p><p>Firstly, GPT performs a quick analysis of the dataset. Despite never having ‘seen’ the dataset before, it is able to interpret the data immediately. Figure 4 shows that it understands the schema of the data and what each column represents with a clear description of each.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lHVwqt2OgitORwSA" /><figcaption><em>Figure 4 — Intuition and understanding displayed</em></figcaption></figure><p>GPT shows an understanding of typical data quality issues and outlines factors that could contribute to poor-quality data with no prompting. Immediately it shows that the PM2.5, PM10, and CO2_Level columns contain some non-numeric values — this is not expected and worth investigating further. It fails to recognise however that the Sensor_ID is just an ID therefore summary statistics are not going to be useful for this showing its potential fallibility. This initial analysis can be seen in Figure 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4xXJkjuoTpsOJkUz" /><figcaption><em>Figure 5 — Initial Data Quality Issues</em></figcaption></figure><p>As the purpose of this project is to assess which sensor array is best, we can ask GPT to do a direct comparison between the sensors of the two manufacturers to help us make a decision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8sLihEHC77Jw5wzB" /><figcaption><em>Figure 6 — Always be polite</em></figcaption></figure><p>Figure 7 shows that GPT understands that missing data is bad and prevents us from doing good quality analytics and as such dives into a comparison of missing data. The response determines that AeroSenseTech has a lower percentage of missing values overall, indicating you would get a more consistent reading from this array.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1rt1HWVX5Yq8Vbtl" /><figcaption><em>Figure 7 — Comparison of missing values</em></figcaption></figure><p>The next insight is truly astounding, as GPT moves onto data consistency. Here it is looking for instances where PM2.5 is greater than PM10 — why is this I hear you ask? Well as particles with a size less than 2.5 nanometers are a subset of particles with a size less than 10 nanometers the concentration of PM2.5 particles should always be less than or equal to the concentration of PM10 particles. This shows an unprompted understanding of the physical phenomena of particulate measurement and indicates that this is caused by a faulty sensor measurement — a truly profound insight into the data, that could surely be missed by a human interpreter.</p><p>Figure 8 shows that AeroSenseTech sensors record more inconsistent data which could indicate the sensors are less accurate or susceptible to changing environmental conditions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rk2uFwOftNvSPlB3" /><figcaption><em>Figure 8 — Data consistency</em></figcaption></figure><p>The final area of consideration is outliers in the data. Another feature of the code interpreter is that it is able to create plots of the data directly in the app (and also gives you the code!) which can be used for reporting and visual analysis. An example of a box plot to highlight anomalies can be seen in Figure 9. This is a highly powerful feature as reports can be rapidly made and distributed to business users to expedite the decision-making process.</p><p>Here we see that AeroSenseTech has some drastic outliers, indicating that it has produced erroneous measurements. Figure 10 gives us the details on these outliers thus concluding our analysis. We see that AeroSenseTech has a higher number of outliers which could indicate a lower-quality sensor with more incorrect readings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*av3j6YmUADW-nCDo" /><figcaption><em>Figure 9 — GPT generated Box Plot</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*miFsmM8nAgKcwFs4" /><figcaption><em>Figure 10 — A comparison of outliers</em></figcaption></figure><h4>The Decision</h4><p>So the power of GPT to provide rapid data analytics insights into our data has been demonstrated, but (as things stand) there needs to be a human in the loop to make a specific business decision. Luckily GPT comes to the rescue once again to help summarise the findings and make an informed data-based recommendation. It shows its intuition and balance once again by not just blindly offering a recommendation but outlining further considerations that should be made before making the call on which manufacturer to go with. This is where finally human knowledge, intuition and heuristics can be called upon to make a decision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C3eawTmKlEP54Joh" /><figcaption><em>Figure 11 — GPT Makes the Call</em></figcaption></figure><p>To complete the decision-making process, it must be reiterated that LLMs can suffer from various biases and hallucinations. In a technical topic such as data quality, these are less apparent, but it is important to check GPT’s homework. Thankfully another nice feature is that the Code Interpreter allows you to see and copy the code that it’s running. This allows you to sense-check what is being done, as well as copy to your own environment to edit and run as desired.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oe1CD-c6xxqaR_6h" /><figcaption><em>Figure 11 — Checking GPT’s Homework</em></figcaption></figure><h4>The Impact</h4><p>The impact of what has just been demonstrated is undeniable. Having a virtual assistant which is not only capable of performing rapid data quality analytics, but can also act as a domain expert with both technical and business expertise is transformative to a process like data quality assessment.</p><p>Nevertheless, it is important to remain human in this world of AI and remember that whilst these current technologies are extremely powerful, they are not perfect and should be used with caution. When performing data quality analysis using GPT it is paramount that this is used as a foundational tool to enable quicker analysis rather than as a replacement for human judgement. It should be ensured that there is always a human in the loop and that findings are double-checked for errors.</p><p>The foundations of how we operate in the world of Data is changing rapidly and it is important to maintain a competitive advantage. Leveraging these tools can be scary and incredible at the same time, but in the right hands will change the way we work forever.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=520a1350c76e" width="1" height="1" alt=""><hr><p><a href="https://blog.mesh-ai.com/unlocking-rapid-data-quality-assessment-with-chatgpts-code-interpreter-520a1350c76e">Unlocking Rapid Data Quality Assessment with ChatGPT’s Code Interpreter</a> was originally published in <a href="https://blog.mesh-ai.com/">Mesh-AI Technology &amp; Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark & Databricks: Important Lessons from My First Six Months]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/spark-databricks-important-lessons-from-my-first-six-months-d9b26847f45d?source=rss-468ea54eadf8------2"><img src="https://cdn-images-1.medium.com/max/2600/1*0NUhmAP6ON6h3QC03ih13g.jpeg" width="4700"></a></p><p class="medium-feed-snippet">&#x2026;and how you can avoid them</p><p class="medium-feed-link"><a href="https://medium.com/data-science/spark-databricks-important-lessons-from-my-first-six-months-d9b26847f45d?source=rss-468ea54eadf8------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/spark-databricks-important-lessons-from-my-first-six-months-d9b26847f45d?source=rss-468ea54eadf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9b26847f45d</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[getting-started]]></category>
            <dc:creator><![CDATA[Daniel Harrington]]></dc:creator>
            <pubDate>Wed, 23 Sep 2020 17:31:14 GMT</pubDate>
            <atom:updated>2020-09-25T14:46:04.967Z</atom:updated>
        </item>
    </channel>
</rss>