<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hazzlenut on Medium]]></title>
        <description><![CDATA[Stories by Hazzlenut on Medium]]></description>
        <link>https://medium.com/@hazzlenut?source=rss-18402a6b874------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TjqJ7wTeFjkBwueoGaZ9rA.png</url>
            <title>Stories by Hazzlenut on Medium</title>
            <link>https://medium.com/@hazzlenut?source=rss-18402a6b874------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:20:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hazzlenut/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI’s Impact on Internet Search and Economy]]></title>
            <link>https://medium.com/architectural-ai/ais-impact-on-internet-search-and-economy-9c3f78d90b82?source=rss-18402a6b874------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c3f78d90b82</guid>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[perplexity]]></category>
            <category><![CDATA[ads]]></category>
            <dc:creator><![CDATA[Hazzlenut]]></dc:creator>
            <pubDate>Thu, 13 Jun 2024 18:23:07 GMT</pubDate>
            <atom:updated>2024-11-13T10:57:04.500Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*2E2PoFQB6ErKKjPk2xKYdg.png" /></figure><p>The rise of AI-powered search engines like Perplexity AI and Google’s AI Overview is transforming how people discover information online, significantly impacting the internet economy. AI’s ability to provide users with synthesized answers from multiple sources threatens the traditional attention-based ad revenue model, potentially resulting in billions of dollars in lost revenue for publishers.</p><h3>Impact on Ad Revenue</h3><p>The emergence of AI-powered search engines like Google’s AI Overview is poised to drastically alter the digital advertising landscape. As AI delivers concise, synthesized answers directly within search results, the traditional model of attracting users to websites through search engine optimization (SEO) is under threat.</p><p>Studies by Raptive, an ad sales company representing various publishers, suggest that Google’s Search Generative Experience (SGE) could lead to a 20% to 60% decline in search traffic for publishers. This significant drop in traffic is expected to translate into a sharp decrease in digital ad revenue, with estimates suggesting losses of up to $2 billion annually across the publishing industry.</p><p>The shift towards AI-generated search results means that content is no longer the primary differentiating factor in online visibility. Instead, search engines will increasingly rely on AI to generate results, favoring websites and brands that produce engaging, valuable content. Businesses heavily dependent on search engine traffic for ad revenue must adapt their strategies, focusing on creating unique, high-quality content to capture user attention.</p><p>Publishers and content creators are exploring alternative monetization strategies to adapt to this changing landscape. Some are partnering with AI companies like OpenAI, which offer payments for access to their content for training AI models. Others are considering subscription models, data monetization, and microtransactions to diversify their revenue streams and reduce reliance on traditional ad-based models.</p><h3>AI-Driven Content Shift</h3><p>As AI-powered search engines gain prominence, the focus of content creation is shifting from traditional SEO strategies to capturing the attention of AI algorithms. This shift requires content creators and publishers to adapt their strategies to remain competitive in an AI-driven search landscape.</p><p>One key aspect of this transition is the emphasis on creating high-quality, engaging content that provides unique value to users. Rather than focusing on keyword optimization and producing large volumes of content, publishers must prioritize creating content that is truly informative, well-researched, and valuable to their target audience. This approach not only helps capture the attention of AI algorithms but also fosters trust and loyalty among users.</p><p>Moreover, as AI search engines become more adept at understanding context and user intent, content creators must focus on providing comprehensive, in-depth coverage of topics. This may involve exploring niche areas or addressing gaps in the existing information landscape. By offering unique insights and perspectives, publishers can differentiate themselves and increase their chances of being featured in AI-generated search results.</p><p>Another crucial aspect of the AI-driven content shift is the importance of brand authority and reputation. As AI algorithms increasingly rely on factors like user engagement, social shares, and brand mentions to determine the relevance and credibility of content, building a strong brand presence becomes essential. Publishers must invest in brand marketing and public relations efforts to establish themselves as trusted sources of information in their respective fields.</p><p>Additionally, the rise of AI-powered search is likely to accelerate the adoption of multimedia content formats, such as videos, podcasts, and interactive experiences. As AI algorithms become more sophisticated in analyzing and indexing non-textual content, publishers who embrace these formats and create engaging, informative multimedia content may gain a competitive edge in the AI-driven search landscape.</p><h3>New Monetization Strategies</h3><p>As AI-powered search engines disrupt traditional ad-based revenue models, publishers and content creators are exploring new monetization strategies to adapt to the changing landscape. These strategies aim to diversify revenue streams and reduce reliance on search engine traffic and advertising income.</p><p>One emerging trend is partnering with AI companies to monetize content. For example, OpenAI has been offering publishers between $1 million and $5 million annually to access their content for training its AI models. This arrangement provides a new revenue stream for publishers while helping AI companies improve their algorithms and services.</p><p>Another potential monetization strategy is the adoption of subscription-based models. By offering premium, high-quality content behind a paywall, publishers can generate a steady income stream from loyal readers who value their expertise and unique insights. This approach can help insulate publishers from the volatility of ad-based revenue and foster a more direct relationship with their audience.</p><p>Data monetization is also gaining traction as a viable strategy for publishers. By collecting and analyzing user data, publishers can create valuable insights and target audiences for advertisers and other businesses. This approach requires a delicate balance between data privacy and monetization, but when done responsibly, it can provide a significant revenue stream.</p><p>Microtransactions and pay-per-article models are another option for publishers looking to diversify their income. By allowing users to access individual articles or pieces of content for a small fee, publishers can generate revenue from casual readers who may not be willing to commit to a full subscription. This model can be particularly effective for niche or specialized content that appeals to a specific audience.</p><p>Finally, some publishers are exploring innovative revenue models such as sponsored content, affiliate marketing, and e-commerce integration. By partnering with brands to create sponsored content or earning commissions from product sales, publishers can tap into new income streams while providing value to their audience.</p><h3>Sustainable AI Practices</h3><p>As AI-powered search engines become more prevalent, it is crucial for businesses and content creators to adopt sustainable AI practices to ensure the long-term viability of their online presence and the overall health of the internet ecosystem. Some key sustainable AI practices include:</p><p>1. <strong>Prioritizing Transparency and Accountability</strong>: Companies developing AI search engines should be transparent about their algorithms, data sources, and content moderation practices. This helps build trust with users and allows for greater accountability in addressing potential biases or inaccuracies in AI-generated results.</p><p>2. <strong>Ensuring Data Privacy and Security</strong>: As AI search engines rely heavily on user data to improve their algorithms, it is essential to implement robust data privacy and security measures. This includes obtaining user consent for data collection, anonymizing data where possible, and implementing strict security protocols to prevent data breaches or misuse.</p><p>3. <strong>Promoting Diversity and Inclusion in AI Development</strong>: To minimize the risk of biases in AI search results, it is important to foster diversity and inclusion in the teams developing these algorithms. This includes hiring diverse talent, conducting regular bias audits, and incorporating feedback from a wide range of stakeholders.</p><p>4. <strong>Collaborating with Content Creators and Publishers</strong>: AI search engine developers should work closely with content creators and publishers to ensure that their algorithms accurately attribute content and provide fair compensation for the use of their intellectual property. This can involve developing new revenue-sharing models or offering tools to help content creators optimize their content for AI-driven search.</p><p>5. <strong>Investing in AI Literacy and User Education</strong>: As AI-powered search becomes more prevalent, it is crucial to invest in AI literacy initiatives to help users understand how these systems work and how to critically evaluate the information they provide. This can include developing educational resources, partnering with schools and universities, and promoting media literacy campaigns.</p><p>By adopting these sustainable AI practices, businesses and content creators can help ensure that the rise of AI-powered search benefits all stakeholders in the internet ecosystem. This approach fosters trust, transparency, and collaboration, ultimately leading to a more robust and equitable digital landscape where high-quality content and innovative AI technologies can coexist and thrive.</p><p><strong><em>Consider subscribing to receive more </em>articles<em> like this in your inbox. Cheers!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c3f78d90b82" width="1" height="1" alt=""><hr><p><a href="https://medium.com/architectural-ai/ais-impact-on-internet-search-and-economy-9c3f78d90b82">AI’s Impact on Internet Search and Economy</a> was originally published in <a href="https://medium.com/architectural-ai">Architectural AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Embrace the LLM Mayhem : From RAG to rags.]]></title>
            <link>https://medium.com/architectural-ai/embrace-the-llm-mayhem-from-rag-to-rags-b04409762181?source=rss-18402a6b874------2</link>
            <guid isPermaLink="false">https://medium.com/p/b04409762181</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Hazzlenut]]></dc:creator>
            <pubDate>Mon, 06 Nov 2023 18:32:23 GMT</pubDate>
            <atom:updated>2024-11-13T10:55:39.072Z</atom:updated>
            <content:encoded><![CDATA[<h3>Embrace the LLM Mayhem: From RAG to rags.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kvNdvxqh5Ay6frnN.jpg" /></figure><p>Enterprises are increasingly recognizing the potential of integrating Large Language Models (LLMs) into their mission-critical applications. However, the unpredictable behavior of LLMs can lead to challenges such as hallucinations, resulting in inaccuracies and errors that are detrimental to the precision, comprehensibility, and dependability of these applications.</p><p>To overcome these challenges, the prominent solution is Retrieval Augmented Generation (RAG), a technique that anchors LLMs in factual information. Developers often opt for vector databases to implement RAG due to their simplicity and effectiveness. Some of the most used vector databases right now in the industry are Pinecone, Chroma, etc.</p><p>In this article, we will delve into the use of vector databases for RAG and highlight some of the associated issues. We will also explore the potential of Graph databases (Graph DBs like Neo4j and Janus DB) to address these problems when customized to specific requirements. Additionally, we will unveil an innovative approach to rescue enterprises from the challenges posed by LLMs.</p><p>Now, let’s address these issues one by one:</p><h3>1. Tackling Complex Questions</h3><p>As the complexity of a question increases, it becomes progressively challenging for a vector database to swiftly and accurately provide results. Adding more subjects to a query compounds the difficulty in locating the desired information.</p><p>For instance, both a knowledge graph and a vector database can readily answer a question like “Who is the CEO of my company?” However, when confronted with a more intricate query such as “Which board meetings in the last twelve months had at least two members abstain from a vote?” a knowledge graph outperforms a vector database. The vector database is likely to retrieve an answer from within the vector space, often lacking precision, while a knowledge graph seeks and delivers precise information by traversing a relationship-connected graph.</p><h3>2. Obtaining Comprehensive Responses</h3><p>Vector databases are prone to providing incomplete or irrelevant results when delivering answers, primarily due to their reliance on similarity scoring and predefined result limits.</p><p>For example, when you inquire, “List all the books written by John Smith,” a vector database may yield one of the following outcomes:</p><p>— An incomplete list of titles (result limit set too low).</p><p>— All titles by John Smith and some by other authors (result limit set too high).</p><p>— The exact answer (result limit appropriately configured).</p><p>Since developers cannot predict the ideal result limit for every possible query, obtaining a precise answer from a vector database is often nearly impossible. In contrast, knowledge graphs, characterized by direct relationships among entities, consistently retrieve and provide the exact answer. In this scenario, a knowledge graph query would return all books authored by John Smith and nothing more.</p><h3>3. Obtaining Credible Responses</h3><p>Vector databases may connect two factual pieces of information and deduce something inaccurately.</p><p>For example, if you were to ask, “Who is on the product management team?” a vector database might erroneously conclude that someone is on the product team due to frequent commenting access to documents produced by the product team, leading to their name being included in the results. In contrast, a knowledge graph, which employs nodes and relationships to identify organizational connections, would exclusively list those who are members of the product team. Knowledge graph queries follow a logical flow of interconnected information, ensuring consistently accurate and explicable responses.</p><h3>4. Rectifying LLM Hallucinations</h3><p>Knowledge graphs provide a human-readable representation of data, whereas vector databases offer an opaque, black-box approach.</p><p>For example, when a member of the product team is misidentified, a vector database cannot identify the facts it used to generate the erroneous inference. This makes it challenging to rectify or even comprehend the source of the error. In contrast, knowledge graph users can easily identify and correct such inaccuracies resulting from LLM inferences. Knowledge graphs offer full transparency, enabling the identification of misinformation in data, retracing the query pathway, and making specific corrections to enhance LLM accuracy. Vector databases, in contrast, provide minimal transparency and lack the capacity for precise corrections.</p><h3>The Ideal Solution: A Fusion of Vector and Graph Databases</h3><p>So, what’s the ideal solution for integrating with RAG systems? The answer lies in combining the strengths of a vector database and a Graph database. A graph database capable of storing information in an embedding vector format can effectively address the limitations of a vector database.</p><p>In conclusion, the integration of Large Language Models (LLMs) into mission-critical applications presents both remarkable opportunities and formidable challenges. The unpredictable nature of LLMs can lead to inaccuracies and errors, which is a matter of serious concern for enterprises seeking accuracy, comprehensibility, and reliability in their applications.</p><p>However, the solution lies in the fusion of Retrieval Augmented Generation (RAG) techniques, combining the strengths of vector databases and Graph databases. While vector databases excel in simplicity and efficiency, they often fall short in handling complex queries and ensuring the completeness, credibility, and correctness of responses. On the other hand, Graph databases provide a structured approach to data representation, ensuring accurate, understandable, and transparent results.</p><p>The combination of these two database technologies represents a promising path forward. By leveraging the strengths of both vector databases and Graph databases, we can overcome the challenges posed by LLMs and enhance the accuracy and reliability of our applications.</p><p>In the coming years, we anticipate the emergence of open-source database solutions that embrace this hybrid approach, offering a robust foundation for enterprises to navigate the LLM landscape effectively. As we continue to develop and refine these solutions, we can look forward to a future where LLMs seamlessly integrate with our applications, providing both powerful capabilities and dependable results.</p><p><strong><em>Consider subscribing to receive more articles like this in your inbox. Cheers!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b04409762181" width="1" height="1" alt=""><hr><p><a href="https://medium.com/architectural-ai/embrace-the-llm-mayhem-from-rag-to-rags-b04409762181">Embrace the LLM Mayhem : From RAG to rags.</a> was originally published in <a href="https://medium.com/architectural-ai">Architectural AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[F1 Score is Overrated. Instead, use this!]]></title>
            <link>https://medium.com/architectural-ai/f1-score-is-overrated-instead-use-this-96fbfc5e3b2?source=rss-18402a6b874------2</link>
            <guid isPermaLink="false">https://medium.com/p/96fbfc5e3b2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[accuracy]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[f1]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Hazzlenut]]></dc:creator>
            <pubDate>Thu, 17 Aug 2023 05:13:34 GMT</pubDate>
            <atom:updated>2024-10-09T07:19:40.263Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*etOZJy9hsaMTaYZc.jpg" /></figure><p>When it comes to evaluating the performance of machine learning models, the F1 score has long been hailed as the go-to metric. However, I dare to challenge the status quo and argue that the F1 score is overrated. In this article, we will explore an alternative metric that deserves more attention and recognition: <strong>the Matthews Correlation Coefficient (MCC).</strong></p><p>Brace yourself as we dissect the F1 score, uncover its limitations, and unveil the hidden gem that is MCC.</p><h4>The F1 Score: A Flawed Hero</h4><p>Ah, the F1 score — the metric that seems to be on every machine learning practitioner’s lips. Calculated by harmonizing precision and recall, the F1 score is intended to strike a balance between the two. But does it always deliver accurate results? Let’s find out.</p><p><strong>Precision, Recall, and a Harmonious Blend<br></strong>To understand the F1 score, we need to take a step back and look at its components: precision and recall. Precision measures how many correctly predicted positive instances out of all predicted positive instances. Recall, on the other hand, quantifies how many positive instances were correctly identified out of all actual positive instances. Both metrics have their merits, but blending them together might not always be the best idea.</p><p><strong>Example: When F1 Score Falls Short<br></strong>Consider a hypothetical scenario where you’re building a model to detect a rare disease. Suppose we have a dataset with the following confusion matrix for a binary classification problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*VlGlDOqzPfzbLiamXZyT3w.png" /></figure><p>In this scenario, the dataset represents a medical test for a rare disease where only a small number of positive cases exist compared to negative cases. The confusion matrix suggests that the model has a high True Negative (TN) rate but a low True Positive (TP) rate. Here are the calculations for precision, recall, and F1 score:</p><ul><li>Precision = TP / (TP + FP) = 25 / (25 + 10) ≈ 0.714</li><li>Recall = TP / (TP + FN) = 25 / (25 + 5) = 0.833</li><li>F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.769</li></ul><p>In this case, the F1 score is around 0.769, which might seem like a reasonable performance. However, the low number of True Positives is a concern, especially in the context of medical diagnostics. The rare nature of the disease means that even a small number of missed positive cases can have significant real-world consequences.</p><p>F1 score, are you really the superhero we thought you were?</p><h4>Introducing the Matthews Correlation Coefficient (MCC)</h4><p>Amidst the F1 score’s limitations, a silent hero awaits its time to shine: the Matthews Correlation Coefficient. Named after its creator, Brian Matthews, this coefficient ranges from -1 to +1, providing a holistic measure of a model’s performance. It takes into account true positives, true negatives, false positives, and false negatives, all wrapped up in a single number. Isn’t that impressive?</p><p>Formulating the MCC: Let’s break down the MCC formula step by step:</p><ol><li>Calculate the true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) from your model’s predictions.</li><li>Use these values to calculate the numerator: (TP * TN) — (FP * FN).</li><li>Compute the denominator: √((TP + FP) * (TP + FN) * (TN + FP) * (TN + FN)).</li><li>Divide the numerator by the denominator to obtain the MCC.</li></ol><pre>MCC = (TP * TN - FP * FN) / sqrt((TP + FP) * (TP + FN) * (TN + FP) * (TN + FN))</pre><p>Let’s take the same example above and see how MCC is better than F1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*j7RrzwJAL0Q80EXgE-baKg.png" /></figure><p>Using the values from the confusion matrix:</p><ul><li>TP = 25</li><li>TN = 9000</li><li>FP = 10</li><li>FN = 5</li></ul><p>Substituting these values into the MCC formula:</p><pre>MCC = (25 * 9000 - 10 * 5) / sqrt((25 + 10) * (25 + 5) * (9000 + 10) * (9000 + 5))<br>MCC ≈ 0.517</pre><p>The MCC value is approximately 0.517.</p><p>The Matthews Correlation Coefficient considers all four elements of the confusion matrix and considers the balance between true positives, true negatives, false positives, and false negatives. Unlike the F1 score, the MCC considers the relative sizes of different classes in the dataset and is suitable for imbalanced datasets or situations where the consequences of false positives and false negatives are not equal.</p><p>As we bid farewell to the F1 score, let’s acknowledge its merits as a starting point for evaluation. However, in the ever-evolving landscape of machine learning, we must embrace alternatives that can better capture the nuances of our models’ performance. The Matthews Correlation Coefficient (MCC) shines as a comprehensive metric, accommodating imbalanced datasets and considering all possible outcomes. So, let’s set aside our F1 score goggles and give MCC the recognition it deserves.</p><p><strong><em>Consider subscribing to </em></strong><a href="http://karam.shine@gmail.com/"><strong><em>email</em></strong></a><strong><em> notifications to receive more </em>nuts<em> like this in your inbox. Cheers!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96fbfc5e3b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/architectural-ai/f1-score-is-overrated-instead-use-this-96fbfc5e3b2">F1 Score is Overrated. Instead, use this!</a> was originally published in <a href="https://medium.com/architectural-ai">Architectural AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Challenges Associated With Building Products Using Large Language Models (LLMs)]]></title>
            <link>https://medium.com/architectural-ai/the-challenges-associated-with-building-products-using-large-language-models-llms-a0573dcfdf5f?source=rss-18402a6b874------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0573dcfdf5f</guid>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Hazzlenut]]></dc:creator>
            <pubDate>Sat, 03 Jun 2023 09:02:32 GMT</pubDate>
            <atom:updated>2023-06-08T02:00:01.131Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2ELf3TZ8-TfzBfJs.png" /></figure><p>Hello there.</p><p>I hope you all are doing well. It has been quite some time since my last publication on Medium. Recently, I’ve dedicated some time to thoroughly explore the newly found love of every serial startup founders out there, popularly known as Large Language Models, or LLMs.</p><p>Now, I don’t want to talk about what are LLMs, how they work, how to build your own LLMs using HuggingFace and LangChain etc. You will find plenty of them with a simple google search.</p><p>Rather, I want to put some light into a lesser discussed topic regarding the LLMs : <strong><em>The challanges of putting LLMs into work</em></strong>.</p><p>In recent years, large language models have emerged as a groundbreaking technology with significant potential across various industries. These models, such as GPT-3.5, have demonstrated remarkable abilities in natural language processing, opening up new avenues for innovation.</p><p>However, while these models offer immense opportunities, they also present economic and operational challenges that businesses must navigate to harness their full potential. In this article, we will explore the key hurdles faced when building products using large language models and discuss strategies to overcome them.</p><p>Now, you may ask, why we need to build products using LLMs when I can simply subscribe to OpenAI chatGPT. The simple answer is <strong>Data privacy and IP protection</strong>.</p><p>Are you comfortable sharing your personal data to Google, Microsoft, Meta etc? I am sure you are not. Infact, from time to time, we have seen so many data privacy related ‘renaissance’. But reccently, in last 6 months, these magical LLMs (chatGPT, BARD etc) controlled by few giants have gained so much popularity and users have almost forgot about data privacy and IP protection. People are passing confidential information to these ‘magical models’ without understanding how they work.</p><p>My point is, what happened in last 6 months, will not continue to happen in the long run. There will be a ‘renaissance’ again and then we have to build purpose driven ‘<strong><em>private chatGPT</em></strong>’. Let’s deep dive.</p><h4>I. Availability of Infrastructure :</h4><p>Yes. A100. That’s what every founder is looking everywhere. A100 GPU card that has 80 GB of GPU memory is the best NVIDIA is offering right now. AWS, Azure, GCP all of them has NVIDIA A100 GPU card powered machines. But the catch is it’s not readily available for everyone. There will be a long thread of emails and long waiting time with customer support explaining the reason, usage, business information, future plan to unlock a machine like that. When you get one, it will start costing you <strong>$8 USD per card per hour</strong> on average.</p><h4>II. Ethical Considerations and Bias Mitigation :</h4><p>Large language models have garnered attention due to their ability to generate human-like text. However, they can also inadvertently perpetuate biases present in the training data, leading to ethical concerns. Addressing bias and ensuring fairness in language models requires dedicated efforts, often involving additional time, resources, and expertise. One of the best practices to reduce bias is <a href="https://huggingface.co/blog/rlhf"><strong>Reinforcement Learning from Human Feedback (RLHF)</strong></a>. Companies must invest in rigorous data screening, algorithmic auditing, and ethical guidelines to mitigate bias. Failure to do so can have long-term consequences, including damage to brand reputation, legal repercussions, and loss of customer trust. Incorporating ethical considerations from the early stages of product development can help prevent such issues and minimize impact. To further reduce the impact, you can definately do effective <strong>prompt enginering</strong>.</p><h4>III. Cost of Training :</h4><p>So after reading the above section, you are convinced that we need to retrain the open source LLMs with purpose driven dataset. One of the primary economic challenges of utilizing large language models is the substantial cost associated with training. Training these models requires significant computational resources and it’s a time-consuming processes. The hardware requirements and energy consumption can strain budgets, particularly for smaller businesses. Moreover, as the demand for computational resources increases, the associated costs can skyrocket. To tackle this challenge, companies can explore strategies for optimizing training pipelines to maximize efficiency and minimize expenses. To give you an approximate idea, reatraining a <strong>12B parameter model would cost around $300 USD</strong>. Higher the no. of parameters, higher the cost. To reduce the retraining cost, you can definately do effective <strong>grounding</strong>.</p><h4>IV. Data Acquisition and Preparation for Retraining :</h4><p>Large language models thrive on vast amounts of high-quality data. Acquiring and preparing such data can present financial and legal hurdles for businesses. Building comprehensive and diverse datasets demands extensive resources, both in terms of time and money. Additionally, the costs associated with data cleaning, preprocessing, and annotation can be significant. Startups and smaller companies often face difficulties in accessing and curating large-scale datasets, potentially limiting their ability to fully utilize language models. Collaborations, partnerships, and open-source initiatives can provide avenues for shared data resources, reducing costs and enabling broader access to high-quality datasets. By the way, if you have just lit an ‘60W vintage bulb’ in your head and go use chatGPT to create training data, blow the bulb immediately. <a href="https://www.lexology.com/library/detail.aspx?g=33bf4b4f-ffd9-4bf1-bfc1-7c790d86a22f">They will sue you</a>.</p><h4><strong>V. Expertise and Talent :</strong></h4><p>Developing products using large language models requires expertise in machine learning, natural language processing, and software development. Acquiring and retaining skilled professionals in these domains can be a considerable challenge. The demand for experts in the field often outpaces supply, leading to high salary expectations and intense competition for top talent. Building a capable team capable of maximizing the potential of language models can strain budgets, particularly for smaller organizations. One solution is to use open-source tools and libraries which can help bridge the expertise gap, enabling developers to leverage pre-existing solutions and reduce development costs.</p><h4>VI. Deployment and Scaling:</h4><p>Deploying large language models in real-world scenarios and scaling them to handle increased user demand can pose financial challenges. The computational requirements and infrastructure costs associated with serving predictions in real-time can be substantial. Companies need to strike a balance between performance and cost to ensure the efficient and cost-effective deployment of language models. Or simply, charge your customers more.</p><p>Leveraging large language models presents businesses with transformative potential but also economic and operational obstacles. To overcome these challenges, a comprehensive strategy is needed, encompassing strategic planning, resource optimization, collaboration, and ethical considerations. This entails addressing costs associated with training and infrastructure, acquiring and preparing data, acquiring expertise and talent, and considering ethical implications. By effectively managing deployment and scaling, businesses can fully tap into the benefits of large language models, leading to innovation across industries.</p><p><strong><em>Consider subscribing to </em></strong><a href="http://hi@hazzlenut.com/"><strong><em>email</em></strong></a><strong><em> notification to receive more </em>Hazzlenuts<em> like this in your inbox. Cheers! Happy weekend.</em></strong></p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>.</em></p><p><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>, </em><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><em>, and </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0573dcfdf5f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/architectural-ai/the-challenges-associated-with-building-products-using-large-language-models-llms-a0573dcfdf5f">The Challenges Associated With Building Products Using Large Language Models (LLMs)</a> was originally published in <a href="https://medium.com/architectural-ai">Architectural AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Tips to Use GIT Like a Senior Developer]]></title>
            <link>https://medium.com/architectural-ai/5-tips-to-use-git-like-a-senior-developer-ff0b7cda9a27?source=rss-18402a6b874------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff0b7cda9a27</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[bitbucket]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Hazzlenut]]></dc:creator>
            <pubDate>Sat, 25 Mar 2023 09:10:44 GMT</pubDate>
            <atom:updated>2023-04-05T14:44:04.048Z</atom:updated>
            <content:encoded><![CDATA[<p>Git is a distributed version control system (DVCS) that allows developers to keep track of changes made to their codebase over time. Git was created by Linus Torvalds in 2005 and has since become one of the most widely used version control systems in the world.</p><p>Git is an essential tool for any developer who wants to work collaboratively, keep track of changes made to their codebase, and maintain a well-organized development workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/0*ayDE5XHx_WU8p3WH.png" /></figure><p>Without further ado, here are five tips for making the best use of Git.</p><h4>1. Commit Frequently and Keep Commit Messages Descriptive</h4><p>Make sure to commit your changes frequently, so that you can keep track of the changes you made to your codebase. Additionally, it’s important to write descriptive commit messages that explain what changes were made in each commit. This helps others who are working on the same project to understand the purpose of each commit, making collaboration much easier.</p><pre>git add .<br>git commit -m &quot;Added feature X, which allows users to do Y&quot;<br><br># Example of a good commit message:<br>feat: add login functionality for users<br><br># Example of a bad commit message:<br>update</pre><h4>2. Use Git hooks to automate tasks</h4><p>Use Git hooks to automate tasks such as running tests or formatting code. This helps ensure that code quality standards are maintained and reduces the risk of human error.</p><pre># pre-commit hook to run tests before committing changes<br># add this to .git/hooks/pre-commit<br><br>#!/bin/sh<br>npm test</pre><h4>3. Use Git Rebase to Keep Your Commits Clean</h4><p>Git rebase is a powerful tool that allows you to rewrite the history of your commits. This can be useful for cleaning up your commit history, especially when you have multiple small commits that could be combined into a larger, more meaningful commit.</p><pre># Start an interactive rebase:<br>git rebase -i HEAD~5<br><br># Edit, delete, or squash commits as needed:<br>pick 12345 Add new feature<br>s 23456 Fix bug<br>pick 34567 Update documentation<br><br># Save and close the file to complete the rebase.</pre><h4>4. Use Git aliases to save time</h4><p>Git has many powerful features, but some commands can be quite lengthy to type out. Git aliases allow you to create shortcuts for frequently used commands.</p><pre># Create an alias for &#39;git status&#39;:<br>git config --global alias.st status<br><br># Use the alias:<br>git st</pre><h4>5. Use Git Tags to Mark Important Releases</h4><p>Git tags allow you to mark important releases of your codebase. This can be useful for keeping track of major milestones or for marking releases that have been thoroughly tested and are ready for production.</p><pre>git tag -a v1.0 -m &quot;Version 1.0 Release&quot;<br>git push --tags</pre><p>I hope these tips help you to make the most of Git in your development workflow!</p><p><strong><em>Consider subscribing to email notification to receive more </em>Hazzlenuts<em> like this in your inbox. Cheers! Happy weekend.</em></strong></p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>.</em></p><p><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>, </em><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><em>, and </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></p><p><strong><em>Interested in scaling your software startup</em></strong><em>? Check out </em><a href="https://circuit.ooo/?utm=publication-post-cta"><strong><em>Circuit</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff0b7cda9a27" width="1" height="1" alt=""><hr><p><a href="https://medium.com/architectural-ai/5-tips-to-use-git-like-a-senior-developer-ff0b7cda9a27">5 Tips to Use GIT Like a Senior Developer</a> was originally published in <a href="https://medium.com/architectural-ai">Architectural AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Data Fabric Can Help Your Company Become Data-Driven in 2023]]></title>
            <link>https://medium.com/architectural-ai/how-data-fabric-can-help-your-company-become-data-driven-in-2023-573812192f55?source=rss-18402a6b874------2</link>
            <guid isPermaLink="false">https://medium.com/p/573812192f55</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[enterprise]]></category>
            <category><![CDATA[entrepreneurship]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Hazzlenut]]></dc:creator>
            <pubDate>Tue, 21 Mar 2023 16:57:17 GMT</pubDate>
            <atom:updated>2023-03-22T07:49:30.789Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l5JVvTFwKq9rq-MQ.png" /></figure><p>In today’s world, data analytics and artificial intelligence (AI) are playing a crucial role in the success of businesses. However, with the exponential growth of data, it has become challenging for organizations to manage and utilize their data effectively. The global effort to create new data privacy laws and increased attention on biases in AI models has added to the complexity. Therefore, businesses need a solution that can help them navigate this new data and AI ecosystem and become data-driven organizations. Enter data fabric.</p><p>Data fabric is an architecture that enables organizations to elevate the value of their enterprise data by providing the right data at the right time, regardless of where it resides. To help businesses become data-driven with data fabric, this article focuses on the four most common entry points aligned with common data and AI stakeholder challenges that businesses face in 2023.</p><h4>Data Governance</h4><p>Data governance is an essential capability of a data fabric. It focuses on the collection, management, and automation of an organization’s data. Automated metadata generation is crucial in turning a manual process into one that is better controlled. This approach helps avoid human error and tags data so that policy enforcement can be achieved at the point of access rather than individual repositories. A data-driven approach makes it easier for business users to find the data that best fits their needs. Moreover, this capability enables business users to quickly and easily find the quality data that conforms to regulatory requirements.</p><h4>Data Integration</h4><p>The rapid growth of data has resulted in not only the issue of siloed data but also a plethora of different repositories across numerous clouds. More data allows the opportunity to provide more accurate data-driven insights, while using multiple clouds helps avoid vendor lock-in and allows data to be stored where it best fits. However, the challenge is the added complexity of data management that hinders the actual use of that data for better decisions, analysis, and AI. A data fabric can help organizations simplify the process of data integration by providing a centralized platform for managing data across multiple environments.</p><h4>Data Science and MLOps</h4><p>AI is no longer experimental. It is becoming mainstream across industries and is proving to be a key driver of enterprise innovation and growth, leading to more accurate, quicker strategic decisions. When AI is done right, enterprises are seeing increased revenues, improved customer experiences, and faster time-to-market, all of which lead to revenue gains and improvements in their competitive positioning.</p><p>The data science and MLOps capability provides data science tools and solutions that enable enterprises to accelerate AI-driven innovation, simplify the MLOps lifecycle, and run any AI model with flexible deployment. With this capability, not only can data-driven companies operationalize data science models on any cloud while instilling trust in AI outcomes, but they are also in a position to improve the ability to manage and govern the AI lifecycle to optimize business decisions with prescriptive analytics.</p><h4>AI Governance</h4><p>Successful AI initiatives require governance, compliance with corporate and ethical principles, laws, and regulations. A data fabric addresses the need for AI governance by providing capabilities to direct, manage, and monitor the AI activities of an organization. AI governance is not just a “nice to have.” It is an integral part of an organization adopting a data-driven culture. It is critical to avoid audits, hefty fines, or damage to the organization’s reputation.</p><blockquote>If you are interested in any of these four use cases, we would be happy to answer any questions you may have as you explore these topics further.</blockquote><p><strong><em>Consider subscribing to email notification to receive more </em>Hazzlenuts<em> like this in your inbox. Cheers!</em></strong></p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>.</em></p><p><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>, </em><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><em>, and </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></p><p><strong><em>Interested in scaling your software startup</em></strong><em>? Check out </em><a href="https://circuit.ooo/?utm=publication-post-cta"><strong><em>Circuit</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=573812192f55" width="1" height="1" alt=""><hr><p><a href="https://medium.com/architectural-ai/how-data-fabric-can-help-your-company-become-data-driven-in-2023-573812192f55">How Data Fabric Can Help Your Company Become Data-Driven in 2023</a> was originally published in <a href="https://medium.com/architectural-ai">Architectural AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Utilizing AWS Auto Scaling and Containerization for Scalable and Cost-Effective Applications]]></title>
            <link>https://medium.com/architectural-ai/utilizing-aws-auto-scaling-and-containerization-for-scalable-and-cost-effective-applications-a50e81507be6?source=rss-18402a6b874------2</link>
            <guid isPermaLink="false">https://medium.com/p/a50e81507be6</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[orchestration]]></category>
            <dc:creator><![CDATA[Hazzlenut]]></dc:creator>
            <pubDate>Sun, 19 Mar 2023 12:42:45 GMT</pubDate>
            <atom:updated>2023-03-19T12:42:45.176Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s digital age, cloud computing has become an essential part of modern businesses. Amazon Web Services (AWS) is a cloud provider that offers a vast range of services tailored to specific needs. In this article, we will discuss how AWS Auto Scaling and containerization can be utilized to create scalable and cost-effective applications.</p><h4>The Benefits of AWS Auto Scaling</h4><p>AWS Auto Scaling is a service that automatically adjusts resources to maintain application performance and optimize costs. It ensures that applications run smoothly and efficiently during high-traffic periods by launching new instances to handle increased workloads. As demand decreases, AWS Auto Scaling automatically terminates instances to save costs, ensuring that you only pay for the resources you require.</p><h4>The Benefits of Containerization</h4><p>Containerizing an application involves packaging the application code, libraries, and dependencies into a container image that can run consistently across different environments. Containerization eliminates cross-platform dependencies and reduces deployment time. Containerization also enables organizations to deploy and manage applications at scale and easily update them without disrupting other components.</p><h4>AWS Services for Auto Scaling and Containerization</h4><p>In this article, we will explore how to leverage AWS’s comprehensive range of services to create a scalable application with ease. Our focus will be on using a simple and hassle-free combination of services to auto-scale a containerized application.</p><ol><li><strong>AWS Fargate</strong>: A serverless compute engine for containers that makes it easy to run and manage Docker containers on AWS. Fargate is used to launch containers in a fully-managed compute environment that automatically scales to meet application needs. You only pay for the resources that your containers use, so you can save money compared to running and managing your own EC2 instances.</li><li><strong>AWS ECS</strong>: A fully-managed container orchestration service that simplifies the deployment, management, and scaling of Docker containers on AWS. ECS integrates with other AWS services, such as Amazon CloudWatch, AWS Identity and Access Management (IAM), Amazon Elastic Load Balancing (ELB), and Amazon Route 53, making it easy to build highly available and scalable containerized applications.</li><li><strong>AWS ECR</strong>: A fully-managed, secure container registry that makes it easy to store, manage, and deploy Docker container images.</li><li><strong>AWS ELB</strong>: A fully-managed service that automatically distributes incoming application traffic across multiple targets, such as containerized applications. ELB provides a highly available, fault-tolerant architecture that can handle millions of requests per second. ELB automatically scales up and down to meet incoming traffic demands and can be configured to detect and route traffic to healthy targets to ensure high availability of applications.</li><li><strong>AWS CloudWatch</strong>: A monitoring and observability service that provides real-time insights into applications hosted in the AWS ecosystem. CloudWatch also offers event management capabilities, which provides information to track and respond to changes in your AWS resources. You can use CloudWatch Events to trigger automated actions, such as scaling up or down a containerized application.</li></ol><h4>CI/CD Automation (Optional)</h4><p>Continuous integration and continuous deployment (CI/CD) automation can be used to automate the process of triggering updates in the production server as soon as a code release occurs. This can be achieved through tools such as Github, AWS, GitLab, and more. CI/CD automation ensures faster deployment of code changes, reduces the risk of errors, and enables organizations to respond quickly to customer needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sJykzyPQ59thjWIB" /><figcaption>Architecture Diagram</figcaption></figure><p>From the diagram, shown above, it can be clearly understood the flowchart of this auto scalling architecture.</p><p><strong><em>Consider subscribing to email notification to receive more </em>Hazzlenuts<em> like this in your inbox. Cheers! Happy weekend.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a50e81507be6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/architectural-ai/utilizing-aws-auto-scaling-and-containerization-for-scalable-and-cost-effective-applications-a50e81507be6">Utilizing AWS Auto Scaling and Containerization for Scalable and Cost-Effective Applications</a> was originally published in <a href="https://medium.com/architectural-ai">Architectural AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Killer Tips to Become a Pro Python Programmer.]]></title>
            <link>https://medium.com/architectural-ai/5-killer-tips-to-become-a-pro-python-programmer-1f3a614e986d?source=rss-18402a6b874------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f3a614e986d</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Hazzlenut]]></dc:creator>
            <pubDate>Sat, 11 Mar 2023 18:47:58 GMT</pubDate>
            <atom:updated>2023-03-11T18:47:58.586Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4NUQp4doW2UgJb9vkwGccw.png" /><figcaption>Credit : DALL-E</figcaption></figure><p>Python is a powerful programming language that has gained popularity in recent years for its simplicity, ease of use, and versatility. Whether you’re a beginner or an experienced programmer, there are always ways to improve your Python skills. In this article, we will share five killer tricks to help you become a pro Python programmer.</p><h4>Tip #1 : Use List Comprehensions for Faster and More Efficient Code</h4><p>List comprehensions are a powerful feature in Python that allow you to create a new list based on an existing list, using a concise and efficient syntax. They can be used to replace traditional for loops and conditional statements, resulting in faster and more readable code.</p><p>Here’s an example of using a list comprehension to create a new list of even numbers from an existing list:</p><pre># Using a for loop<br>numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]<br>even_numbers = []<br>for num in numbers:<br>  if num % 2 == 0:<br>  even_numbers.append(num)<br>print(even_numbers)<br><br># Using a list comprehension<br>numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]<br>even_numbers = [num for num in numbers if num % 2 == 0]<br>print(even_numbers)</pre><p>In this example, the list comprehension is much more concise and easier to read than the traditional for loop. Additionally, it is faster and more efficient, particularly for large lists.</p><h4>Tip #2 : Master Python’s Built-in Functions for Greater Efficiency</h4><p>Python has a wide range of built-in functions that can save you time and effort when programming. These functions perform common tasks, such as sorting, filtering, and mapping, without the need for complex code.</p><p>Here’s an example of using Python’s built-in sorted() function to sort a list of strings in alphabetical order:</p><pre>words = [&quot;apple&quot;, &quot;banana&quot;, &quot;cherry&quot;, &quot;date&quot;, &quot;elderberry&quot;]<br>sorted_words = sorted(words)<br>print(sorted_words)</pre><p>This code sorts the list of words in alphabetical order without requiring a custom function. By mastering Python’s built-in functions, you can write code more efficiently and effectively.</p><h4>Tip #3 : Use Generators</h4><p>Generators are another powerful tool in the hands of a Python programmer. They are a special kind of function that can be used to create iterators. Generators allow you to write iterators in a more concise and elegant way.</p><p>Generators are used when you want to iterate over a large dataset, but you don’t want to load the entire dataset into memory at once. They are also useful when you want to create an infinite sequence of values.</p><p>Example:</p><p>Suppose you want to create an iterator that generates all the Fibonacci numbers. Here’s how you can do it using a generator:</p><pre>def fibonacci():<br>    a, b = 0, 1<br>    while True:<br>        yield a<br>        a, b = b, a + b<br><br>fib = fibonacci()<br>for i in range(10):<br>    print(next(fib))</pre><p>In the above example, the fibonacci() function is a generator that generates the Fibonacci sequence. The yield keyword is used to return a value from the generator. The next() function is used to get the next value from the generator.</p><p>Generators are a powerful tool for creating iterators in Python. They allow you to write iterators in a more concise and elegant way.</p><h4>Tip #4 : Decorators</h4><p>Decorators are a powerful feature of Python that allow you to modify the behavior of functions or classes without changing their source code. A decorator is a function that takes another function as input, and returns a new function that may or may not modify the original function’s behavior.</p><p>Here’s an example of how to define a decorator:</p><pre>def my_decorator(func):<br>    def wrapper():<br>        print(&quot;Before the function is called.&quot;)<br>        func()<br>        print(&quot;After the function is called.&quot;)<br>    return wrapper</pre><p>This decorator defines a new function wrapper that prints a message before and after calling the original function func.</p><p>Here’s an example of how to use the decorator:</p><pre>@my_decorator<br>def say_hello():<br>    print(&quot;Hello!&quot;)</pre><p>This code uses the @my_decorator syntax to apply the my_decorator function to the say_hello function. When the say_hello function is called, it will now execute the code in the wrapper function defined by the decorator, which will print the &quot;Before&quot; message, call the original say_hello function, and then print the &quot;After&quot; message.</p><pre>Before the function is called.<br>Hello!<br>After the function is called.</pre><p>Decorators can be used for a wide range of purposes, including logging, authentication, caching, and more. They are a powerful tool for writing modular and reusable code.</p><h4>Tip #5 : Use Python’s Built-in Modules and Libraries</h4><p>Python has a large and active community that has developed a wide variety of modules and libraries for various purposes, such as scientific computing, web development, data processing, and more. By using these modules and libraries, you can avoid reinventing the wheel and focus on solving the specific problem at hand.</p><p>Here are some examples of built-in modules and libraries that can be useful for various tasks:</p><ul><li>math module: provides mathematical functions and constants, such as sin, cos, pi, and sqrt.</li><li>datetime module: provides classes for working with dates, times, and time intervals.</li><li>re module: provides regular expression matching operations for pattern searching and string manipulation.</li><li>os module: provides a portable way of using operating system dependent functionality, such as reading or writing files, manipulating paths, and executing shell commands.</li><li>csv module: provides classes for reading and writing comma-separated value (CSV) files.</li><li>requests library: provides a simple and elegant way to send HTTP/1.1 requests using Python.</li><li>numpy library: provides a powerful N-dimensional array object, numerical routines, and tools for working with arrays.</li><li>pandas library: provides data structures and tools for data analysis and manipulation.</li><li>matplotlib library: provides a plotting system for creating a wide range of static, animated, and interactive visualizations in Python.</li></ul><p>By familiarizing yourself with Python’s built-in modules and libraries, you can become a more efficient in coding.</p><p><strong><em>Consider subscribing to email notification to receive more </em>Hazzlenuts<em> like this in your inbox. Cheers! Happy weekend.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f3a614e986d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/architectural-ai/5-killer-tips-to-become-a-pro-python-programmer-1f3a614e986d">5 Killer Tips to Become a Pro Python Programmer.</a> was originally published in <a href="https://medium.com/architectural-ai">Architectural AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Governance: Importance and Best Practices]]></title>
            <link>https://medium.com/architectural-ai/data-governance-importance-and-best-practices-b488a18b2017?source=rss-18402a6b874------2</link>
            <guid isPermaLink="false">https://medium.com/p/b488a18b2017</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Hazzlenut]]></dc:creator>
            <pubDate>Thu, 02 Mar 2023 07:24:17 GMT</pubDate>
            <atom:updated>2023-03-02T07:24:17.745Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*OZxYrfgATPK3LqWpTQrp7Q.png" /></figure><p>Data is one of the most important assets of an organization, and it needs to be managed properly to ensure accuracy, reliability, and security. This is where data governance comes in.</p><p>Data governance is the process of managing the availability, usability, integrity, and security of the data used in an organization. In this article, we will discuss the importance of data governance and the best practices to follow.</p><h3>Importance of Data Governance</h3><p>Data governance is crucial for the success of any organization. It ensures that data is being used in a way that is consistent with the organization’s goals and values. Without proper data governance, an organization might face several problems, such as <strong>inconsistent or conflicting data</strong>, <strong>data security breaches</strong>, and <strong>loss of credibility</strong>.</p><p>Inconsistent or conflicting data can lead to costly mistakes, and unreliable or incorrect data can seriously damage an organization’s reputation. Data security breaches can result in the loss of sensitive data, leading to legal and financial consequences. Therefore, it is essential to establish policies, procedures, and standards for managing data within an organization.</p><h3>Best Practices for Data Governance</h3><h4>Establish Clear Roles and Responsibilities</h4><p>Everyone in an organization should know their roles and responsibilities when it comes to managing data. This includes defining who is responsible for collecting, analyzing, and reporting data. Providing the necessary training and resources can help people do their jobs effectively.</p><h4>Define Policies and Procedures</h4><p>Clear policies and procedures should be set for how data is collected, stored, and used within an organization. This will help ensure that everyone is on the same page and that data is being managed consistently.</p><h4>Ensure Data Quality</h4><p>Regularly checking data for accuracy and completeness is important. Data that is incomplete or inaccurate can lead to wrong conclusions and bad decisions. Processes should be put in place to fix any issues that are found.</p><h4>Break Down Data Silos</h4><p>Data silos are isolated areas of data within an organization. These silos can cause problems such as inconsistent or conflicting data. By breaking down these silos and encouraging data sharing and collaboration, a more complete picture of what’s going on within an organization can be obtained.</p><h4>Use the Right Tools</h4><p>Using the right tools can make data governance easier. Tools such as data cataloging and data lineage can help with data discovery and tracking. Data profiling tools can help identify data quality issues.</p><h3>Challenges in Implementing Data Governance</h3><p>Implementing a successful data governance program is not always easy. Some of the challenges an organization might face include:</p><h4>Getting Buy-In from Stakeholders</h4><p>Data governance is a cross-functional effort that requires buy-in from stakeholders across an organization. Convincing stakeholders of the importance of data governance can be a challenge.</p><h4>Dealing with Resistance to Change</h4><p>Introducing a new data governance program can cause resistance from employees who are used to doing things a certain way. It is important to provide adequate training and support to help employees adapt to the new program.</p><h4>Managing Data Quality</h4><p>Ensuring data quality can be a challenging task. Data quality issues can be caused by various factors, such as incorrect data entry, inconsistent data sources, and outdated data.</p><p>Data governance is essential for managing data within an organization. It ensures that data is accurate, up-to-date, and secure. By following best practices such as establishing clear roles and responsibilities, defining policies and procedures, ensuring data quality, breaking down data silos, and using the right tools, organizations can create a successful data governance program. While implementing a data governance program can be challenging, the benefits of having an effective program are numerous.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b488a18b2017" width="1" height="1" alt=""><hr><p><a href="https://medium.com/architectural-ai/data-governance-importance-and-best-practices-b488a18b2017">Data Governance: Importance and Best Practices</a> was originally published in <a href="https://medium.com/architectural-ai">Architectural AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to Data Engineering — Part 3 : ETL , ELT and Data Engineering.]]></title>
            <link>https://medium.com/architectural-ai/intro-to-data-engineering-part-3-etl-elt-and-data-engineering-e685226b5eb?source=rss-18402a6b874------2</link>
            <guid isPermaLink="false">https://medium.com/p/e685226b5eb</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Hazzlenut]]></dc:creator>
            <pubDate>Thu, 02 Mar 2023 07:03:53 GMT</pubDate>
            <atom:updated>2023-03-02T07:03:53.636Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro to Data Engineering — Part 3 : <strong>ETL , ELT and Data Engineering.</strong></h3><blockquote>In the <a href="https://medium.com/@hazzlenutconsultancy/intro-to-data-engineering-part-2-database-data-warehouse-and-data-lake-ac8df9c175c4">second part</a> of this series, we have got the basic understanding of Database, Data Warehouse and Data Lake. In this article, we will try to understand what is ETL, ELT and Data Engineering.</blockquote><p>Do you often get confused by the terms ETL, ELT, and Data Engineering? These terms are sometimes used interchangeably, but there are some key differences.</p><p><strong>What is Data engineering? </strong>It<strong> </strong>is the practice of designing, building, and maintaining the infrastructure that enables organizations to collect, store, process, and analyze large volumes of data. This includes creating and managing data pipelines, designing and implementing data warehouses and data lakes, and ensuring data quality and reliability. As simple as that!</p><p>Now to start, let’s differentiate between <strong>ELT and ETL</strong>.</p><p><strong>ETL stands for Extract, Transform, Load, while ELT is Extract, Load, Transform</strong>. Both processes involve extracting data from multiple sources, but the main difference lies in what happens after extraction. With ELT, the data is loaded into a data lake and then transformed, while with ETL the data is transformed into a structured schema before being loaded into a data warehouse.</p><p>ELT is more favored by analysts and data scientists as they have the flexibility to transform the data directly from the data sources. The below diagram illustrates the difference between ETL and ELT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*9RMQwiDwoie8_3xv.jpg" /><figcaption>Credit : Saras Analytics</figcaption></figure><p>Now, how is Data Engineering different from ETL and ELT? Views on this question may vary. Some developers argue that there is no difference, while others believe there is.</p><p>In our opinion, the difference between ETL/ELT and Data Engineering depends on the approach taken by the developer. ETL/ELT developers often use GUI-based tools like Informatica, while Data Engineers work with programming languages like Scala, Java, or Python, using cloud or open-source frameworks such as Glue or Spark. This gives Data Engineers more flexibility and a deeper understanding of what is happening under the hood.</p><p>In contrast, ETL/ELT developers are limited by the options of their GUI-based third-party tools. To summarize, a Data Engineer is an ELT/ETL developer, but not vice versa.</p><p>Here comes the main question. Which one is better? ELT or ETL?</p><p>To answers this question, we have to look at different factors of both approach. Both has there advantages and disadvantage. We will explore few of them below.</p><h3>ETL</h3><h4>Advantages:</h4><ol><li>Can handle a wide range of data sources and formats.</li><li>Enables data cleansing and normalization.</li><li>Processing occurs before data is loaded into the warehouse.</li><li>Provides a structured approach to data integration.</li></ol><h4>Disadvantages:</h4><ol><li>Requires more upfront data preparation and cleansing.</li><li>Processing can be slower due to data transformation.</li><li>Can be more expensive due to the need for ETL tools.</li></ol><h3>ELT</h3><h4>Advantages:</h4><ol><li>Can handle large and complex data sets.</li><li>Allows for more flexible data analysis.</li><li>Processing occurs on the data warehouse.</li><li>Can be more cost-effective than ETL.</li></ol><h4>Disadvantages:</h4><ol><li>Requires a powerful data warehouse platform.</li><li>Data quality issues can be harder to detect.</li><li>Data governance and security can be challenging.</li></ol><p>Overall, the choice between ELT and ETL depends on the specific needs and constraints of a given organization. ELT is often more suitable for large and complex data sets, and can be more cost-effective, while ETL is typically more appropriate for traditional data warehousing environments and provides a structured approach to data integration. Both approaches have their advantages and disadvantages, and organizations should carefully consider their requirements before making a decision.</p><p><strong><em>That’s all for today. Next article in this series, we will explore some open source as well as cloud based tools and solutions that can be used for Data Engineering.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e685226b5eb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/architectural-ai/intro-to-data-engineering-part-3-etl-elt-and-data-engineering-e685226b5eb">Intro to Data Engineering — Part 3 : ETL , ELT and Data Engineering.</a> was originally published in <a href="https://medium.com/architectural-ai">Architectural AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>