<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Suraj Batuwana on Medium]]></title>
        <description><![CDATA[Stories by Suraj Batuwana on Medium]]></description>
        <link>https://medium.com/@suraj-batuwana?source=rss-b587db5cb48------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*W0igMZvDREpzx2e4mAVfcg.jpeg</url>
            <title>Stories by Suraj Batuwana on Medium</title>
            <link>https://medium.com/@suraj-batuwana?source=rss-b587db5cb48------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 18:57:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@suraj-batuwana/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why AI Doesn’t Know Everything: The Limits of Large Language Models (LLMs)]]></title>
            <link>https://suraj-batuwana.medium.com/why-ai-doesnt-know-everything-the-limits-of-large-language-models-llms-30b0d257f229?source=rss-b587db5cb48------2</link>
            <guid isPermaLink="false">https://medium.com/p/30b0d257f229</guid>
            <category><![CDATA[human-intelligence]]></category>
            <category><![CDATA[leadership-skills]]></category>
            <category><![CDATA[medical]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[humanity]]></category>
            <dc:creator><![CDATA[Suraj Batuwana]]></dc:creator>
            <pubDate>Tue, 22 Jul 2025 00:40:18 GMT</pubDate>
            <atom:updated>2025-07-22T00:40:18.398Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*86Q2cjnU7baS9Lio" /></figure><h3><strong>Introduction: The LLM Knowledge Gap You Need to Know About</strong></h3><p>Large Language Models (LLMs) have revolutionised artificial intelligence and transformed how we interact with technology. From ChatGPT to Claude, these AI systems demonstrate remarkable capabilities in generating human-like text, solving complex problems, and providing detailed explanations on countless topics. However, a critical misconception persists: that LLMs possess comprehensive, unlimited knowledge about everything in the world.</p><p>This belief couldn’t be further from the truth. While LLMs represent significant technological achievements, they operate within substantial constraints that prevent them from having complete knowledge. More importantly, human intelligence continues to surpass these AI systems in numerous fundamental ways that are often overlooked in our AI-dominated discourse.</p><p>Understanding these limitations isn’t about diminishing the impressive capabilities of LLMs, but rather about developing a realistic perspective on their role in our information ecosystem. This comprehensive analysis will explore why LLMs lack complete knowledge, examine their fundamental constraints, and demonstrate why human intelligence remains irreplaceable in our quest for understanding and innovation.</p><h3>The Training Data Limitation: Why LLMs Can’t Know Everything</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2jEN0oQMfSuyMghF" /></figure><h4>The Finite Nature of Training Datasets</h4><p>The most fundamental reason why LLMs don’t possess all the knowledge in the world lies in their training data limitations. These models learn from massive datasets, typically comprising billions of text documents scraped from the internet, books, articles, and other written sources. However, this training data represents only a fraction of human knowledge and experience.</p><p><strong>Data Collection Boundaries</strong>: Training datasets have specific cutoff dates, meaning LLMs lack knowledge about events, discoveries, and developments that occurred after their training completion. This creates immediate knowledge gaps that grow larger over time. For instance, an LLM trained on data up to 2021 would have no information about significant events, scientific discoveries, or cultural developments that happened in 2022 and beyond.</p><p><strong>Language and Cultural Bias</strong>: The majority of LLM training data comes from English-language sources, predominantly from Western perspectives. This creates significant blind spots regarding non-English knowledge, indigenous wisdom, oral traditions, and cultural insights that exist outside the digital, text-based realm. Vast amounts of human knowledge remain embedded in local communities, specialised practices, and cultural traditions that never make it into digital formats.</p><p><strong>Quality vs. Quantity Paradox</strong>: While LLMs train on enormous amounts of text, quantity doesn’t equal quality or comprehensiveness. Much of the internet contains redundant, outdated, or incorrect information. Simultaneously, high-quality, specialised knowledge often remains locked behind paywalls, in private databases, or within expert communities that don’t publish their insights online.</p><h4>The Accessibility Problem</h4><p>Not all human knowledge is accessible to LLM training processes. Significant portions of our collective understanding remain in formats or locations that AI systems cannot access:</p><p><strong>Proprietary and Classified Information</strong>: Vast amounts of research, particularly in fields like medicine, engineering, and scientific research, remain in proprietary databases, classified documents, or behind restrictive access controls. Corporate research and development, government intelligence, and specialised industry knowledge represent enormous knowledge repositories that LLMs cannot access.</p><p><strong>Experiential and Tacit Knowledge</strong>: Much of human understanding comes from direct experience, intuition, and tacit knowledge that’s difficult or impossible to articulate in text. A master craftsperson’s understanding of their trade, a surgeon’s intuitive feel for a complex operation, or a parent’s instinctive understanding of their child represents knowledge that cannot be easily transferred to LLMs through text-based training.</p><p><strong>Real-time and Dynamic Information</strong>: The world constantly changes, with new discoveries, events, and developments occurring every moment. LLMs cannot access real-time information during their standard operation, creating an inherent lag between reality and their knowledge base. This limitation becomes particularly significant in rapidly evolving fields like technology, politics, and current events.</p><h3>Knowledge Representation vs. Understanding: The Fundamental Difference</h3><h4>Statistical Patterns vs. True Comprehension</h4><p>LLMs operate fundamentally differently from human cognition. They identify and reproduce statistical patterns in text rather than developing genuine understanding or knowledge. This distinction is crucial for understanding their limitations.</p><p><strong>Pattern Matching vs. Meaning</strong>: When an LLM generates text about quantum physics, it’s not demonstrating understanding of quantum mechanics principles. Instead, it’s reproducing patterns it learned from physics texts during training. The model can generate coherent explanations without truly comprehending the underlying concepts, much like how someone might recite a poem in a foreign language without understanding its meaning.</p><p><strong>Contextual Limitations</strong>: Human understanding includes rich contextual awareness that LLMs struggle to replicate. While humans can understand how the same information applies differently across various situations, cultures, and timeframes, LLMs often miss these nuanced contextual applications. They may provide technically accurate information that’s contextually inappropriate or culturally insensitive.</p><p><strong>The Chinese Room Problem</strong>: Philosopher John Searle’s Chinese Room argument illustrates this limitation perfectly. An LLM processing information about advanced mathematics is like someone in a room following complex rules to manipulate Chinese characters without understanding Chinese. The output appears knowledgeable, but true comprehension is absent.</p><p><strong>Synthesis vs. Aggregation</strong></p><p>Creative Knowledge Generation: Humans excel at synthesising disparate pieces of information to create new knowledge and insights. We can combine experiences from different domains, recognise patterns across seemingly unrelated fields, and generate novel solutions. LLMs primarily aggregate and recombine existing information rather than creating genuinely new knowledge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*L8UNPAlx0st-knMDc6WlNQ.png" /></figure><p><strong>Intuitive Leaps</strong>: Human innovation often involves intuitive leaps that seem to come from nowhere but actually represent unconscious processing of vast amounts of experiential data. Scientists like Einstein or Newton made breakthrough discoveries through intuitive insights that LLMs cannot replicate through pattern matching alone.</p><p><strong>Emotional and Social Intelligence</strong>: Human knowledge includes deep understanding of emotions, social dynamics, and interpersonal relationships. While LLMs can discuss these topics academically, they lack the experiential foundation that gives humans genuine insight into human nature and social interactions.</p><h3>The Dynamic Nature of Knowledge: Why Static Models Fall Short</h3><h4>Knowledge as a Living, Evolving Entity</h4><p>Human knowledge isn’t static — it’s a dynamic, ever-evolving entity that grows, changes, and adapts continuously. LLMs, however, represent frozen snapshots of information from their training period, creating fundamental mismatches with how knowledge actually works.</p><p><strong>Paradigm Shifts and Revolutionary Discoveries</strong>: Throughout history, major breakthroughs have completely overturned existing knowledge paradigms. From the geocentric to heliocentric model of the solar system, from Newtonian to quantum physics, human knowledge experiences revolutionary changes that require abandoning previously accepted facts. LLMs cannot participate in these paradigm shifts; they can only reflect the knowledge state that existed during their training.</p><blockquote>The Byzantine Knowledge Migration (1453)</blockquote><blockquote>When Constantinople fell to the Ottoman Empire, many Byzantine scholars fled westward to Italy and other parts of Europe, carrying with them.</blockquote><p><strong>Emergent Knowledge</strong>: New knowledge often emerges from the intersection of multiple disciplines, collaborative research, and cross-pollination of ideas. While LLMs can identify existing interdisciplinary connections in their training data, they cannot participate in the dynamic process of knowledge creation that happens in laboratories, research institutions, and collaborative environments worldwide.</p><p><strong>Local and Situational Knowledge</strong>: Much valuable knowledge is highly contextual and location-specific. A farmer’s understanding of local soil conditions, a urban planner’s insight into community dynamics, or a teacher’s knowledge of how specific students learn — this situated knowledge cannot be adequately captured in generalised training datasets.</p><h4>The Verification and Validation Challenge</h4><p><strong>Source Reliability</strong>: LLMs cannot independently verify the accuracy of information in their training data. They learn from a mixture of accurate information, outdated facts, misconceptions, and deliberate misinformation. Without the ability to cross-reference sources or conduct independent verification, they may confidently present incorrect information.</p><p><strong>Confidence Calibration</strong>: Humans develop sophisticated mechanisms for calibrating confidence in their knowledge. We understand when we’re certain, when we’re speculating, and when we simply don’t know. LLMs often struggle with this calibration, sometimes presenting uncertain information with inappropriate confidence or hedging unnecessarily when providing well-established facts.</p><h3>Domain-Specific Limitations: Where LLMs Fall Particularly Short</h3><p><strong>Scientific and Technical Knowledge</strong></p><p><strong>Experimental and Laboratory Knowledge</strong>: Scientific understanding often requires hands-on experimentation, laboratory work, and direct observation. A chemist’s knowledge of how different compounds react comes from years of laboratory experience that cannot be fully captured in text descriptions. LLMs miss the tactile, visual, and sensory components of scientific knowledge.</p><p><strong>Cutting-Edge Research</strong>: The most advanced scientific knowledge often exists in recently published papers, ongoing experiments, and preliminary results that may not be included in LLM training data. Researchers working at the frontier of their fields possess knowledge that hasn’t yet been published or disseminated widely.</p><p><strong>Mathematical Intuition</strong>: While LLMs can manipulate mathematical symbols and reproduce proof steps, they lack the deep mathematical intuition that guides human mathematicians. The aesthetic sense that helps mathematicians choose productive research directions, the intuitive understanding of when a proof approach will succeed, and the creative insight that leads to breakthrough theorems all remain uniquely human capabilities.</p><figure><img alt="While LLMs can manipulate mathematical symbols and reproduce proof steps, they lack the deep mathematical intuition that guides human mathematicians. The aesthetic sense that helps mathematicians choose productive research directions, the intuitive understanding of when a proof approach will succeed, and the creative insight that leads to breakthrough theorems all remain uniquely human capabilities." src="https://cdn-images-1.medium.com/max/1024/1*Pbk8Lb1GSzjlqM7arxhbLg.png" /></figure><p><strong>Engineering Judgment</strong>: Professional engineers develop judgment through years of experience with real-world constraints, material properties, and system behaviours. This practical knowledge includes understanding how theoretical principles apply in messy real-world conditions, recognising when approximations are appropriate, and developing intuitive feel for system performance that no amount of textbook knowledge can provide.</p><h4>Medical and Healthcare Knowledge</h4><p><strong>Clinical Experience</strong>: Medical practitioners develop diagnostic skills through direct patient interaction, pattern recognition from examining thousands of cases, and intuitive understanding of how diseases manifest differently in individual patients. While LLMs can access medical literature, they cannot replicate the clinical judgment that comes from years of hands-on patient care.</p><p><strong>Personalized Treatment Understanding</strong>: Effective medical treatment requires understanding each patient as a unique individual with specific medical history, genetic factors, lifestyle considerations, and personal preferences. Human physicians excel at integrating these multiple factors to develop personalized treatment strategies that LLMs cannot fully replicate.</p><p><strong>Ethical Medical Decision-Making</strong>: Medical practice involves complex ethical decisions that require understanding of patient values, cultural considerations, family dynamics, and quality-of-life factors. These deeply human elements of medical knowledge cannot be reduced to algorithmic decision-making processes.</p><h4>Creative and Artistic Knowledge</h4><p><strong>Artistic Intuition and Aesthetic Judgment</strong>: Artists develop sophisticated understanding of composition, color, form, and emotional expression through years of practice and aesthetic development. This knowledge includes not just technical skills but also deep understanding of how artistic choices affect human emotional responses — knowledge that emerges from lived experience and cultural immersion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OMcHsbpgGZQ73Ffk" /></figure><p><strong>Cultural Context and Meaning</strong>: Artistic and creative knowledge is deeply embedded in cultural context, historical understanding, and social meaning. While LLMs can analyze artistic techniques and historical information, they lack the cultural intuition and emotional understanding that drives truly meaningful creative expression.</p><p><strong>Innovation in Creative Fields</strong>: Creative breakthroughs often come from artists pushing beyond established conventions, combining influences in unexpected ways, or expressing unique personal perspectives. This type of creative innovation requires the kind of genuine originality and personal vision that LLMs cannot authentically replicate.</p><h3>Why Human Intelligence Remains Superior: The Irreplaceable Advantages</h3><h4>Consciousness and Self-Awareness</h4><p><strong>Metacognitive Abilities</strong>: Humans possess sophisticated metacognitive skills — the ability to think about thinking. We can monitor our own thought processes, recognize when we don’t understand something, identify gaps in our knowledge, and develop strategies for learning and problem-solving. This self-awareness enables continuous improvement and adaptive learning that LLMs cannot achieve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6Gn75Dhp_Jq20Q0iLmtvA.png" /></figure><p><strong>Intentionality and Purpose</strong>: Human knowledge is driven by genuine intentions, goals, and purposes. When we seek knowledge, we do so with specific objectives, personal motivations, and meaningful goals. This intentionality shapes how we acquire, organize, and apply knowledge in ways that purely pattern-matching systems cannot replicate.</p><h4>Embodied Intelligence and Sensory Integration</h4><p><strong>Multisensory Knowledge Integration</strong>: Human knowledge integrates information from all our senses — sight, sound, touch, taste, and smell — creating rich, multidimensional understanding of the world. A chef’s knowledge of cooking combines visual assessment, aromatic evaluation, tactile understanding of textures, and taste memory in ways that text-based LLMs cannot access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C9nV348y6qnvTfFn83Nr5w.png" /></figure><p><strong>Spatial and Physical Intelligence</strong>: Human intelligence includes sophisticated understanding of physical space, movement, and embodied interaction with the world. Athletes, craftspeople, surgeons, and dancers possess forms of embodied knowledge that cannot be fully captured in linguistic descriptions.</p><p><strong>Environmental Adaptation</strong>: Humans excel at adapting their knowledge and behavior to specific environmental contexts. We automatically adjust our understanding and responses based on physical location, social setting, cultural context, and situational demands in ways that demonstrate flexible, adaptive intelligence.</p><h4>Emotional Intelligence and Social Understanding</h4><p>Empathy and Emotional Resonance: Human emotional intelligence includes genuine empathy — the ability to understand and share the emotional experiences of others. This creates deep understanding of human motivation, behavior, and social dynamics that cannot be replicated through pattern analysis of text descriptions of emotions.</p><p><strong>Nonverbal Communication Mastery</strong>: Humans excel at reading and interpreting nonverbal cues — facial expressions, body language, tone of voice, and subtle social signals. This knowledge includes understanding cultural variations in nonverbal communication, recognizing deception, and interpreting emotional states through physical manifestations. LLMs, operating purely in text, cannot access this crucial dimension of human communication.</p><p><strong>Relationship Dynamics</strong>: Human social intelligence includes deep understanding of relationship patterns, family dynamics, group behavior, and interpersonal chemistry. We develop sophisticated models of individual personalities, predict how different people will interact, and navigate complex social situations using knowledge that emerges from years of social experience.</p><p><strong>Cultural Nuance and Context</strong>: Humans possess intuitive understanding of cultural subtleties, social norms, and contextual appropriateness that varies across different communities and situations. This cultural intelligence enables effective communication and relationship-building across diverse social contexts.</p><h4>Creative Problem-Solving and Innovation</h4><p><strong>Analogical Reasoning</strong>: Humans excel at drawing analogies between seemingly unrelated domains, recognizing abstract patterns across different contexts, and applying insights from one field to solve problems in another. This analogical thinking drives much scientific discovery and creative innovation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fEEhRVFgJiQIySoz3VwZeg.png" /></figure><blockquote>A teacher is explaining how the human heart works by comparing it to a water pump.</blockquote><p><strong>Lateral and Divergent Thinking</strong>: Human creativity includes the ability to think <strong>outside the box,</strong> generate multiple solution approaches, and explore unconventional possibilities. While LLMs can recombine existing ideas, they struggle with the kind of genuinely original thinking that leads to breakthrough innovations.</p><p><strong>Risk Assessment and Judgment</strong>: Humans develop sophisticated judgment about risk, uncertainty, and decision-making under incomplete information. This includes not just analytical risk assessment but also intuitive understanding of when to trust gut feelings, when to take calculated risks, and how to navigate uncertain situations.</p><h4>Moral Reasoning and Ethical Judgment</h4><p><strong>Values-Based Decision Making</strong>: Human moral reasoning integrates personal values, cultural principles, situational ethics, and emotional considerations to make complex ethical decisions. This includes understanding how moral principles apply differently across various contexts and cultures.</p><p><strong>Moral Intuition</strong>: Humans possess moral intuitions that help guide ethical decision-making even in novel situations. These intuitions emerge from deep understanding of human values, social consequences, and the inherent worth and dignity of individuals.</p><p><strong>Responsibility and Accountability</strong>: Humans can take genuine responsibility for their decisions and actions, understanding the consequences of their choices and being held accountable for their impact on others. This creates a foundation for ethical behavior that goes beyond rule-following.</p><h3>The Inherent Biases and Inaccuracies in LLM Knowledge</h3><h4>Training Data Biases</h4><p><strong>Demographic and Cultural Representation</strong>: LLM training data often over-represents certain demographic groups, geographical regions, and cultural perspectives while under-representing others. This creates systematic biases in how knowledge is presented and which viewpoints are considered authoritative.</p><p><strong>Temporal Bias</strong>: Training data reflects the knowledge, attitudes, and understanding prevalent during specific time periods. Historical biases, outdated scientific understanding, and past social attitudes become embedded in LLM responses, even when human knowledge has evolved beyond these positions.</p><p><strong>Source Quality Variations</strong>: Training datasets include information from sources of widely varying quality and reliability. Academic papers, Wikipedia articles, blog posts, and social media content are often treated with equal weight, creating potential for misinformation to be presented alongside authoritative knowledge.</p><h4>Amplification of Existing Problems</h4><p><strong>Misinformation Propagation</strong>: LLMs can inadvertently amplify misinformation that was present in their training data. Without the ability to fact-check or verify information independently, they may confidently present incorrect facts or perpetuate harmful misconceptions.</p><p><strong>Confirmation Bias</strong>: Training data often contains more discussion of popular or controversial topics, potentially creating bias toward mainstream viewpoints or sensationalized information over nuanced, expert perspectives.</p><p><strong>Language and Translation Limitations</strong>: Information that originates in languages other than English may be poorly represented or incorrectly translated in training data, leading to cultural misunderstandings or loss of nuanced meaning.</p><h3>Real-Time Knowledge Acquisition: The Human Advantage</h3><h4>Continuous Learning and Adaptation</h4><p><strong>Experiential Learning</strong>: Humans continuously update their knowledge through direct experience, learning from mistakes, and adapting their understanding based on new information. Every conversation, observation, and experience potentially contributes to our evolving knowledge base.</p><p><strong>Social Learning Networks</strong>: Human knowledge grows through dynamic social networks where information is shared, debated, refined, and validated through collaborative processes. Scientists collaborate across institutions, professionals share insights within their communities, and individuals learn from mentors and peers in ways that create living, evolving knowledge ecosystems.</p><p><strong>Contextual Adaptation</strong>: Humans excel at adapting their knowledge application to new and changing contexts. When faced with novel situations, we can draw upon our existing knowledge base while recognizing what might not apply and what new information we need to gather.</p><p><strong>Active Knowledge Seeking</strong>: Unlike LLMs which are limited to their training data, humans actively seek out new information when gaps in knowledge are identified. We can pose research questions, conduct experiments, interview experts, and pursue learning opportunities to fill knowledge gaps in real-time.</p><h4>Sensory and Environmental Updates</h4><p><strong>Multi-Modal Information Processing</strong>: Humans continuously integrate information from multiple sensory channels — visual observations, auditory inputs, tactile feedback, and environmental cues — creating rich, dynamic understanding that updates moment by moment.</p><p><strong>Environmental Responsiveness</strong>: Human knowledge includes immediate awareness of changing environmental conditions, social dynamics, and contextual factors that influence decision-making and understanding. This real-time environmental intelligence cannot be replicated by systems operating from static training data.</p><p><strong>Feedback Loop Integration</strong>: Humans naturally incorporate feedback from their actions and decisions, learning from successes and failures to refine their knowledge and improve future performance. This creates iterative knowledge improvement that LLMs cannot achieve.</p><h3>The Wisdom Gap: Experience vs. Information</h3><h4>Practical Wisdom and Judgment</h4><p><strong>Life Experience Integration</strong>: Human wisdom emerges from years of lived experience, including mistakes, successes, relationships, and personal growth. This experiential knowledge provides depth of understanding that cannot be captured in text-based training data.</p><p><strong>Situational Judgment</strong>: Humans develop sophisticated judgment about what knowledge applies in specific situations, when to trust different sources of information, and how to weigh competing considerations. This situational wisdom guides effective decision-making in complex, real-world scenarios.</p><p><strong>Emotional Maturity</strong>: Human understanding includes emotional maturity — the ability to manage emotions, understand their role in decision-making, and integrate emotional and rational considerations effectively. This emotional wisdom contributes to more balanced and effective knowledge application.</p><h4>Tacit Knowledge and Intuition</h4><p><strong>Professional Expertise</strong>: Master practitioners in every field develop tacit knowledge — deep, intuitive understanding that cannot be easily articulated or transferred through explicit instruction. A master chess player’s intuitive evaluation of positions, a experienced physician’s diagnostic instincts, or a skilled teacher’s understanding of how to reach different students represents knowledge that emerges from extensive practice and experience.</p><p><strong>Pattern Recognition</strong>: Human expertise includes sophisticated pattern recognition abilities that operate below conscious awareness. These patterns are learned through extensive experience and enable rapid, accurate judgments in complex situations.</p><p><strong>Intuitive Integration</strong>: Humans can integrate multiple sources of information intuitively, arriving at insights and conclusions that they may not be able to fully explain logically. This intuitive integration often leads to creative breakthroughs and innovative solutions.</p><h3>Scientific Method and Knowledge Validation: Where Humans Excel</h3><h3>Hypothesis Formation and Testing</h3><p><strong>Scientific Creativity</strong>: Human scientists excel at forming creative hypotheses, designing innovative experiments, and asking novel research questions. This scientific creativity requires the ability to imagine possibilities that go beyond existing knowledge and design ways to test these possibilities.</p><p><strong>Experimental Design</strong>: Effective scientific research requires understanding of experimental controls, statistical significance, potential confounding variables, and methodological rigor. Human researchers bring creativity, skepticism, and methodological expertise to experimental design that LLMs cannot replicate.</p><p><strong>Peer Review and Scientific Discourse</strong>: The scientific method relies on peer review, debate, and collaborative validation of findings. Human scientists can critically evaluate research methods, identify potential flaws, and engage in the kind of rigorous academic discourse that ensures knowledge quality.</p><h4>Critical Thinking and Skepticism</h4><p><strong>Source Evaluation</strong>: Humans can develop sophisticated abilities to evaluate source credibility, identify potential conflicts of interest, and assess the reliability of different types of evidence. This critical evaluation goes beyond simple fact-checking to include understanding of methodology, potential biases, and contextual factors.</p><p><strong>Logical Reasoning</strong>: While LLMs can follow logical patterns present in their training data, humans can engage in genuine logical reasoning, identifying fallacies, constructing valid arguments, and recognizing when conclusions don’t follow from premises.</p><p><strong>Healthy Skepticism</strong>: Human knowledge includes appropriate skepticism — knowing when to question claims, demand evidence, and withhold judgment pending further information. This skeptical thinking protects against misinformation and ensures that knowledge claims are properly validated before acceptance.</p><p><strong>Contextual Questioning</strong>: Humans excel at asking the right questions in specific contexts, recognizing what information is missing, and identifying the most important aspects to investigate further. This questioning ability drives scientific progress and knowledge advancement.</p><p><strong>Philosophical Reflection</strong>: Human knowledge includes the ability to reflect on the nature of knowledge itself — understanding epistemological questions about how we know what we know, what constitutes valid evidence, and how to handle uncertainty and ambiguity.</p><h3>Cultural and Contextual Knowledge: The Human Advantage</h3><h4>Deep Cultural Understanding</h4><p><strong>Cultural Immersion</strong>: Humans who grow up within specific cultural contexts develop deep, intuitive understanding of cultural norms, values, communication styles, and social dynamics that cannot be fully captured in textual descriptions. This cultural knowledge influences every aspect of understanding and interpretation.</p><p><strong>Cross-Cultural Competence</strong>: Experienced multicultural individuals develop sophisticated abilities to navigate between different cultural contexts, understanding how the same information or behavior might be interpreted differently across cultures. This cross-cultural competence enables more nuanced and appropriate knowledge application.</p><p><strong>Historical Context</strong>: Human understanding of historical events includes not just factual information but also understanding of historical significance, causation, and continued relevance. This historical consciousness shapes how current events and information are interpreted and understood.</p><h4>Local and Situated Knowledge</h4><p><strong>Community-Specific Understanding</strong>: Humans possess deep knowledge of their local communities, including informal networks, local history, environmental conditions, and social dynamics that rarely appear in training datasets but are crucial for effective local action and decision-making.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HTwUgIIb2W4gofF4t37Hfw.png" /></figure><p><strong>Professional Context</strong>: Within professional fields, humans develop understanding of informal hierarchies, unwritten rules, political dynamics, and practical considerations that influence how formal knowledge is actually applied in workplace contexts.</p><p><strong>Personal Relationship Knowledge</strong>: Humans excel at understanding individual personalities, relationship dynamics, and personal histories that influence how to communicate effectively with specific people and navigate particular social situations.</p><h3>The Innovation and Discovery Process: Uniquely Human</h3><h4>Creative Breakthrough Processes</h4><p><strong>Insight and “Aha” Moments</strong>: Human innovation often involves sudden insights or breakthrough moments where disparate pieces of information suddenly connect in new ways. These creative leaps represent genuine knowledge creation that goes beyond recombining existing information.</p><p><strong>Persistence and Dedication</strong>: Major discoveries often require years of persistent effort, the ability to continue working despite failures and setbacks, and the dedication to pursue questions even when progress seems impossible. This kind of sustained creative effort reflects uniquely human characteristics.</p><p><strong>Risk-Taking and Exploration</strong>: Innovation requires willingness to explore unknown territories, take intellectual risks, and pursue ideas that may not work out. Human researchers and innovators can make judgment calls about which risky directions are worth pursuing.</p><h4>Collaborative Knowledge Creation</h4><p><strong>Team Dynamics</strong>: Effective collaborative research and innovation requires understanding of team dynamics, communication skills, and the ability to build on others’ ideas while contributing unique perspectives. Human collaboration creates synergistic knowledge creation that exceeds individual capabilities.</p><p><strong>Mentorship and Knowledge Transfer</strong>: The apprenticeship model of knowledge transfer — where experienced practitioners guide newcomers through hands-on learning — represents a uniquely human form of knowledge transmission that includes not just information but also judgment, wisdom, and professional culture.</p><p><strong>Cross-Disciplinary Integration</strong>: Human innovators excel at bringing together insights from multiple disciplines, recognizing connections that aren’t obvious, and creating new knowledge at the intersections of existing fields.</p><h3>Emotional and Interpersonal Intelligence: The Human Edge</h3><h4>Understanding Human Motivation</h4><p><strong>Motivational Complexity</strong>: Humans possess deep understanding of what motivates other people, including conscious and unconscious motivations, competing desires, and the complex interplay between rational and emotional factors in decision-making.</p><p><strong>Behavioral Prediction</strong>: Experienced individuals can predict how specific people are likely to respond in various situations, understanding personality patterns, emotional triggers, and behavioral tendencies that influence human action.</p><p><strong>Persuasion and Influence</strong>: Effective human communication involves understanding how to present information in ways that resonate with specific audiences, addressing not just logical arguments but also emotional appeals and values-based reasoning.</p><h4>Therapeutic and Counseling Knowledge</h4><p><strong>Healing Relationships</strong>: Mental health professionals develop deep understanding of how therapeutic relationships facilitate healing and growth. This knowledge includes not just techniques but also intuitive understanding of timing, emotional attunement, and the healing power of genuine human connection.</p><p><strong>Trauma-Informed Understanding</strong>: Working with individuals who have experienced trauma requires deep understanding of how trauma affects the nervous system, relationships, and worldview. Mental health professionals develop sophisticated knowledge about creating safety, building trust, and facilitating healing that goes far beyond textbook knowledge of trauma symptoms.</p><p><strong>Developmental Understanding</strong>: Human professionals who work with children, adolescents, or adults in transition develop nuanced understanding of developmental processes, individual variation, and how to support growth at different life stages. This knowledge emerges from extensive observation and interaction with real people over time.</p><p><strong>Crisis Intervention Skills</strong>: Professionals who work in crisis situations develop intuitive abilities to assess risk, de-escalate dangerous situations, and provide appropriate support under high-stress conditions. These skills require reading subtle cues and making split-second decisions based on deep understanding of human psychology.</p><h4>Leadership and Management Wisdom</h4><p><strong>Organizational Dynamics</strong>: Effective leaders develop sophisticated understanding of organizational culture, power dynamics, and change processes that cannot be fully captured in management textbooks. This knowledge includes understanding informal networks, cultural patterns, and how to navigate complex political environments.</p><p><strong>Team Psychology</strong>: Successful managers understand team dynamics, how to motivate different personality types, and how to create productive working relationships among diverse individuals. This knowledge requires ongoing observation and adaptation to specific team characteristics.</p><p><strong>Decision-Making Under Pressure</strong>: Leadership roles require making difficult decisions with incomplete information, managing competing stakeholder interests, and taking responsibility for outcomes. This decision-making expertise develops through experience and cannot be replicated through pattern matching.</p><h3>Practical Problem-Solving: Where Experience Trumps Data</h3><h4>Real-World Application Knowledge</h4><p><strong>Implementation Challenges</strong>: Professionals across all fields understand the difference between theoretical knowledge and practical implementation. Engineers know how theoretical designs must be modified for real-world conditions, teachers understand how lesson plans must be adapted for specific students, and managers know how policies must be adjusted for particular organizational contexts.</p><p><strong>Resource Constraints</strong>: Human knowledge includes understanding of resource limitations — time, money, personnel, materials — and how to achieve objectives within these constraints. This practical knowledge involves trade-offs, prioritization, and creative problem-solving that LLMs cannot fully appreciate.</p><p><strong>Stakeholder Considerations</strong>: Real-world problem-solving requires understanding of multiple stakeholder perspectives, competing interests, and political considerations that influence what solutions are feasible and acceptable. This stakeholder awareness shapes how knowledge is applied in practice.</p><h4>Troubleshooting and Diagnostic Skills</h4><p><strong>Pattern Recognition in Complex Systems</strong>: Experienced professionals develop abilities to diagnose problems in complex systems by recognizing subtle patterns, unusual behaviors, and interconnected causes. A master mechanic’s ability to diagnose engine problems, a physician’s diagnostic instincts, or an IT professional’s troubleshooting skills represent knowledge that emerges from extensive hands-on experience.</p><p><strong>Intuitive Problem-Solving</strong>: Many professionals develop intuitive approaches to problem-solving that they cannot fully articulate but that prove consistently effective. This intuitive knowledge guides them toward promising solution directions and helps them avoid unproductive approaches.</p><p><strong>Adaptive Strategy Development:</strong> Humans excel at adapting their problem-solving strategies when initial approaches don’t work, learning from failed attempts, and developing new approaches based on emerging understanding of the problem. This adaptive capacity enables persistent problem-solving in complex situations.</p><h4>The Future of Human-AI Collaboration: Leveraging Complementary Strengths</h4><h4>Optimal Integration Models</h4><p><strong>Human Oversight and Validation</strong>: The most effective applications of LLMs involve human experts who can evaluate, validate, and contextualize AI-generated information. Humans provide the critical thinking, source evaluation, and contextual judgment that ensure appropriate application of AI capabilities.</p><p><strong>Creative Collaboration</strong>: LLMs can serve as powerful tools for human creativity, generating initial ideas, providing research assistance, and offering different perspectives that humans can then evaluate, modify, and build upon. This collaborative model leverages AI capabilities while maintaining human creative control and judgment.</p><p><strong>Complementary Problem-Solving</strong>: Humans and LLMs have complementary strengths — LLMs excel at processing large amounts of information quickly, while humans excel at creative thinking, contextual judgment, and complex reasoning. Effective collaboration leverages these complementary capabilities.</p><h4>Maintaining Human Knowledge Primacy</h4><p><strong>Critical Evaluation Skills</strong>: As LLMs become more prevalent, human skills in critical evaluation, source verification, and contextual judgment become increasingly valuable. These skills ensure that AI-generated information is appropriately evaluated and applied.</p><p><strong>Domain Expertise</strong>: Deep human expertise in specific fields remains essential for understanding when and how to apply AI capabilities effectively. Expert knowledge provides the context and judgment needed to interpret AI outputs appropriately and identify when AI-generated information may be incorrect or incomplete.</p><p><strong>Ethical Decision-Making</strong>: As AI systems become more powerful, human ethical reasoning becomes increasingly important for ensuring that these tools are used responsibly. Humans must maintain ultimate responsibility for decisions that affect people’s lives, applying moral reasoning and values-based judgment that AI cannot provide.</p><p><strong>Innovation Leadership</strong>: While AI can assist with various aspects of research and development, human creativity, intuition, and visionary thinking remain essential for driving genuine innovation and breakthrough discoveries. Humans must continue to lead the direction of scientific inquiry and technological development.</p><h3>The Importance of Preserving Human Knowledge Systems</h3><h4>Educational Implications</h4><p><strong>Critical Thinking Education</strong>: Educational systems must emphasize critical thinking skills, source evaluation abilities, and independent reasoning capabilities to ensure that humans can effectively work with and evaluate AI systems. These skills become more, not less, important in an AI-enhanced world.</p><p><strong>Experiential Learning</strong>: Hands-on experience, internships, and practical application remain crucial for developing the kind of deep, contextual knowledge that cannot be replicated by AI systems. Educational programs must maintain emphasis on real-world application and experiential learning.</p><p><strong>Cultural Knowledge Preservation</strong>: As digital systems become more prevalent, preserving cultural knowledge, oral traditions, and local expertise becomes increasingly important. These forms of knowledge represent irreplaceable human heritage that must be maintained and transmitted.</p><h4>Professional Development Priorities</h4><p><strong>Uniquely Human Skills</strong>: Professionals should focus on developing skills that remain uniquely human — creativity, emotional intelligence, ethical reasoning, cultural competence, and complex problem-solving. These capabilities become increasingly valuable as AI handles more routine tasks.</p><p><strong>AI Collaboration Competency</strong>: Understanding how to work effectively with AI systems — knowing their capabilities and limitations, how to prompt them effectively, and how to evaluate their outputs — becomes an essential professional skill across many fields.</p><p><strong>Continuous Learning Adaptation</strong>: As AI capabilities evolve rapidly, humans must maintain strong capabilities for continuous learning, adaptation, and skill development to remain relevant and effective in their professional roles.</p><h3>Quality Control and Knowledge Verification: Human Oversight Remains Essential</h3><h4>Fact-Checking and Verification</h4><p><strong>Multi-Source Validation</strong>: Humans excel at cross-referencing information across multiple sources, identifying discrepancies, and resolving conflicting information through additional research and investigation. This verification process requires judgment about source reliability and methodological rigor that AI cannot fully provide.</p><p><strong>Context-Dependent Accuracy</strong>: The accuracy of information often depends heavily on context — the same fact might be true in one situation but not another, or accurate for one time period but not another. Human knowledge includes sophisticated understanding of these contextual factors.</p><p><strong>Bias Detection</strong>: Experienced human evaluators can identify subtle biases in information presentation, recognize potential conflicts of interest, and understand how different perspectives might shape the presentation of information. This bias detection requires cultural awareness and critical thinking skills.</p><h4>Quality Assurance Processes</h4><p><strong>Peer Review Excellence</strong>: Human peer review processes in academic and professional contexts involve sophisticated evaluation of methodology, reasoning, and conclusions that require deep expertise and critical judgment. These quality control mechanisms remain essential for maintaining knowledge standards.</p><p><strong>Editorial Judgment</strong>: Human editors bring contextual understanding, audience awareness, and quality judgment to information presentation that ensures appropriate communication for specific contexts and purposes.</p><p><strong>Professional Standards</strong>: Human professional communities maintain standards, ethics, and best practices that guide appropriate application of knowledge within specific fields. These professional standards require human judgment and community consensus.</p><h3>Conclusion: Embracing Human Intellectual Superiority in the AI Age</h3><p>The examination of Large Language Models’ limitations reveals a fundamental truth: despite their impressive capabilities, LLMs represent sophisticated pattern-matching systems rather than genuine knowledge repositories. They lack the comprehensive, dynamic, and contextual understanding that characterizes human intelligence.</p><h4>Key Takeaways for the AI Era</h4><p><strong>LLMs Are Tools, Not Oracles</strong>: Understanding LLMs as powerful tools rather than authoritative knowledge sources helps establish appropriate expectations and use cases. Like any tool, their effectiveness depends on skilled human operators who understand their capabilities and limitations.</p><p><strong>Human Knowledge Remains Irreplaceable</strong>: The depth, context, creativity, and wisdom that characterize human knowledge cannot be replicated by current AI systems. Human intelligence encompasses consciousness, experience, emotion, and judgment that provide qualitatively different types of understanding.</p><p><strong>Complementary Strengths Create Optimal Outcomes</strong>: The future lies not in human-AI competition but in thoughtful collaboration that leverages the unique strengths of both. Humans provide creativity, judgment, ethical reasoning, and contextual understanding, while AI systems offer rapid information processing and pattern recognition capabilities.</p><p><strong>Critical Thinking Becomes More Valuable</strong>: As AI-generated content becomes more prevalent, human abilities to evaluate, verify, and contextualize information become increasingly precious. The capacity for independent critical thinking and source evaluation represents a crucial human advantage that must be cultivated and preserved.</p><h4>The Enduring Value of Human Intelligence</h4><p><strong>Experiential Wisdom</strong>: Human knowledge is enriched by lived experience, emotional understanding, and the accumulated wisdom that comes from navigating real-world challenges. This experiential foundation provides depth and meaning that cannot be captured in training datasets or replicated through pattern matching.</p><p><strong>Moral and Ethical Leadership</strong>: In an age of powerful AI systems, human moral reasoning and ethical judgment become more crucial than ever. The decisions about how to develop, deploy, and regulate AI technologies require the kind of values-based reasoning and ethical wisdom that only humans can provide.</p><p><strong>Creative Innovation</strong>: Breakthrough discoveries, artistic achievements, and innovative solutions continue to emerge from uniquely human capabilities — creative insight, imaginative thinking, and the ability to make unexpected connections across disparate domains of knowledge.</p><p><strong>Cultural and Social Intelligence</strong>: Understanding human nature, social dynamics, and cultural contexts requires the kind of empathetic intelligence and interpersonal wisdom that develops through genuine human relationships and social experience.</p><h4>Moving Forward with Confidence</h4><p>Rather than feeling threatened by AI capabilities, we should recognize that human intelligence possesses irreplaceable qualities that become more valuable, not less, as AI systems become more prevalent. The limitations of LLMs — their lack of genuine understanding, their inability to access real-time information, their dependence on static training data, and their absence of consciousness and experience — highlight the continued superiority of human intelligence in crucial domains.</p><p><strong>Embrace Your Unique Capabilities</strong>: Every human possesses forms of knowledge and understanding that no AI system can replicate — personal experience, cultural insight, emotional intelligence, moral reasoning, and creative potential. These capabilities represent genuine intellectual advantages that should be recognized and developed.</p><p><strong>Maintain Intellectual Independence</strong>: While AI tools can provide valuable assistance, maintaining the ability to think independently, evaluate information critically, and make reasoned judgments remains essential. Human intellectual autonomy ensures that we remain the authors of our own understanding rather than passive consumers of AI-generated content.</p><p><strong>Cultivate Uniquely Human Skills</strong>: Focus on developing the capabilities that distinguish human intelligence — creativity, empathy, ethical reasoning, cultural competence, and the ability to find meaning and purpose in knowledge and experience.</p><h4>Final Thoughts: The Irreplaceable Human Advantage</h4><p>Large Language Models, impressive as they are, represent elaborate systems for processing and recombining existing information. They lack the consciousness, experience, creativity, and wisdom that make human knowledge genuinely transformative. While they can serve as valuable tools for human knowledge work, they cannot replace the fundamental human capabilities that drive genuine understanding, innovation, and progress.</p><p>Your intelligence — with its capacity for consciousness, creativity, emotional understanding, moral reasoning, and experiential wisdom — remains superior to current AI systems in the most important ways. The future belongs not to artificial intelligence alone, but to humans who understand both their own irreplaceable capabilities and how to leverage AI tools effectively while maintaining intellectual leadership.</p><p>As we navigate the AI revolution, remember that human intelligence represents billions of years of evolutionary development, thousands of years of cultural accumulation, and the unique perspective that comes from conscious, embodied experience in the world. These advantages cannot be replicated by pattern-matching algorithms, no matter how sophisticated they become.</p><p>The goal is not to compete with AI systems but to recognize and develop your unique human capabilities while using AI tools to enhance and amplify your natural intelligence. In this collaboration, you remain the creative director, critical evaluator, and wise decision-maker — roles that require the irreplaceable qualities of human consciousness and understanding.</p><p>Your intelligence, with all its complexity, creativity, and wisdom, remains the most remarkable information processing system we know. Embrace that advantage, develop it further, and use it to shape a future where human intelligence continues to lead the way toward greater understanding, innovation, and human flourishing.</p><p>This comprehensive analysis demonstrates why human intelligence continues to reign supreme over artificial intelligence systems, despite their impressive capabilities. By understanding both the limitations of LLMs and the unique strengths of human cognition, we can approach the AI age with confidence in our irreplaceable intellectual capabilities</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30b0d257f229" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Compelling Benefits of Using Existing LLMs Instead of Building from Scratch]]></title>
            <link>https://suraj-batuwana.medium.com/10-compelling-benefits-of-using-existing-llms-instead-of-building-from-scratch-6410c1037662?source=rss-b587db5cb48------2</link>
            <guid isPermaLink="false">https://medium.com/p/6410c1037662</guid>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[how-to-build-llm]]></category>
            <category><![CDATA[finetune-llm]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Suraj Batuwana]]></dc:creator>
            <pubDate>Sat, 24 May 2025 00:22:10 GMT</pubDate>
            <atom:updated>2025-05-24T00:30:07.246Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gAUx0cN2CHBywWRH" /></figure><p><strong>Introduction</strong><br>Large Language Models (LLMs) like GPT-4, Claude, and PaLM 2 are revolutionizing industries — from customer service to content creation. However, businesses face a critical decision: <em>build a custom LLM from scratch</em> or <em>leverage existing models</em>. While the allure of a bespoke AI might seem tempting, this 3000-word guide explores why <strong>using pre-trained LLMs</strong> is often the smarter choice. We’ll break down 10 compelling advantages, from cost savings to ethical compliance, and showcase real-world success stories.</p><h3>1. Cost-Effectiveness: Avoid Millions in Development</h3><p>Building an LLM requires staggering financial investment. Let’s break down the costs:</p><h4>Data Acquisition and Processing</h4><ul><li>Collecting diverse, high-quality datasets can cost <strong>$500k–$1M+</strong>.</li><li>Cleaning and labeling data adds another 20–30% in time and resources.</li></ul><h4>Infrastructure and Training</h4><ul><li>Training GPT-3 reportedly cost <strong>$4.6M</strong> in compute alone.</li><li>Hardware (GPUs/TPUs) and cloud costs can exceed <strong>$10,000/day</strong>.</li></ul><h4>Talent and Maintenance</h4><ul><li>Hiring ML engineers and researchers: <strong>$150k–$300k/year per specialist</strong>.</li><li>Ongoing tuning and bug fixes: <strong>20%+ of initial costs annually</strong>.</li></ul><p><strong>Existing LLMs</strong>, like OpenAI’s API, operate on pay-as-you-go pricing. For example, GPT-4 costs <strong>$0.03–$0.12 per 1k tokens</strong>, making it accessible for startups and enterprises alike.</p><h3>2. Time Savings: Deploy in Days, Not Years</h3><p>Training an LLM from scratch takes <strong>6–12 months</strong> for data preparation and <strong>3–6 months</strong> for training. In contrast, platforms like Hugging Face or Azure AI offer pre-trained models via APIs, enabling integration in <strong>under a week</strong>.</p><h3>3. Access to State-of-the-Art Technology</h3><p>Leading LLM providers invest billions in R&amp;D. For instance:</p><ul><li><strong>GPT-4</strong> leverages 1.76 trillion parameters and multimodal capabilities.</li><li><strong>Google’s PaLM 2</strong> excels in multilingual tasks across 100+ languages.</li></ul><p>By using these models, you inherit cutting-edge advancements without reinventing the wheel.</p><h3>4. Scalability and Infrastructure Handled for You</h3><p>Cloud-based LLM providers automatically manage:</p><ul><li><strong>Server load balancing</strong> during traffic spikes.</li><li><strong>GPU/TPU clusters</strong> for low-latency inference.</li><li><strong>Global CDNs</strong> to serve users worldwide.</li></ul><p>Building equivalent infrastructure could cost <strong>$1M+ annually</strong>.</p><h3>5. Pre-Trained on Massive, Diverse Datasets</h3><p>Existing LLMs are trained on datasets impractical for most organizations:</p><ul><li><strong>Common Crawl</strong> (250+ billion web pages).</li><li><strong>BooksCorpus</strong> (11,000+ unpublished books).</li><li><strong>Multilingual sources</strong> (e.g., Wikipedia in 20+ languages).</li></ul><p>Replicating this diversity is financially and logistically unfeasible for individual teams.</p><h3>6. Reduced Technical Complexity</h3><p>Maintaining an LLM in production involves:</p><ul><li><strong>Model drift monitoring.</strong></li><li><strong>Security patches and updates.</strong></li><li><strong>Optimizing inference speed.</strong></li></ul><p>Providers like Anthropic handle these challenges, letting you focus on application logic.</p><h3>7. Thriving Ecosystem and Community Support</h3><p>Pre-trained LLMs benefit from robust ecosystems:</p><ul><li><strong>Hugging Face Transformers</strong> (50k+ pre-trained models).</li><li><strong>TensorFlow/PyTorch integrations.</strong></li><li><strong>Forums and tutorials</strong> for troubleshooting.</li></ul><p>Custom models lack this collaborative advantage.</p><h3>8. Customization Without the Heavy Lifting</h3><p>Fine-tuning existing LLMs requires <strong>100x less data</strong> than training from scratch. Techniques include:</p><ul><li><strong>Prompt Engineering:</strong> Guide outputs with tailored instructions.</li><li><strong>LoRA Adapters:</strong> Modify behavior with lightweight modules.</li><li><strong>Domain-Specific Tuning:</strong> E.g., medical chatbots via BioBERT.</li></ul><h3>9. Built-In Ethical and Compliance Safeguards</h3><p>Leading LLMs incorporate safeguards:</p><ul><li><strong>Bias Mitigation:</strong> Tools to reduce harmful outputs.</li><li><strong>GDPR Compliance:</strong> Data anonymization features.</li><li><strong>Content Filtering:</strong> API-level toxicity checks.</li></ul><p>Building these responsibly from scratch is a legal and ethical minefield.</p><h3>10. Focus on Core Business Innovation</h3><p>Instead of allocating 70% of resources to model development, teams can:</p><ul><li>Enhance user experience.</li><li>Develop niche applications (e.g., legal document analyzers).</li><li>Iterate faster based on customer feedback.</li></ul><p><strong>Example:</strong> Jasper.ai scaled its marketing content tool using GPT-3, avoiding AI infrastructure costs.</p><h3>Addressing Potential Drawbacks</h3><p>While existing LLMs have limitations (e.g., less control over data), solutions include:</p><ul><li><strong>Hybrid Models:</strong> Combine open-source LLMs (LLaMA) with proprietary data.</li><li><strong>Enterprise Agreements:</strong> Negotiate data privacy terms with providers.</li></ul><h4>Here are 10 startups leveraging existing LLMs to solve real-world problems</h4><p><strong>1. Perplexity AI</strong> — Uses LLMs to provide AI-powered search with cited sources, transforming how people find and verify information online.</p><p><strong>2. Jasper</strong> — Applies LLMs for content marketing automation, helping businesses generate blog posts, ad copy, and social media content at scale.</p><p><strong>3. Harvey</strong> — Integrates LLMs into legal workflows, assisting lawyers with contract review, legal research, and document drafting to increase efficiency.</p><p><strong>4. Tome</strong> — Leverages LLMs to automatically generate presentation decks and storytelling content, streamlining the creation process for professionals.</p><p><strong>5. Copy.ai</strong> — Uses LLMs for sales and marketing copy generation, helping teams create email campaigns, product descriptions, and sales materials.</p><p><strong>6. Hebbia</strong> — Applies LLMs to financial and consulting research, enabling analysts to quickly process and synthesize large volumes of documents.</p><p><strong>7. Avanade (Accenture)</strong> — Built solutions using LLMs for enterprise automation, particularly in customer service chatbots and internal knowledge management.</p><p><strong>8. DoNotPay</strong> — Integrates LLMs to help consumers fight bureaucracy, from contesting parking tickets to canceling subscriptions automatically.</p><p><strong>9. Mem</strong> — Uses LLMs to create an AI-powered note-taking and knowledge management system that understands context and relationships between ideas.</p><p><strong>10. Writer</strong> — Applies LLMs for enterprise content creation and brand consistency, helping large organizations maintain voice and tone across all communications.</p><p>These companies demonstrate how existing LLM capabilities can be specialized and applied to specific industry problems rather than building foundation models from scratch.</p><h3>Conclusion</h3><p>For 95% of businesses, existing LLMs offer unparalleled cost, time, and efficiency benefits. By leveraging these tools, companies can deploy AI faster, reduce risk, and focus on solving real-world problems — not reinventing infrastructure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6410c1037662" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Regression error?]]></title>
            <link>https://suraj-batuwana.medium.com/what-is-regression-error-3e7bf712b5e1?source=rss-b587db5cb48------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e7bf712b5e1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Suraj Batuwana]]></dc:creator>
            <pubDate>Fri, 02 May 2025 13:15:13 GMT</pubDate>
            <atom:updated>2025-05-02T13:15:13.487Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Regression error</strong> refers to the difference between the predicted values and the actual (true) values in a regression problem. Regression is a type of supervised machine learning task where the goal is to predict a continuous numerical value (e.g., house prices, temperature, or sales figures). The error quantifies how well (or poorly) the regression model is performing.</p><h4>Types of regression errors:</h4><p>Regression errors are typically measured using various metrics, each of which provides a different perspective on the model’s performance. Here are the most common types of regression errors:</p><ol><li><strong>Mean Absolute Error (MAE)</strong>:<br>The average of the absolute differences between predicted and actual values.<br>Formula:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*wKNXcD-zQY9_Z2jMZ1-r8w.png" /></figure><p><strong>Interpretation</strong>: A lower MAE indicates better performance. It is easy to understand but does not penalize large errors heavily.</p><p>2. <strong>Mean Squared Error (MSE)</strong>:<br>The average of the squared differences between predicted and actual values.<br>Formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*UIAFlQr8prClKJK8A3JLwA.png" /></figure><p><strong>Interpretation</strong>: MSE penalizes larger errors more heavily, making it sensitive to outliers.</p><p>3. <strong>Root Mean Squared Error (RMSE)</strong>:<br>The square root of MSE, which brings the error back to the original units of the target variable.<br>Formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*8Y33ZKwGmFZOzPon_41vfg.png" /></figure><p><strong>Interpretation</strong>: Like MSE, RMSE penalizes larger errors but is more interpretable because it is in the same units as the target variable.</p><p>4. <strong>Mean Absolute Percentage Error (MAPE)</strong>:<br>The average of the absolute percentage differences between predicted and actual values.<br>Formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*8Y33ZKwGmFZOzPon_41vfg.png" /></figure><p><strong>Interpretation</strong>: MAPE expresses errors as percentages, making it easier to compare across different datasets. However, it can be problematic if the actual values (yi) are zero or very close to zero.</p><p>5. <strong>R-squared (Coefficient of Determination)</strong>:<br>Measures the proportion of variance in the target variable that is explained by the model.<br>Formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*fWldxL3iWV8ZC6ECSCTgqQ.png" /></figure><p><strong>Interpretation</strong>: Ranges from 0 to 1, where 1 indicates a perfect fit and 0 indicates no explanatory power.</p><p><strong>Causes of regression errors</strong>:<br>1. <strong>Model Bias</strong>:<br>The model is too simple and cannot capture the underlying patterns in the data (underfitting).</p><p>2. <strong>Model Variance</strong>:<br>The model is too complex and captures noise in the training data, leading to poor generalization (overfitting).</p><p>3. <strong>Noisy Data</strong>:<br>Outliers or errors in the dataset can distort the model’s predictions.</p><p>4. <strong>Insufficient Features</strong>:<br>The model lacks important predictors (features) that are necessary to make accurate predictions.</p><p>5. <strong>Nonlinear relationships</strong>:<br>The relationship between the features and the target variable is nonlinear, but the model assumes a linear relationship.</p><p>6. <strong>Multicollinearity</strong>:<br>High correlation between independent variables can make the model unstable and less interpretable.</p><p><strong>How to Reduce Regression Errors</strong>:<br>1. <strong>Feature Engineering</strong>:<br>Select relevant features and create new ones that better capture the relationships in the data.</p><p>2. <strong>Regularization</strong>:<br>Use techniques like Lasso (L1) or Ridge (L2) regression to penalize overly complex models and reduce overfitting.</p><p>3. <strong>Cross-Validation</strong>:<br>Use cross-validation to evaluate the model’s performance on unseen data and ensure it generalizes well.</p><p>4. <strong>Outlier Handling</strong>:<br>Identify and handle outliers in the dataset to reduce their impact on the model.</p><p>5. <strong>Model Selection</strong>:<br>Experiment with different regression algorithms (e.g., linear regression, decision trees, support vector regression) to find the best-performing one.</p><p>6. <strong>Hyperparameter Tuning</strong>:<br>Optimize the model’s <a href="https://suraj-batuwana.medium.com/what-are-models-hyperparameters-d8e3b56b0419">hyperparameters</a> to improve performance.</p><p>7. <strong>Ensemble Methods</strong>:<br>Combine multiple models (e.g., using bagging or boosting) to improve prediction accuracy.</p><p><strong>Example of Regression Error</strong>:<br>Suppose you are predicting house prices based on features like size, location, and number of bedrooms. The actual and predicted prices for 3 houses are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kEiKE1SEqp01K9JgT0a9Zw.png" /></figure><p>By analyzing and minimizing regression errors, you can build more accurate and reliable models for predicting continuous outcomes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e7bf712b5e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-Tuning Foundation Models with Amazon Bedrock: A Step-by-Step Guide]]></title>
            <link>https://suraj-batuwana.medium.com/fine-tuning-foundation-models-with-amazon-bedrock-a-step-by-step-guide-9c36e2b954b5?source=rss-b587db5cb48------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c36e2b954b5</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[finetune-llm]]></category>
            <category><![CDATA[bedrock]]></category>
            <dc:creator><![CDATA[Suraj Batuwana]]></dc:creator>
            <pubDate>Fri, 02 May 2025 13:13:36 GMT</pubDate>
            <atom:updated>2025-05-02T13:13:36.211Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mtPxLPdjXme5U66-" /></figure><h4>Introduction</h4><p>In the rapidly evolving field of artificial intelligence, foundation models like those offered by Amazon Bedrock provide powerful capabilities out of the box. However, to make these models truly effective for specific tasks or domains, fine-tuning is often necessary. This blog post will walk you through the process of fine-tuning a foundation model using Amazon Bedrock, ensuring you have the knowledge to customize these models for your unique needs.</p><h4>Why Fine-Tuning?</h4><p>Fine-tuning allows you to adapt a pre-trained foundation model to better understand and generate responses for specialized tasks. Whether you’re working in healthcare, finance, or customer service, fine-tuning ensures the model aligns with your domain-specific requirements.</p><h4><strong>Prerequisites</strong></h4><p>Before diving into fine-tuning, ensure you have the following:</p><p>1. <strong>AWS Account</strong>: Access to Amazon Web Services (AWS) with permissions for Amazon Bedrock and S3. <br>2. <strong>Dataset</strong>: A properly formatted dataset for training and validation. <br>3. <strong>Basic Knowledge of AWS Services</strong>: Familiarity with S3 and IAM roles will be helpful.</p><h4><strong>Step 1: Setting Up Your S3 Bucket</strong></h4><p>The first step in fine-tuning a foundation model is preparing your data storage. Amazon S3 (Simple Storage Service) will host your training and validation datasets, as well as store the output metrics.</p><p><strong>Creating an S3 Bucket</strong></p><p>1. Navigate to the <strong>AWS Management Console</strong> and open <strong>Amazon S3</strong>. <br>2. Click <strong>Create bucket</strong>. <br>3. Name your bucket (e.g., `bedrock-data-12345`) and leave other settings as default. <br>4. Click <strong>Create bucket</strong>.</p><p><strong>Organizing Your Data</strong></p><p>Inside your bucket, create two folders: <br>- <strong>Input Data</strong>: Contains your training and validation datasets. <br>- <strong>Output Data</strong>: Stores metrics and logs from the fine-tuning process.</p><h4>Step 2: Uploading Your Dataset</h4><p>Your dataset must follow a specific JSON format for Amazon Bedrock to process it correctly. Here’s an example of the required structure:</p><pre>[<br> {&quot;prompt&quot;: &quot;CLASSIFY: Patient has persistent high fever (&gt;102°F), cough, and fatigue. No recent travel.&quot;, &quot;completion&quot;: &quot;Possible Influenza&quot;}<br> {&quot;prompt&quot;: &quot;CLASSIFY: 45-year-old with crushing chest pain radiating to left arm, sweating, and nausea.&quot;, &quot;completion&quot;: &quot;Suspected Myocardial Infarction (Heart Attack)&quot;}<br> {&quot;prompt&quot;: &quot;CLASSIFY: Rash with red circular patches, central clearing, and history of tick bite.&quot;, &quot;completion&quot;: &quot;Likely Lyme Disease&quot;}<br> {&quot;prompt&quot;: &quot;CLASSIFY: Fatigue and weight loss. No other symptoms.&quot;, &quot;completion&quot;: &quot;Broad Differential (Diabetes, Hyperthyroidism, Cancer)&quot;}<br> {&quot;prompt&quot;: &quot;CLASSIFY: Patient reports chest pain but normal EKG and troponin.&quot;, &quot;completion&quot;: &quot;Non-Cardiac Chest Pain&quot;}<br> {&quot;prompt&quot;: &quot;CLASSIFY: Patient reports chest pain but normal EKG and troponin.&quot;, &quot;completion&quot;: &quot;Non-Cardiac Chest Pain&quot;}<br>]</pre><p><strong>Key Points About the Dataset Format: </strong><br>- <strong>Prompt</strong>: Introduces the task. <br>- <strong>Instruction</strong>: Specifies what the model should do. <br>- <strong>Completion</strong>: Provides the correct answer or output.</p><p><strong>Uploading Files to S3</strong></p><p>1. Navigate to your <strong>Input Data</strong> folder in S3. <br>2. Click <strong>Upload</strong> and select your training and validation JSON files. <br>3. Confirm the upload was successful.</p><h4>Step 3: Configuring Amazon Bedrock for Fine-Tuning</h4><p>With your data ready, the next step is to set up the fine-tuning job in Amazon Bedrock.</p><p><strong>Accessing Amazon Bedrock</strong></p><p>1. Open the <strong>AWS Management Console</strong> and navigate to <strong>Amazon Bedrock</strong>. <br>2. Select <strong>Custom Models</strong> from the sidebar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S1xD3xmTDYCfkZVkuPDpnQ.png" /></figure><p><strong>Creating a Fine-Tuning Job</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NaPPuhW2fzD0cso1H4TANA.png" /></figure><p>1. Click <strong>Customize Model</strong> and then <strong>Create Fine-Tuning Job</strong>. <br>2. Select the foundation model you want to fine-tune (e.g., <strong>Titan Text G1 Light</strong>). <br>3. Name your fine-tuned model (e.g., `fine-tune-titan`). <br>4. Provide a name for the job configuration (e.g., `fine-tuning-job`).</p><p><strong>Specifying Data Locations</strong></p><p>1. <strong>Training Data</strong>: Browse your S3 bucket and select the training JSON file. <br>2. <strong>Validation Data</strong>: Select the validation JSON file from the same bucket. <br>3. <strong>Output Data</strong>: Choose the <strong>Output Data</strong> folder in S3 for storing metrics.</p><p><strong>Hyperparameter Configuration (Optional)</strong></p><p>Amazon Bedrock provides default hyperparameters, but you can adjust <br>- <strong>Epochs</strong>: Number of times the model processes the entire dataset. <br>- <strong>Batch Size</strong>: Number of samples processed before updating the model. <br>- <strong>Learning Rate</strong>: Speed at which the model learns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nXu9pFb6O9LNnpiINFG_bA.png" /></figure><p>For beginners, sticking with defaults is recommended.</p><p><strong>Setting Up IAM Permissions</strong></p><p>Bedrock needs permissions to read from and write to S3.</p><p>1. Click <strong>Create New Role</strong>. <br>2. Name the role (e.g., `BedrockRole`). <br>3. AWS will automatically configure the necessary permissions.</p><h4>Step 4: Launching the Fine-Tuning Job</h4><p>Once everything is configured:</p><p>1. Review your settings. <br>2. Click <strong>Start Fine-Tuning Job</strong>.</p><p>The process may take several hours, depending on the dataset size and model complexity.</p><p><strong>Monitoring Progress</strong></p><p>You can track the job’s progress in the <strong>Custom Models</strong> section. Metrics and logs will be saved in your S3 <strong>Output Data</strong> folder.</p><h4>Step 5: Using Your Fine-Tuned Model</h4><p>After completion, your fine-tuned model will appear under <strong>Custom Models</strong>. You can now deploy it for inference in your applications.</p><p><strong>Testing the Model</strong></p><p>Use the Test Bedrock feature to input prompts and evaluate responses.</p><p>![Testing Fine-Tuned Model](<a href="https://example.com/test-model.png">https://example.com/test-model.png</a>)</p><p><strong>Best Practices for Fine-Tuning</strong></p><p>1. <strong>Data Quality</strong>: Ensure your dataset is clean and well-structured. <br>2. <strong>Hyperparameter Tuning</strong>: Experiment with different settings for optimal performance. <br>3. <strong>Monitoring</strong>: Regularly check metrics to avoid overfitting or underfitting.</p><p><strong>Conclusion</strong></p><p>Fine-tuning foundation models with Amazon Bedrock empowers you to create highly specialized AI solutions tailored to your needs. By following this guide, you’ve learned how to prepare data, configure a fine-tuning job, and deploy a custom model.</p><p>By following this guide, you’ll be well-equipped to harness the full potential of Amazon Bedrock for your AI projects. Happy fine-tuning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c36e2b954b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are Model’s hyperparameters?]]></title>
            <link>https://suraj-batuwana.medium.com/what-are-models-hyperparameters-d8e3b56b0419?source=rss-b587db5cb48------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8e3b56b0419</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[hyperparameter]]></category>
            <dc:creator><![CDATA[Suraj Batuwana]]></dc:creator>
            <pubDate>Tue, 04 Mar 2025 02:39:12 GMT</pubDate>
            <atom:updated>2025-03-04T02:39:12.231Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*nx02JxNgfhcwQc0PaWDOqw.png" /></figure><p>Hyperparameters are settings or configurations that are external to the model and cannot be learned directly from the data during training. They are set before the training process begins and control the behavior, complexity, and performance of the model. Unlike model parameters (e.g., weights in a neural network or coefficients in linear regression), hyperparameters are not updated during training.</p><p><strong>Examples of Hyperparameters</strong></p><p>Here are some common examples of hyperparameters across different machine learning models:</p><p>1. <strong>Linear regression</strong>:<br>Regularization Strength (Alpha):<br> — Controls the amount of regularization (e.g., in Ridge or Lasso regression).<br> — Example: `alpha = 0.1` in Ridge regression.</p><p>2. <strong>Decision Trees</strong>:<br>Maximum Depth:<br> — Limits the depth of the tree to prevent overfitting.<br> — Example: `max_depth = 5`.<br> Minimum Samples per Leaf:<br> — Specifies the minimum number of samples required to be at a leaf node.<br> — Example: `min_samples_leaf = 10`.</p><p>3. <strong>Random Forest</strong>:<br> Number of Trees (n_estimators):<br> — Controls the number of trees in the forest.<br> — Example: `n_estimators = 100`.<br> Maximum Features:<br> — Limits the number of features considered for splitting a node.<br> — Example: `max_features = ‘sqrt’`.</p><p>4. <strong>Support Vector Machines (SVM)</strong>:<br>Kernel Type:<br> — Specifies the type of kernel function (e.g., linear, polynomial, radial basis function).<br> — Example: `kernel = ‘rbf’`.<br>Regularization Parameter ©:<br> — Controls the trade-off between achieving a low training error and a low testing error.<br> — Example: `C = 1.0`.</p><p>5. <strong>Neural Networks:</strong><br>Learning Rate:<br> — Controls the step size during weight updates.<br> — Example: `learning_rate = 0.001`.<br>Number of Hidden Layers:<br> — Specifies the number of hidden layers in the network.<br> — Example: `num_hidden_layers = 3`.<br> Number of Neurons per Layer:<br> — Controls the number of neurons in each hidden layer.<br> — Example: `neurons_per_layer = 64`.</p><p>6. <strong>k-Nearest Neighbors (k-NN)</strong>:<br>Number of Neighbors (k):<br> — Specifies the number of neighbors to consider for predictions.<br> — Example: `n_neighbors = 5`.</p><p>7. <strong>Gradient Boosting Machines (e.g., XGBoost, LightGBM)</strong>:<br> Learning Rate:<br> — Controls the contribution of each tree to the final model.<br> — Example: `learning_rate = 0.01`.<br>Number of Boosting Stages:<br> — Specifies the number of boosting stages (trees) to build.<br> — Example: `n_estimators = 200`.</p><p><strong>Why Are Hyperparameters Important?</strong></p><p>Hyperparameters play a crucial role in determining the performance of a machine learning model. They influence:<br><strong>Model Complexity</strong>: Controls how complex the model can be (e.g., tree depth in decision trees).<br><strong>Training Speed</strong>: Affects how quickly the model learns (e.g., learning rate in neural networks).<br><strong>Generalization</strong>: Helps prevent overfitting or underfitting (e.g., regularization strength).</p><p><strong>How to Tune Hyperparameters</strong></p><p>Hyperparameter tuning is the process of finding the best set of hyperparameters for a model. Common methods include:</p><p>1. <strong>Grid Search</strong>:<br>Exhaustively searches through a predefined set of hyperparameter values.<br>Example: Trying all combinations of `max_depth = [3, 5, 7]` and `min_samples_leaf = [1, 2, 4]`.</p><p>2. <strong>Random Search</strong>:<br> Randomly samples hyperparameter values from a predefined distribution.<br> Example: Randomly selecting `learning_rate` from a range of `[0.001, 0.1]`.</p><p>3. Bayesian Optimization:<br>Uses probabilistic models to find the best hyperparameters efficiently.<br> — Example: Using libraries like `Optuna` or `Hyperopt`.</p><p>4. Automated Tools:<br>Libraries like `scikit-learn` (e.g., `GridSearchCV`, `RandomizedSearchCV`) or `Keras Tuner` can automate hyperparameter tuning.</p><p><strong>Example of Hyperparameter Tuning</strong></p><p>Suppose you are training a Random Forest model for a classification task. You want to tune the following hyperparameters:<br> `n_estimators`: Number of trees in the forest.<br> `max_depth`: Maximum depth of each tree.<br>`min_samples_split`: Minimum number of samples required to split a node.</p><p>Using Grid Search in `scikit-learn`:</p><pre>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.model_selection import GridSearchCV<br><br># Define the model<br>model = RandomForestClassifier()<br><br># Define the hyperparameter grid<br>param_grid = {<br>    &#39;n_estimators&#39;: [50, 100, 200],<br>    &#39;max_depth&#39;: [None, 10, 20],<br>    &#39;min_samples_split&#39;: [2, 5, 10]<br>}<br><br># Perform grid search<br>grid_search = GridSearchCV(estimator=model, param_grid=param_grid, cv=5)<br>grid_search.fit(X_train, y_train)<br><br># Best hyperparameters<br>print(&quot;Best Hyperparameters:&quot;, grid_search.best_params_)</pre><p>Output:</p><pre>Best Hyperparameters: {&#39;max_depth&#39;: 10, &#39;min_samples_split&#39;: 2, &#39;n_estimators&#39;: 200}</pre><p>By tuning hyperparameters, you can optimize your model’s performance and ensure it generalizes well to new data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8e3b56b0419" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Neptune Date Search]]></title>
            <link>https://suraj-batuwana.medium.com/aws-neptune-date-search-33ea0b07cc0e?source=rss-b587db5cb48------2</link>
            <guid isPermaLink="false">https://medium.com/p/33ea0b07cc0e</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[date]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[neptune]]></category>
            <dc:creator><![CDATA[Suraj Batuwana]]></dc:creator>
            <pubDate>Fri, 12 Jul 2024 07:04:59 GMT</pubDate>
            <atom:updated>2024-07-12T07:04:59.024Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Mjg-uB7QbN11XCOK.png" /></figure><p>Gremlin does not currently support mathematical operations on dates, such as “today minus 40 days”. However, you can still compare dates using the Gremlin datetime function. For example:</p><pre>g.V().hasLabel(&#39;User&#39;).has(&#39;created&#39;,lte(datetime(&#39;2024-07-11T16:27:24.398Z&#39;))).elementMap().toList()<br></pre><p>OR</p><pre>g.V().hasLabel(&quot;User&quot;).has(&quot;created&quot;,between(datetime(&quot;2024-07-11T00:35:44.741Z&quot;),datetime(&quot;2024-07-12T00:35:44.741Z&quot;))).elementMap().toList()</pre><p>For this to work, you need to have created the vertex using:</p><pre> g.addV(&quot;User&quot;)<br>                .property(&quot;id&quot;, 1))<br>                .property(&quot;firstName&quot;, &quot;Suraj)<br>                .property(&quot;lastName&quot;, &quot;Batuwana&quot;)<br>                .property(single,&quot;created&quot;, datetime(&quot;2024-07-11T16:27:24.398Z))<br>                .next();</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33ea0b07cc0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Cognito authentication with Amplify 1]]></title>
            <link>https://suraj-batuwana.medium.com/aws-cognito-authentication-with-amplify-1-570d0a1789bc?source=rss-b587db5cb48------2</link>
            <guid isPermaLink="false">https://medium.com/p/570d0a1789bc</guid>
            <category><![CDATA[cognito]]></category>
            <category><![CDATA[cognito-user-pools]]></category>
            <category><![CDATA[amplify]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Suraj Batuwana]]></dc:creator>
            <pubDate>Wed, 10 Jul 2024 14:00:41 GMT</pubDate>
            <atom:updated>2024-07-10T14:00:41.952Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>What is AWS Cognito?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rwuum2jLljuiU-UM83BV5A.jpeg" /></figure><p>AWS Cognito is a powerful authentication and user management service provided by Amazon Web Services (AWS). It simplifies user sign-up, sign-in, and access control for web and mobile applications. With AWS Cognito, developers can seamlessly integrate user authentication and authorization into their applications, enhancing security and user experience.</p><p><strong>Key Features of AWS Cognito</strong></p><ol><li><strong>User sign-up and sign-in</strong>:<br><strong>User Pools</strong>: AWS Cognito user pools enable developers to manage user registration, authentication, and account recovery. Users can sign in using social identity providers (like Google, Facebook, and Amazon), enterprise identity providers (via SAML and OIDC), or directly through the user pool.<br><strong>Identity Pools</strong>: AWS Cognito identity pools provide temporary AWS credentials, allowing users to access AWS services. These credentials can be granted to both authenticated and unauthenticated users.</li></ol><p>2. <strong>Multi-Factor Authentication (MFA)</strong>:<br> Enhance security by requiring multiple verification factors for user authentication.</p><p>3. <strong>Federation</strong>:<br> Support for federated identities allows users to sign in using their existing credentials from social or enterprise identity providers.</p><p>4. <strong>User Management</strong>:<br>Comprehensive user management features, including sign-up, sign-in, account verification, password recovery, and user profile updates.</p><p>5. <strong>Advanced Security Features</strong>:<br> Built-in monitoring and response capabilities for suspicious sign-in attempts, along with customizable security policies for passwords and MFA.</p><p>6. <strong>Customizable authentication flows</strong>:<br> Use AWS Lambda triggers to customize authentication and authorization workflows.</p><p>7. <strong>Scalability</strong>:<br> Designed to handle millions of users, making AWS Cognito suitable for applications of any size.</p><p><strong>How AWS Cognito Works</strong></p><ol><li><strong>User Pools</strong>:<br>Users register and sign in to the user pool.<br>Sign in through social and enterprise identity providers is also supported.<br>AWS Cognito issues JSON Web Tokens (JWTs) after successful authentication, which can be used to secure API calls to your backend services.</li></ol><p>2. <strong>Identity Pools</strong>:<br>Provides temporary AWS credentials for accessing AWS services after user authentication.<br> Can be integrated with user pools or external identity providers for federated access.</p><p><strong>Use Cases for AWS Cognito</strong></p><p>1. <strong>Mobile and Web Applications</strong>:<br> Manage user authentication and maintain user sessions across different devices and platforms seamlessly.</p><p>2. <strong>APIs and Backend Services</strong>:<br>Secure API endpoints using tokens issued by AWS Cognito user pools, ensuring only authenticated users can access your backend services.</p><p>3. <strong>Enterprise Applications</strong>:<br> Integrate with enterprise identity providers for single sign-on (SSO) capabilities, streamlining user access and improving security.</p><p>4. <strong>Social Login</strong>:<br>Allow users to log in with their existing social media accounts, reducing friction and enhancing user experience.</p><p><strong>Example Workflow with AWS Cognito</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hm0nbFPBKKe89sYPPD51Bw.jpeg" /></figure><p>1. <strong>User registration</strong>:<br> The user signs up via a web or mobile app.<br> AWS Cognito stores user credentials in a user pool.</p><p>2. <strong>User Authentication</strong>:<br> User signs in with their credentials.<br> AWS Cognito verifies credentials and issues tokens (ID, access, and refresh tokens).</p><p>3. <strong>Access Control</strong>:<br> Use tokens to authenticate API calls or access other AWS resources via an identity pool.</p><p><strong>Benefits of Using AWS Cognito</strong></p><p>By leveraging AWS Cognito, developers can offload the complexities of user authentication and focus on building and enhancing application features. AWS Cognito provides robust security features, scalability, and ease of integration, making it an ideal choice for managing user identities in modern web and mobile applications.</p><h3>Steps to Create a Cognito user pool</h3><p>Creating a Cognito User Pool with images involves a series of steps that can be performed using the AWS Management Console. Here’s a detailed guide to create a Cognito User Pool with images for user profiles:</p><h4>Step 1: Sign in to the AWS Management Console</h4><ol><li>Open the <a href="https://aws.amazon.com/console/">AWS Management Console</a>.</li><li>Sign in with your AWS account credentials.</li></ol><h4>Step 2: Navigate to Cognito</h4><ol><li>In the AWS Management Console, search for “Cognito” in the services search bar.</li><li>Select “Amazon Cognito” from the list.</li></ol><h4>Step 3: Create a User Pool</h4><ol><li>Click on “Manage User Pools” to start the process.</li><li>Click on “Create a user pool.”</li></ol><h4>Configure sign-in experience</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Nr3yrqMG7uShNbaOYxV4A.png" /></figure><p>In the “Configure sign-in experience” section, I selected “Cognito user pool,” as shown in Figure 2. Under the “Cognito user pool sign-in” options, I chose “User name.” This allows users to sign in using their “User Name.” After making this selection, I clicked the “Next” button.</p><p>You can also choose the federated providers you want to use with this user pool. For more information on this topic, refer to the “Adding User Pool Sign-in Through a Third Party” link.</p><blockquote>Remember, “<strong>Cognito user pool sign-in options can’t be changed after the user pool has been created.</strong>”</blockquote><h4>Configure security requirements</h4><p>In the “Password Policy” section, I selected “Cognito defaults.” This means users will follow the default password requirements when creating their passwords.</p><p>In the “Multi-factor authentication” section, I selected “No MFA.” Alternatively, you can choose other options to enable multi-factor authentication (MFA) .</p><p>In the “User account recovery” section, I configured the method for users to recover their accounts if they forget their passwords. I selected email because SMS messages incur charges, as shown in the below image. Then, I clicked the “Next” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*aMwgFhaO0dj-DvMmZjjDfg.png" /></figure><h4>Configure sign-up experience</h4><p>In the “Configure sign-up experience” section, I selected “Enable self-registration,” as shown below. This option displays a “Sign up” link on the sign-in page in the hosted UI and allows the use of public APIs to create new user accounts.</p><blockquote><strong>Note</strong>: Enabling user sign-up in your user pool allows anyone on the internet to sign up for an account and sign in to your apps. Do not enable self-registration in your user pool until you are ready to open your app to public sign-up.</blockquote><p>In the “Attribute verification and user account confirmation” section, I selected “Allow Cognito to automatically send messages to verify and confirm” and “Send email message to verify email address,” as shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aHke7c-DJSsyqvhtxF7kTA.png" /></figure><p>In the “Required attributes” section, I chose “email,” “given_name,” and “family_name,” as shown below. Cognito will still function even if no attributes are selected here. Then, I clicked the “Next” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yIyyUqk_2m72ANRpZF48iA.png" /></figure><h4>Configure message delivery</h4><p>In the “Configure message delivery” section, I selected “Send email with Cognito,” as shown in below image. By default, emails will be sent to users from the address “<a href="/service/mailto:no-reply@verificationemail.com">no-reply@verificationemail.com</a>.” Then, I clicked the “Next” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nb91iEPIXHsJ6IZzXUbxog.png" /></figure><h4>Integrate your app</h4><p>In the “Integrate your app” section, I entered my user pool name as “application-user-pool,” as depicted in the below image. Once created, the user pool name cannot be changed, and it must be unique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FA273b5z70x26jWJ9iOTIQ.png" /></figure><p>In the “Hosted authentication” section, I selected “Use the Cognito Hosted UI.”</p><blockquote>The Cognito Hosted UI is a customizable(<strong>not 100% </strong>) authentication interface provided by Amazon Cognito. It allows developers to easily add user sign-up, sign-in, and account recovery to web and mobile apps without writing backend code or managing infrastructure for these functionalities.</blockquote><p>In the “Domain” section, I selected “Use a Cognito domain” and entered a domain prefix as “suraj-application-pool.&quot;. If you opt for “<a href="https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-add-custom-domain.html">Use a custom domain</a>,” you can use your own domain name, but you must also possess a valid certificate for that domain, as shown in the same figure. For simplicity, I proceeded by choosing “Use a Cognito domain” in this instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ugQfxTuPT985r1LDZNAGnQ.png" /></figure><p>In the “Initial app client” section, I selected “Public client” and named the app client “my-application-client”. I entered the allowed callback URL as “<a href="https://mr.cmakkaya-awsdevops.link/">http://</a>localhost:8080/welcome, as displayed below.</p><p>The “Allowed callback URLs” field redirects users to this address after authentication, which corresponds to where our application is hosted.</p><blockquote>Amazon Cognito mandates HTTPS for callback URLs, except for <a href="http://localhost/">http://localhost</a>, which can be used for testing purposes only.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*dFKHbN-q6Dle2Btb5dCEgw.png" /></figure><p>On the “Review and Create” page, after reviewing the entered values, I clicked the “Create user pool” button, as depicted in the below image.</p><p>On the “Review and Create” page, after reviewing the entered values, I clicked the “Create user pool” button, as depicted below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*14R79cDwb-_5F4vaz3pD6A.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=570d0a1789bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Storing and Querying User Geo Data in AWS Neptune with Springboot]]></title>
            <link>https://suraj-batuwana.medium.com/storing-and-querying-user-geo-data-in-aws-neptune-with-springboot-ca50ed545c99?source=rss-b587db5cb48------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca50ed545c99</guid>
            <category><![CDATA[graphdb]]></category>
            <category><![CDATA[neptune]]></category>
            <category><![CDATA[spring-boot]]></category>
            <dc:creator><![CDATA[Suraj Batuwana]]></dc:creator>
            <pubDate>Wed, 10 Jul 2024 13:56:42 GMT</pubDate>
            <atom:updated>2024-07-10T13:56:42.662Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="In today’s data-driven landscape, numerous AWS customers are increasingly turning to purpose-built databases as a strategic solution to tackle their complex business challenges. These databases offer tailored environments that enable efficient storage, integration, and management of diverse datasets. One prominent area where these solutions shine is in handling geospatial data, such as latitude and longitude coordinates, within AWS Neptune." src="https://cdn-images-1.medium.com/max/1024/0*mTWumR9inBy5gKcr.jpeg" /></figure><p>In today’s data-driven landscape, numerous AWS customers are increasingly turning to purpose-built databases as a strategic solution to tackle their complex business challenges. These databases offer tailored environments that enable efficient storage, integration, and management of diverse datasets. One prominent area where these solutions shine is in handling geospatial data, such as latitude and longitude coordinates, within AWS Neptune.</p><h3>Introduction to Purpose-Built Databases</h3><p>AWS customers face diverse and evolving business requirements that demand robust data management solutions. Traditional relational databases, while versatile, may struggle to efficiently handle specialized data types and access patterns. Purpose-built databases, on the other hand, are designed to excel in specific use cases, offering optimized performance, scalability, and flexibility tailored to the data they manage.</p><h3>The Appeal of Purpose-Built Databases</h3><p><strong>1. Specialized Capabilities</strong>: Purpose-built databases are crafted to address specific challenges, such as handling geospatial data, time-series data, graph data, or unstructured data. This specialization allows them to employ unique data structures, indexing methods, and query languages optimized for their intended workload.</p><p><strong>2. Performance and Scalability</strong>: By focusing on specific data types and access patterns, these databases can achieve superior performance and scalability compared to general-purpose solutions. They are engineered to efficiently manage and query large volumes of data, ensuring responsive applications and analytics.</p><p><strong>3. Flexibility in Data Modeling</strong>: Purpose-built databases often support flexible data modeling approaches that align closely with the data’s inherent structure and relationships. This enables developers to design schemas that maximize performance and simplify application logic.</p><h3>Geospatial Data in AWS Neptune</h3><p>AWS Neptune is Amazon’s managed graph database service, designed to store and query highly connected data. While traditionally used for managing complex relationships in data, Neptune is also adept at handling geospatial data through its graph data model.</p><ol><li><strong>Graph-Based Approach</strong>: Geospatial data, such as latitude and longitude coordinates, can be represented as vertices (nodes) and edges (relationships) in a graph database like Neptune. This representation allows for efficient storage and traversal of spatial relationships, such as proximity queries or routing algorithms.</li><li><strong>Spatial Indexing</strong>: Neptune supports spatial indexing capabilities, which enhance query performance for geospatial data. These indexes enable fast retrieval of vertices based on their geographic proximity, ensuring that applications can quickly identify nearby locations or analyze spatial patterns.</li><li><strong>Integration with AWS Services</strong>: AWS Neptune seamlessly integrates with other AWS services, such as Amazon S3 for data storage and retrieval, Amazon CloudWatch for monitoring, and AWS IAM for security management. This integration streamlines data workflows and enhances operational efficiency.</li></ol><h3>Use Cases and Benefits</h3><ol><li><strong>Location-Based Services</strong>: Applications leveraging geospatial data, such as mapping applications, logistics optimization tools, or IoT (Internet of Things) platforms, benefit from Neptune’s ability to model and query complex spatial relationships.</li><li><strong>Network Analysis</strong>: Geospatial data often underpins network analysis scenarios, such as social network analysis or supply chain optimization. Neptune’s graph capabilities enable sophisticated analysis of interconnected data points, revealing insights that drive strategic decisions.</li><li><strong>Real-Time Analytics</strong>: With its scalable architecture and support for SPARQL and Gremlin query languages, Neptune empowers real-time analytics on geospatial data. Organizations can derive actionable insights from dynamic datasets, enhancing responsiveness and competitiveness.</li></ol><p>Let’s consider storing a dataset like the following in Neptune:</p><pre>{<br>    &quot;id&quot;: &quot;edcb5ea3-e1b3-4727-b5e0-596fe8c5db36&quot;,<br>    &quot;apiKey&quot;: &quot;apiKey-215&quot;,<br>    &quot;firstName&quot;: &quot;FirstName-215&quot;,<br>    &quot;lastName&quot;: &quot;LastName-215&quot;,<br>    &quot;email&quot;: &quot;user215@example.com&quot;,<br>    &quot;latitude&quot;: -35.34682739710193,<br>    &quot;longitude&quot;: 149.1357153157512<br>}</pre><p>To create an endpoint in a Java Spring Boot application for searching by latitude and longitude using Apache TinkerPop Gremlin, you’ll need to set up a few components: Spring Boot application structure, endpoint definition, service layer for handling queries, and configuration for Apache TinkerPop Gremlin. Here’s a step-by-step guide:</p><h3>1. Project Setup</h3><p>Assuming you have a Spring Boot project set up with Gradle and the necessary dependencies (org.apache.tinkerpop:gremlin-driver, org.apache.tinkerpop:gremlin-core, org.apache.tinkerpop:tinkergraph-gremlin), ensure your build.gradle file includes these dependencies correctly:</p><pre>dependencies {<br>    implementation &#39;org.springframework.boot:spring-boot-starter-web&#39;<br>    implementation &#39;org.apache.tinkerpop:gremlin-driver:3.4.10&#39;<br>    implementation &#39;org.apache.tinkerpop:gremlin-core:3.4.10&#39;<br>    implementation &#39;org.apache.tinkerpop:tinkergraph-gremlin:3.4.10&#39;<br>    implementation &#39;com.fasterxml.jackson.core:jackson-databind&#39;<br>    implementation &#39;org.slf4j:slf4j-simple:1.7.32&#39; // Optional for logging<br>}</pre><h3>2. Define the User Entity</h3><p>Create a User class to represent your data model. Include fields for id, apiKey, firstName, lastName, displayName, profileType, email, latitude, and longitude. Here&#39;s a basic example:</p><pre>import lombok.Data;<br><br>@Data<br>public class User {<br>    private String id;<br>    private String apiKey;<br>    private String firstName;<br>    private String lastName;<br>    private String displayName;<br>    private String profileType;<br>    private String email;<br>    private double latitude;<br>    private double longitude;<br>}</pre><h3>3. Create the Service</h3><p>Create a service class where you’ll implement the logic to search for users based on latitude and longitude. This example assumes a simple in-memory approach using TinkerGraph for demonstration purposes:</p><pre>@Service<br>public class GremlinService {<br><br>    private static final double EARTH_RADIUS = 6371.0;<br><br>    private GraphTraversalSource g;<br><br>    private Cluster cluster;<br><br>    public GremlinService(Client gremlinClient) {<br>        Cluster.Builder builder = Cluster.build();<br>        builder.addContactPoint(&quot;neptune connection string&quot;);<br>        builder.port(8182);<br>        builder.enableSsl(true);<br>        builder.serializer(new GraphBinaryMessageSerializerV1());<br>        this.g = traversal().withRemote(DriverRemoteConnection.using(cluster));<br>    }<br><br>    public List&lt;User&gt; searchUsersByRadius(double centerLat, double centerLon, double radiusInKm) {<br><br>        double[] boundingBox = getBoundingBox(centerLat, centerLon, radiusInKm);<br>        double minLat = boundingBox[0];<br>        double maxLat = boundingBox[1];<br>        double minLon = boundingBox[2];<br>        double maxLon = boundingBox[3];<br><br>        // Step 1: Retrieve users within the bounding box<br>        List&lt;Map&lt;Object, Object&gt;&gt; candidateUsers = g.V()<br>                .hasLabel(&quot;User&quot;)<br>                .has(&quot;latitude&quot;, P.between(minLat, maxLat))<br>                .has(&quot;longitude&quot;, P.between(minLon, maxLon)<br>                .elementMap()<br>                .toList();<br><br>        // Step 2: Filter users within the radius using Haversine formula<br>        return candidateUsers.stream()<br>                .map(map -&gt; mapToClass(map, User.class))<br>                .filter(user -&gt; isWithinRadius(user, centerLat, centerLon, radiusInKm))<br>                .collect(Collectors.toList());<br>    }<br><br>    public static double[] getBoundingBox(double centerLat, double centerLon, double radius) {<br>        double deltaLat = Math.toDegrees(radius / EARTH_RADIUS);<br>        double deltaLon = Math.toDegrees(radius / (EARTH_RADIUS * Math.cos(Math.toRadians(centerLat))));<br><br>        double minLat = centerLat - deltaLat;<br>        double maxLat = centerLat + deltaLat;<br>        double minLon = centerLon - deltaLon;<br>        double maxLon = centerLon + deltaLon;<br><br>        return new double[]{minLat, maxLat, minLon, maxLon};<br>    }<br><br>    public static &lt;T&gt; T mapToClass(Map&lt;Object, Object&gt; map, Class&lt;T&gt; clazz) {<br>        try {<br>            T instance = clazz.getDeclaredConstructor().newInstance();<br>            for (Map.Entry&lt;Object, Object&gt; entry : map.entrySet()) {<br>                String fieldName = entry.getKey().toString();<br>                Object fieldValue = entry.getValue();<br><br>                try {<br>                    Field field = clazz.getDeclaredField(fieldName);<br>                    field.setAccessible(true);<br><br>                    // Handle special case for lists<br>                    if (fieldValue instanceof List &amp;&amp; !((List&lt;?&gt;) fieldValue).isEmpty()) {<br>                        fieldValue = ((List&lt;?&gt;) fieldValue).get(0);<br>                    }<br><br>                    // Cast fieldValue to the correct type<br>                    if (field.getType().isAssignableFrom(fieldValue.getClass())) {<br>                        field.set(instance, fieldValue);<br>                    } else if (field.getType() == double.class || field.getType() == Double.class) {<br>                        field.set(instance, Double.valueOf(fieldValue.toString()));<br>                    } else if (field.getType() == long.class || field.getType() == Long.class) {<br>                        field.set(instance, Long.valueOf(fieldValue.toString()));<br>                    } else if (field.getType() == int.class || field.getType() == Integer.class) {<br>                        field.set(instance, Integer.valueOf(fieldValue.toString()));<br>                    } else if (field.getType() == String.class) {<br>                        field.set(instance, fieldValue.toString());<br>                    }<br>                } catch (NoSuchFieldException ignored) {<br>                    // Skip unknown fields<br>                }<br>            }<br>            return instance;<br>        } catch (Exception e) {<br>            throw new RuntimeException(&quot;Error mapping to class &quot; + clazz.getName(), e);<br>        }<br>    }<br>    private boolean isWithinRadius(User user, double centerLat, double centerLon, double radiusInKm) {<br>        double latDistance = Math.toRadians(user.getLatitude() - centerLat);<br>        double lonDistance = Math.toRadians(user.getLongitude() - centerLon);<br>        double a = Math.sin(latDistance / 2) * Math.sin(latDistance / 2) +<br>                Math.cos(Math.toRadians(centerLat)) * Math.cos(Math.toRadians(user.getLatitude())) *<br>                        Math.sin(lonDistance / 2) * Math.sin(lonDistance / 2);<br>        double c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));<br>        double distance = EARTH_RADIUS * c;<br>        return distance &lt;= radiusInKm;<br>    }<br>}</pre><p>In your GremlinService class, the Haversine formula would typically be used to calculate distances between geographic points stored in a database (like AWS Neptune) or provided as input. This is useful, for example, in querying nearby locations or calculating distances for location-based recommendations or optimizations.</p><p>Here’s a basic example of how you might use the Haversine formula in Java to calculate distances:</p><pre>private boolean isWithinRadius(User user, double centerLat, double centerLon, double radiusInKm) {<br>        double latDistance = Math.toRadians(user.getLatitude() - centerLat);<br>        double lonDistance = Math.toRadians(user.getLongitude() - centerLon);<br>        double a = Math.sin(latDistance / 2) * Math.sin(latDistance / 2) +<br>                Math.cos(Math.toRadians(centerLat)) * Math.cos(Math.toRadians(user.getLatitude())) *<br>                        Math.sin(lonDistance / 2) * Math.sin(lonDistance / 2);<br>        double c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));<br>        double distance = EARTH_RADIUS * c;<br>        return distance &lt;= radiusInKm;<br>    }</pre><h4>What is Haversine formula</h4><p>The Haversine formula is a mathematical formula used to calculate the shortest distance between two points on the surface of a sphere, given their latitudes and longitudes. It’s commonly used in navigation and geographic applications where calculating distances on the Earth’s surface is necessary.</p><h4>Understanding the Formula</h4><p>The Haversine formula is based on spherical trigonometry and calculates the great-circle distance between two points on the surface of a sphere, such as the Earth.</p><h4>Why Use Haversine Formula?</h4><ul><li><strong>Accuracy</strong>: It provides reasonably accurate results for short distances (up to a few thousand kilometers/miles) on the Earth’s surface, considering the spherical shape of the Earth.</li><li><strong>Geospatial Applications</strong>: It is widely used in geospatial applications such as location-based services (LBS), mapping, and GPS navigation systems to compute distances between coordinates.</li></ul><h3>4. Create the Controller</h3><pre>@GetMapping(&quot;/search&quot;)<br>    public ResponseEntity&lt;List&lt;User&gt;&gt; searchUsersByRadius(@RequestParam double latitude,<br>                                                          @RequestParam double longitude,<br>                                                          @RequestParam double radius) {<br>        List&lt;User&gt; users = gremlinService.searchUsersByRadius(latitude, longitude, radius);<br>        return ResponseEntity.ok(users);<br>    }</pre><h3>5. Run and Test</h3><p>Run your Spring Boot application and test the endpoint using tools like Postman or curl:</p><pre>http://localhost:8080/users/search?latitude=-35.30489&amp;longitude=149.14121&amp;radius=5</pre><h3>7. Performace</h3><p>When executing this operation against a dataset of one million records, it required approximately 17 to 20 seconds to retrieve all the data.</p><p>Improving the performance of spatial searches in a large dataset such as 3,000,000 records involves a combination of better querying techniques and leveraging spatial indexing. Here are several strategies to improve the performance:</p><h4>Ensure Spatial Indexing</h4><p>Ensure that your latitude and longitude properties are indexed. This can significantly speed up range queries:</p><pre>// Create spatial index in Gremlin<br>schema.propertyKey(&#39;latitude&#39;).Double().ifNotExists().create();<br>schema.propertyKey(&#39;longitude&#39;).Double().ifNotExists().create();<br>schema.vertexLabel(&#39;User&#39;).properties(&#39;latitude&#39;, &#39;longitude&#39;).index(&#39;search&#39;).ifNotExists().create()</pre><h4>Refine Bounding Box Query</h4><p>Make sure the initial bounding box query is as precise as possible to reduce the number of candidates before applying the Haversine filter.</p><pre>g.V().hasLabel(&quot;User&quot;)<br>                .has(&quot;latitude&quot;, P.between(minLat, maxLat))<br>                .has(&quot;longitude&quot;, P.between(minLon, maxLon))<br>                .skip(offset)<br>                .limit(batchSize)<br>                .elementMap()<br>                .toList();</pre><p>Based on the above approach, I incorporated pagination using offset and limit, significantly reducing the dataset size and improving performance to 600 milliseconds.</p><h4>Issues</h4><p>The code above will encounter errors when executed locally because AWS Neptune clusters typically do not have public accessibility enabled by default. To address this challenge, I will outline solutions in a separate blog post.</p><h3>Dataset</h3><p>Now, the significant question arises: How should we efficiently insert a dataset of over one million records into AWS Neptune, as per your preference?</p><p>To create a Neptune notebook with Gremlin and Python 3, you’ll need to set up an environment where you can interact with Neptune using Jupyter notebooks. Here’s a step-by-step guide to get you started:</p><h4>Prerequisites</h4><ul><li>Python 3.7 or higher</li><li>Jupyter Notebook (optional, for interactive testing)</li><li>AWS account with access to AWS Neptune</li><li>Gremlin Python driver (gremlin-python) installed (pip install gremlinpython)</li></ul><h4>Step-by-Step Guide</h4><h4>1. Access Neptune Cluster Endpoint</h4><p>First, locate and note down the endpoint of your Neptune cluster. You can find this in the AWS Management Console under “Neptune” &gt; “Clusters” &gt; Select your cluster &gt; “Properties” tab.</p><h4>2. Install Required Libraries</h4><p>If you are using a local Jupyter notebook server, ensure you have the necessary Python libraries installed:</p><pre>!pip install gremlinpython</pre><p>Here is the sample python script to generate 3,000,000 data</p><pre>from gremlin_python.structure.graph import Graph<br>from gremlin_python.process.graph_traversal import __<br>from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection<br>import random<br>import math<br>import uuid<br><br>def generate_random_lat_lon(center_lat, center_lon, radius):<br>    earth_radius = 6371.0  # km<br>    distance = radius * math.sqrt(random.random())<br>    angle = random.random() * 2 * math.pi<br><br>    delta_lat = distance * math.cos(angle) / earth_radius<br>    delta_lon = distance * math.sin(angle) / (earth_radius * math.cos(math.radians(center_lat)))<br><br>    new_lat = center_lat + math.degrees(delta_lat)<br>    new_lon = center_lon + math.degrees(delta_lon)<br><br>    return new_lat, new_lon<br><br>graph = Graph()<br>connection = DriverRemoteConnection(&#39;wss://neptune-connection-details:8182/gremlin&#39;, &#39;g&#39;)<br>g = graph.traversal().withRemote(connection)<br><br>center_lat = -35.30489 <br>center_lon = 149.14121<br>radius = 5.0<br><br>for i in range(1, 3000000):<br>    new_lat, new_lon = generate_random_lat_lon(center_lat, center_lon, radius)<br>    g.addV(&#39;User&#39;)\<br>        .property(&#39;id&#39;, str(uuid.uuid4()))\<br>        .property(&#39;apiKey&#39;, f&#39;apiKey-{i}&#39;)\<br>        .property(&#39;firstName&#39;, f&#39;FirstName-{i}&#39;)\<br>        .property(&#39;lastName&#39;, f&#39;LastName-{i}&#39;)\<br> <br>        .property(&#39;email&#39;, @example.com&quot;&gt;f&#39;user{i}@example.com&#39;)\<br>        .property(&#39;latitude&#39;, new_lat)\<br>        .property(&#39;longitude&#39;, new_lon)\<br>        .next()<br><br>connection.close()</pre><p>By change range(1, 3000000), you can create any big data set</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca50ed545c99" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrate BitBucket Pipelines with SonarCloud]]></title>
            <link>https://suraj-batuwana.medium.com/integrate-bitbucket-pipelines-with-sonarcloud-2c1023af57f9?source=rss-b587db5cb48------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c1023af57f9</guid>
            <category><![CDATA[bitbucket]]></category>
            <category><![CDATA[bitbucket-pipelines]]></category>
            <category><![CDATA[sonarqube]]></category>
            <category><![CDATA[sonarcloud]]></category>
            <dc:creator><![CDATA[Suraj Batuwana]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 03:07:23 GMT</pubDate>
            <atom:updated>2024-05-01T12:05:33.979Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/805/0*DLtfHfHy9xWs2gIw.jpeg" /></figure><p>BBucket pipelines are an essential tool for modern software development, facilitating continuous integration and deployment (CI/CD) workflows. These pipelines automate the process of building, testing, and deploying code changes, ensuring a smooth and efficient development lifecycle. Let’s delve into a hypothetical scenario where a software team utilizes Bitbucket pipelines to streamline their workflow.</p><p>Pipeline Configuration:</p><pre>image: node:14.15.1<br><br>pipelines:<br>  branches:<br>    master:<br>      - step:<br>          name: Build and Test<br>          script:<br>            - npm install<br>            - npm run build<br>            - npm test<br>      - step:<br>          name: Deploy to Production<br>          script:<br>            - npm run deploy</pre><p>Explanation:</p><p><strong>Build and Test</strong>: Upon pushing changes to the master branch, Bitbucket triggers the first step of the pipeline. This step installs dependencies, builds the project, and runs automated tests to ensure code quality and functionality.</p><p><strong>Deploy to Production</strong>: Assuming the build and tests are successful, the pipeline proceeds to deploy the changes to the production environment. This step could involve various deployment strategies, such as rolling updates or blue-green deployments, to minimize downtime and ensure a seamless release.</p><p><strong>Benefits</strong>: Automation: Bitbucket pipelines automate repetitive tasks, reducing manual intervention and potential errors. Developers can focus on writing code rather than managing deployments manually.</p><p><strong>Consistency</strong>: By defining the pipeline stages in code, teams ensure consistency across environments. Everyone follows the same deployment process, mitigating configuration drift and environment inconsistencies.</p><p><strong>Faster Feedback</strong>: Pipelines provide instant feedback on code changes. If a build or test fails, developers are immediately notified, enabling them to address issues promptly and maintain a high level of code quality.</p><p><strong>Scalability</strong>: As the project grows, Bitbucket pipelines scale effortlessly. Whether there are ten or a hundred developers working on the project, the pipeline adapts to the increased workload, ensuring efficient CI/CD processes.</p><p>In conclusion, Bitbucket pipelines revolutionize the software development lifecycle by automating repetitive tasks, ensuring consistency, providing fast feedback, and enabling scalable CI/CD workflows. By incorporating pipelines into their workflow, development teams can deliver high-quality software faster and more reliably.</p><h3>Sonarcloud</h3><p>Integrating Bitbucket Pipelines with SonarCloud can significantly enhance code quality by performing static code analysis on your project’s codebase. SonarCloud provides comprehensive insights into code quality, security vulnerabilities, and code smells, enabling teams to identify and address issues early in the development process. Here’s how you can seamlessly integrate Bitbucket Pipelines with SonarCloud:</p><p>1. Set up SonarCloud:</p><ul><li>Begin by signing up for a SonarCloud account if you haven’t already. You can log in using your Bitbucket credentials.</li><li>Create a new project in SonarCloud and generate an authentication token. This token will be used to authenticate the analysis in your Bitbucket pipelines.</li></ul><p>2. Configure Bitbucket Pipelines:</p><ul><li>Navigate to your Bitbucket repository and create or edit your bitbucket-pipelines.yml file to include SonarCloud analysis.</li><li>Add the necessary environment variables to securely pass the SonarCloud token and project key to your pipeline. You can store these variables securely in the Bitbucket Pipelines settings.</li></ul><p><strong>Example Bitbucket Pipelines Configuration:</strong></p><pre>image: node:14.15.1<br><br>pipelines:<br>  branches:<br>    master:<br>      - step:<br>          name: Build and Test<br>          script:<br>            - npm install<br>            - npm run build<br>            - npm test<br>            # SonarCloud Analysis<br>            - pipe: sonarsource/sonarcloud-scan:1.2.0<br>              variables:<br>                SONAR_TOKEN: $SONAR_TOKEN<br>                EXTRA_ARGS: &#39;-Dsonar.projectKey=your_project_key&#39;</pre><p>In this example, the SonarCloud analysis is integrated as a step in the Bitbucket pipeline. TheSONAR_TOKEN environment variable securely passes the authentication token, and the EXTRA_ARGS parameter specifies the project key.</p><p>Run SonarCloud Analysis:</p><ul><li>With the pipeline configured, push your changes to the repository’s master branch to trigger the pipeline.</li><li>Bitbucket Pipelines will execute the build and tests, and subsequently run the SonarCloud analysis.</li><li>The analysis results are then sent to SonarCloud for processing.</li></ul><p>Review SonarCloud Results:</p><ul><li>Once the analysis is complete, navigate to your SonarCloud project dashboard to view the results.</li><li>SonarCloud provides detailed reports on code quality, including metrics such as code duplication, code coverage, security vulnerabilities, and code smells.</li><li>Use these insights to identify areas for improvement and prioritize code refactoring or security fixes.</li></ul><p>Benefits of Integrating Bitbucket Pipelines with SonarCloud:</p><ul><li>Automated Code Analysis: SonarCloud seamlessly integrates with Bitbucket Pipelines to automate static code analysis, ensuring consistent code quality checks with every code change.</li><li>Early Issue Detection: By analyzing code as part of the CI/CD process, teams can identify and address code quality issues early in the development lifecycle, minimizing technical debt and improving overall code maintainability.</li><li>Actionable Insights: SonarCloud provides actionable insights into code quality, security vulnerabilities, and best practices, empowering teams to make informed decisions and continuously improve their codebase.</li><li>Integration Flexibility: Bitbucket Pipelines and SonarCloud offer flexible integration options, allowing teams to customize analysis settings and incorporate additional quality gates or checks as needed.</li></ul><p>In summary, integrating Bitbucket Pipelines with SonarCloud enhances code quality by automating static code analysis and providing actionable insights into code issues. By leveraging this integration, teams can ensure consistent code quality standards and deliver high-quality software with confidence.</p><h4>How to handle “The bridge server is unresponsive” and “Analysis of JS/TS files failed”</h4><p>“The bridge server is unresponsive” is a common issue encountered when using SonarQube or SonarCloud integrations in Bitbucket Pipelines. This error typically occurs when the connection between the pipeline and the Sonar server is interrupted or when there are issues with the Sonar server itself. Here’s how you can handle this issue:</p><p>1. Check SonarQube/SonarCloud status:</p><p>Before troubleshooting further, ensure that SonarQube or SonarCloud services are up and running. Check the status of the Sonar server by accessing its web interface or respective status pages.</p><p>2. Verify Connection Configuration:</p><ul><li>Review the configuration in your Bitbucket Pipelines script (bitbucket-pipelines.yml) to ensure that the Sonar server URL, authentication credentials, and other settings are correct.</li><li>Make sure that the Sonar server URL specified in the pipeline configuration is accessible from the pipeline environment.</li></ul><p>3. Retry the connection:</p><ul><li>Sometimes, the “bridge server is unresponsive” error could be transient. Retry the sonar analysis step in the Bitbucket pipeline to see if the issue resolves itself.</li></ul><p>4. Check network connectivity:</p><ul><li>Verify that there are no network issues preventing the Bitbucket Pipelines environment from connecting to the Sonar server. Check for firewall rules, proxy configurations, or network restrictions that might block the connection.</li></ul><p>5. Review SonarQube/SonarCloud Logs:</p><ul><li>Check the logs on the Sonar server to identify any errors or issues that might be causing it to become unresponsive. Look for relevant error messages or stack traces that could provide clues about the root cause.</li></ul><p>6. Upgrade SonarQube/SonarCloud:</p><ul><li>Ensure that you are using the latest version of SonarQube or SonarCloud. Sometimes, issues with connectivity or stability are addressed in newer versions of the software.</li></ul><p>7. Contact Support:</p><ul><li>If you’ve exhausted all troubleshooting steps and are still experiencing the issue, reach out to SonarQube or SonarCloud support for assistance. Provide them with relevant details such as error messages, logs, and configuration settings for faster resolution.</li></ul><p>8. Temporary Workaround:</p><ul><li>As a temporary workaround, you can consider disabling the Sonar analysis step in the Bitbucket Pipelines configuration until the issue with the Sonar server is resolved. This will allow other pipeline steps to continue running without interruption.</li></ul><p>By following these steps, you can effectively handle the “bridge server is unresponsive” error when integrating SonarQube or SonarCloud with Bitbucket Pipelines, ensuring smooth and reliable code analysis within your CI/CD workflow.</p><p>Sample Bitbucket pipeline</p><pre>image: maven:3.3.9<br><br>definitions:<br>  services:<br>      docker:<br>        memory: 3072<br>      docker-4g:<br>        memory: 2048<br>        type: docker<br>  steps:<br>    step: &amp;build-step<br>        name: SonarQube analysis<br>        size: 2x<br>        max-time: 40<br>        services:<br>          - docker-4g<br>        script:<br>          - pipe: sonarsource/sonarcloud-scan:2.0.0<br>          pipe: sonarsource/sonarcloud-quality-gate:0.1.6<br>            variables:<br>              SONAR_TOKEN: $SONAR_TOKEN<br>              EXTRA_ARGS: &#39;-Dsonar.log.level=TRACE -Dsonar.verbose=true -Dsonar.javascript.node.maxspace=2048 -Dsonar.sourceEncoding=UTF-8 -Dsonar.javascript.maxFileSize=12000 -Dsonar.javascript.detectBundles=false -Dsonar.javascript.node.debugMemory=true -Dsonar.exclusions=node_modules/**/*,  -Dsonar.test.inclusions=**/*.test.*,**/*.spec.*,**.int.* -Dsonar.cpd.exclusions=**&#39;<br>  caches:<br>    sonar: ~/.sonar<br><br>clone:<br>  depth: full<br><br>pipelines:<br>  branches:<br>    master:<br>      - step: *build-step<br>    develop:<br>      - step: *build-step<br><br></pre><p>If need to use branch pattern, you can write like this</p><pre>pipelines:<br>  branches:<br>      &#39;*&#39;: // for any branch<br>        - step: *build-step<br>      &#39;features/*&#39;: // for any branch starts with features/<br>        - step: *build-step  <br>      &#39;bugfixs/*&#39;: for any branch starts with bugfixs/<br>        - step: *build-step </pre><p>The above Bitbucket Pipeline configuration is designed to perform SonarQube analysis on a Maven project using a Docker container. Below is a breakdown of the configuration:</p><p>1. Docker Image:</p><ul><li>The pipeline is using the Maven 3.3.9 Docker image as the base image for running the pipeline steps.</li></ul><p>2. Service Definitions:</p><ul><li>Two Docker services are defined:</li><li>docker with a memory limit of 3072 MB. These values can be changed</li><li>docker-4g with a memory limit of 2048 MB and specified as the service for the SonarQube analysis step. These values can be changed</li></ul><p>3. Pipeline Steps:</p><ul><li>A single step named “SonarQube analysis” is defined using the YAML anchor (&amp;build-step) and reference (*build-step) syntax.</li><li>This step uses a 2x size and a maximum execution time of 40 minutes.</li><li>It specifies the docker-4g service for execution.</li><li>The script for this step involves two pipe commands:</li><li>The first pipe command (sonarsource/sonarcloud-scan:2.0.0) performs the SonarQube analysis.</li><li>The second pipe command (sonarsource/sonarcloud-quality-gate:0.1.6) checks the quality gate status.</li><li>Various variables are passed to the second pipe command, including the Sonar token (SONAR_TOKEN) and additional arguments for configuring SonarQube analysis.</li></ul><p>4. Caches:</p><ul><li>A cache namedsonar is defined to store SonarQube analysis data in the ~/.sonar directory for future pipeline runs.</li></ul><p>5. Clone Depth:</p><ul><li>Full clone depth is specified to ensure that the entire Git history is cloned for each branch.</li></ul><p>6. Branches:</p><ul><li>Two branches (masterand)develop) are configured to trigger the pipeline. Both branches execute the same SonarQube analysis step defined in thebuild-step.</li></ul><p>In summary, this Bitbucket Pipeline configuration runs SonarQube analysis on Maven projects for both the master and develop branches using a Docker container with specified memory limits. It ensures code quality and compliance with quality gates as part of the CI/CD process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c1023af57f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to check If All the Variables of an Object Are Null]]></title>
            <link>https://suraj-batuwana.medium.com/how-to-check-if-all-the-variables-of-an-object-are-null-88ad67a70ea2?source=rss-b587db5cb48------2</link>
            <guid isPermaLink="false">https://medium.com/p/88ad67a70ea2</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[npe]]></category>
            <category><![CDATA[stream]]></category>
            <category><![CDATA[check]]></category>
            <dc:creator><![CDATA[Suraj Batuwana]]></dc:creator>
            <pubDate>Thu, 09 Nov 2023 05:56:32 GMT</pubDate>
            <atom:updated>2023-11-09T05:56:32.795Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why</h3><p>Checking if variables are null in Java is a common practice for several reasons:</p><p>1. <strong>Avoiding NullPointer Exceptions:</strong><br>The primary reason for checking if variables are null is to prevent `NullPointerExceptions` (NPE). If you attempt to access or manipulate an object or variable that is null, it will result in a runtime exception. By checking for null before performing operations, you can gracefully handle the situation and avoid unexpected crashes in your program.</p><p>2. <strong>Ensuring a Valid State:</strong><br>Checking for null is a way to ensure that your program operates on valid and expected data. It helps maintain the integrity of your application’s state and prevents unintended consequences that may arise from using null values in inappropriate contexts.</p><p>3. <strong>Robust Error Handling:</strong><br>By explicitly checking for null, you can implement robust error-handling mechanisms. Rather than letting the program fail with an unhandled exception, you can gracefully handle the null case, log relevant information, and provide meaningful error messages to users or developers.</p><p>4. <strong>Preventing Logical Errors</strong>:<br>Checking for null can help catch logical errors early in the development process. If a variable is supposed to contain a non-null value at a certain point in the program’s execution, checking for null ensures that this expectation is met.</p><p>5. <strong>Enhancing code readabilit</strong>y:<br>Explicitly checking for null makes the code more readable and self-explanatory. It communicates the developer’s intent to handle the null case and promotes code clarity for both the original developer and others who may read or maintain the code.</p><p>6. <strong>Compatibility with APIs:</strong><br>When working with external libraries or APIs, it’s common to check for null values returned by methods or functions. This is a defensive programming practice to handle situations where the external code may return null to indicate a specific condition or absence of a value.</p><p>7. <strong>Avoiding Unintended Side Effects</strong>:<br>If null values are not handled properly, they can lead to unintended side effects or incorrect behavior in your program. Checking for null allows you to control the flow of execution and handle special cases accordingly.</p><p>Here’s a simple example of checking for null:</p><pre>String text = // some operation that may return null<br>if (text != null) {<br> // perform operations on text<br>} else {<br> // handle the case where text is null<br>}</pre><p>In summary, checking for null in Java is a fundamental practice for building robust, reliable, and maintainable software by preventing runtime errors, improving code readability, and enhancing overall program stability.</p><h3>Working example</h3><p>Here&#39;s an example of how you can use Java streams for this task:</p><pre>import java.lang.reflect.Field;<br>import java.util.Arrays;<br><br>public class NullChecker {<br><br>    public static &lt;T&gt; boolean areAllVariablesNull(T obj) {<br>        return Arrays.stream(obj.getClass().getDeclaredFields())<br>                .peek(field -&gt; field.setAccessible(true))<br>                .allMatch(field -&gt; {<br>                    try {<br>                        return field.get(obj) == null;<br>                    } catch (IllegalAccessException e) {<br>                        e.printStackTrace(); // Handle the exception as needed<br>                        return false;<br>                    }<br>                });<br>    }<br><br>    public static void main(String[] args) {<br>        YourClass instance = new YourClass();<br>        boolean allNull = NullChecker.areAllVariablesNull(instance);<br>        System.out.println(&quot;All variables are null: &quot; + allNull);<br>    }<br>}</pre><p>In this example, Arrays.stream(obj.getClass().getDeclaredFields()) converts the array of fields into a stream. The peek operation is used to set the accessibility of each field to true. The allMatch operation checks if all elements of the stream satisfy the given condition, which is specified using a lambda expression. The lambda expression checks whether the value of each field is null, handling any exceptions that may occur during the reflection process.</p><p>Using streams can make the code more concise, but keep in mind that this approach may be less efficient than a traditional for loop, especially when dealing with a large number of fields. Choose the approach that best fits your specific use case and performance requirements.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88ad67a70ea2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>