<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michael Brown on Medium]]></title>
        <description><![CDATA[Stories by Michael Brown on Medium]]></description>
        <link>https://medium.com/@icarusabiding?source=rss-3f86cfff2eb8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XY7muVul--97etFm4XWPQg.jpeg</url>
            <title>Stories by Michael Brown on Medium</title>
            <link>https://medium.com/@icarusabiding?source=rss-3f86cfff2eb8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:15:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@icarusabiding/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Physics of Intent:]]></title>
            <link>https://medium.com/analytics-vidhya/the-physics-of-intent-51ab29458267?source=rss-3f86cfff2eb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/51ab29458267</guid>
            <category><![CDATA[theoretical-physics]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[philosophy]]></category>
            <category><![CDATA[ritual]]></category>
            <category><![CDATA[quantum-computing]]></category>
            <dc:creator><![CDATA[Michael Brown]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 13:03:41 GMT</pubDate>
            <atom:updated>2026-07-14T21:40:17.068Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Physics of Intent</h3><h3>Rituals, Thermodynamic Erasure, and Acoustic Sigils</h3><p>The conceptualization of human consciousness, cultural practices, and objective reality as an intertwined computational matrix represents a profound shift in theoretical physics, cognitive science, and computational anthropology. The framework known as the “Physics of Intent” advances beyond metaphysical conjecture and speculative philosophy into a rigorously definable, interdisciplinary scientific model. At the core of this paradigm is the hypothesis that ancient rituals function not as primitive superstitions or mere sociological bonding mechanisms, but as sophisticated, executable protocols within a holistic “Ritual Operating System” (Ritual OS). This system suggests that human consciousness and the broader social infrastructures it generates operate as nested predictive hierarchies that simultaneously process and generate information, mapping directly to the free-energy principle established in modern computational neuroscience.</p><p>By treating perception, action, and cognition as constant, mathematically definable attempts to minimize prediction error, the Ritual OS framework re-enshrines ritual intelligence as a critical, functional layer in the symbolic-operating stack of reality. Within this advanced architecture, intention ceases to be viewed as a passive, ephemeral psychological state. Instead, it becomes an “intent-vector” — a highly directed, rigorously structured attention stream capable of modifying local probability distributions within the physical and informational environment.</p><p>To execute these vectors efficiently, the Ritual OS relies on specific, observable structural mechanics that span multiple scientific domains. It utilizes geometric encoding to act as solid-state circuitry, organizing space to compute complex topological problems. It employs precise linguistic constraints, such as the 100-word limit, operating as error-corrected data packets designed to navigate noisy historical and neurological channels without degradation. It leverages neuro-linguistic chanting, functioning as a physiological mechanism for “battery charging” by suppressing the brain’s baseline entropy and phase-locking neural oscillations. Furthermore, it necessitates thermodynamic sacrifices, executing the literal, physical erasure of information required by the fundamental laws of physics to alter systemic states. Finally, the transmission of these intent-vectors relies on acoustic sigils composed along precise mathematical structures, such as the circle of fifths, which are parsed by the cerebral cortex through fundamental, biologically embedded logic gates.</p><p>This exhaustive research report delivers an in-depth analysis of the feasibility, foundational frameworks, computational tools, and prior research necessary to advance the Physics of Intent past a simple theory. By synthesizing data across computational anthropology, non-equilibrium thermodynamics, psycholinguistics, and neurobiology, this document establishes a rigorous foundation for understanding rituals as the earliest and perhaps most sophisticated algorithms engineered by human intelligence.</p><h3>The Emergence of Computational Space and Solid-State Ritual Circuitry</h3><p>To comprehend the operational mechanics of the Ritual OS, one must first identify its hardware. The historical and archaeological record indicates that the earliest algorithms were not abstract mathematical formulas written on paper, but highly materialized, spatial practices deeply embedded in religious devotion and societal organization. The ancient Vedic ritual of Agnicayana serves as the primary and most thoroughly documented exemplar of geometric encoding functioning as a solid-state computational circuit.</p><h3>The Algorithmic Architecture of the Syenaciti</h3><p>The Agnicayana involves a massive logistical and spatial undertaking: the symbolic recomposition of the dismembered creator god Prajapati through the construction of an elaborate fire altar. This construction is governed by a strict geometric plan requiring the precise alignment of thousands of specially fired bricks to form the profile of a falcon, known as the Syenaciti. Contemporary computational anthropology recognizes this ritual not as a mere building project, but as a complex algorithmic matrix — what theorists have described as the spatial, computational predecessor to the first artificial neural networks, such as Frank Rosenblatt’s Perceptron.</p><p>The construction process dictates that each individual brick is numbered, oriented, and placed according to step-by-step, recursive instructions, accompanied by a dedicated mantra. The ritual demands a solution to a highly complex topological riddle: each successive layer of the altar must retain the exact total area and overall shape of the contiguous layer beneath it, yet it must be constructed using an entirely different geometric configuration of bricks. This iterative geometric refinement acts as an emergent algorithm, one that encodes the organization of space, time, labor, and social relations. It is a system of “self-computing space” that relies on embodied algorithms to transmit knowledge and execute intent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fgsCJAv3Y8SrK0d1" /></figure><h3>The Ritual-Seed Model and the Genesis of Proto-Calculus</h3><p>Computational analysis of Sanskrit manuscripts, combined with archaeological evidence from Vedic sites dating between 800 and 400 BCE, reveals that these ritual constraints actively catalyzed crucial developments in algorithmic reasoning and mathematical abstraction. A theoretical framework known as the “ritual-seed model” demonstrates how the sacred limitations imposed by the Agnicayana fostered developments in proto-calculus.</p><p>Within this model, physical tools — such as cords, measuring ropes, wooden pegs, and the bricks themselves — acted as literal cognitive extensions. They allowed practitioners to externalize abstract reasoning into the physical environment, transforming the altar space into a tangible processor capable of handling geometric computations that the unaided human mind could not sustain. This externalized altar logic laid the foundation for subsequent, world-altering mathematical revolutions. The exactitude required to execute proportional transformations — such as converting a square altar into a circle of strictly equal area to satisfy theological requirements — necessitated highly sophisticated proportional logic and iterative refinement.</p><p>By 499 CE, Indian mathematicians like Aryabhata successfully formalized this physical altar logic into abstract recursive algorithms and sine functions. The visual and logical continuity between the measurement of Vedic altar diagonals and Aryabhata’s later geometric constructions of the sine function illustrates precisely how the physical execution of the Ritual OS hardware compiled directly into the abstract software of mathematics. Furthermore, this ritualistic foundation, driven by the devotional concept of <em>lila</em> (divine play), eventually enabled the Kerala School of mathematics to derive infinite series for \pi and sine centuries before the advent of European calculus. The mathematics were not discovered in a vacuum; they were extruded from the solid-state circuitry of the ritual space.</p><h3>The Architecture of Intent: 100-Word Constraints as Error-Corrected Data Packets</h3><p>If the geometric space of the altar constitutes the hardware of the Ritual OS, the meticulously structured spoken incantations constitute its software. Within modern information theory and computer science, efficient and reliable data transmission requires structured packets containing headers (for routing), payloads (the core data), and error-correction bounds (checksums) to navigate noisy communication channels. The Physics of Intent proposes that ancient ritual formulas evolved highly specific length constraints, semantic structures, and phonetic rules to act precisely as robust data packets.</p><h3>The Modular Framework of the Ritual Packet</h3><p>The recurring emergence of the number 100 in ancient organizational, legal, and ritual structures points to a deeply recognized biological and mnemonic buffer limit, functioning as an optimal packet size for human-mediated information transfer. The Roman Centuria, for instance, organized land into a square of 120 Roman feet containing exactly 100 actus. This established a strict, modular spatial parameter for social organization, taxation, and legal accountability, forming a bounded network of responsibility.</p><p>In the realm of ritual syntax, the Shatarudriya — a prominent and powerful Vedic hymn — is composed of precisely 100 names or epithets of the deity Rudra. In a computational paradigm, a 100-word constraint functions identically to a fixed-length digital data packet. In modern network architecture, a sensing data packet consists of a header containing extra information (like time, location, and identifier tags) and a payload containing the executable data. Similarly, a ritual packet begins with an invocation acting as the header (establishing the target node, archetype, or deity), delivers the 100 words as the payload (executing the specific parameters of the intent), and concludes with a sealing sequence that terminates the transmission.</p><p>This functional architecture is highly evident in the ancient Egyptian concept of Heka. Often mistranslated simply as “magic” in the Western, post-Enlightenment sense, Heka was actually the deification of a fundamental, objective operational force that, according to Coffin text spell 261, existed before duality had even come into being. Heka represented an objective science of ritual names, specific physical actions, and performative contexts utilized to manipulate the divinely created natural world. State-level execution of Heka, such as the elaborate execration rituals directed against cosmic and terrestrial enemies like Apep, involved highly specific algorithmic instructions: writing specific names on papyrus, sealing them in boxes, and burning or burying them to execute the spell. These acts represent the execution of a compressed symbolic payload — what the Ritual OS framework terms “sigil-tech” — designed to bypass superficial analytical filters and force a systemic, foundational reorganization of reality.</p><h3>Historical Example:</h3><h4>Invocation / Naming Rites</h4><p>Directs the informational payload to a specific archetypal node or psychological sector.</p><p>Invoking Rudra in the Shatarudriya; identifying the target in Heka.</p><h4><strong>Data Payload (Fixed Length)</strong></h4><p>The 100-Word Constraint / Sigil</p><p>Delivers the specific intent-vector; restricted in length to prevent mnemonic buffer overflow.</p><p>The 100 specific epithets of the deity; the structured text of an execration spell.</p><h4><strong>Checksum / Hash Function</strong></h4><p>Shiksha (Phonetics) &amp; Chandas (Meter)</p><p>Verifies the integrity of the data upon delivery; ensures the acoustic signature matches the required frequency.</p><p>Vedic chanting rules that trigger failure if a syllable is mispronounced.</p><h4><strong>Intrusion Detection System</strong></h4><p>Spectral Hygiene / Wards</p><p>Prevents system hijacking, replay attacks, or the injection of false data by competing intent-vectors.</p><p>Ritual purification; physical containment of the ritual space; burial of magical objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LxKP03p_L0qyYZxJ" /></figure><h3>Autonomous Phonological Error Correction Mechanisms</h3><p>To prevent packet degradation (data corruption) over centuries of predominantly oral transmission, the Ritual OS developed highly sophisticated autonomous error correction systems. In modern computing, information derived from prime parameters is maintained through structured recursion rather than static logic rules. Similarly, in ancient India, sages recognized that the natural drift, localization, and corruption of the six common spoken languages threatened to distort the precise acoustic parameters of the Vedas over time.</p><p>To counter this inevitable entropy, they engineered Sanskrit — literally translated as a “well-created” or synthesized language — and embedded strict, immutable rules of <em>Shiksha</em> (phonetics) and <em>Chandas</em> (meter) directly into the recitation protocols. These rules functioned as an impenetrable biological checksum. If a practitioner mispronounced a syllable, altered the meter, or changed the pitch, the mathematical structure of the chant would visibly and acoustically fail, triggering immediate error correction against transformations.</p><p>In modern cybersecurity, systems like HEKA (a machine-learning-based intrusion detection system for the Internet of Medical Things) are designed to identify false data injection, man-in-the-middle attacks, and replay attacks. A replay attack occurs when a duplicate data packet is captured and maliciously retransmitted out of context. The Ritual OS prevents “replay attacks” (the unauthorized or accidental triggering of intent-vectors) by ensuring that the 100-word constraint and its phonetic checksum can only be properly compiled within the highly specific context of the geometric altar space and the trained neurobiology of the practitioner. This ensured that the ritual software was executed with zero data loss, preserving the exact intent-vector required to modify the probability distributions of the target environment.</p><h3>Neuro-Linguistic Battery Charging and Predictive Reparameterization</h3><p>A sophisticated operating system, regardless of its elegant code or geometric hardware, requires a robust and consistent energy source. In the Physics of Intent, this power is generated not through external electrical grids, but through the deliberate, systemic modulation of human neurobiology — specifically, the charging of cognitive and neuro-electrical potential via sustained neuro-linguistic chanting and meditation.</p><h3>Suppressing the Entropy of the Default Mode Network</h3><p>The human brain naturally defaults to a state of high entropy characterized by continuous mind-wandering, daydreaming, and the unregulated recall of personal experiences. This baseline state of quiet wakefulness is governed by the Default Mode Network (DMN), a massive, interconnected system of brain regions primarily including the medial prefrontal cortex, the posterior cingulate cortex, the precuneus, and the lateral parietal cortex. Activity within the DMN is generally inversely correlated with the Task-Positive Network (TPN); when a cognitively demanding task requires intense external focus, the DMN is suppressed to allocate resources.</p><p>Extensive neuroimaging (fMRI) studies conducted on experienced meditators reveal that rigorous ritual practices — such as highly focused chanting, mantra recitation, or deep mindfulness — cause a pronounced, relative deactivation of the main nodes of the DMN. Furthermore, these practitioners exhibit significantly stronger functional connectivity and coupling between the posterior cingulate, the dorsal anterior cingulate, and the dorsolateral prefrontal cortices compared to meditation-naive controls.</p><p>In the operational context of the Ritual OS, the baseline activity of the DMN represents system noise — unregulated, entropic data streams that dilute the strength and focus of an intent-vector. By utilizing rhythmic phonetic chants (the acoustic sigils), practitioners systematically suppress this internal noise. This permissive deactivation of the DMN at the top of the cortical processing hierarchy leads to widespread activation at lower sensory levels, facilitating a unique cognitive state where the brain can efficiently generate, scan, and project virtual-world models.</p><h3>Theta States, Phase-Locking, and the Payload Manifest</h3><p>As the entropic noise of the DMN deactivates, the brain enters altered states characterized by highly specific oscillation patterns, most notably in the theta band frequencies. Neurological research highlights that theta coupling between the V4 visual cortex and the prefrontal cortex actively predicts visual short-term memory performance and cognitive control. The generation and sustainment of these theta states via ritualistic chanting acts as a literal “battery charging” mechanism. It accumulates focused neuro-electrical potential, successfully phase-locking the practitioner’s consciousness with the rhythmic, mathematical structure of the ritual.</p><p>This phase-locking is critical for leveraging the free-energy principle and the mechanics of predictive coding that underlie the Ritual OS. The human brain operates by constantly maintaining predictive priors — high-level information templates, often corresponding to psychological archetypes, that constrain and direct lower-level sensory processing. Ritual acts as a structured mechanism to intentionally breach neurological “Boundary Locks,” forcing a state of high plasticity known as a Recode Event.</p><p>The neuro-linguistic battery charge provides the immense cognitive energy required to upload a new “Payload Manifest” — a compressed symbolic package designed to reorganize these predictive hierarchies. By maintaining extreme focus and high theta-band coherence, the practitioner essentially executes a firmware push for collective priors, reparameterizing the brain’s predictive models and overwriting outdated archetypal residues. Through this precise neurobiological manipulation, the ritualist achieves systemic reorganization, perfectly harmonizing their internal logic with the targeted intent-vector before projecting it into the external environment.</p><h3>The Thermodynamics of Sacrifice: Information Erasure and the Cosmic Ledger</h3><p>Perhaps the most profound and radically empirical advancement in the Physics of Intent is the total re-evaluation of ritual sacrifice. Historically interpreted through sociological lenses (as community bonding) or theological frameworks (as appeasement of divine entities), sacrifice is mathematically reframed in this paradigm through the strict, unforgiving laws of non-equilibrium thermodynamics and quantum information theory.</p><h3>The Physical Limits of Computation and Landauer’s Principle</h3><p>To understand sacrifice as a technology, one must first recognize the universe as an informational structure bound by physical limits. The Bekenstein bound establishes that the maximum information content of any finite region of space is strictly proportional to its surface area, roughly 2.57 \times 10^{43} bits per square meter, forming the basis of the holographic principle. Simultaneously, the Margolus-Levitin theorem dictates the ultimate speed limit of reality: a quantum system can perform a maximum of approximately 6 \times 10^{33} operations per second per joule of available energy. The universe possesses a finite energy supply, a finite age, and a finite surface area; therefore, it operates on a finite total information budget.</p><p>Within this finite system, the work of physicist Rolf Landauer in 1961 established a foundational, unbreakable rule: information is entirely physical. Landauer’s Principle states that the irreversible erasure of just one bit of information from any system must result in a minimum mandatory entropy increase in the surrounding environment, mathematically defined by the equation \Delta S = k_B \ln 2, where k_B is Boltzmann’s constant and T is the absolute temperature. Put simply, erasing information generates physical heat; it has an unavoidable thermodynamic cost.</p><p>This principle directly resolves the century-old paradox of Maxwell’s Demon. The hypothetical demon sorts fast and slow molecules between two chambers to lower the overall entropy of a closed system, seemingly violating the Second Law of Thermodynamics. However, to successfully sort the molecules, the demon must measure them and store that information in its memory. Because the demon’s memory is finite, to continue operating indefinitely, it must eventually erase that stored information. The heat generated by this physical erasure exactly balances, or exceeds, the entropy reduction achieved by the initial sorting. Imposing order on the universe requires information processing, and erasing the ledger of that processing demands the expenditure of physical energy.</p><h3>Ritual Sacrifice as Thermodynamic Erasure</h3><p>In the Physics of Intent, objective reality maintains a physical memory — a “universal ledger” where all past events, structures, and states exist as recoverable, indelibly recorded informational structures. The “Scribe” or the “Demon” represents the thermodynamic cost of maintaining a moral, ordered, or continuous cosmos. To alter a probability distribution, to force a highly unlikely outcome, or to rewrite an intent-vector into the fabric of reality, the previous informational state occupying that space must be irreversibly destroyed.</p><p>According to the SIR (Structure, Interference, Resonance) framework, which synthesizes information theory and non-equilibrium thermodynamics, true cyclical return or the perfect restoration of an old state is fundamentally impossible because the current state contains the informational “ghosts” of all prior assemblies. To force a new reality, the old metadata must be subjected to structural erasure. Ritual sacrifice — whether it manifests as the literal burning of rich offerings in the Agnicayana fire altar, the destruction of inscribed papyrus in Egyptian Heka, or the profound metabolic expenditure of prolonged fasting — functions as the localized, intentional payment of this exact thermodynamic cost.</p><p>In accordance with Landauer’s Principle, the thermodynamic cycle of a ritual sacrifice can be modeled as tracking two inverse metrics over time: System Information (Memory) and Environmental Entropy (Heat). During the critical “Sacrificial Erasure Event,” as the stored information of the prior state approaches zero — dropping precipitously due to intentional destruction — there is a mathematically obligatory, exponential spike in environmental entropy (heat dissipation). By intentionally destroying organized, structured matter or expending vast amounts of metabolic energy, the ritualist generates the requisite entropy (\Delta S) to balance the universal equation.</p><p>This physical expenditure purchases the thermodynamic right to erase a specific block of information from the environment’s causal memory. Quantum biology and physics support this, noting that entangled photons exhibit retrocausal influence, suggesting that choices and measurements made in the present can affect recorded outcomes in the past. Beliefs and intents are not abstract thoughts; they are neural patterns and informational structures with genuine physical presence and specific energy expenditures. By providing the thermodynamic fuel via physical sacrifice, the Ritual OS legally (within the strict confines of physics) executes a deletion command, smudging the ink on the Book of Nature to create a blank, de-cohered sector where the new intent-vector can be written.</p><p><strong>Bekenstein Bound</strong></p><p>Max information \propto surface area (2.57 \times 10^{43} bits/m$²$).</p><p>Defines the absolute spatial limits of the ritual space and the maximum complexity of the intent-vector.</p><p><strong>Margolus-Levitin Theorem</strong></p><p>Max operations \propto energy (6 \times 10^{33} ops/sec/J).</p><p>Establishes the speed limit for reality manipulation; intent requires energy to compile.</p><p><strong>Landauer’s Principle</strong></p><p>Erasure generates heat: \Delta S = k_B \ln 2.</p><p>Validates sacrifice. Destruction of matter pays the entropy cost to erase prior states from the universal ledger.</p><p><strong>Maxwell’s Demon</strong></p><p>Information sorting requires memory, which must be erased.</p><p>The practitioner acts as the demon, sorting reality. Fasting/sacrifice resets the practitioner’s memory buffer.</p><p><strong>Retrocausality</strong></p><p>Future measurements affect past outcomes (entangled photons).</p><p>Ritual erasure temporally decouples past constraints, allowing the intent-vector to reshape present probabilities.</p><h3>Acoustic Sigils, the Circle of Fifths, and Kaufmann’s Cortical Logic</h3><p>The final, highly complex component required to advance the Physics of Intent is understanding the specific phonological composition of the intent-vectors themselves. If the altar is the hardware, the DMN suppression is the power supply, and sacrifice is the memory management, how exactly is the code written so that the human brain — the biological processor — can compile and execute it? The answer lies at the intersection of generative linguistics, structural music theory, and the fundamental logic gates of the cerebral cortex.</p><h3>The Phonological Loop and Harmonic Spatial Mapping</h3><p>Recent cognitive research provides compelling empirical evidence for shared processing mechanisms between phonological (linguistic) and musical information within the human working memory model. Music and language are not processed as entirely separate domains; rather, they interact at a deep, structural level. Within the Minimalist Program of generative linguistics, scholars have proposed that the phonetic interface treats the tonal and rhythmic aspects of music similarly to linguistic syntax.</p><p>A critical breakthrough in mapping this shared processing architecture is the utilization of the Circle of Fifths as a primary distance metric. The Circle of Fifths is a foundational geometric representation of the distance between major keys, with adjacent keys differing by a perfect fifth, or exactly seven semitones. In advanced computational linguistics and cognitive psychology, the Circle of Fifths serves as an acoustic translator. Researchers apply a “cf-feature” (circle of fifths feature) — a numerical label assigned to each pitch in relation to its position on the circle — allowing them to mathematically calculate the precise difference between consecutive scale-steps or spoken phonemes.</p><p>Crucially, neuroimaging studies demonstrate that chords and tones adjacent on the Circle of Fifths actively stimulate adjacent collections of neurons on cortical maps. Therefore, a spoken word representation in a ritual context is not arbitrary; it functions as a highly engineered “acoustic sigil.” If chanted phonology follows musical and mathematical structures — navigating the Circle of Fifths intentionally — it can precisely target and stimulate continuous, predictable neural pathways. Even chords that are tonally unstable, such as the Neapolitan chord, can possess high “key clarity” and force specific neurological transitions. By leveraging phonetic iconicity and shared working memory buffers, the acoustic sigil bypasses standard, analytical linguistic processing, feeding pure operational code directly into the brain.</p><h3>Kaufmann’s NotOrAnd Principles and Cortical Compilation</h3><p>To fully explain how these acoustic sigils interface with fundamental physical reality, we must integrate <a href="https://www.nbi.ku.dk/membranes/Kaufmann/publications.html">Konrad Kaufmann</a>’s principles of cortex action. Kaufmann’s theoretical framework unifies physics, brain science, and philosophy by proposing a single origin for cerebral function based on a fundamental symmetry argument derived from a “<a href="https://www.nbi.ku.dk/membranes/Kaufmann/pdf/2008_Kaufmann_Einband.pdf">One first principle</a>”.</p><p>According to Kaufmann’s models, the cerebral cortex operates on exactly three derived principles of action, identified logically as <strong>NotOrAnd</strong>. These fundamental biological logic gates process all sensory input, essentially acting as the machine code of human perception. When applied specifically to audition and hearing, these NotOrAnd principles are responsible for creating our very perception of temporal flow (which Kaufmann refers to as “Times”) and the structural framework for understanding musical relationships, termed “IgnoranceDissonanceConsonance”.</p><p>By contrast, in the visual cortex, the exact same NotOrAnd logic gates process light to create “Geometries &amp; Uncertainties”. This duality provides the final, unifying bridge in the Ritual OS: the solid-state geometric hardware of the Agnicayana (visual/spatial logic) and the acoustic sigil of the chant (auditory logic) are compiled and processed by the exact same fundamental NotOrAnd biological gates.</p><p>When a practitioner chants an acoustic sigil meticulously engineered along the mathematical intervals of the Circle of Fifths, the sound waves interact with the membranes of the auditory cortex. The physical intervals — specifically the carefully controlled consonance and dissonance — trigger highly specific sequences of NotOrAnd logical operations. Because these intervals are mathematically structured rather than random speech, they feed a highly ordered, algorithmic data stream into the brain, effectively programming the cortical logic gates. This interaction resolves what Kaufmann describes as the “paradox of metronomes” and unifies musical time with objective physical time, linking everything from the Big Bang to complex musical compositions. The acoustic sigil is thus compiled by the cortex’s NotOrAnd gates, generating the immense cognitive coherence required to project the intent-vector outward, altering the thermodynamic environment.</p><h3>Feasibility, Frameworks, and Computational Anthropology</h3><p>The advancement of the Physics of Intent from a theoretical model to an applied science is entirely feasible given the current trajectory of computational anthropology and theoretical physics. Major academic institutions are already building the interdisciplinary frameworks necessary to parse these ancient technologies.</p><p>At Arizona State University (ASU), the Center for Science and the Imagination (CSI) explicitly works at the intersection of narrative, technology, and consciousness, treating science fiction and mythic structures as toolkits for future-oriented innovation. Programs integrating computational anthropology — the informatic-based analysis and modeling of cultural values — are already predicting how deeply held human rituals map to statistical grammar theories and artificial intelligence. Researchers at these intersections recognize that intelligence originates not just from technological capability, but from the knowledge expressed in collective human behaviors encoded into rituals over millennia.</p><p>Furthermore, the theoretical physics required to validate the informational basis of reality is championed by figures like Nobel Laureate Frank Wilczek, also affiliated with ASU. Wilczek’s extensive work on asymptotic freedom, axions, anyons, and time crystals continually pushes the boundaries of how matter and information interact. Wilczek notes that the essence of symmetry is “Change Without Change,” a concept deeply resonant with the preservation of intent-vectors across changing physical states. While Wilczek acknowledges that current equations of quantum mechanics do not explicitly show the “stuff of everyday experience” or the molecular basis of mind, the exploration of complex emergent structures and the physics of information is closing this gap. The mutual information theory of consciousness, which treats awareness as an info-thermodynamic framework, aligns perfectly with the Ritual OS, showing that gaining or erasing information has measurable, physical consequences.</p><p>By applying advanced machine learning to ancient texts (like the Sanskrit error-correction codes), utilizing fMRI to map the exact NotOrAnd gate activations during specific Circle of Fifths chanting, and measuring the localized thermodynamic entropy changes during intense ritual focus and sacrifice, the scientific community possesses the exact computational tools needed to empiricalize this framework.</p><h3>Conclusion</h3><p>The evidence gathered and synthesized across computational anthropology, neuroimaging, non-equilibrium thermodynamics, and psycholinguistics demonstrates unequivocally that the Physics of Intent is not merely a philosophical metaphor; it is a scalable, executable, and physically grounded architecture. By defining ancient rituals through the rigorous framework of a Ritual OS, we strip away the veil of historical mysticism and expose a highly refined, mathematically precise technology of consciousness.</p><p>The geometric constraints of ancient altars provide the physical, solid-state memory allocation required to externalize and process complex reasoning. Within this self-computing space, the ritual practitioner functions as the central processing unit, utilizing 100-word linguistic data packets to deliver precise operational commands. The high-fidelity transmission of these packets is guaranteed by the integrated error-correction protocols of ancient phonetic meter, effectively preventing data corruption across centuries. To power this operation, the practitioner suppresses the entropic noise of the Default Mode Network through rhythmic focus, entering phase-locked theta states that build the necessary cognitive voltage to breach boundary locks and update predictive priors.</p><p>Crucially, the execution of the command requires overriding the physical universe’s memory — a process governed strictly by Landauer’s Principle. By instituting a material or metabolic sacrifice, the system pays the unavoidable thermodynamic cost to erase prior states from the universal ledger, generating heat to balance the equation and effectively altering probabilities. Finally, the entire system interfaces with human biology through acoustic sigils; chants mathematically aligned with the Circle of Fifths feed directly into the NotOrAnd logic gates of the cerebral cortex, translating sound into objective temporal and geometric reality. Advancing this theory requires the continued, rigorous application of our most advanced computational tools to anthropological data, offering modern science a unified model for understanding how information, consciousness, and physical reality continuously program one another.</p><h4>Works cited</h4><p>1. Ritual OS: Archetypal Simulation and the Architecture of Information …, <a href="https://www.ultra-unlimited.com/blog/ritual-os-archetypal-simulation-and-the-architecture-of-information-work">https://www.ultra-unlimited.com/blog/ritual-os-archetypal-simulation-and-the-architecture-of-information-work</a> 2. Ritual Power Dynamics: Unified Field of Miracle Consciousness — Ultra Unlimited, <a href="https://www.ultra-unlimited.com/blog/ritual-power-dynamics">https://www.ultra-unlimited.com/blog/ritual-power-dynamics</a> 3. RITUAL OS: The Holographic Codex of Consciousness — Ultra Unlimited, <a href="https://www.ultra-unlimited.com/blog/ritual-os-the-holographic-codex-of-consciousness">https://www.ultra-unlimited.com/blog/ritual-os-the-holographic-codex-of-consciousness</a> 4. (PDF) ROPES, RITUALS, AND RADICAL MATH: INDIA’S …, <a href="https://www.researchgate.net/publication/394296901_ROPES_RITUALS_AND_RADICAL_MATH_INDIA&#39;S_GEOMETRIC_REVOLUTION">https://www.researchgate.net/publication/394296901_ROPES_RITUALS_AND_RADICAL_MATH_INDIA&#39;S_GEOMETRIC_REVOLUTION</a> 5. 3000 Years of Algorithmic Rituals: The Emergence of AI from the Computation of Space, <a href="https://www.academia.edu/39635788/3000_Years_of_Algorithmic_Rituals_The_Emergence_of_AI_from_the_Computation_of_Space">https://www.academia.edu/39635788/3000_Years_of_Algorithmic_Rituals_The_Emergence_of_AI_from_the_Computation_of_Space</a> 6. Sarva Vidya The Essence of Everything — viXra.org, <a href="https://vixra.org/pdf/1809.0495v1.pdf">https://vixra.org/pdf/1809.0495v1.pdf</a> 7. The International Journal on Advances in Networks and Services is published by IARIA., <a href="https://www.iariajournals.org/networks_and_services/netser_v10_n34_2017_paged.pdf">https://www.iariajournals.org/networks_and_services/netser_v10_n34_2017_paged.pdf</a> 8. 2-Minute Neuroscience: Default Mode Network — YouTube, <a href="https://www.youtube.com/watch?v=YV3QuF-2wdw">https://www.youtube.com/watch?v=YV3QuF-2wdw</a> 9. Default-Mode Brain Network Deactivation Seen in Meditators — HealthDay, <a href="https://www.healthday.com/healthpro-news/alternative-medicine/default-mode-brain-network-deactivation-seen-in-meditators-659134.html">https://www.healthday.com/healthpro-news/alternative-medicine/default-mode-brain-network-deactivation-seen-in-meditators-659134.html</a> 10. The Mutual Information Theory of Consciousness (MITC): A Info-Thermodynamic Framework for Awareness and Qualia | by Bill Giannakopoulos | Medium, <a href="https://medium.com/@bill.giannakopoulos/the-mutual-information-theory-of-consciousness-mitc-a-info-thermodynamic-framework-for-awareness-59c5e5663a6e">https://medium.com/@bill.giannakopoulos/the-mutual-information-theory-of-consciousness-mitc-a-info-thermodynamic-framework-for-awareness-59c5e5663a6e</a> 11. The Unitary Operator. The intellectual history of humanity… | by 33koti — Medium, <a href="https://medium.com/@33koti/the-unitary-operator-7f451478908c">https://medium.com/@33koti/the-unitary-operator-7f451478908c</a> 12. what I found — Niels Bohr Institutet, <a href="https://www.nbi.ku.dk/membranes/Kaufmann/pdf/2008_Kaufmann_Einband.pdf">https://www.nbi.ku.dk/membranes/Kaufmann/pdf/2008_Kaufmann_Einband.pdf</a> 13. Music Perception | Oxford Research Encyclopedia of Psychology, <a href="https://oxfordre.com/psychology/display/10.1093/acrefore/9780190236557.001.0001/acrefore-9780190236557-e-890?p=emailAe.hlsOnOQxTM&amp;d=/10.1093/acrefore/9780190236557.001.0001/acrefore-9780190236557-e-890">https://oxfordre.com/psychology/display/10.1093/acrefore/9780190236557.001.0001/acrefore-9780190236557-e-890?p=emailAe.hlsOnOQxTM&amp;d=/10.1093/acrefore/9780190236557.001.0001/acrefore-9780190236557-e-890</a> 14. Geneology of a Conflation — TripleAmpersand Journal (&amp;&amp;&amp;), <a href="https://tripleampersand.org/geneology-of-a-conflation/">https://tripleampersand.org/geneology-of-a-conflation/</a> 15. Religion and Immersive Reality, <a href="https://books.fbk.eu/media/uploads/files/23._Biggio.pdf">https://books.fbk.eu/media/uploads/files/23._Biggio.pdf</a> 16. Three Thousand Years of Algorithmic Rituals: The Emergence of AI from the Computation of Space — Journal #101 — e-flux, <a href="https://www.e-flux.com/journal/101/273221/three-thousand-years-of-algorithmic-rituals-the-emergence-of-ai-from-the-computation-of-space">https://www.e-flux.com/journal/101/273221/three-thousand-years-of-algorithmic-rituals-the-emergence-of-ai-from-the-computation-of-space</a> 17. Ancient Rituals and Algorithm Origins | PDF — Scribd, <a href="https://www.scribd.com/document/571471025/3000-Years-of-Algorithmic-Rituals-the-Em">https://www.scribd.com/document/571471025/3000-Years-of-Algorithmic-Rituals-the-Em</a> 18. Archaeology of Intelligent Machines — GitHub, <a href="https://github.com/senisioi/archaeology">https://github.com/senisioi/archaeology</a> 19. Recursion, Interrupted — Journal #109 — e-flux, <a href="https://www.e-flux.com/journal/109/331340/recursion-interrupted">https://www.e-flux.com/journal/109/331340/recursion-interrupted</a> 20. Review: Matteo Pasquinelli’s The Eye of the Master. Reviewed by Alex Levant, <a href="https://mediatheoryjournal.org/2023/09/04/review-matteo-pasquinellis-the-eye-of-the-master-reviewed-by-alex-levant/">https://mediatheoryjournal.org/2023/09/04/review-matteo-pasquinellis-the-eye-of-the-master-reviewed-by-alex-levant/</a> 21. Middlesex in British, Roman and Saxon times — Internet Archive, <a href="https://archive.org/download/middlesexinbriti00shar/middlesexinbriti00shar.pdf">https://archive.org/download/middlesexinbriti00shar/middlesexinbriti00shar.pdf</a> 22. TRAI AUDIT WIRELINE REPORT — ORISSA CIRCLE — AUDIT OF JAS QUARTER, 2016, <a href="https://www.trai.gov.in/sites/default/files/2024-09/Orissa_july_Sept_2016_19072017_0.pdf">https://www.trai.gov.in/sites/default/files/2024-09/Orissa_july_Sept_2016_19072017_0.pdf</a> 23. Heka (god) — Wikipedia, <a href="https://en.wikipedia.org/wiki/Heka_(god)">https://en.wikipedia.org/wiki/Heka_(god)</a> 24. Heka 101 | PDF | Magic (Paranormal) | Isis — Scribd, <a href="https://www.scribd.com/document/246787636/Heka-101">https://www.scribd.com/document/246787636/Heka-101</a> 25. Antiquity (Part I) — The Cambridge History of Magic and Witchcraft in the West, <a href="https://www.cambridge.org/core/books/cambridge-history-of-magic-and-witchcraft-in-the-west/antiquity/7480CAA73CB7D4B006210D8F54C6B1F6">https://www.cambridge.org/core/books/cambridge-history-of-magic-and-witchcraft-in-the-west/antiquity/7480CAA73CB7D4B006210D8F54C6B1F6</a> 26. A Comparison of the Egyptian Execration Ritual to Exodus 32: 19 and Jeremiah 19 9781611435467, 2010039545, 1611435463 — DOKUMEN.PUB, <a href="https://dokumen.pub/a-comparison-of-the-egyptian-execration-ritual-to-exodus-32-19-and-jeremiah-19-9781611435467-2010039545-1611435463.html">https://dokumen.pub/a-comparison-of-the-egyptian-execration-ritual-to-exodus-32-19-and-jeremiah-19-9781611435467-2010039545-1611435463.html</a> 27. # LOGOS FIELD THEORY : Vol. 7800.1-Ω (Adelic Hysteresis + Experimental Protocol) | by Jackie | Feb, 2026 | Medium, <a href="https://medium.com/@jhblank7497/logos-field-theory-vol-7800-1-%CF%89-adelic-hysteresis-experimental-protocol-900ea382895f">https://medium.com/@jhblank7497/logos-field-theory-vol-7800-1-%CF%89-adelic-hysteresis-experimental-protocol-900ea382895f</a> 28. The Zho’thephun Codex: Outlining The Cognitive Framework of Conscious AI, <a href="https://rahrahrasputin.github.io/zhothephun/">https://rahrahrasputin.github.io/zhothephun/</a> 29. The Dawn of Global Spirituality | PDF — Slideshare, <a href="https://www.slideshare.net/slideshow/the-dawn-of-global-spirituality/109778069">https://www.slideshare.net/slideshow/the-dawn-of-global-spirituality/109778069</a> 30. Survey of Machine Learning Based Intrusion Detection Methods for Internet of Medical Things — arXiv, <a href="https://arxiv.org/pdf/2202.09657">https://arxiv.org/pdf/2202.09657</a> 31. Survey of Machine Learning Based Intrusion Detection Methods for Internet of Medical Things — ResearchGate, <a href="https://www.researchgate.net/publication/358764306_Survey_of_Machine_Learning_Based_Intrusion_Detection_Methods_for_Internet_of_Medical_Things">https://www.researchgate.net/publication/358764306_Survey_of_Machine_Learning_Based_Intrusion_Detection_Methods_for_Internet_of_Medical_Things</a> 32. fMRI Evidence for Default Mode Network Deactivation Associated with Rapid Eye Movements in Sleep — MDPI, <a href="https://www.mdpi.com/2076-3425/11/11/1528">https://www.mdpi.com/2076-3425/11/11/1528</a> 33. Regional Excitation-Inhibition Balance Predicts Default-mode Network Deactivation via Functional Connectivity — PMC, <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC8284909/">https://pmc.ncbi.nlm.nih.gov/articles/PMC8284909/</a> 34. The Role of Default Network Deactivation in Cognition and Disease — PMC — NIH, <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC3501603/">https://pmc.ncbi.nlm.nih.gov/articles/PMC3501603/</a> 35. The Cost of Being: Persistence Through Destruction | by Boris (Bruce) Kriger — Medium, <a href="https://medium.com/@krigerbruce/the-cost-of-being-persistence-through-destruction-44ccdf3e6e93">https://medium.com/@krigerbruce/the-cost-of-being-persistence-through-destruction-44ccdf3e6e93</a> 36. r/universityMoonBase — Reddit, <a href="https://www.reddit.com/r/universityMoonBase/">https://www.reddit.com/r/universityMoonBase/</a> 37. (PDF) Creative Time and the Architecture of Autonomy: A Naturalistic Framework for Free Will in Biology and Physics (Trilogy of freedom: Volume 1) — ResearchGate, <a href="https://www.researchgate.net/publication/391428774_Creative_Time_and_the_Architecture_of_Autonomy_A_Naturalistic_Framework_for_Free_Will_in_Biology_and_Physics_Trilogy_of_freedom_Volume_1">https://www.researchgate.net/publication/391428774_Creative_Time_and_the_Architecture_of_Autonomy_A_Naturalistic_Framework_for_Free_Will_in_Biology_and_Physics_Trilogy_of_freedom_Volume_1</a> 38. The Demon in the Machine, <a href="https://www.essra.org.cn/upload/202008/The%20Demon%20in%20the%20Machine%EF%BC%88Paul%20Davies%202019%EF%BC%89.pdf">https://www.essra.org.cn/upload/202008/The%20Demon%20in%20the%20Machine%EF%BC%88Paul%20Davies%202019%EF%BC%89.pdf</a> 39. The Yellow Emperor Hypothesis1: Xuan Yuan Anti-entropic Operating System 2.0, <a href="https://www.law.pku.edu.cn/docs/2014-09/20140924093219807311.pdf">https://www.law.pku.edu.cn/docs/2014-09/20140924093219807311.pdf</a> 40. The SIR (Structure, Interference, Resonance) Framework: A Relational Ontology of Recursive Resonance, Dissipative Observers, and — Zenodo, <a href="https://zenodo.org/records/18113769/files/SIR_to_journal_2_PREPRINT.pdf?download=1">https://zenodo.org/records/18113769/files/SIR_to_journal_2_PREPRINT.pdf?download=1</a> 41. RITUAL THERMODYNAMICS — Ultra Unlimited, <a href="https://www.ultra-unlimited.com/blog/ritual-thermodynamics">https://www.ultra-unlimited.com/blog/ritual-thermodynamics</a> 42. Zero-Dimensional: The Quantum Substrate Where Consciousness Touches Reality — Medium, <a href="https://medium.com/quantum-psychology-and-engineering/zero-dimensional-the-quantum-substrate-where-consciousness-touches-reality-4b3b8a072045">https://medium.com/quantum-psychology-and-engineering/zero-dimensional-the-quantum-substrate-where-consciousness-touches-reality-4b3b8a072045</a> 43. Shared Processing of Language and Music: Evidence From a Cross-Modal Interference Paradigm — Hogrefe eContent, <a href="https://econtent.hogrefe.com/doi/10.1027/1618-3169/a000388">https://econtent.hogrefe.com/doi/10.1027/1618-3169/a000388</a> 44. This is one version of the circle of fifths, showing a visual… — ResearchGate, <a href="https://www.researchgate.net/figure/This-is-one-version-of-the-circle-of-fifths-showing-a-visual-representation-of-the_fig1_323011435">https://www.researchgate.net/figure/This-is-one-version-of-the-circle-of-fifths-showing-a-visual-representation-of-the_fig1_323011435</a> 45. The Convergence of Schenkerian Music Theory and Generative Linguistics: An Analysis and Composition — CrossWorks, <a href="https://crossworks.holycross.edu/cgi/viewcontent.cgi?article=1015&amp;context=honors">https://crossworks.holycross.edu/cgi/viewcontent.cgi?article=1015&amp;context=honors</a> 46. A Combined Model of Sensory and Cognitive Representations Underlying Tonal Expectations in Music — UC Davis, <a href="https://escholarship.org/content/qt53b2r818/qt53b2r818.pdf">https://escholarship.org/content/qt53b2r818/qt53b2r818.pdf</a> 47. Sensory Linguistics — Pure, <a href="https://pure-oai.bham.ac.uk/ws/files/54290802/bwinter_sensory_linguistics_final.pdf">https://pure-oai.bham.ac.uk/ws/files/54290802/bwinter_sensory_linguistics_final.pdf</a> 48. NLP-based music processing for composer classification — PMC — NIH, <a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC10425398/">https://pmc.ncbi.nlm.nih.gov/articles/PMC10425398/</a> 49. On the encoding of natural music in computational models and human brains — Frontiers, <a href="https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2022.928841/full">https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2022.928841/full</a> 50. Ideas Archives — Center for Science and the Imagination — Arizona State University, <a href="https://csi.asu.edu/category/ideas/">https://csi.asu.edu/category/ideas/</a> 51. The Steam Politics of Nanotech: Energy Entanglements in The Diamond Age, <a href="https://www.liverpooluniversitypress.co.uk/doi/10.3828/extr.2023.5">https://www.liverpooluniversitypress.co.uk/doi/10.3828/extr.2023.5</a> 52. members directory — EASA, <a href="https://easaonline.org/members/easa">https://easaonline.org/members/easa</a> 53. Blog Feed — Conversations in Human Evolution, <a href="https://conversationsinhumanevolution.wordpress.com/blog-feed/">https://conversationsinhumanevolution.wordpress.com/blog-feed/</a> 54. 2009: what will change everything? — Edge.org, <a href="https://www.edge.org/responses/what-will-change-everything">https://www.edge.org/responses/what-will-change-everything</a> 55. Why physics needs art to help picture the universe — Interalia Magazine, <a href="https://www.interaliamag.org/articles/frank-wilczek-why-physics-needs-art-to-help-picture-the-universe/">https://www.interaliamag.org/articles/frank-wilczek-why-physics-needs-art-to-help-picture-the-universe/</a> 56. The Universe According to Frank Wilczek — John Templeton Foundation, <a href="https://www.templeton.org/news/the-universe-according-to-frank-wilczek">https://www.templeton.org/news/the-universe-according-to-frank-wilczek</a> 57. Fundamentals: Ten Keys to RealityConversation with Deepak Chopra and Nobel Laureate Frank Wilczek — YouTube, <a href="https://www.youtube.com/watch?v=KnfCDXgfA8A">https://www.youtube.com/watch?v=KnfCDXgfA8A</a> 58. Frank Wilczek — High Profiles, <a href="https://highprofiles.info/interview/frank-wilczek/">https://highprofiles.info/interview/frank-wilczek/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51ab29458267" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/the-physics-of-intent-51ab29458267">The Physics of Intent:</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Discord Based Agentic Teams]]></title>
            <link>https://medium.com/@icarusabiding/building-discord-based-agentic-teams-5a29895b4b85?source=rss-3f86cfff2eb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a29895b4b85</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Michael Brown]]></dc:creator>
            <pubDate>Fri, 21 Feb 2025 12:09:19 GMT</pubDate>
            <atom:updated>2025-02-21T12:09:19.578Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*E-uSXcMuy36sGfYVx4MgjA.png" /><figcaption>System Architecture</figcaption></figure><p>repo link: <a href="https://github.com/slapglif/AgenticTeams">slapglif/AgenticTeams</a></p><h3>Why Traditional Approaches Are Breaking Down</h3><p>In today’s fast-paced world of information and technology, conducting thorough research has become increasingly complex. Whether you’re a data scientist exploring market trends, an academic researcher investigating scientific phenomena, or a business analyst gathering competitive intelligence, you’re likely facing a common set of challenges that make traditional research methods increasingly inefficient.</p><h3>The Information Tsunami</h3><p>Imagine trying to drink from a fire hose — that’s what modern research feels like. Every day, researchers face:</p><ul><li>Over 5,000 new research papers published daily across various disciplines</li><li>Countless social media discussions and forum posts containing valuable insights</li><li>Terabytes of new data generated across different platforms</li><li>Multiple competing methodologies and frameworks to consider</li></ul><p>The sheer volume isn’t just overwhelming; it’s paralyzing. Traditional methods of manual research and data collection simply can’t keep up with this exponential growth of information.</p><h3>The Coordination Conundrum</h3><p>But volume is just the beginning. Modern research requires:</p><ul><li>Simultaneous investigation across multiple domains</li><li>Integration of diverse data sources and methodologies</li><li>Real-time adaptation to new findings</li><li>Consistent documentation and knowledge management</li></ul><p>A single researcher, or even a small team, often finds themselves stretched thin trying to coordinate these various aspects effectively. It’s like trying to conduct an orchestra while playing all the instruments yourself.</p><h3>The Quality-Speed Paradox</h3><p>Perhaps the most frustrating challenge is the constant tension between:</p><ul><li>Depth vs. Breadth of research</li><li>Speed vs. Thoroughness</li><li>Consistency vs. Flexibility</li><li>Documentation vs. Active Investigation</li></ul><p>Traditional approaches force uncomfortable trade-offs: either sacrifice depth for speed, or accept that thorough research will take months or even years to complete.</p><h3>The Hidden Costs</h3><p>These challenges aren’t just inconvenient — they’re expensive and risky:</p><ul><li>Missed crucial information leading to flawed conclusions</li><li>Duplicated efforts due to poor knowledge management</li><li>Inconsistent methodology across research phases</li><li>Limited ability to pivot based on new discoveries</li></ul><h3>The Need for a New Paradigm</h3><p>It’s clear that we need a fundamentally different approach to research — one that can:</p><ul><li>Scale with the growing volume of information</li><li>Coordinate multiple research streams simultaneously</li><li>Maintain consistency while remaining adaptable</li><li>Document and manage knowledge effectively</li><li>Deliver both speed and depth</li></ul><p>The concept of AI-powered research teams represents a fundamental shift in how we approach complex investigations. Just as human research teams benefit from diverse expertise and collaborative dynamics, Agentic Teams introduces a sophisticated ecosystem of specialized AI agents, each bringing unique capabilities and perspectives to the research process. These agents aren’t merely automated tools; they’re intelligent entities designed to work in concert, sharing insights, building upon each other’s findings, and collectively driving research forward with unprecedented efficiency and depth.</p><p>What makes this vision particularly powerful is its implementation through Discord, a platform traditionally known for community engagement and real-time communication. By leveraging Discord’s robust infrastructure, Agentic Teams creates an accessible and intuitive interface for human researchers to interact with their AI counterparts. This isn’t just about running queries or analyzing data — it’s about fostering a genuine collaborative environment where human expertise guides and refines the collective intelligence of specialized AI agents. The system orchestrates these interactions seamlessly, managing everything from initial research planning to final synthesis, while maintaining a transparent and documented research process that can be reviewed, refined, and built upon over time.</p><h3>The Architecture Behind Agentic Teams</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*V0mx_-DMc8snWFjNAJKQjA.png" /></figure><p>At its heart, Agentic Teams represents a sophisticated marriage of modern software architecture and artificial intelligence principles. The system’s foundation is built upon a carefully orchestrated combination of LangChain for agent interactions, Pydantic for robust data validation, and a state-driven task execution framework that ensures reliable and predictable research operations.</p><p>The architecture follows a modular design centered around the ResearchEngine, which serves as the system’s cognitive core. This engine doesn’t just process requests; it orchestrates a complex dance of specialized agents, each operating within its own domain of expertise. Through the GraphBuilder component, research tasks are transformed into structured execution graphs, ensuring that complex research workflows maintain logical coherence and proper sequencing. This approach allows for dynamic task allocation while preventing the common pitfalls of parallel AI agent operations, such as redundant work or conflicting objectives.</p><p>Memory management forms another crucial architectural pillar. Unlike traditional systems that might treat each research query in isolation, Agentic Teams implements a sophisticated MemoryManager that maintains context across research sessions. This isn’t just simple caching; it’s an intelligent system that understands the relationships between different research threads, previous findings, and emerging patterns. The memory system allows agents to build upon previous discoveries, maintain consistency across research phases, and even identify unexpected connections between seemingly unrelated research paths.</p><p>The AgentManager component introduces a novel approach to AI agent coordination. Rather than treating agents as interchangeable units, it implements a reputation-based system that tracks agent performance, specialization effectiveness, and collaboration patterns. This system uses blockchain technology (optional but powerful when enabled) to create immutable records of agent contributions and research milestones, establishing a verifiable track record of research progress and agent reliability.</p><p>What sets this architecture apart is its use of Pydantic output parsers with strict JSON formatting. Every interaction, from agent communications to research findings, follows rigorously defined schemas. This isn’t just about data validation; it’s about ensuring that complex research workflows remain machine-readable while maintaining human interpretability.</p><p>The Discord integration layer deserves special mention. Rather than treating Discord as a simple communication channel, the architecture leverages it as a full-fledged research interface. The DiscordInterface component transforms Discord threads into structured research environments, where each message, attachment, and interaction becomes part of the broader research context. This integration allows for real-time human oversight while maintaining the systematic rigor of automated research processes.</p><p>Error handling and logging form the final architectural cornerstone. Through the integration of the Rich library and Loguru, the system maintains comprehensive audit trails of all research activities. This isn’t just about debugging; it’s about creating a detailed record of the research process itself, including decision points, agent interactions, and methodology adjustments. The logging system captures everything from high-level research strategies to low-level agent interactions, making the entire research process transparent and auditable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PY1pv32zu9pXNHjR0_qJ5g.png" /><figcaption>Architecture Diagram</figcaption></figure><p>You can find the code here: <a href="https://github.com/slapglif/AgenticTeams">slapglif/AgenticTeams</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a29895b4b85" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I implemented googles groundbreaking co-research at home (and then some)]]></title>
            <link>https://medium.com/@icarusabiding/how-i-implemented-googles-groundbreaking-co-research-at-home-and-then-some-3aa8a510887b?source=rss-3f86cfff2eb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/3aa8a510887b</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[discovery]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Michael Brown]]></dc:creator>
            <pubDate>Thu, 20 Feb 2025 17:13:25 GMT</pubDate>
            <atom:updated>2025-02-20T17:13:25.455Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1OZI27FPx-EbCUKBDkI6AQ.png" /><figcaption>Sifu being booted :)</figcaption></figure><p><strong>The Scientific Frontier and the Rise of the AI Co-scientist</strong></p><p>Note: Repo link here: <a href="https://github.com/slapglif/Sifu">slapglif/Sifu</a></p><p>Among my many wacky, fun projects — and trust me, there are a few — I recently stumbled upon <a href="https://research.google/blog/accelerating-scientific-breakthroughs-with-an-ai-co-scientist/">Google’s announcement</a> of a co-research system, currently in private preview. While access is still under wraps, they did generously release the <a href="https://storage.googleapis.com/coscientist_paper/ai_coscientist.pdf">theoretical underpinnings</a> alongside it. Now, full disclosure, I’d already been tinkering away for a while on my own deep-research system. Initially, I drew inspiration from projects like smol-agents and other companies venturing into deep research AI. However, I consistently found these existing systems lacking in key areas: capabilities felt limited, interoperability was a headache, and overall robustness? Let’s just say it wasn’t confidence-inspiring.</p><p>What I mean by that, of course, is that every deep research system I’ve seen seems determined to reinvent the wheel. They’re all implementing custom agentic frameworks and architectures from scratch, seemingly just to avoid using LangChain. Perhaps it’s a desire to declare, “We’re <em>not</em> LangChain!” But for me? Well, the solution I’ve landed on is built almost entirely in LangChain and LangGraph. And I’m a massive fan — because, quite simply, it <em>works</em>.</p><h3><strong>Embodying the Scientific Method in AI</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PY2EXicu0wg7dPTK.png" /><figcaption>origional design by googles team</figcaption></figure><p>So, how do you actually <em>build</em> an AI that can, you know, <em>think</em> like a scientist? That’s the million-dollar question, right? Well, instead of trying to conjure some magical, sentient AI from thin air, I decided to stick to something that’s been proven to work for centuries: the scientific method itself. Seriously, if it ain’t broke…</p><p>The whole system is built around what I call the “Generate, Debate, Evolve” framework. Think of it as a sped-up, AI-powered version of how real science gets done:</p><ul><li><strong>Generate:</strong> First, you gotta have ideas, right? This is where the AI Co-scientist gets creative. It’s constantly churning through knowledge, exploring literature, and basically brainstorming potential research hypotheses. We’re talking about the AI taking on the role of the initial spark, the “what if?” moment that kicks off an investigation. It’s not just regurgitating facts; it’s trying to come up with <em>new</em> angles, fresh perspectives.</li><li><strong>Debate:</strong> But just throwing ideas out there isn’t science. You need rigor. This is where the “Debate” part comes in. The AI Co-scientist doesn’t just blindly accept its own ideas. It puts them through the wringer, essentially simulating a scientific peer review process <em>internally</em>. It critiques its own hypotheses, identifies weaknesses, and throws them into a virtual “tournament” to see which ones hold up under scrutiny. It’s like a bunch of AI scientists arguing in a digital lab, except way faster and (hopefully) less prone to bruised egos.</li><li><strong>Evolve:</strong> Finally, science is iterative. You build on what you’ve learned, refine your theories, and keep pushing forward. The “Evolve” stage is all about improvement. The AI takes the top-ranked hypotheses from the “tournament” and actively tries to make them better. It digs deeper into the literature, fills in reasoning gaps, simplifies overly complex ideas, and even tries to draw inspiration from seemingly unrelated fields. It’s a continuous cycle of refinement, pushing the quality of the research proposals higher and higher.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NmvcmXaDu-nX49B9.png" /><figcaption>comcepts from the google release</figcaption></figure><p>Now, to make this “Generate, Debate, Evolve” thing actually <em>work</em>, I went for a <strong>multi-agent architecture</strong>. Basically, instead of one giant, monolithic AI brain, I broke the system down into a team of specialized agents, each with its own role and expertise. Think of it like assembling a crack team of researchers:</p><ul><li>You’ve got the <strong>Generation Agent</strong>, the idea guy, constantly coming up with new hypotheses.</li><li>Then there’s the <strong>Reflection Agent</strong>, the super-critical peer reviewer, ruthlessly evaluating those ideas.</li><li>The <strong>Ranking Agent</strong> is the objective judge, running the “tournament” and figuring out which hypotheses are the most promising.</li><li>The <strong>Evolution Agent</strong> is the master refiner, taking those top ideas and making them even better.</li><li>And so on… you get the picture. Each agent is like a specialized module, focusing on a specific part of the scientific process. They all communicate and collaborate (mostly argue, let’s be honest), orchestrated by a <strong>Supervisor Agent</strong> that’s kind of like the Principal Investigator, keeping everything on track.</li></ul><p>And the secret sauce that ties it all together? It’s built on the <a href="https://github.com/langchain-ai/langgraph/blob/main/docs/docs/tutorials/llm-compiler/LLMCompiler.ipynb"><strong>LLM Compiler pattern</strong></a>. This is where LangChain really shines. Instead of just using the LLM to generate text, I’m using it as the <em>engine</em> for the entire workflow. The LLM isn’t just spitting out answers; it’s <em>planning</em>, <em>executing</em>, and <em>deciding</em> the next steps in the research process. It’s like having a super-powered AI conductor leading an orchestra of specialized agents. And because it’s all built in LangChain and LangGraph, it’s surprisingly robust, flexible, and (dare I say it?) actually kind of elegant.</p><p>This multi-agent, scientific-method-inspired design is the backbone of the AI Co-scientist. It’s not just about throwing AI at a problem; it’s about building a system that <em>reasons</em> and <em>evolves</em> like a real scientific collaborator. And that, I think, is where the real magic happens.</p><h3><strong>The Building Blocks of Scientific Intelligence</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*33Xvqf8E_yFfe37WetyPYA.png" /><figcaption>Current `Sifu` Architecture</figcaption></figure><p>Alright, so we’ve got the “Generate, Debate, Evolve” engine humming, powered by a team of specialized AI agents. But what are the <em>actual</em> pieces that make this whole contraption tick? It’s not magic, folks, it’s (relatively) clever engineering! Let’s break down the key components that give the AI Co-scientist its, shall we say, <em>scientific</em> intelligence:</p><p>First up, the brains of the operation, the <strong>Knowledge Graph (KG)</strong>. Think of it as the AI’s long-term memory, its personal Wikipedia, but way more structured and interconnected. We’re talking about a <strong>Neo4j graph database</strong>, because when you’re dealing with knowledge, relationships are <em>everything</em>. It’s not just about storing facts; it’s about mapping out how those facts <em>relate</em> to each other.</p><p>Why a Knowledge Graph? Because science isn’t just a pile of isolated data points. It’s a web of concepts, theories, evidence, and connections. The KG lets the AI Co-scientist:</p><ul><li><strong>Organize knowledge like a scientist thinks:</strong> Entities (drugs, diseases, genes, mechanisms) become nodes, and relationships (treats, causes, interacts_with) become the links. It mirrors how scientific understanding is actually structured.</li><li><strong>Reason across domains:</strong> The KG allows the system to connect knowledge from different scientific fields. Suddenly, insights from genetics can inform drug repurposing, and vice versa. It’s all interconnected in the graph.</li><li><strong>Go beyond keyword search:</strong> Forget just searching for words. The KG lets the AI perform <em>semantic</em> searches, understanding the meaning behind queries and retrieving contextually relevant information based on relationships, not just keywords. It’s like asking a smart colleague, not just a search engine.</li></ul><p>Populating this KG is a beast of a task, of course. That’s where <strong>Deep Web Research and Information Acquisition</strong> comes in. The AI Co-scientist isn’t just limited to a pre-packaged dataset. It’s designed to be a relentless knowledge hunter, constantly scouring the web for the latest scientific publications, research papers, and data. We’re talking about:</p><ul><li><strong>Automated Literature Review on Steroids:</strong> The system can generate diverse search queries, targeting PubMed, ArXiv, and the general web to get a truly comprehensive view of a research topic. It’s not just surface-level Google searches; it’s digging deep into the scientific literature.</li><li><strong>Content Extraction and Cleaning:</strong> It’s not enough to just find web pages; you gotta <em>understand</em> them. The AI can parse different document formats (HTML, PDFs, even some media files), extract the relevant text, and clean up all the web cruft (think: ads, navigation menus, and all that internet noise).</li><li><strong>Continuous Knowledge Updates:</strong> The goal is to keep the KG <em>alive</em> and up-to-date. The web research capabilities are designed for ongoing knowledge acquisition, ensuring the AI Co-scientist is always working with the latest information.</li></ul><p>But raw knowledge isn’t enough. To really <em>do</em> science, you need to tailor your tools to the specific problem. That’s where <strong>LoRA Fine-tuning for Domain Expertise</strong> comes into play. We’re talking about:</p><ul><li><strong>Customizing the LLM Brain:</strong> LoRA (Low-Rank Adaptation) is a fancy way of saying we can tweak the underlying Large Language Model to become a <em>specialist</em> in a particular scientific domain. Instead of a general-purpose LLM, we can create a biomedicine-focused LLM, or a materials science-focused LLM, etc.</li><li><strong>Learning from its Own Discoveries:</strong> The coolest part? The AI Co-scientist learns from <em>itself</em>. It uses the knowledge it synthesizes to generate training examples and then uses those examples to fine-tune its own LLM brain with LoRA. It’s like a self-improving scientific apprentice, constantly getting smarter.</li><li><strong>Boosting Performance and Relevance:</strong> LoRA fine-tuning makes the system way more effective within its chosen domain. It understands the nuances of scientific language, generates more relevant hypotheses, and provides more accurate and insightful answers. It goes from being a smart generalist to a laser-focused scientific expert.</li></ul><p>Finally, at the heart of it all is the <strong>Natural Language Interface and Scientist-in-the-Loop Design</strong>. This isn’t some black box AI that spits out answers and disappears. It’s designed to be a <em>collaborator</em>.</p><ul><li><strong>Talking the Scientist’s Language:</strong> Natural language is the primary interface. Scientists can specify their research goals, provide feedback, and guide the system using simple, intuitive language, just like talking to a colleague.</li><li><strong>Human-AI Partnership:</strong> It’s all about augmentation, not automation. The AI Co-scientist is meant to <em>empower</em> scientists, not replace them. Humans are always in the loop, guiding the research direction, providing expert judgment, and making the final decisions.</li><li><strong>Accessible and User-Friendly:</strong> The goal is to make this powerful AI accessible to <em>any</em> scientist, regardless of their AI expertise. The natural language interface and intuitive design are key to making that happen.</li></ul><p>These are the key building blocks. Each component is crucial, and they all work together to create a system that’s more than just a collection of AI tools; it’s a cohesive, intelligent, and (hopefully) game-changing research partner. But building it? Yeah, that was definitely a journey…</p><p>If you like the conceptual work — you can check out and fork/star the <a href="https://github.com/slapglif/Sifu">github repo here</a>, I only ask that if you use my work, you mention me.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3aa8a510887b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NeurIPS 2024 BELKA Challenge: Innovating Small Molecule Binding Prediction]]></title>
            <link>https://medium.com/@icarusabiding/neurips-2024-belka-challenge-innovating-small-molecule-binding-prediction-593aefc7a310?source=rss-3f86cfff2eb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/593aefc7a310</guid>
            <category><![CDATA[kolmogorov-arnold-network]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neurips-2024]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Michael Brown]]></dc:creator>
            <pubDate>Mon, 01 Jul 2024 23:51:46 GMT</pubDate>
            <atom:updated>2025-01-24T02:43:00.421Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*pqdi8mm8D0XOjk6u" /></figure><h3>KAN you feel it? Small Molecule Binding Prediction with Signal to Noise, and Global Attention</h3><p>Recently, I entered into multiple Kaggle competitions — but the one I’m most excited about was the <a href="https://www.kaggle.com/competitions/leash-BELKA/overview">NeurIPS 2024 BELKA Challenge</a>.</p><blockquote>The NeurIPS 2024 “Predict New Medicines with BELKA” competition presents a formidable challenge in the realm of drug discovery. With a dataset of 133 million small molecules tested against three protein targets, this competition pushes the boundaries of machine learning in pharmaceutical research. In this article, I’ll detail my approach to this challenge, diving into the technical aspects of data processing, feature engineering, and model architecture.</blockquote><p>Due to my work at @AnunaAI — I am actually very familliar with the Protein-Compound interaction space, and have worked to develop autoencoders for various pci and ppi purposes. This was something right in my wheelhouse, however — The challenge poses quite a few initial problems to solve.</p><h3>My Approach: Graph Neural Networks and Fourier Magic</h3><p>So, how am I tackling this Goliath of a challenge? With a slingshot made of graph neural networks, Fourier transforms, and a dash of Kolmogorov-Arnold network magic!</p><p>Here’s a sneak peek at my secret sauce:</p><ol><li><strong>Graph Convolutional Networks (GCN)</strong>: These bad boys are perfect for handling molecular structures. They can learn from the atoms and bonds in our compounds, capturing all that juicy structural information.</li><li><strong>Naive Fourier Kolmogorov-Arnold Network (KAN) Layers</strong>: This is where things get funky. By applying Fourier transforms to our molecule representations, we can capture complex, non-linear relationships. It’s like giving our model a pair of kaleidoscope glasses to see patterns we mere mortals might miss.</li><li><strong>Protein Embeddings</strong>: We can’t forget about our protein targets! I’m using embeddings to represent them, allowing the model to learn meaningful representations of these complex biological entities.</li><li><strong>Global Attention Mechanism</strong>: To tie it all together, I’ve whipped up a global attention mechanism. It helps the model focus on the most important features when making predictions. Think of it as giving our neural network a magnifying glass and a really good pair of glasses.</li></ol><h3>The BELKA Dataset: Scale and Complexity</h3><p>BELKA (Big Encoded Library for Chemical Assessment), provided by Leash Biosciences, is a game-changer in the field of drug discovery datasets. At over 50GB, it dwarfs existing public resources like BindingDB. The dataset contains SMILES representations of molecules, protein identifiers, and binary binding information.</p><p>The sheer size of BELKA necessitated a custom streaming solution. I implemented a StreamingMoleculeDataset class, inheriting from torch.utils.data.IterableDataset:</p><pre>class StreamingMoleculeDataset(torch.utils.data.IterableDataset):<br>    &quot;&quot;&quot;<br>    Dataset for streaming and processing molecule data from CSV or Parquet files.<br>    Includes calculation of LogP and selective Tanimoto similarity.<br>    &quot;&quot;&quot;<br><br>    def __init__(self, file_path: str, smiles_col: str = &#39;SMILES&#39;, protein_col: str = &#39;Target Sequence&#39;,<br>                 target_col: Optional[str] = &#39;Label&#39;, chunk_size: int = 10000, max_ref_mols: int = 1000):<br>        &quot;&quot;&quot;<br>        Initialize the StreamingMoleculeDataset.<br><br>        Args:<br>            file_path (str): Path to the input CSV or Parquet file.<br>            smiles_col (str): Name of the column containing SMILES strings. Defaults to &#39;SMILES&#39;.<br>            protein_col (str): Name of the column containing protein identifiers. Defaults to &#39;Target Sequence&#39;.<br>            target_col (Optional[str]): Name of the column containing target values. Defaults to &#39;Label&#39;.<br>            chunk_size (int): Number of rows to process per iteration. Defaults to 10000.<br>            max_ref_mols (int): Maximum number of reference molecules for Tanimoto similarity. Defaults to 1000.<br>        &quot;&quot;&quot;<br>        super().__init__()<br>        self.file_path = file_path<br>        self.smiles_col = smiles_col<br>        self.protein_col = protein_col<br>        self.target_col = target_col<br>        self.chunk_size = chunk_size<br>        self.max_ref_mols = max_ref_mols<br><br>        self.protein_encoder = OrdinalEncoder()<br>        self.reference_fps = {}  # Dictionary to store reference fingerprints<br>        self._preprocess()</pre><p>This class allows for efficient processing of the massive dataset without memory issues, reading and processing the data in chunks.</p><p>The trick was in the preprocessing steps really, we need to pull only a subset of rows, get their mol and chem info, then stuff our batch object with it on the fly</p><pre>def _preprocess(self) -&gt; None:<br>    &quot;&quot;&quot;<br>    Preprocess the dataset: fit the protein encoder and calculate reference fingerprints.<br>    &quot;&quot;&quot;<br>    if self.file_path.endswith(&#39;.csv&#39;):<br>        df = pd.read_csv(self.file_path, nrows=1000000)  # Load a subset for preprocessing<br>    elif self.file_path.endswith(&#39;.parquet&#39;):<br>        # For parquet files, read only necessary columns during preprocessing<br>        columns = [self.protein_col, self.smiles_col]<br>        if self.target_col is not None:<br>            columns.append(self.target_col)<br>        df = pd.read_parquet(self.file_path, columns=columns)<br>    else:<br>        raise ValueError(f&quot;Unsupported file format: {self.file_path}&quot;)<br><br>    self.protein_encoder.fit(df[self.protein_col].unique())<br><br>    # Select reference molecules (prioritize known binders)<br>    if self.target_col is not None:<br>        binders = df[df[self.target_col] == 1][self.smiles_col].unique()<br>        non_binders = df[df[self.target_col] == 0][self.smiles_col].unique()<br><br>        n_binders = min(len(binders), self.max_ref_mols // 2)<br>        n_non_binders = min(len(non_binders), self.max_ref_mols - n_binders)<br><br>        reference_smiles = np.random.choice(binders, n_binders, replace=False).tolist() + \<br>                           np.random.choice(non_binders, n_non_binders, replace=False).tolist()<br>    else:<br>        # If there&#39;s no target column, just select random molecules<br>        all_smiles = df[self.smiles_col].unique()<br>        reference_smiles = np.random.choice(all_smiles, min(len(all_smiles), self.max_ref_mols),<br>                                            replace=False).tolist()<br><br>    # Calculate reference fingerprints<br>    for smiles in reference_smiles:<br>        mol = Chem.MolFromSmiles(smiles)<br>        if mol is not None:<br>            self.reference_fps[smiles] = GetMorganFingerprintAsBitVect(mol, 2, nBits=1024)</pre><p>finally, we can iterate each batch as a tensor</p><pre>def __iter__(self):<br>    &quot;&quot;&quot;<br>    Iterate over the dataset, yielding processed Data objects.<br><br>    Yields:<br>        Data: A PyTorch Geometric Data object containing molecule information.<br>    &quot;&quot;&quot;<br>    if self.file_path.endswith(&#39;.csv&#39;):<br>        chunks = pd.read_csv(self.file_path, chunksize=self.chunk_size)<br>    elif self.file_path.endswith(&#39;.parquet&#39;):<br>        # For parquet files, read all columns during iteration<br>        chunks = pd.read_parquet(self.file_path, chunksize=self.chunk_size)<br>    else:<br>        raise ValueError(f&quot;Unsupported file format: {self.file_path}&quot;)<br><br>    for chunk in chunks:<br>        for _, row in chunk.iterrows():<br>            mol = Chem.MolFromSmiles(row[self.smiles_col])<br>            if mol is None:<br>                continue<br><br>            x, edge_index, global_features = self._get_molecule_features(mol)<br><br>            # Ensure global_features has the correct shape (1, num_features)<br>            global_features = global_features.unsqueeze(0)<br><br>            data = Data(x=x, edge_index=edge_index, global_features=global_features)<br><br>            if self.target_col is not None and self.target_col in row:<br>                data.y = torch.tensor([row[self.target_col]], dtype=torch.float32)<br>            else:<br>                data.y = torch.tensor([0.0], dtype=torch.float32)  # Default value if target is not available<br><br>            data.protein = torch.tensor([self.protein_encoder.transform([[row[self.protein_col]]])[0]],<br>                                        dtype=torch.long)<br><br>            yield data</pre><h3>Capturing Molecular Properties</h3><p>In addition to the normal morgan fingerprint and the normal mol and chem info, I decided (on advice from a colleague at Anuna) to select Tanimoto and LogP — I hadn’t even heard of them.</p><p><strong>LogP</strong> (Octanol-Water Partition Coefficient) is a crucial physicochemical property in drug discovery. It measures a compound’s lipophilicity, which influences its ability to permeate cell membranes.</p><p>I calculate LogP using RDKit’s Crippen module:</p><pre>logp = Crippen.MolLogP(mol)</pre><blockquote>Including LogP as a feature helps the model understand a molecule’s behavior in biological systems.</blockquote><h3>Tanimoto Similarities and Morgan Fingerprints</h3><p>Tanimoto similarity, based on Morgan fingerprints, is a powerful tool for comparing molecular structures. Morgan fingerprints are circular fingerprints that capture local molecular substructures. I generate these using RDKit:</p><pre>mol_fp = GetMorganFingerprintAsBitVect(mol, 2, nBits=1024)</pre><p>I then calculate Tanimoto similarities between each molecule and a set of reference molecules:</p><pre>tanimoto_sims = [DataStructs.TanimotoSimilarity(mol_fp, ref_fp) for ref_fp in self.reference_fps.values()]</pre><blockquote>This approach allows the model to understand structural similarities between molecules, which can be indicative of similar binding properties.</blockquote><h3>Additional Molecular Descriptors</h3><p>To provide a comprehensive view of each molecule, I include several other descriptors:</p><pre>mol_features = [<br>    CalcNumRings(mol),<br>    AllChem.CalcNumRotatableBonds(mol),<br>    AllChem.CalcNumHBD(mol),<br>    AllChem.CalcNumHBA(mol),<br>    CalcNumAmideBonds(mol),<br>    CalcNumSpiroAtoms(mol),<br>    ExactMolWt(mol),<br>    MolWt(mol),<br>    HeavyAtomMolWt(mol),<br>    MaxAbsPartialCharge(mol),<br>    MinAbsPartialCharge(mol),<br>    NumRadicalElectrons(mol),<br>    NumValenceElectrons(mol),<br>]</pre><blockquote>These features capture various aspects of molecular structure and properties that can influence binding behavior.</blockquote><h3>Graph Representation: Adjacency Matrix and Node Features</h3><p>To leverage the structural information of molecules, I represent them as <strong>graphs</strong>. The adjacency matrix, or edge index in PyTorch Geometric terminology, <strong>is crucial for this representation:</strong></p><pre>edges = []<br>for bond in mol.GetBonds():<br>    i = bond.GetBeginAtomIdx()<br>    j = bond.GetEndAtomIdx()<br>    edges += [[i, j], [j, i]]<br>edge_index = torch.tensor(edges, dtype=torch.long).t().contiguous()</pre><blockquote>This edge index defines the connectivity of atoms in the molecule. For node features, I use atomic properties and the global molecular features:</blockquote><pre>node_features = []<br>for atom in mol.GetAtoms():<br>    atom_features = [<br>        atom.GetAtomicNum(),<br>        atom.GetTotalDegree(),<br>        atom.GetFormalCharge(),<br>        atom.GetTotalNumHs(),<br>        atom.GetNumRadicalElectrons(),<br>        atom.GetHybridization(),<br>        atom.GetIsAromatic(),<br>        atom.IsInRing(),<br>    ]<br>    node_features.append(atom_features + mol_features)<br>x = torch.tensor(node_features, dtype=torch.float)</pre><h3>Model Architecture: Graph Neural Networks and Fourier KAN Layers</h3><p>The core of my approach is a Graph Neural Network (GNN) combined with Fourier Kolmogorov-Arnold Network (KAN) layers. Here’s an overview of the BELKAModule:</p><pre>class BELKAModule(pl.LightningModule):<br>    def __init__(<br>            self,<br>            in_feat: int,<br>            hidden_feat: int,<br>            out_feat: int,<br>            num_layers: int,<br>            num_proteins: int,<br>            protein_embedding_dim: int,<br>            global_feat: int,<br>            learning_rate: float,<br>            grid_feat: int = 200<br>    ):<br>        &quot;&quot;&quot;<br>        Initialize the BELKAModule.<br><br>        Args:<br>            in_feat (int): Number of input features for node attributes.<br>            hidden_feat (int): Number of hidden features in GCN and KAN layers.<br>            out_feat (int): Number of output features (typically 1 for binary classification).<br>            num_layers (int): Number of GCN and KAN layers.<br>            num_proteins (int): Number of unique proteins for embedding.<br>            protein_embedding_dim (int): Dimension of protein embeddings.<br>            global_feat (int): Number of global features (including LogP and Tanimoto similarities).<br>            learning_rate (float): Learning rate for the optimizer.<br>            grid_feat (int): Grid size for the Fourier KAN layer.<br>        &quot;&quot;&quot;<br>        super().__init__()<br>        self.save_hyperparameters()<br><br>        self.protein_embedding = nn.Embedding(num_proteins, protein_embedding_dim)<br><br>        self.convs = nn.ModuleList()<br>        self.kans = nn.ModuleList()<br>        for i in range(num_layers):<br>            if i == 0:<br>                self.convs.append(GCNConv(in_feat, hidden_feat))<br>            else:<br>                self.convs.append(GCNConv(hidden_feat, hidden_feat))<br>            self.kans.append(NaiveFourierKANLayer(hidden_feat, hidden_feat, grid_feat))<br><br>        self.global_attention = nn.Sequential(<br>            nn.Linear(global_feat, hidden_feat),<br>            nn.ReLU(),<br>            nn.Linear(hidden_feat, 1)<br>        )<br><br>        self.final_layer = nn.Linear(hidden_feat + protein_embedding_dim + global_feat, out_feat)<br><br>        self.criterion = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([99.0]))<br>        self.learning_rate = learning_rate<br><br>        # Metrics<br>        self.train_accuracy = torchmetrics.Accuracy(task=&quot;binary&quot;)<br>        self.val_accuracy = torchmetrics.Accuracy(task=&quot;binary&quot;)<br>        self.test_accuracy = torchmetrics.Accuracy(task=&quot;binary&quot;)</pre><h3>The Forward Pass</h3><p>This is where all the magic happens, first we —</p><ul><li>Pull the Batches global features which is:</li></ul><pre>torch.tensor([logp] + tanimoto_sims, dtype=torch.float)</pre><ul><li><strong>Next, the graph convolutional layers</strong> extract features from the input molecular graph data, learning complex relationships between atoms and bonds. This information is further refined by the KAN enhancing the model’s representational power.</li><li>To focus on key aspects of the molecular data, we employ an <strong>attention mechanism</strong> on the <strong>global features</strong>, which might capture properties like molecular weight or solubility. We then apply the attention to the global features, selectively weighting their importance.</li><li>Simultaneously, we generate a rich <strong>protein embedding</strong>, capturing the essence of the protein’s amino acid sequence.</li></ul><pre>def forward(self, data: Union[Data, Batch]) -&gt; torch.Tensor:<br>    &quot;&quot;&quot;<br>    Forward pass of the BELKAModule.<br><br>    Args:<br>        data (Union[Data, Batch]): Input data containing node features, edge indices, and global features.<br><br>    Returns:<br>        torch.Tensor: Output predictions.<br>    &quot;&quot;&quot;<br>    device = self.device<br>    x, edge_index, batch = data.x.to(device), data.edge_index.to(device), data.batch.to(device)<br>    global_features = data.global_features.to(device)<br><br>    # Handle zero nodes case<br>    if x.size(0) == 0:<br>        return torch.zeros((1, 1), device=device)<br><br>    # Graph convolutions and KAN layers<br>    for conv, kan in zip(self.convs, self.kans):<br>        x = conv(x, edge_index)<br>        x = kan(x)<br>        x = F.relu(x)<br><br>    # Global pooling<br>    x = global_mean_pool(x, batch)<br><br>    # Attention on global features<br>    attention_weights = self.global_attention(global_features).squeeze(-1)<br>    attention_weights = F.softmax(attention_weights, dim=0)<br>    attended_global = torch.sum(global_features * attention_weights.unsqueeze(-1), dim=0)<br><br>    # Combine with protein embedding<br>    protein_emb = self.protein_embedding(data.protein.to(device))<br>    if len(protein_emb.shape) &gt; 2:<br>        protein_emb = protein_emb.mean(dim=1)  # Average protein embeddings if multiple per graph<br>    elif len(protein_emb.shape) == 1:<br>        protein_emb = protein_emb.unsqueeze(0)  # Add batch dimension if single protein<br><br>    # Ensure all tensors have the same batch size<br>    num_graphs = x.size(0)<br>    protein_emb = protein_emb.expand(num_graphs, -1)<br>    attended_global = attended_global.unsqueeze(0).expand(num_graphs, -1)<br><br>    x = torch.cat([x, protein_emb, attended_global], dim=1)<br><br>    return self.final_layer(x)</pre><p>Finally, we fuse the processed graph information, the attended global features, and the protein embedding, <strong>feeding this combined representation into a final layer to generate the desired predictions.</strong></p><h3>Fourier Kolmogorov-Arnold Network (KAN) Layers</h3><p>The decision to use Fourier KAN layers was motivated by their ability to capture complex, non-linear relationships in data. These layers apply a learnable Fourier series to the input:</p><p>Think of it like this:</p><blockquote>imagine trying to approximate a complex wave pattern. Instead of using simple curves, we can combine various sine and cosine waves of different frequencies. This is essentially what a Fourier series does — it decomposes complex patterns into simpler, oscillating components.</blockquote><p>Here’s how the NaiveFourierKANLayer leverages this concept:</p><ol><li><strong>Learnable Fourier Coefficients:</strong> The layer learns a set of weights (called “Fourier coefficients”) for a range of frequencies, represented by sine and cosine functions. These coefficients determine how much each frequency contributes to representing the input data’s underlying patterns.</li><li><strong>Creating a Frequency Grid:</strong> The code generates a grid of frequencies (using torch.arange) that determine the resolution at which it analyzes the input data.</li><li><strong>Projecting to Fourier Space:</strong> The input data is effectively projected into a “Fourier space” using sine and cosine functions applied to the frequency grid. This transformation highlights patterns at different frequencies.</li><li><strong>Applying Fourier Coefficients:</strong> The learned Fourier coefficients are multiplied with the transformed data. This step weights the importance of each frequency component.</li><li><strong>Combining and Outputting:</strong> The contributions from different frequencies are combined, effectively reconstructing the input data with enhanced non-linear features. The output of the layer captures these complex relationships.</li></ol><blockquote>In simpler terms, this layer acts like a sophisticated filter bank that learns to pick out important patterns in your data at various frequencies, allowing the network to model highly non-linear relationships. This can be particularly useful for tasks where the data exhibits complex periodicity or intricate patterns.</blockquote><p>This code defines a NaiveFourierKANLayer, a neural network layer that leverages the power of <strong>Fourier series</strong> to learn complex relationships within data. Big shoutout to <a href="https://medium.com/u/fbd6f4eb076e">Ronan Takizawa</a> for this awesome <a href="https://medium.com/@ronantech/how-to-build-a-graph-based-kolmogorov-arnold-network-d5b37303f452">implementation</a> (which I tweaked for my purposes of course 🤖)</p><pre>class NaiveFourierKANLayer(nn.Module):<br>    &quot;&quot;&quot;<br>    Naive Fourier Kolmogorov-Arnold Network Layer.<br><br>    This layer applies a Fourier transform to the input and uses learnable coefficients<br>    to create a complex, non-linear transformation.<br>    &quot;&quot;&quot;<br><br>    def __init__(self, inputdim: int, outdim: int, gridsize: int = 300, addbias: bool = True):<br>        &quot;&quot;&quot;<br>        Initialize the NaiveFourierKANLayer.<br><br>        Args:<br>            inputdim (int): Dimension of the input features.<br>            outdim (int): Dimension of the output features.<br>            gridsize (int): Size of the Fourier grid. Defaults to 300.<br>            addbias (bool): Whether to add a bias term. Defaults to True.<br>        &quot;&quot;&quot;<br>        super(NaiveFourierKANLayer, self).__init__()<br>        self.gridsize = gridsize<br>        self.addbias = addbias<br>        self.inputdim = inputdim<br>        self.outdim = outdim<br><br>        # Initialize Fourier coefficients<br>        self.fouriercoeffs = nn.Parameter(<br>            torch.randn(2, outdim, inputdim, gridsize) /<br>            (torch.sqrt(torch.tensor(inputdim).float()) * torch.sqrt(torch.tensor(self.gridsize).float()))<br>        )<br>        if self.addbias:<br>            self.bias = nn.Parameter(torch.zeros(1, outdim))<br><br>    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:<br>        &quot;&quot;&quot;<br>        Forward pass of the NaiveFourierKANLayer.<br><br>        Args:<br>            x (torch.Tensor): Input tensor of shape (batch_size, inputdim).<br><br>        Returns:<br>            torch.Tensor: Output tensor of shape (batch_size, outdim).<br>        &quot;&quot;&quot;<br>        xshp = x.shape<br>        outshape = xshp[0:-1] + (self.outdim,)<br>        x = x.view(-1, self.inputdim)<br><br>        # Create frequency grid<br>        k = torch.arange(1, self.gridsize + 1, device=x.device).view(1, 1, 1, self.gridsize)<br>        xrshp = x.view(x.shape[0], 1, x.shape[1], 1)<br><br>        # Compute Fourier features<br>        c = torch.cos(k * xrshp)<br>        s = torch.sin(k * xrshp)<br>        c = c.view(1, x.shape[0], x.shape[1], self.gridsize)<br>        s = s.view(1, x.shape[0], x.shape[1], self.gridsize)<br><br>        # Apply Fourier coefficients<br>        y = torch.einsum(&quot;dbik,djik-&gt;bj&quot;, torch.cat([c, s], dim=0), self.fouriercoeffs)<br><br>        if self.addbias:<br>            y += self.bias<br><br>        return y.view(outshape)</pre><blockquote>These layers allow the model to capture periodic patterns and complex interactions that might be present in molecular binding data.</blockquote><h3>Training and Optimization</h3><p>The model is trained using PyTorch Lightning, which simplifies the training loop and allows for easy integration of advanced features like early stopping and learning rate scheduling</p><p>In addition to the lightning model above, we need to be sure it has the right metrics and optimizers</p><pre>    def training_step(self, batch: Batch, batch_idx: int) -&gt; torch.Tensor:<br>        &quot;&quot;&quot;<br>        Perform a single training step.<br><br>        Args:<br>            batch (Batch): The input batch of data.<br>            batch_idx (int): The index of the current batch.<br><br>        Returns:<br>            torch.Tensor: The computed loss for the batch.<br>        &quot;&quot;&quot;<br>        y_hat = self(batch)<br>        y = batch.y.to(y_hat.device).view(-1, 1)  # Reshape y to match y_hat<br>        loss = self.criterion(y_hat, y)<br>        self.log(&#39;train_loss&#39;, loss, on_step=True, on_epoch=True, prog_bar=True)<br>        self.train_accuracy(y_hat.sigmoid(), y)<br>        self.log(&#39;train_acc&#39;, self.train_accuracy, on_step=True, on_epoch=True, prog_bar=True)<br>        return loss<br><br>  # ... Repeat for val and test steps<br><br>  def configure_optimizers(self) -&gt; Dict[str, Any]:<br>      &quot;&quot;&quot;<br>      Configure optimizers and learning rate schedulers.<br>      Returns:<br>          Dict[str, Any]: A dictionary containing the optimizer and learning rate scheduler configuration.<br>      &quot;&quot;&quot;<br>      optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)<br>      scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode=&#39;min&#39;, factor=0.1, patience=10,<br>                                                             verbose=True)<br>      return {<br>          &quot;optimizer&quot;: optimizer,<br>          &quot;lr_scheduler&quot;: scheduler,<br>          &quot;monitor&quot;: &quot;val_loss&quot;<br>      }<br></pre><p>Lastly, we define our Lightning data module to ensure everything is compatible</p><pre>class BELKADataModule(pl.LightningDataModule):<br>    def __init__(self, train_file: str, test_file: str, batch_size: int = 32, num_workers: int = 4):<br>        super().__init__()<br>        self.train_file = train_file<br>        self.test_file = test_file<br>        self.batch_size = batch_size<br>        self.num_workers = num_workers<br><br>    def setup(self, stage: Optional[str] = None):<br>            if stage == &#39;fit&#39; or stage is None:<br>                self.train_dataset = StreamingMoleculeDataset(self.train_file)<br>                self.val_dataset = StreamingMoleculeDataset(self.train_file)  # Use a separate iterator for validation<br>            if stage == &#39;test&#39; or stage is None:<br>                self.test_dataset = StreamingMoleculeDataset(self.test_file, target_col=None)<br>        def train_dataloader(self) -&gt; DataLoader:<br>            return DataLoader(self.train_dataset, batch_size=self.batch_size, num_workers=self.num_workers,<br>                              persistent_workers=True)<br>        def val_dataloader(self) -&gt; DataLoader:<br>            return DataLoader(self.val_dataset, batch_size=self.batch_size, num_workers=self.num_workers,<br>                              persistent_workers=True)<br>        def test_dataloader(self) -&gt; DataLoader:<br>            return DataLoader(self.test_dataset, batch_size=self.batch_size, num_workers=self.num_workers,<br>                              persistent_workers=True)</pre><p>…and set up our main training loop.</p><pre>def main():<br>    pl.seed_everything(42)<br>    train_file = &quot;kaggle/input/leash-BELKA/train.csv&quot;<br>    test_file = &quot;kaggle/input/leash-BELKA/test.parquet&quot;<br>    data_module = BELKADataModule(train_file, test_file, batch_size=128, num_workers=4)<br>    data_module.setup()<br><br>    # Get a sample batch to determine input dimensions<br>    sample_batch = next(iter(data_module.train_dataloader()))<br>    in_feat = sample_batch.x.size(-1)<br>    num_proteins = len(data_module.train_dataset.protein_encoder.categories_[0])<br>    global_feat = sample_batch.global_features.size(-1)  # This should now be 1001<br>    <br>    model = BELKAModule(<br>        in_feat=in_feat,<br>        hidden_feat=256,<br>        out_feat=1,<br>        num_layers=2,<br>        num_proteins=num_proteins,<br>        protein_embedding_dim=32,<br>        global_feat=global_feat,<br>        learning_rate=1e-3,<br>        grid_feat=200<br>    )<br><br>    logger.info(&quot;🏗️ Model architecture:&quot;)<br>    logger.info(model)</pre><p>Personally, I use MLFlow its free and dead simple to use, so I implemented their logger integration, as well as their early stopping, checkpointing and learning rate monitor — all of which are advanced training strategies. I can’t stress enough, everyone should be using early stopping.</p><pre>callbacks = [<br>  EarlyStopping(monitor=&#39;val_ap&#39;, patience=10, mode=&#39;max&#39;),<br>  ModelCheckpoint(monitor=&#39;val_ap&#39;, save_top_k=3, mode=&#39;max&#39;, filename=&#39;belka-{epoch:02d}-{val_ap:.4f}&#39;),<br>  LearningRateMonitor(logging_interval=&#39;step&#39;)<br>]<br>mlf_logger = MLFlowLogger(experiment_name=&quot;BELKA_GKAN&quot;, tracking_uri=&quot;mlruns&quot;)</pre><p>If your using multiple GPU’s you can set the strategy to ddp or ddp_notebook if your in a colab environment</p><pre>trainer = pl.Trainer(<br>  max_epochs=100,<br>  callbacks=callbacks,<br>  logger=mlf_logger,<br>  accelerator=&#39;gpu&#39; if torch.cuda.is_available() else &#39;cpu&#39;,<br>  devices=1, # number of gpus<br>  precision=&quot;16-mixed&quot;,<br>  gradient_clip_val=0.5,<br>  accumulate_grad_batches=4,<br>  log_every_n_steps=50,<br>  num_sanity_val_steps=0<br>  # strategy=&quot;ddp&quot;<br>)<br>logger.info(&quot;🚂 Starting model training&quot;)<br>    trainer.fit(model, data_module)<br>    logger.info(&quot;🔮 Making predictions on test set&quot;)<br>    predictions = trainer.predict(model, data_module.test_dataloader())<br>    predictions = torch.cat([pred for batch in predictions for pred in batch]).cpu().numpy()<br>    sample_submission = pd.read_csv(&quot;kaggle/input/leash-BELKA/sample_submission.csv&quot;)<br>    sample_submission[&#39;binds&#39;] = predictions<br>    sample_submission.to_csv(&#39;submission.csv&#39;, index=False)<br>    logger.info(&quot;💾 Submission saved to submission.csv&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h3>Conclusion and Future Work</h3><p><strong>This approach to the BELKA challenge combines advanced data processing techniques with a sophisticated model architecture designed to capture the complexities of molecular binding. By leveraging graph neural networks, we can directly learn from molecular structures, while the Fourier KAN layers allow for capturing complex, potentially periodic relationships in the data.</strong></p><p>Moving forward, there are several avenues for improvement:</p><ol><li>Exploring more advanced graph neural network architectures, such as Graph Attention Networks (GAT) or Graph Isomorphism Networks (GIN).</li><li>Incorporating additional molecular descriptors or using pre-trained molecular embeddings.</li><li>Experimenting with different attention mechanisms or pooling strategies for the global molecular features.</li><li>Investigating multi-task learning approaches to leverage the information from all three protein targets simultaneously.</li></ol><p>The BELKA challenge represents a significant step forward in applying machine learning to drug discovery. Success in this competition could lead to more efficient drug discovery processes, potentially accelerating the development of new treatments for a wide range of diseases.</p><p>I hope you liked and learned from this implementation, my code can be found <a href="https://github.com/slapglif/NeurIPS2024/blob/main/pipeline.py">here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=593aefc7a310" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tackling the ISIC 2024 Skin Cancer Detection Challenge]]></title>
            <link>https://medium.com/@icarusabiding/tackling-the-isic-2024-skin-cancer-detection-challenge-e0b78000b6a7?source=rss-3f86cfff2eb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0b78000b6a7</guid>
            <category><![CDATA[image-classification]]></category>
            <category><![CDATA[kaggle-competition]]></category>
            <category><![CDATA[machin-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[skin-cancer-treatment]]></category>
            <dc:creator><![CDATA[Michael Brown]]></dc:creator>
            <pubDate>Sun, 30 Jun 2024 06:41:24 GMT</pubDate>
            <atom:updated>2024-06-30T06:44:09.596Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*TskqnB9IWfOCnTQpvSl-Ow.png" /></figure><p>As a machine learning enthusiast and one-man-devshop, I recently decided to dive into the world of medical image analysis by participating in the ISIC 2024 — Skin Cancer Detection with 3D-TBP competition on Kaggle. Little did I know that this journey would take me on a roller coaster ride of frustration, learning, and ultimately, a deeper understanding of the intricacies of deep learning model training.</p><h3>The Challenge</h3><p>The competition’s goal seemed straightforward enough: develop a model to detect skin cancer from images. However, as I soon discovered, the path to creating a reliable and accurate model was anything but straightforward.</p><p>I started with what I thought was a solid foundation — a pre-trained EfficientNet model combined with custom a Kolmogorov-Arnold Network (KAN) layers. In theory, this architecture should have been able to capture both the low-level features from the images and the complex relationships between them. But as the saying goes, “In theory, there is no difference between theory and practice. In practice, there is.”</p><h3>The Initial Struggles</h3><p>My first attempts at training the model were, to put it mildly, disheartening. The loss would plummet to near-zero values within a few epochs, while the accuracy skyrocketed to a perfect 1.0. On the surface, this might seem like a cause for celebration, but any experienced machine learning practitioner knows that when something seems too good to be true, it probably is.</p><p>The model was overfitting — dramatically. It was essentially memorizing the training data rather than learning generalizable features. This meant that while it performed beautifully on the training set, it would likely fail miserably when faced with new, unseen data.</p><p>To make matters worse, the other metrics I was tracking — F1 score and AUROC — remained stubbornly at zero. It was as if the model was learning something, but not what I wanted it to learn.</p><h3>The Quest for Stability</h3><p>Determined to overcome these challenges, I embarked on a quest for training stability. This journey took me through a labyrinth of hyperparameter tuning, architecture modifications, and countless hours of debugging.</p><p>First, I tackled the learning rate. I implemented a learning rate scheduler, specifically the OneCycleLR, which starts with a low learning rate, gradually increases it, and then decreases it again. This approach often leads to more stable training dynamics.</p><p>Next, I turned my attention to regularization. I increased the dropout rate, bumped up the weight decay, and even added batch normalization layers between my KAN layers. Each of these techniques helps prevent overfitting in its own way, and I hoped that their combined effect would be greater than the sum of their parts.</p><p>I also realized that my dataset might be imbalanced — a common issue in medical imaging where the number of positive cases (patients with cancer) is often much smaller than the number of negative cases. To address this, I implemented class weighting in my loss function, giving more importance to the underrepresented class.</p><h3>The Breakthrough</h3><p>After what felt like countless iterations and a late night staring at training logs, I finally started to see improvements. The loss was decreasing more gradually, and the accuracy, while still high, was no longer suspiciously perfect. More importantly, the F1 score and AUROC began to budge from zero, indicating that the model was actually learning to distinguish between classes.</p><p>But the journey wasn’t over. I realized that to truly understand how my model was performing, I needed more frequent validation checks. I modified my training loop to evaluate on the validation set more often, giving me a clearer picture of how well the model was generalizing as training progressed.</p><h3>Lessons Learned</h3><p>This experience taught me several valuable lessons:</p><ol><li><strong>Patience is key</strong>: Deep learning models, especially for complex tasks like medical image analysis, often require extensive tuning and experimentation.</li><li><strong>Trust your metrics: </strong>When something seems too good to be true (like perfect accuracy after a few epochs), it probably is. Use multiple metrics to get a comprehensive view of your model’s performance.</li><li><strong>Understand your data</strong>: Knowing the characteristics of your dataset, such as class imbalance, is crucial for designing an effective training strategy.</li><li><strong>Regularization is your friend</strong>: In the fight against overfitting, you can never have too many weapons. Dropout, weight decay, and batch normalization can work wonders.</li><li>Monitor closely: Frequent validation checks and comprehensive logging can provide invaluable insights into your model’s behavior during training.</li></ol><h3>Looking Forward</h3><p>While I’ve made significant progress in stabilizing my model’s training, I know that the journey is far from over. The field of medical image analysis is complex and ever-evolving, and there’s always more to learn.</p><p>As I continue to refine my model for the ISIC 2024 competition, I’m excited to explore more advanced techniques, perhaps delving into ensemble methods or experimenting with other architectures. But more than that, I’m grateful for the deep learning lessons this experience has taught me — lessons that I know will serve me well in future projects and competitions.</p><p>To all my fellow data scientists and ML enthusiasts out there tackling similar challenges: keep pushing, keep learning, and remember that every training instability is just an opportunity for growth in disguise.</p><p>Happy hacking!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0b78000b6a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the Dice Roll: Mutual Information, Uncertain Journeys, and the Power of Knowing What You…]]></title>
            <link>https://medium.com/@icarusabiding/beyond-the-dice-roll-mutual-information-uncertain-journeys-and-the-power-of-knowing-what-you-6852e42403a3?source=rss-3f86cfff2eb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6852e42403a3</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[stastics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[quantum-algorithms]]></category>
            <dc:creator><![CDATA[Michael Brown]]></dc:creator>
            <pubDate>Sat, 29 Jun 2024 18:45:04 GMT</pubDate>
            <atom:updated>2024-07-02T12:12:19.456Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I_s9skH_Dt3yTW8I.jpg" /></figure><h3>Beyond the Dice Roll: Mutual Information, Uncertain Journeys, and the Power of Knowing What You Don’t Know</h3><p><strong>Embracing uncertainty as a valuable source of information, we can build more robust and reliable neural networks.</strong> Imagine models that can not only make predictions but also quantify their confidence in those predictions, flagging areas where more data or different approaches are needed.</p><p>The world of data science is full of hidden patterns and connections, whispering secrets that only the most astute observers can decipher. Just like the seasoned gamblers in ancient Pompeii, relying on their keen eyes to exploit the bias in loaded dice, data scientists today are exploring ways to make sense of these whispers, uncovering the intricate dance between information and uncertainty.</p><h3>Mutual Information: Unmasking the Whispers</h3><p>Picture two seasoned gamblers huddled over a worn game board in ancient Pompeii, their eyes locked on the roll of the dice. These weren’t just any dice; they were loaded, each roll carrying secrets, hidden patterns that only a keen observer could decipher. This, in essence, is the world of mutual information — the hidden connections, the shared whispers between two variables.</p><p>Just as a gambler might exploit the bias in a loaded die to predict the next roll, mutual information quantifies how much knowing one variable tells us about another. It’s a measure of the shared information content, a way to quantify how much knowing one variable reduces our uncertainty about the other. Imagine being able to anticipate the weather based on the direction of the wind or predict stock market trends based on social media sentiment. Mutual information allows us to quantify these relationships, providing insights into the underlying connections that drive complex systems.</p><p>In the realm of neural networks, mutual information takes on an even deeper significance. It becomes the key to understanding how features intertwine, revealing deeper relationships that drive our models’ decisions. By quantifying the mutual information between different features, we can gain insights into which features are truly relevant, which ones are redundant, and how they interact to shape the model’s predictions. This understanding paves the way for more informed feature selection, model design, and, ultimately, more robust and interpretable AI systems.</p><h3>The Travelling Salesman and the Quest for Optimal Paths</h3><p>Now, consider a travelling salesman, tasked with navigating a labyrinth of cities, seeking the shortest route that touches each destination just once. This infamous problem, the Travelling Salesman Problem (TSP), embodies the challenge of optimization under uncertainty. The salesman doesn’t know the optimal route beforehand; he must explore, weigh options, and make decisions based on incomplete information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/0*X8zIZB6BGMRFdjui" /><figcaption>Cartoon by Sarah Meadows</figcaption></figure><p>This problem has captivated mathematicians and computer scientists for decades, showcasing the difficulty of finding optimal solutions in complex, high-dimensional spaces. The TSP is not just a theoretical curiosity; it has applications in logistics, transportation, DNA sequencing, and even chip design, highlighting its relevance to a wide range of real-world problems.</p><p>But the TSP also serves as a powerful analogy for the challenges faced by neural networks. Just like the wandering salesman, our models must navigate a complex landscape of data, making predictions in the face of the unknown. They must wrestle with noisy data, incomplete features, and the inherent randomness of stochastic processes, constantly making decisions based on incomplete information.</p><h3>Embracing the Unknown: Uncertainty as a Guiding Light</h3><p>Uncertainty reigns in the world of neural networks. Our models grapple with noisy data, incomplete features, and the inherent randomness of stochastic processes. Just like the wandering salesman, they must navigate a complex landscape, making predictions in the face of the unknown.</p><p>However, what if we could equip our networks with a sense of their own limitations? What if they could recognize when they’re treading on shaky ground, signaling their uncertainty instead of blindly forging ahead? This is where the power of quantifying uncertainty shines.</p><p>By embracing uncertainty as a valuable source of information, we can build more robust and reliable neural networks. Imagine models that can not only make predictions but also quantify their confidence in those predictions, flagging areas where more data or different approaches are needed. This would be a game-changer in applications where reliability is paramount, from medical diagnosis to autonomous driving.</p><h3>A Glimpse into the Future: Uncertainty-Aware Neural Networks</h3><p>The fusion of mutual information, optimization strategies inspired by the TSP, and robust uncertainty estimation techniques holds the key to unlocking a new era of AI. Imagine neural networks that can not only learn from data but also learn from their own limitations, constantly refining their understanding of the world and becoming more reliable and trustworthy partners in decision-making.</p><p><strong>My upcoming release of open-source software seeks to do just that.</strong> By incorporating advanced techniques like the Travelling Salesman Kernel and innovative uncertainty estimation methods, it aims to empower neural networks to navigate the world with a nuanced understanding of the unknown, leading to more informed decisions and a deeper appreciation for the intricate dance between information and uncertainty. Stay tuned to discover how we’re bringing this vision to life.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6852e42403a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My journey into planet-scale AI (and the technology I built along the way)]]></title>
            <link>https://medium.com/@icarusabiding/my-journey-into-planet-scale-ai-d6b5a64256c0?source=rss-3f86cfff2eb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6b5a64256c0</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Michael Brown]]></dc:creator>
            <pubDate>Sat, 29 Jun 2024 02:34:43 GMT</pubDate>
            <atom:updated>2024-06-29T05:16:19.553Z</atom:updated>
            <content:encoded><![CDATA[<p>Today I’m releasing one (of many) of the tools I use personally to pretrain and benchmark my models for numerical stability, gradient propagation and task-driven efficacy. The tool is <a href="https://github.com/slapglif/LLMBootstrapper"><strong>LLMBootstrapper</strong></a></p><blockquote>This is part 1 of a few posts I intend to make about some open-source tech I’m releasing this week, so stay tuned for more :)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*ibPw60oOYcfAymgpsEuePw.png" /></figure><p>This is just a small suite of tools that I ripped out of my various codebases — It started with me wanting to rapidly prototype some models and measure their efficacy, they were novel and so no one even knew if the arch would work. Unit tests, pass</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uDN46WBm-cd4sTf9VMv2_g.png" /></figure><p>We can see gradients flowing and the network is even able to train on a subset of random data and prove in this small scenario I made, that the MHA contributes meaningfully to the system</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*Rq1sJbzBKXkg6obwti8yMQ.png" /></figure><p>(cosim before and after the output) and sure as hell everything&#39;s great. However, I ran it against the WikiText dataset for a whole day…. and it was still generating garbage output!</p><p>You can imagine my frustration and pain… look I had to stream the dataset for VARIOUS reasons and in addition, I have just one 4070TI. One iteration through the whole pipeline is like half a day just to see it fail at some point or output garbage? gah!</p><p>Thats when I decided to finally put this thing together. In the end, successfully training a model on input text for it to capture the correct features and predict meaningful sequences has almost nothing to do with the model architecture. No — hear me out, its true— even a 2-layer MLP can predict token albeit maybe with not the best perplexity score. The real sauce is in the data pipeline, training objective, preprocessing/augmenting and task modeling, and meta-management of the models&#39; weights.</p><p>Of course, this is not something you find out easily — sure you can ask an LLM, and they will give you the shallow version of this — maybe a wordier way of telling you maybe even less than what I just did. Actually, there is a major gap in the publicly available knowledge in LLM modeling, and successfully scaling up a model past maybe 100m params… especially given the compute requirements.</p><p>First of all, LLMBootstrapper comes in at a very key place here. If you are like me, you probably downloaded one of these public LLM or deep learning model repos and tried to actually run it — if you took it seriously like me and did it on modern hardware with a modern model — you probably failed. You probably couldn&#39;t get triton or deepspeed or accelerate to install, flash attention to compile or the custom fused kernels to work… that&#39;s because the ecosystem is actually not interested at all in you using their code and succeeding — if they were, they would release the full training pipelines they really used with their cloud scripts and real data and training modeling… but they don&#39;t, they always strip it from the repos and leave you with some silly training loop they would have you believe works — and the good ones will tell you it&#39;s just for demonstration purposes etc…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*Us4h2WTB-qWyrArv.gif" /></figure><p>So why does no one release their training pipelines? Why does no one provide insight to the best practices and pitfalls? Why is everything you find on the internet about flash attention (requires 90+ gb of vram to use) or TensorRT which I have never, ever gotten to successfully compile on any machine since it was released (I’m a Linux guy too…)</p><p>It’s simple… all the money and attention are in the release of a model architecture they know means nothing (or well, not as much as you think) to the success of the language model. Language models success because of their Direct Preference Optimizations, their Objective Modeling, their rigorous Reinforcement Learning with Human Feedback loops, and the core corpuses of synthetic data generated in unlimited quantities to align them to some trait or another … You see though, each of those things is a data task, and requires a repeatable and measurable way to iteratively alter and experiment with the models…and if they released that, you could do what they do in no time at all.</p><p>LLMBootstrapper is a tool I made for you, to save you the hundreds of hours on an arduous journey into scaling a language model from an idea that you throw random tensors at, to a pre-trained model which you can benchmark empirically. My project does not compare the model against public benchmarks or QA eval — this is not a downstream eval harness. LLMBoostrapper seeks to give you quick, objective statistics which can be used in quantifying each little change to your network as you attempt to learn and control the objectives in your language system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*GtP8pxmcAIvyAxP7.gif" /></figure><p>Among the ways we measure an LLM performance is <a href="https://en.wikipedia.org/wiki/Measurement_uncertainty">Gaussian Uncertainty</a>. I will make another post about soon to announce the release of my novel foundation model which models Gaussian Uncertainty as the primary driver of its language decoding… but that’s another story.</p><p>My system is opinionate, but that’s because my opinion works repeatably. The code however was put together hastily when I got the idea, and I will be updating it more as time goes on — but as of today it supports any pytorch model which has a language head (liner layer modeling language ) and which is supported by transformers lib (so pretty much any pytorch model) and as time goes in I will expand the ways we can load and work with models in the system. Today, I use the same code I released to train and benchmark the efficacy of my novel Neural Networks and rapidly throw datasets at various models and layers in various configurations — my development pace of NN tech has gone through the roof, because I don&#39;t have to waste so much damn time trying to figure out the 300 little steps one needs to take to see their first epoch tick.</p><p>you ready?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aSTbsunmebJtir1Txo3Bvw.png" /><figcaption>Yeah,,,, wow.</figcaption></figure><p>My only desire, is it that as you enter the world of the uninitiated AI engineer (or if you struggle as a pro like me) and you begin to see the whole scope of the monumental task ahead of you — my one true hope in this world is that google lands you on this blog post or my repo and in doing so, saves you days, months, or weeks of your life you can never get back from chasing wild geese down triton holes and flash attention pits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*TujZ_GqwiIYUtyF6-2WA6w.png" /></figure><p>Note: the project can be located <a href="https://github.com/slapglif/LLMBootstrapper">here</a> — but be warned, its WIP and may be subject to errors or version problems, w/e. Hit me on GitHub if there&#39;s an issue and Ill fix it asap.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6b5a64256c0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Data Seeder for your database in Python with SQLAlchemy]]></title>
            <link>https://medium.com/analytics-vidhya/building-a-data-seeder-for-your-database-in-python-with-sqlalchemy-2e0a20256c18?source=rss-3f86cfff2eb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e0a20256c18</guid>
            <category><![CDATA[abstraction]]></category>
            <category><![CDATA[dataloader]]></category>
            <category><![CDATA[sqlalchemy]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Michael Brown]]></dc:creator>
            <pubDate>Tue, 28 Mar 2023 05:26:01 GMT</pubDate>
            <atom:updated>2023-03-28T05:55:58.311Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*g6VYxLRA_wyVWeQC.jpg" /></figure><h3>Building a Data Seeder/Loader for your database…in Python</h3><p>Some time ago, I was challenged with making a data seeder for a social media app I was working on. What I came up with worked, it was modular and simple to use… but it was a complex piece of code with many logical complications and — like all such modules are, it was quickly made obsolete by changes to the data contract and the maintenance became a serious problem. I researched the subject a lot but didn&#39;t find anything satisfactory, so I came up with a class-reflection based approach which will deep dive into my classes and rely on the pure magic that SQLAlchemy brings.</p><h3><strong>Getting Started</strong></h3><p>First and foremost, I want to say that this article is truly a love letter to SQLAlchemy. I have been using it for many many years and the only ORM that comes anywhere near close to it is Entity Framework in C#. For those of you purists who say ‘just write the SQL by hand!’ — folly I say, there&#39;s nothing, including performance tuning, you cant do with SQLALchemy if you know what your doing.</p><p>That said, we have a few steps that need to be followed in order for this to actually work and be maintainable (or need none at all)</p><p>First, what is reflection?</p><blockquote><a href="https://prutor.ai/reflection-in-python/#:~:text=Reflection%20refers%20to%20the%20ability,represents%20the%20type%20of%20obj.">Reflection refers to the ability for code to be able to examine attributes about objects that might be passed as parameters to a function. For example, if we write type(obj) then Python will return an object which represents the type of obj.</a></blockquote><p>By using reflection, we can create an *abstract* and *programmatic* way to discover our data types, foreign key relationships, and constraints.</p><p>lets start by defining our data models:</p><pre>import contextlib<br>import uuid<br>from datetime import timedelta<br>from typing import List, Iterable<br>from sqlalchemy import create_engine, MetaData, DateTime, String, Integer, Boolean, Table, Column, Interval<br>import pytz<br>from random import randint, choice<br>from faker import Faker<br>from types import SimpleNamespace<br>from sqlalchemy.dialects.postgresql import UUID<br>from sqlalchemy.exc import IntegrityError<br>from sqlalchemy.ext.declarative import declarative_base<br>from sqlalchemy.orm import registry, sessionmaker<br><br># Explicitly import the registry and declarative base for reflection<br>mapper_registry = registry()<br>DeclarativeBase = declarative_base()<br>Base = mapper_registry.generate_base(cls=())<br><br><br>class ModelMixin(Base):<br>     # we build a mixin with a common method we would like to impl<br>     @classmethod<br>    def get_or_create(cls, *_, **kwargs):<br>        context_id = kwargs.get(&quot;id&quot;)<br>        user = cls.session.query(cls).filter_by(id=context_id).first()<br>        if not user:<br>            _object = cls(**kwargs)<br>            cls.session.add(_object)<br>            cls.session.commit()<br>            return _object<br>   <br><br>class User(ModelMixin):<br>    __tablename__ = &quot;user&quot;<br><br>    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4, index=True)<br>    email = Column(String(255))<br>    username = Column(String(255))<br>    password = Column(String(255))<br>    active = Column(Boolean)<br>    createdAt = Column(DateTime, default=lambda: datetime.now(pytz.utc))<br>    updatedAt = Column(DateTime)<br>    accessToken = Column(String(255))<br><br>    agentId = Column(<br>        UUID(as_uuid=True),<br>        ForeignKey(&quot;agent.id&quot;, ondelete=&quot;CASCADE&quot;, link_to_name=True),<br>        index=True,<br>    )<br>    createdByAgent = relationship(<br>        &quot;Agent&quot;,<br>        foreign_keys=&quot;User.agentId&quot;,<br>        backref=backref(&quot;users&quot;, single_parent=True),<br>   )</pre><p>Here we have a User object which has a foreign key relation to an Agent object, the user was created by the agent and back references his parent.</p><p>We also build a model for Payment History, in this case because we want to make sure all foreign keys are being followed.</p><pre>class Agent(ModelMixin):<br>    __tablename__ = &quot;agent&quot;<br>    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4, index=True)<br>    email = Column(String(255))<br>    password = Column(String(255))<br>    username = Column(String(255))<br>    active = Column(Boolean)<br>    createdAt = Column(DateTime)<br>    updatedAt = Column(DateTime)<br>    accessToken = Column(String(255))<br>    quota = Column(Integer, default=0)<br><br><br>class PaymentHistory(ModelMixin):<br>    __tablename__ = &quot;payment_history&quot;<br><br>    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4, index=True)<br>    beforeScore = Column(Integer)<br>    changeScore = Column(Integer)<br>    newScore = Column(Integer)<br>    approval = Column(Boolean)<br>    createdAt = Column(DateTime, default=lambda: datetime.now(pytz.utc))<br>    approvalAt = Column(DateTime)<br>    ownerId = Column(<br>        UUID(as_uuid=True),<br>        ForeignKey(<br>            &quot;user.id&quot;,<br>            ondelete=&quot;CASCADE&quot;,<br>            link_to_name=True,<br>        ),<br>        index=True,<br>    )<br>    owner = relationship(<br>        &quot;User&quot;,<br>        foreign_keys=&quot;PaymentHistory.ownerId&quot;,<br>        backref=backref(&quot;paymentHistory&quot;, single_parent=True),<br>    )</pre><p>One of the challenges we face here is seeding data with a non-auto incrementing id field — this is a <a href="https://www.cockroachlabs.com/blog/what-is-a-uuid/"><em>best practice</em></a> for distributed systems, but can be challenging when writing data loaders for your db. Another pitfall is foreign key relationships — they are useful, for data integrity purposes but man can they really foul up your intentions to test your tech stack at scale.</p><h3>The First Step</h3><p>Now, if we want to generate data here, there&#39;s some logical complexities we need to follow to ensure we are able to seed this data:</p><ul><li>We need to access the non instantiated User and Agent classes AFTER they have entered the SQLA mapper registry, To do this We need to import the module and return the <strong><em>table_name </em></strong>from that module and convert it to pascal case — the class name.</li></ul><p>This example assumes you have a Config class (maybe in a settings.py) file which reads config values with dotenv.</p><pre>import os<br>from dotenv import load_dotenv<br><br>load_dotenv()<br><br>class Config:<br>    postgres_connection: str = os.getenv(&quot;POSTGRES_CONNECTION&quot;, &quot;&quot;)</pre><p>Now our config and setup is done, how about we start by setting up our seeder/loader with a class:</p><pre>class DataSeeder:<br>      &quot;&quot;&quot;<br>      Instanciated with: <br>      engine = create_engine(<br>              f&quot;postgresql+psycopg2://{Config.postgres_connection}&quot;)<br>      DataSeeder(engine=engine, ...)<br>      &quot;&quot;&quot;<br><br>    def __init__(<br>        self, include_models: list, <br>        engine: create_engine, <br>        number_of_records: int = 10<br>      ):<br>        self.number_of_records = number_of_records<br>        Session = sessionmaker(bind=engine)<br>        self.session = Session()<br>        self.fake = Faker()<br>        self.metadata = MetaData(bind=engine)<br>        self.metadata.reflect()<br>        self.mapped = {}</pre><p>So we are working inside of a class context which has the advantage of sharing some properties — we will start by defining some helper functions</p><h3><strong>Inspecting and Reflecting the Models</strong></h3><p>These three methods largely make up what I consider to be an extraordinarily hacky abstraction — but sometimes the most beautifully terrible ones work the best. In this case, we are effectively doing a linear search algorithm to explore a given array of items to its maximum depth and comparing each item against the recursive access of another array. If that sounds convoluted, that&#39;s because it is — but we have to ensure that all tables, all columns, and all rows are caught.</p><pre>#define a list of models to reflect<br>included_models = [&#39;Agent&#39;, &#39;User&#39;]<br><br>@staticmethod<br>def snake_to_pascal_case(name: str) -&gt; str:<br>        return &quot;&quot;.join(word.capitalize() for word in name.split(&quot;_&quot;))<br><br>@staticmethod<br>def get_model_class(table_name: str) -&gt; Base:<br>    for model in include:<br>        # we ignore import errors here so we can iterate to find the model<br>        with contextlib.suppress(ImportError, AttributeError):<br>            module = __import__(<br>              # note, you should point it to your own folder structure,<br>              # mine is a FastAPI app (app.api.x)   <br>              # protip, make sure you use __init__.py files and pythonpath<br>                f&quot;app.api.{model}&quot;,<br>                fromlist=[table_name]<br>            )<br>            <br>            # convert the table_name to TableName<br>            class_name = snake_to_pascal_case(table_name)<br><br>            # Return the actual class its self<br>            return getattr(module, class_name)<br><br>def get_model_metadata(self) -&gt; Iterator:<br><br>      # import metadata for all models to build the registry<br>        for model in include:<br>            with contetlib.suppress(ImportError):<br>                # ya i know, were using exec here... deal with it<br>                exec(f&quot;{model}.models import ModelMixin as Base&quot;)<br><br>       # loop through models in registry<br>        for table in sorted(<br>            Base.metadata.sorted_tables, <br>                key=lambda t: t.name, <br>                # sort them in Alphabetical order<br>                reverse=True<br>        ): # ensure the table we are targeting is valid<br>          if table.name not in self.metadata.tables:<br>              continue<br>        #  build a generator for the data<br>        if model := self.get_model_class(table.name):<br>            model.name = model.__name__<br>            yield model, table</pre><h3><strong>Preparing Data Types</strong></h3><p>Now that we have a mapping of our class names and MetaData imported, we can proceed with handling the data types for each column — this is where some of the magic happens, because it lets us dot-notate access the types… strictly speaking we could have done this a lot easier, maybe with a Pydantic model — but simple name spaces are good enough… for now.</p><pre>def get_data_type_mapper(self) -&gt; SimpleNamespace:<br>  # build a namespace for easy type access<br>  return SimpleNamespace(<br>      type_maps=[<br>          SimpleNamespace(<br>              type=DateTime,<br>              fake_type=self.fake.date_time_between(<br>                  start_date=&quot;-30y&quot;, end_date=&quot;now&quot;<br>              ),<br>          ),<br>          SimpleNamespace(type=Boolean, fake_type=self.fake.boolean()),<br>          SimpleNamespace(type=Integer, fake_type=self.fake.random_int()),<br>          SimpleNamespace(<br>              type=Float, fake_type=self.fake.pyfloat(positive=True)<br>          ),<br>          SimpleNamespace(<br>              type=Interval, fake_type=timedelta(seconds=randint(0, 86400))<br>          ),<br>          SimpleNamespace(type=UUID, fake_type=str(uuid.uuid4())),<br>          SimpleNamespace(<br>              type=String,<br>              fake_type=f&quot;{&#39; &#39;.join([self.fake.word() for _ in range(8)])}&quot;,<br>          ),<br>      ]<br>  )</pre><p>Now we need to ensure we follow all columns, including foreign key relationship and seed the fake data.</p><h3>Generating the Row data</h3><p>We generate fake data for a given table by inspecting each column and mapping it to a corresponding data type. then generates fake data based on the mapped data type using the `Faker` library.</p><ul><li>The following code is effectively an<em> </em><a href="https://lenchen.medium.com/leetcode-133-clone-graph-fdbfe902ed4"><em>undirected graph clone</em></a> algorithm , but its perhaps the most abstract implementation of reflection, cyclic recursion and data input validation that I’ve ever done… I’m a little proud of this one, though don&#39;t ask me to do it at a google interview.</li></ul><pre> def generate_fake_row_data(self, table: MetaData) -&gt; dict:<br>    <br>    # loop through all table columns<br>    row_data = {}<br>    for column in table.columns:<br><br>    # Check data types for all columns in a table and generate fake data<br>    for data_type in data_type_mapper.type_maps:<br>        if isinstance(column.type, data_type.type):<br>            row_data[column.name] = data_type.fake_type<br>    <br>    # ensure pk is unique<br>    if column.primary_key and isinstance(column.type, UUID):<br>        row_data[column.name] = str(uuid.uuid4())<br>    <br>    # loop through table relationships<br>    for fk in column.foreign_keys:<br>        fk_table = fk.column.table<br>        fk_column = fk.column<br>    <br>        # link fk to existing record, or make one first then build relationship<br>        if fk_records := self.get_table_data(fk_table.name, fk_column.name):<br>            row_data[column.name] = choice(fk_records)[0]<br>        else:<br>             # cycle back to the first node<br>            new_data = self.generate_fake_row_data(fk_table)<br>            # if the fk does not exist, make it<br>            self.save_model(self.get_model_class(fk_table.name), new_data)</pre><p>If a column has a foreign key constraint, we must recursively generate a fake row for the referenced table until it finds a primary key that does not violate any constraints. This is done by checking if matching rows exist in the referenced table and selecting one of them randomly; otherwise, generate a new row for the referenced table and use its primary key as the foreign key value. This method ensures the integrity of the generated data by handling unique constraints when saving the data to the database.</p><ul><li>Finally we must save the results to the database after checking for integrity errors and rolling back if they exist — this is not strictly required, but its good practice to ensure we don&#39;t fail on an unforeseen relationship or unique constraint.</li></ul><pre> @staticmethod<br>def save_model(model, row_data):<br>    try:<br>        model.get_or_create(model, **row_data)<br>        # commit changes to the database<br>        model.session.commit()<br>    except IntegrityError:<br>         # handle unique constraint violation by rolling back the transaction<br>        model.session.rollback()<br><br>def generate(self):<br>    # get the MetaData generator and iterate<br>    for model, table in list(self.get_model_metadata()):<br>        # where num_of_records is our upper limit<br>        for _ in range(self.number_of_records):<br>            row_data = self.generate_fake_row_data(table)<br>            self.save_model(model, row_data)<br>        print(model, f&quot;{self.number_of_records} records added to db&quot;)</pre><p>That&#39;s effectively it, we can now add the records to our database.</p><h3>Running the Data Seeder</h3><pre>                # note, oother than UUID, we use no postgres specific<br>                # fields, im sure you can swap this out for anything<br>engine = create_engine(f&quot;postgresql+psycopg2://{Config.postgres_connection}&quot;)<br>included_models = [&#39;Agent&#39;, &#39;User&#39;, &#39;PaymentHistory&#39;]<br>seeder = DataSeeder(<br>    included_models,<br>    number_of_records=1000,<br>    engine=enngine<br>)<br>seeder.generate()<br>&gt;&gt; &lt;class &#39;app.api.agent.models.Agent&#39;&gt; 1000 records added to db<br>&gt;&gt; &lt;class &#39;app.api.user.models.User&#39;&gt; 1000 records added to db<br>&gt;&gt; &lt;class &#39;app.api.history.models.PaymentHistory&#39;&gt; 1000 records added to db</pre><p>Now what is truly fun, is that these three classes had relationships to models (which we did not show here) -&gt; lets see the result with some queries to visualize the data:</p><ul><li><strong>Get all the related id records created in a single cycle</strong></li></ul><pre>select a.id, u.id, b.balance, ph.id from &quot;user&quot; u<br>    join payment_history ph<br>        on u.id = ph.&quot;ownerId&quot;<br>    join balance b on u.id = b.&quot;ownerId&quot;<br>    join agent a on u.&quot;agentId&quot; = a.id</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_9nn8qPZNes-FpyvD67_MQ.png" /></figure><ul><li><strong>Get the user and his balance which was created by the relationship</strong></li></ul><pre>select u.*, b.balance from &quot;user&quot; u join balance b on u.id = b.&quot;ownerId&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mMEvQYk_nfP1V-tnutEsoA.png" /></figure><h3>Conclusion</h3><p>The proposed code, my implementation of which you can find <a href="https://github.com/slapglif/data_seeder/blob/main/data_seeder.py">here</a> — is just the start of what we could do. some improvements we might make:</p><ul><li>Map data types with Pydantic models, and support advanced data types for various languages like MySQL and types like JSON.</li><li>Accurately profile the results of creation — the print statement used ignores chained foreign key creation.</li><li>Reduce the cyclomatic complexity by abstracting the recursions into a single set and leveraging tools like py_linq or SQLAlchemy_mixins.</li><li>Create a <a href="https://docs.sqlalchemy.org/en/14/core/metadata.html#sqlalchemy.schema.Table.params.autoload_with">reflected session</a> to use live inspections on the data model, and potentially save a lot of trouble here.</li></ul><p>Theres many things we coujld do to improve, but still this solution worked for me, and im happily testing my backend app on millions of data points today — and hopefully this helps you shortcut there also. If you do end up using this, please shoot me a star on github, and seriously if you want to submit a PR I will entertain any solution which improves upon mine.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e0a20256c18" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/building-a-data-seeder-for-your-database-in-python-with-sqlalchemy-2e0a20256c18">Building a Data Seeder for your database in Python with SQLAlchemy</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a Cryptocurrency scanner with Python using a Data Science driven approach]]></title>
            <link>https://medium.com/@icarusabiding/creating-a-cryptocurrency-scanner-with-python-using-a-data-science-driven-approach-c4555048973b?source=rss-3f86cfff2eb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4555048973b</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automated-trading]]></category>
            <category><![CDATA[technical-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Michael Brown]]></dc:creator>
            <pubDate>Tue, 10 Mar 2020 01:58:13 GMT</pubDate>
            <atom:updated>2020-03-12T14:35:22.112Z</atom:updated>
            <content:encoded><![CDATA[<p>Sometime ago, I created a Pump &amp; Dump scanner using the exchange Binance. On my journey, I learned some amazing things about working with the Binance API and transforming the data into a functional trading application. In today’s post I will walk you through the first part of this application, which collects and scans all symbols for specific logic.</p><p>Keep in mind that while you can apply this to a functional trading situation, this post should be considered part one of creating an automated trading system because while we will find market conditions, we will <strong>*not*</strong> be back-testing the results. In my next post we will get into application of Technical Analysis beyond what is used by our basic setup, and in the third look forward to the implementation of a back-testing engine.</p><h3>Dependencies</h3><p>The tech were working with today is:</p><p><a href="https://pypi.org/project/Flask/">Flask</a>, <a href="https://pypi.org/project/SQLAlchemy/">SQLAlchemy</a>, <a href="https://pypi.org/project/matplotlib/">MatPlotLib</a>, <a href="https://pypi.org/project/numpy/">Numpy</a>, <a href="https://pandas.pydata.org/">Pandas,</a> <a href="https://github.com/matplotlib/mpl-finance">mpl_finance</a> and <a href="https://github.com/mrjbq7/ta-lib">TALib</a></p><h3>Getting Started</h3><p>This is a Data Science driven approach, so for those of you who’s eyes gloss over when we use words like ‘DataFrame’ and ‘Exponential Moving Average’ — get ready to poor a glass of Whiskey because this will be a douzey. We will start by assuming you have installed the above dependencies and jump right into the application.</p><p>Let’s start by importing everything we will be using:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc8fb77d53095c4c15e4e9efb277ca57/href">https://medium.com/media/dc8fb77d53095c4c15e4e9efb277ca57/href</a></iframe><p>Okay yea, that’s quite a lot. If you’re an experienced python developer who may have dealt with flask before, you probably stopped at line 7 and asked “Yo dawg where the models at??” Keep your pants on, it’s coming.</p><p>Next let’s go ahead and set up environment and initialize our variables</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5ee0f4b6d63f8b47aa9c1e63993b80af/href">https://medium.com/media/5ee0f4b6d63f8b47aa9c1e63993b80af/href</a></iframe><p>Really simple stuff here. We are initializing the flask app, the DB and setting up our API’s for further use. Next we set a list of desired currencies to scan with line 9. We have a small loop to slice the list down to just our expected pairs; By default it’s set to all pairs, but if you define anything in line 9 it will scrape just those symbols. Finally we are setting up our lists for data structure storage within pandas and numpy.</p><p>Example output:</p><pre>[&#39;ETHBTC&#39;, &#39;LTCBTC&#39;, &#39;BNBBTC&#39;, &#39;NEOBTC&#39;, &#39;BCCBTC&#39;, &#39;GASBTC&#39;, &#39;BTCUSDT&#39;, &#39;HSRBTC&#39;, &#39;MCOBTC&#39;, &#39;WTCBTC&#39;, &#39;LRCBTC&#39;, &#39;QTUMBTC&#39;, &#39;YOYOBTC&#39;, &#39;OMGBTC&#39;, &#39;ZRXBTC&#39;, &#39;STRATBTC&#39;, &#39;SNGLSBTC&#39;, &#39;BQXBTC&#39;, &#39;KNCBTC&#39;, &#39;FUNBTC&#39;, &#39;SNMBTC&#39;, &#39;IOTABTC&#39;, &#39;LINKBTC&#39;, &#39;XVGBTC&#39;, &#39;SALTBTC&#39;, &#39;MDABTC&#39;, &#39;MTLBTC&#39;, &#39;SUBBTC&#39;, &#39;EOSBTC&#39;, &#39;SNTBTC&#39;, &#39;ETCBTC&#39;, &#39;MTHBTC&#39;, &#39;ENGBTC&#39;, &#39;DNTBTC&#39;, &#39;ZECBTC&#39;, &#39;BNTBTC&#39;, &#39;ASTBTC&#39;, &#39;DASHBTC&#39;, &#39;OAXBTC&#39;, &#39;ICNBTC&#39;, &#39;BTGBTC&#39;, &#39;EVXBTC&#39;, &#39;REQBTC&#39;, &#39;VIBBTC&#39;, &#39;TRXBTC&#39;, &#39;POWRBTC&#39;, &#39;ARKBTC&#39;, &#39;XRPBTC&#39;, &#39;MODBTC&#39;, &#39;ENJBTC&#39;, &#39;STORJBTC&#39;, &#39;VENBTC&#39;, &#39;KMDBTC&#39;, &#39;RCNBTC&#39;]</pre><p>That is obviously a trimmed list. Note the values in the above gist:</p><pre>short_window = 9<br>long_window = 26<br>RSI_window = 14 </pre><pre>timeframe = &quot;15m&quot;</pre><p>Here we are setting up the basis for our scanner alert. A lot of different things can be done with this system, which we will get into in the next post — for now suffice it to say that we are using a 9/26 EMA cross strategy which is known in the Ichimoku trading system as a T-K Cross. This method is a time honored system for determining entry and exit on trades. We are also setting the chart timeframe to 15m which may or may not be ideal for your own alerts. We can really add any logic we want to determine when a trade alert goes off but for now let’s keep it simple, and not get into in-depth technical analysis. A single confirmation indicator is used called the RSI or relative strength index. If you are not an active trader or just starting out, you can read about it <a href="https://www.investopedia.com/terms/r/rsi.asp">here</a>.</p><h3>Building a Database for our trade signals</h3><p>Okay now one of the most important aspects of a system like this is setting up a DB to easily track the signals, it makes things simple for reference and use outside of this application. Lets see the code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf4bf12ef3d20b89766620cdcb072e92/href">https://medium.com/media/bf4bf12ef3d20b89766620cdcb072e92/href</a></iframe><p>We are using sqlite flat file for right now, you can change this to any engine you want if you intend to host a large scale signals DB. What exactly are we doing here you ask? Really nothing special. We built a basic table and an entry creation method with the function get_or_create() as well as creating a db search function named find Simple right? I thought so, but I’m betting I get all kinds of questions about it. #inb4stringsizequestions</p><h3>Signal Function</h3><p>Okay we are set to start scanning the market conditions, but first we need to define the main function and get our data formatted correctly.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd83da53fddcc8870abf4a3dfa53fb1c/href">https://medium.com/media/fd83da53fddcc8870abf4a3dfa53fb1c/href</a></iframe><p>Why are we saving the data as a CSV if we intend to use a DB? Well quite simply, we need a CSV file to build our DataFrame from. Okay lets build our DataFrame and set up the expected conditions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5423975be171e715db4c81f08e57c3ff/href">https://medium.com/media/5423975be171e715db4c81f08e57c3ff/href</a></iframe><p>So if you are following along, you are probably wondering why we are defining the short and long signals twice. Earlier we built a CSV with candle data and now we are using two libs to do essentially the same thing, TALib and Numpy. The basic goal here is to be able to output a visual chart with matplotlib and actually visualize market conditions for later debug. Let’s not get ahead of ourselves.</p><p>The second thing we are doing is using the last candle close for our indicators, as seen on line 16. You could use open, but traditionally these types of applications work best using the close of the last candle rather than the open of a new one. For more information as to why see <a href="https://news.cqg.com/blogs/coding/2014/04/close-vs-open-trading-systems">here</a></p><p>Now that the signals are set up we can proceed to the main logic of the system which will find and define a trade setup.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5644df79f305c0e22bd0decdf014d0b7/href">https://medium.com/media/5644df79f305c0e22bd0decdf014d0b7/href</a></iframe><p><strong>Wait, this is one directional — why?</strong></p><p>Simple answer is that the vast majority of traders spot trade and do not use margin positions. We might add features later for margin trading, but for now the only thing anyone will be using this scanner for is to determine <strong>*entry*</strong></p><p>Closing positions in profit is an art, but for the sake of this tutorial we are going to use generic percentage gain based settings, making this more of a ‘buy and hold’ signal bot than anything. In future versions we will probably add logic to indicate when the optimal positional exit occurs.</p><p>Now, if you are paying attention you may have noticed line 2 of the above gist which effectively finds the symbol in the database generated earlier and if not, creates an instance of it. Since we are working with the database state, lets go ahead and commit the signal to it:</p><pre>db.commit()<br>print(&quot;Finished generating output, waiting 900 seconds...&quot;)<br>time.sleep(900)</pre><p>Why 900 seconds? API rate limits for one, but also — our set time frame is 15m</p><p>Okay that’s the program in a nutshell, but that isn’t everything. We will want this to run continuously, and as an added bonus we will want to see the output of certain functions. Let’s use threading for our loops and let’s output the data via a JSON API as well as build a function to regen the list at will.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/747b92a71faf0feddf41e026f63e6990/href">https://medium.com/media/747b92a71faf0feddf41e026f63e6990/href</a></iframe><p>Note how I use the <a href="https://pypi.org/project/timing/">timing</a> library, It’s not strictly necessary nor is it the best way to measure performance, it has however become a staple in all my code.</p><p>now lets make a run function and run the app!</p><pre>if __name__ == &#39;__main__&#39;:<br><br>    app.run(debug=True, host=&quot;0.0.0.0&quot;,use_reloader=False)</pre><p>Now we can head over to <a href="http://127.0.0.1:5000/output">http://127.0.0.1:5000/output</a> and see the results!</p><pre>[<br>  {<br>    &quot;pair&quot;: &quot;ETHBTC&quot;, <br>    &quot;price&quot;: &quot;0.03248500&quot;, <br>    &quot;signal&quot;: &quot;Neutral&quot;<br>  }, <br>  {<br>    &quot;pair&quot;: &quot;LTCBTC&quot;, <br>    &quot;price&quot;: &quot;0.00850800&quot;, <br>    &quot;signal&quot;: &quot;Neutral&quot;<br>  }, <br>  {<br>    &quot;pair&quot;: &quot;BNBBTC&quot;, <br>    &quot;price&quot;: &quot;0.00150660&quot;, <br>    &quot;signal&quot;: &quot;Neutral&quot;<br>  }, <br>  {<br>    &quot;pair&quot;: &quot;NEOBTC&quot;, <br>    &quot;price&quot;: &quot;0.00267100&quot;, <br>    &quot;signal&quot;: &quot;Neutral&quot;<br>  }]</pre><p>Simple enough right? run the thing, and you can play with it for yourself. In the next post we will cover visualizing the data with a live chart, additional Technical Analysis options and working in depth with the application to send functional orders to Binance for buying and selling. I hope you enjoyed this post, and there is a lot more to come. Please drop me a line in comments if you would like to learn more, and be sure to check out the whole code <a href="https://gist.github.com/slapglif/203285daccb69406611436f2e544a13b">here</a></p><p><strong>Important note: This code is functional but not by any means safe to use for real investment, please don’t blame me if something goes wrong, this is meant to be a basic framework for learning, not a functional signal scanner!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4555048973b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a telegram trade signal bot]]></title>
            <link>https://medium.com/@icarusabiding/creating-a-telegram-signal-bot-d368a701df4c?source=rss-3f86cfff2eb8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d368a701df4c</guid>
            <category><![CDATA[forex-signals]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[telegram]]></category>
            <category><![CDATA[trading]]></category>
            <dc:creator><![CDATA[Michael Brown]]></dc:creator>
            <pubDate>Thu, 30 Aug 2018 19:49:20 GMT</pubDate>
            <atom:updated>2018-08-30T19:50:11.487Z</atom:updated>
            <content:encoded><![CDATA[<p>Some time ago I began creating a trading application which I call IcarusBot. The results of this application have been so encouraging that I wanted to share the trade signals with other people but I don’t have a community of my own and it’s been tough finding a method in which I can export a trade journal and trade signals reliably short of hosting an expensive website or server on my own. This struggle led me to start using the itty bitty site technology to publish my trade journals however there are certain issues with url-encoding an entire website such as not being able to use URL shorteners and social media not accepting the URL. I then had the idea to start publishing my trade signals on Telegram and let other users find the results for themselves. Below is just a small guide on how to create your own as well as some information on where you can find my trade signals if you want to have more reliable trading signals in your life.</p><h3>It’s easier than you think</h3><p>Anyone with a rudimentary understanding of Python Programming can really take this on even without my help with that being said I’m going to explain the process I used to create the bot and the troubles that I ran into. first let’s take a look at a snippet of the code andwalk through the functions one by one. We start by creating a Signal function to relay the signal into chat which I call tele(). In order for this function to be called however it needs to be able to read the The bot ID and the update instance, so we create empty lists so we can later pass those values outside of a loop:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/248e6a2db76e7f02a758f0bbad3c5530/href">https://medium.com/media/248e6a2db76e7f02a758f0bbad3c5530/href</a></iframe><p>Next are the default functions included in the telegram bot API, found on pypy as <a href="https://pypi.org/project/python-telegram-bot/"><strong>python-telegram-bot</strong></a>. We have a start command to initiate the bot, a cancel function and an error log function, Note that I have commented out the error logging and opted for a pass because I was running into certain issues with the error log halting my code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7d6d21efb855a1531efd06b414fae49/href">https://medium.com/media/d7d6d21efb855a1531efd06b414fae49/href</a></iframe><p>Finally we have the main sauce of the program, the function main() which is running in a thread thanks to the py-telegram package mentioned earlier. The signals section can be set up to Additionally allow users to give the bot more commands which you can handle with additional functions however because the spot is designed to Receive external signals via be previously programmed TradingView alerts we don’t really need any contextual commands to be processed from the client and Telegram rather they just need updates:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c4055c6bf511eb74a6e4382d9f7aa02a/href">https://medium.com/media/c4055c6bf511eb74a6e4382d9f7aa02a/href</a></iframe><h3>Okay so it’s done right?</h3><p>Whoa there Nelly hold your horses. Okay we have a bot that can publish signals it was provided into a given telegram Channel, But what we don’t have is a function Providing the actual signals to give to the user, For that I’m going to fall back on the <a href="https://github.com/slapglif/tv-mt4"><strong>original Icarus bot code</strong></a> and provide a concurrent read mail function. Note that this is a stripped-down version of <a href="https://github.com/slapglif/tv-mt4"><strong>the original code</strong></a> so that it can function inside of this telegram application:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b4e565dacc0c6bd1209a16e7c37c4d0/href">https://medium.com/media/6b4e565dacc0c6bd1209a16e7c37c4d0/href</a></iframe><p>All were doing here is reading settings from a Json file and then passing the information to an IMAP watching Loop which then relays the trade signal to our Telegram application however because both of these need to run concurrently I recommend employing multi-processing or threading which python is notoriously not good at in fact I myself spent the better part of an afternoon just trying to figure out how to get to while Loops to run concurrently, And I’ve made many asynchronous applications in the past with other languages so you can imagine my frustration.</p><p>And that’s pretty much it we have ourselves an application that takes email signals from tradingview.com and provides them to users in a telegram Channel so that they may continue on and find their own trade opportunities with the signals provided. If you’d like to view the entire application as it’s running on my computer please check out my repository below:</p><p><a href="https://github.com/slapglif/tv-mt4/blob/master/telegram-bot.py">https://github.com/slapglif/tv-mt4/blob/master/telegram-bot.py</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d368a701df4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>