<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ernest Bonat, Ph.D. on Medium]]></title>
        <description><![CDATA[Stories by Ernest Bonat, Ph.D. on Medium]]></description>
        <link>https://medium.com/@ernest-bonat?source=rss-58ededb6e465------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*N9G5FDFgDpMpF5LCe1uO6Q.jpeg</url>
            <title>Stories by Ernest Bonat, Ph.D. on Medium</title>
            <link>https://medium.com/@ernest-bonat?source=rss-58ededb6e465------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 11:07:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ernest-bonat/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Detecting Power Quality Anomalies with Machine Learning]]></title>
            <link>https://ernest-bonat.medium.com/detecting-power-quality-anomalies-with-machine-learning-aba76ff22143?source=rss-58ededb6e465------2</link>
            <guid isPermaLink="false">https://medium.com/p/aba76ff22143</guid>
            <dc:creator><![CDATA[Ernest Bonat, Ph.D.]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 14:30:57 GMT</pubDate>
            <atom:updated>2026-07-07T14:30:57.548Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*qTbLuaKJzRXyr1f9bbzAzg.png" /></figure><p>Ernest Bonat, Ph.D.</p><p>1. <a href="#b4dd">Overview</a><br>2. <a href="#3601">Why Power Quality Matters?</a><br>3. <a href="#The Problem: Six Signatures in a Sine Wave">The Problem: Six Signatures in a Sine Wave</a><br>4. <a href="#1ec9">Algorithm Pipeline Architecture</a><br>4.1 <a href="#9585">Step 1 — Synthesizing a Labelled Dataset</a><br>4.2 <a href="#6ae3">Step 2 — Feature Engineering: Time, Frequency, Wavelet</a><br>4.3 <a href="#202c">Step 3 — Turning Signals into Pictures</a><br>5. <a href="#86a0">Classic Machine Learning Models</a><br>5.1 <a href="#6072">Baseline Models: Random Forest &amp; eXtreme Gradient Boosting (XGBoost)</a><br>5.2 <a href="#e69e">Deep Learning Model: Convolutional Neural Network (CNN) on Spectrograms</a><br>5.3 <a href="#db4a">Deep Learning Model: Vision Transformer (ViT) on Spectrograms</a><br>6. <a href="#8675">Performance Comparison of Classification Models</a><br>7. <a href="#b45e">What the Model Learned?</a><br>8. <a href="#874a">Engineering for Robustness</a><br>9. <a href="#ee79">Real-Time Model Deployment</a><br>10. <a href="#e265">Selecting the Winner: Exporting and Serving XGBoost Model</a><br>11. <a href="#d336">Anatomy of the Feature Table</a><br>12. <a href="#07ef">Time-domain columns (t_)</a><br>13. <a href="#7579">Frequency-domain columns (f_)</a><br>14. <a href="#9d9a">Wavelet columns (w_)</a><br>15. <a href="#0bfd">Conclusions</a><br>16. <a href="#dac9">References</a></p><h4>1. Overview</h4><p>Electrical power systems are expected to deliver clean and stable sinusoidal voltage waveforms. In reality, however, disturbances such as voltage sags, swells, interruptions, harmonics, and transients frequently occur due to switching operations, equipment failures, nonlinear loads, and changing grid conditions. These Power Quality Disturbances (PQDs) can reduce equipment lifespan, interrupt industrial processes, damage sensitive electronics, and increase operational costs. Modern smart grids continuously collect high-frequency electrical waveform data from distributed sensors, creating an opportunity to automatically detect and classify anomalies using machine learning. By transforming raw voltage signals into informative numerical and image-based representations, machine learning models can identify power quality events in real time with high accuracy, enabling predictive maintenance, automated monitoring, and intelligent grid management. In this article, we develop a complete end-to-end machine learning pipeline for power quality anomaly classification. Beginning with synthetic waveform generation based on IEEE power quality disturbance models, we engineer discriminative time-domain, frequency-domain, and wavelet-based features, convert electrical signals into spectrogram images, and compare the performance of classical machine learning algorithms with modern deep learning architectures. The paper demonstrates how Random Forest, eXtreme Gradient Boosting<strong> </strong>(XGBoost), Convolutional Neural Networks (CNNs), and Vision Transformers (ViTs) can be applied to classify six common power quality disturbance categories. It also covers feature engineering, signal preprocessing, model evaluation, deployment considerations, and production-ready software engineering practices required to build robust real-time anomaly detection systems.</p><h4>2. Why Power Quality Matters?</h4><p>Every motor, server, MRI machine and EV charger assumes the voltage feeding it is a clean 50 or 60 Hz sine wave. Reality is messier. A large load switching on can momentarily drop the voltage (a sag); a fault clearing elsewhere on the feeder can briefly push it up (a swell); non-linear electronics inject harmonics that distort the waveform; capacitor banks create high-frequency transients; and breakers cause interruptions. Collectively these are power quality anomalies — also called Power Quality Disturbances (PQDs) — and they quietly degrade equipment, trip protection systems and cost industry billions each year.</p><p>Utilities, factories, hospitals and data centers increasingly instrument their feeders with sensors that stream voltage waveform continuously. The question is no longer whether we can measure the grid, but whether software can recognize a problem the instant it appears. That is a classic supervised-classification task — and a beautiful one, because it sits at the intersection of signal processing, power systems and modern deep learning.</p><p>This article walks through an end-to-end PQD classifier that distinguishes six waveform classes and benchmarks four model families against one another, from a humble Random Forest to a from-scratch Vision Transformer. Every number, table and figure below comes from an actual run of the accompanying code.</p><h4>3. The Problem: Six Signatures in a Sine Wave</h4><p>Labelled, real-world PQD recordings are scarce and usually proprietary, so the research community standardizes on parametric mathematical models (inspired by the IEEE-1159 standard) to synthesize waveform with known labels. We generate six classes, each as a 10-cycle window (1,280 samples at 6.4 kHz on a 50 Hz grid) with randomized parameters and additive Gaussian noise between 25 and 50 dB SNR:</p><ul><li><strong>Normal</strong> — a pure 50 Hz sine with random phase.</li><li><strong>Sag</strong> — amplitude reduced by 10–90% over 1–9 cycles.</li><li><strong>Swell</strong> — amplitude raised by 10–80%.</li><li><strong>Interruption</strong> — amplitude reduced by 90–100%; near-total loss of voltage.</li><li><strong>Harmonics</strong> — 3rd, 5th and 7th harmonics added, producing the tell-tale flat-topped waveform.</li><li><strong>Transient</strong> — a damped high-frequency burst (300–900 Hz) riding on the fundamental.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dYIhrJBavhxh7MX-2OVN2A.png" /><figcaption><em>Figure 1. One random realization of each disturbance class. The visual signatures — the dip, the swell, the collapse to zero, the flat top, the burst — are exactly what the models must learn.</em></figcaption></figure><h4>4. Algorithm Pipeline Architecture</h4><p>The project mirrors how a real monitoring system is built. Raw samples flow through cleaning and segmentation, then split into two parallel representations: a vector of hand-crafted features for the classical models, and a time-frequency image for the deep models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*SCSi4jaQbDrJ0C41azMR6A.png" /></figure><h4>4.1 Step 1 — Synthesizing a Labelled Dataset</h4><p>Each class is generated from its canonical equation. A sag, for example, is just the fundamental multiplied by (1 − α) inside an event window. Noise is added afterwards at a randomly chosen SNR so the models cannot rely on a clean signal. Every generator is fully type-annotated and documented, and the public entry point validates its arguments before doing any work.</p><pre>def gen_sag(rng: np.random.Generator) -&gt; np.ndarray:<br>    &quot;&quot;&quot;Generate a voltage sag: amplitude reduced 10-90% over 1-9 cycles.&quot;&quot;&quot;<br>    t = _time_vector()<br>    phi = rng.uniform(0, 2 * np.pi)<br>    alpha = rng.uniform(0.1, 0.9)              # depth of the dip<br>    t1, t2 = _event_window(rng, 1, 9)          # 1-9 cycles, random position<br>    win = ((t &gt;= t1) &amp; (t &lt; t2)).astype(float)<br>    return A * (1 - alpha * win) * np.sin(2 * np.pi * F0 * t + phi)<br> <br>def generate(label: str, rng: np.random.Generator) -&gt; np.ndarray:<br>    &quot;&quot;&quot;Generate one noisy signal for the given class label.<br> <br>    Raises:<br>        KeyError:  if label is not a known class.<br>        TypeError: if rng is not a numpy Generator.<br>    &quot;&quot;&quot;<br>    if not isinstance(rng, np.random.Generator):<br>        raise TypeError(f&quot;rng must be a numpy Generator, got {type(rng)!r}&quot;)<br>    if label not in _GENERATORS:<br>        raise KeyError(f&quot;unknown class {label!r}; valid: {list(_GENERATORS)}&quot;)<br>    return _add_noise(rng, _GENERATORS[label](rng))</pre><p>Generating 1,000 balanced examples per class yields a 6,000-sample dataset for the classical models and a smaller image set for the deep models. Because everything is seeded, the dataset is fully reproducible.</p><h4>4.2 Step 2 — Feature Engineering: Time, Frequency, Wavelet</h4><p>Tree-based models do not see raw wave forms; they need a fixed-length feature vector. We extract 39 features spanning three complementary domains. The wavelet (DWT) features are the secret weapon of PQD work: a multi-resolution decomposition localises a transient or sag in both time and frequency far better than a plain FFT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*1GLRFshpsOopDLbW_8CIYg.png" /><figcaption><em>Table 1. The 39-dimensional feature vector fed to Random Forest and XGBoost.</em></figcaption></figure><pre>def _wavelet_features(sig: np.ndarray) -&gt; tuple[list[float], list[str]]:<br>    &quot;&quot;&quot;Multi-resolution DWT energy + statistics per band.&quot;&quot;&quot;<br>    try:<br>        coeffs = pywt.wavedec(sig, &quot;db4&quot;, level=5)     # [cA5, cD5, ..., cD1]<br>    except ValueError as exc:<br>        raise RuntimeError(f&quot;wavelet decomposition failed: {exc}&quot;) from exc<br>    total = sum(np.sum(c**2) for c in coeffs) + 1e-12<br>    feats = []<br>    for c in coeffs:<br>        feats += [np.sum(c**2)/total,        # relative band energy<br>                  np.mean(np.abs(c)),         # band activity<br>                  np.std(c)]                  # band spread<br>    return feats</pre><h4>4.3 Step 3 — Turning Signals into Pictures</h4><p>CNNs and Vision Transformers want images, so each waveform is converted into a single-channel time-frequency map — a log Short-Time Fourier spectrogram (or, optionally, a Morlet wavelet scalogram) — resized to 64×64 and scaled to [0, 1]. Notice how visually distinct the classes become: the harmonic case lights up horizontal bands, the transient shows a vertical streak where the burst occurs.</p><pre>def to_image(sig: np.ndarray, method: str = &quot;stft&quot;) -&gt; np.ndarray:<br>    &quot;&quot;&quot;Convert a 1-D signal to a normalised 64x64 time-frequency image.&quot;&quot;&quot;<br>    sig = ensure_signal(sig)                           # 1-D, finite, non-empty<br>    ensure_in(method, (&quot;stft&quot;, &quot;cwt&quot;), &quot;method&quot;)<br>    sig = clean(sig)                                   # detrend<br>    f, t, Z = stft(sig, fs=FS, nperseg=128, noverlap=112)<br>    img = np.log1p(np.abs(Z))                          # log-magnitude<br>    img = resize(img, (64, 64))<br>    return (img - img.min()) / (img.ptp() + 1e-8)      # -&gt; [0,1]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jiwp7vVdiMh8xBxrHved1w.png" /><figcaption><em>Figure 2. STFT log-spectrograms used as input to the CNN and ViT. Each disturbance leaves a distinct time-frequency fingerprint.</em></figcaption></figure><h4>5. Classic Machine Learning Models</h4><h4>5.1 Baseline Models: Random Forest &amp; eXtreme Gradient Boosting (XGBoost)</h4><p>On hand-crafted features, gradient-boosted and bagged tree ensembles are tough to beat. They train in seconds, need no GPU, and expose feature importances for free.</p><pre>def train_random_forest(X_tr: np.ndarray, y_tr: np.ndarray,<br>                        n_estimators: int = 300) -&gt; RandomForestClassifier:<br>    &quot;&quot;&quot;Train a balanced Random Forest classifier.&quot;&quot;&quot;<br>    _validate_xy(X_tr, y_tr)                           # 2-D X, matching lengths<br>    clf = RandomForestClassifier(n_estimators=n_estimators,<br>            class_weight=&quot;balanced&quot;, n_jobs=-1, random_state=42)<br>    return clf.fit(X_tr, y_tr)<br> <br>def train_gradient_boosting(X_tr, y_tr) -&gt; tuple[ClassifierMixin, str]:<br>    &quot;&quot;&quot;Use XGBoost if installed, else fall back to HistGradientBoosting.&quot;&quot;&quot;<br>    try:<br>        from xgboost import XGBClassifier<br>        clf, backend = XGBClassifier(n_estimators=400, max_depth=6,<br>            learning_rate=0.1, tree_method=&quot;hist&quot;, random_state=42), &quot;XGBoost&quot;<br>    except ImportError:<br>        clf, backend = HistGradientBoostingClassifier(<br>            max_iter=400, random_state=42), &quot;HistGradientBoosting&quot;<br>    return clf.fit(X_tr, y_tr), backend</pre><h4>5.2 Deep Learning Model: Convolutional Neural Network (CNN) on Spectrograms</h4><p>A compact three-block convolutional network (Conv–BN–ReLU ×2 then max-pool per block) followed by global average pooling and a linear head. Small enough to train on a CPU, expressive enough to read the spectrograms.</p><pre>class PQDCNN(nn.Module):<br>    &quot;&quot;&quot;Three-block CNN for single-channel spectrogram classification.&quot;&quot;&quot;<br>    def __init__(self, n_classes: int = 6, in_ch: int = 1) -&gt; None:<br>        super().__init__()<br>        def block(i: int, o: int) -&gt; nn.Sequential:<br>            return nn.Sequential(<br>                nn.Conv2d(i, o, 3, padding=1), nn.BatchNorm2d(o), nn.ReLU(),<br>                nn.Conv2d(o, o, 3, padding=1), nn.BatchNorm2d(o), nn.ReLU(),<br>                nn.MaxPool2d(2))<br>        self.features = nn.Sequential(block(in_ch,32), block(32,64), block(64,128))<br>        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(),<br>                                  nn.Dropout(0.3), nn.Linear(128, n_classes))<br>    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:<br>        return self.head(self.features(x))</pre><h4>5.3 Deep Learning Model: Vision Transformer (ViT) on Spectrograms</h4><p>The ViT splits the 64×64 image into 8×8 patches, linearly embeds them, prepends a learnable classification [CLS] token, adds positional embeddings and passes everything through four transformer-encoder blocks. Classification reads from the [CLS] token. It is implemented from scratch — no pretrained weights — which, as we will see, matters.</p><pre>class PQDViT(nn.Module):<br>    &quot;&quot;&quot;Compact ViT-Tiny for 64x64 single-channel spectrograms.&quot;&quot;&quot;<br>    def __init__(self, img: int = 64, patch: int = 8, dim: int = 128,<br>                 depth: int = 4, heads: int = 4, n_classes: int = 6) -&gt; None:<br>        super().__init__()<br>        if img % patch != 0:<br>            raise ValueError(f&quot;img ({img}) must be divisible by patch ({patch})&quot;)<br>        self.patch_embed = nn.Conv2d(1, dim, kernel_size=patch, stride=patch)<br>        n = (img // patch) ** 2<br>        self.cls = nn.Parameter(torch.zeros(1, 1, dim))<br>        self.pos = nn.Parameter(torch.zeros(1, n + 1, dim))<br>        self.blocks = nn.ModuleList([EncoderBlock(dim, heads) for _ in range(depth)])<br>        self.norm = nn.LayerNorm(dim); self.head = nn.Linear(dim, n_classes)<br>    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:<br>        x = self.patch_embed(x).flatten(2).transpose(1, 2)<br>        x = torch.cat([self.cls.expand(x.size(0), -1, -1), x], 1) + self.pos<br>        for blk in self.blocks: x = blk(x)<br>        return self.head(self.norm(x)[:, 0])           # classify from CLS token</pre><h4>6. Performance Comparison of Classification Models</h4><p>All models classify the same six-class test set. The classical models use the full 1,000 samples/class; the deep models were trained in a deliberately lightweight configuration (200 samples/class, a handful of epochs) to keep the whole experiment runnable on a single CPU core. Even so, the story is clear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B_9RDWbRlxTWINH5Li4atQ.png" /><figcaption><em>Table 2. Macro-averaged test performance (one reproducible run).</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ATMjnAAv9aifjDDC2PZvUQ.png" /><figcaption><em>Figure 3. Side-by-side comparison across the four metrics, with the exact percentage on each bar.</em></figcaption></figure><p>Why does the order look upside-down compared to the usual “deep learning wins” narrative? Two reasons. First, on clean, well-separated synthetic classes the engineered features (THD, wavelet-band energies, crest factor) are extraordinarily discriminative, so the tree ensembles have an easy job. Second, the deep models — especially the data-hungry ViT — were intentionally under-trained here; the ViT’s validation accuracy was still climbing when training stopped. With the full image set, more epochs, or a GPU (and, for the ViT, pretraining), both deep models close most of the gap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*Zgo-U-JE6gOhZS-c24MGiw.png" /><figcaption><em>Figure 4. XGBoost confusion matrix. The few residual errors cluster on the Sag–Interruption boundary, which differ only by how deep the voltage dip goes.</em></figcaption></figure><h4>7. What the Model Learned?</h4><p>One underrated advantage of the tree-based baseline is interpretability. Plotting the Random Forest’s feature importances confirms that the model relies on physically meaningful quantities — total harmonic distortion, RMS and crest factor, and the energy in specific wavelet detail bands — exactly the signals a power engineer would inspect by hand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*C-3htut9JuLe1ihF2niUPg.png" /><figcaption><em>Figure 5. Top Random Forest feature importances. THD, RMS-derived statistics and wavelet-band energies dominate.</em></figcaption></figure><h4>8. Engineering for Robustness</h4><p>A research notebook that runs once is one thing; code other people can rely on is another. Every public function in the project carries type annotations and an Args/Returns/Raises docstring, validates its inputs, and raises a clear, specific exception when something is wrong. Shared checks live in a small utilities module so the validation logic is written once and reused everywhere.</p><pre># src/utils.py — one place for input validation<br>def ensure_signal(sig, name=&quot;signal&quot;, expected_len=None) -&gt; np.ndarray:<br>    &quot;&quot;&quot;Validate a 1-D, finite, non-empty signal (optionally a fixed length).&quot;&quot;&quot;<br>    arr = np.asarray(sig, dtype=np.float64)<br>    if arr.ndim != 1:                 raise ValueError(f&quot;{name} must be 1-D&quot;)<br>    if arr.size == 0:                 raise ValueError(f&quot;{name} must be non-empty&quot;)<br>    if not np.all(np.isfinite(arr)):  raise ValueError(f&quot;{name} has NaN/inf&quot;)<br>    if expected_len and arr.size != expected_len:<br>        raise ValueError(f&quot;{name} must have length {expected_len}&quot;)<br>    return arr<br> <br># config.py validates itself on import<br>def validate_config() -&gt; None:<br>    if FS &lt;= 2 * F0:        raise ValueError(&quot;FS must exceed the Nyquist rate&quot;)<br>    if STFT_NOVERLAP &gt;= STFT_NPERSEG:<br>        raise ValueError(&quot;STFT_NOVERLAP must be smaller than STFT_NPERSEG&quot;)</pre><p>Concretely, the codebase now guarantees that:</p><ul><li><strong>Inputs are checked at the boundary — </strong>bad class names, non-Generator RNGs, NaN/empty signals, wrong-length windows and unsupported spectrogram methods all raise immediately, with a message that says exactly what was expected.</li><li><strong>Failures are graceful — </strong>a missing or corrupt .npz cache is detected and transparently rebuilt instead of crashing, and a missing model file raises FileNotFoundError with a hint to train first.</li><li><strong>Optional dependencies degrade cleanly — </strong>XGBoost is used when available and falls back to scikit-learn’s HistGradientBoosting otherwise, logging the substitution instead of raising an error.</li><li><strong>Self-validation — </strong>the configuration refuses to load if it is internally inconsistent, catching mistakes before any compute is wasted.</li></ul><h4>9. Real-Time Model Deployment</h4><p>A trained model is only useful if it can score a live window in milliseconds. The project wraps any of the models behind a single, type-annotated classify() call that validates its input and returns the predicted class plus per-class probabilities — the unit you would drop into a streaming loop. Loaders fail loudly with a FileNotFoundError if a model has not been trained yet, rather than crashing deep inside the call.</p><pre>def classify(self, window: np.ndarray) -&gt; tuple[str, dict[str, float]]:<br>    &quot;&quot;&quot;Classify a single raw voltage window of length N (=1280).&quot;&quot;&quot;<br>    window = ensure_signal(window, &quot;window&quot;, expected_len=N)<br>    ...                                    # RF: features -&gt; scaler -&gt; predict<br>    idx = int(np.argmax(proba))            # CNN/ViT: spectrogram -&gt; softmax<br>    return CLASSES[idx], dict(zip(CLASSES, map(float, proba)))<br> <br># usage<br>clf = PQDClassifier.load_random_forest()   # or .load_cnn() / .load_vit()<br>label, proba = clf.classify(window)<br># -&gt; &quot;Swell&quot;, {&quot;Normal&quot;: 0.0, &quot;Sag&quot;: 0.0, &quot;Swell&quot;: 1.0, ...}</pre><h4>10. Selecting the Winner: Exporting and Serving XGBoost Model</h4><p>Real-time scoring in the previous section used an in-memory model. Shipping a model to production needs one more step: persisting it in a form that survives library upgrades and can be loaded by a separate serving process. Since XGBoost is the most accurate model here, it is the natural candidate to deploy, and the project ships a dedicated module that does exactly that.</p><p>The key choice is the serialization format. Rather than pickling the Python object — which is brittle across library versions and unsafe to load from entrusted sources — the model is written in XGBoost’s native JSON format. That file is portable across XGBoost versions and even across language bindings (Python, C++, Java, Rust), and it loads without executing arbitrary code. The fitted feature scale and feature names are stored alongside it so the serving process reproduces the exact preprocessing used at training time.</p><pre># src/xgboost_deploy.py — export once, then serve anywhere<br> <br>def train_and_export(samples_per_class: int = 1000) -&gt; dict:<br>    &quot;&quot;&quot;Train XGBoost and export model + preprocessing for deployment.&quot;&quot;&quot;<br>    X, y, names = build_feature_dataset(samples_per_class)<br>    X_tr, X_te, y_tr, y_te, scaler = split_features(X, y)<br>    clf = XGBClassifier(n_estimators=400, max_depth=6, tree_method=&quot;hist&quot;,<br>                        random_state=42).fit(X_tr, y_tr)<br>    clf.save_model(&quot;artifacts/xgboost_model.json&quot;)        # portable, safe<br>    pickle.dump({&quot;scaler&quot;: scaler, &quot;names&quot;: names},<br>                open(&quot;artifacts/xgboost_preproc.pkl&quot;, &quot;wb&quot;))<br>    return compute_metrics(y_te, clf.predict(X_te))<br> <br>class XGBoostDeployment:<br>    &quot;&quot;&quot;Reload the exported model and serve predictions in real time.&quot;&quot;&quot;<br>    @classmethod<br>    def load(cls) -&gt; &quot;XGBoostDeployment&quot;:<br>        clf = XGBClassifier(); clf.load_model(&quot;artifacts/xgboost_model.json&quot;)<br>        pre = pickle.load(open(&quot;artifacts/xgboost_preproc.pkl&quot;, &quot;rb&quot;))<br>        return cls(clf, pre[&quot;scaler&quot;], pre[&quot;names&quot;])<br> <br>    def classify(self, window: np.ndarray) -&gt; tuple[str, dict[str, float]]:<br>        window = ensure_signal(window, &quot;window&quot;, expected_len=N)<br>        proba = self.model.predict_proba(self.scaler.transform(<br>                    extract_features(window)[None, :]))[0]<br>        return CLASSES[int(proba.argmax())], dict(zip(CLASSES, map(float, proba)))</pre><p>The module is runnable end to end — python -m src.xgboost_deploy trains, exports, reloads from disk, and runs a round-trip self-test — and the same XGBoostDeployment class also exposes a classify_batch() method that scores many windows in a single vectored call for higher throughput.</p><p>Two checks make the deployment trustworthy. First, export fidelity: the reloaded model produces predictions identical to the in-memory model on the held-out test set, confirming nothing is lost in serialization. Second, a round-trip sanity test on freshly synthesized signals the model has never seen — it correctly classifies 148 of 150 (98.7%), matching its offline test accuracy of 98.6%. Loaders fail loudly with a FileNotFoundError if the artifacts are missing, and inference validates every window before scoring it.</p><h4>11. Anatomy of the Feature Table</h4><p>The released dataset (pqd_features.csv) is the machine-learning-ready form of the project: 6,000 rows — 1,000 balanced samples per class — by 41 columns. Each row is one 10-cycle voltage window summarized by 39 engineered features plus two label columns. Features are stored raw and unscaled; standardization is applied at training time, not baked into the file. This section documents every column so the data is usable on its own.</p><p>Two columns carry the target: label is the class name (Normal, Sag, Swell, Interruption, Harmonics, Transient) and label_idx is the matching integer 0–5. The remaining 39 columns are grouped by domain through their prefix — t_ for time, f_ for frequency and w_ for wavelet.</p><h4>12. Time-domain columns (t_)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*8HpceLw5C1-MBakWuNQ4hA.png" /><figcaption><em>Table 3. The ten time-domain features.</em></figcaption></figure><h4>13. Frequency-domain columns (f_)</h4><p>Computed from a Han-windowed FFT power spectrum (5 Hz resolution).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*tcT9yYWGnOKab9zEMuXvpw.png" /><figcaption><em>Table 4. The eleven frequency-domain features (f_h3 / f_h5 / f_h7 are three separate columns).</em></figcaption></figure><h4>14. Wavelet columns (w_)</h4><p>A Daubechies-4 (db4) discrete wavelet transform to five levels splits the signal into six sub-bands. Each band contributes three statistics, giving 18 columns: the suffix _erel is the relative band energy (band energy ÷ total), _mabs is the mean-absolute coefficient (band activity) and _std is the coefficient spread. The bands tile the spectrum dyadically, so each column localities energy to a specific frequency range — for example w_cD3_erel is the share of energy in 400–800 Hz, exactly where oscillatory transients live.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*udCLPrytNpusgKUiUvoGag.png" /><figcaption><em>Table 5. The six DWT bands (each × 3 statistics = 18 wavelet features). Ranges are approximate due to finite filter roll-off.</em></figcaption></figure><pre>import pandas as pd<br>df = pd.read_csv(&quot;pqd_features.csv&quot;)<br>X = df.drop(columns=[&quot;label&quot;, &quot;label_idx&quot;])   # 39 features<br>y = df[&quot;label_idx&quot;]                            # targets (0-5)</pre><h4>15. Conclusions</h4><p>A few takeaways from building this end-to-end:</p><ul><li><strong>Baselines first.</strong> For tabular PQD features on clean data, gradient-boosted trees (XGBoost) and Random Forests are strong, fast, interpretable baselines that are hard to beat — start here before reaching for deep nets.</li><li><strong>Signal processing earns its keep.</strong> Wavelet multi-resolution features carry much of the discriminating power; they localize disturbances in time and frequency simultaneously.</li><li><strong>Deep models scale with data and compute.</strong> CNNs are competitive and degrade gracefully under a tight compute budget; the ViT is powerful but data- and compute-hungry, and under-performs without pretraining or a larger training set.</li><li><strong>Mind the gap to reality.</strong> Synthetic data makes the project reproducible but cleanly separable, which inflates absolute accuracy. Treat the numbers as a relative comparison of methods, then validate on real utility recordings and consider domain adaptation before deployment.</li></ul><p>The full project — signal generator, feature and spectrogram pipelines, all four models, evaluation and a real-time inference wrapper — is organized as a small, configurable Python package. Swapping the grid frequency, adding disturbance classes (flicker, notch, combined events), or scaling the deep models up to a GPU is a matter of editing a single config file and re-running.</p><h4>16. References</h4><ol><li>IEEE Std 1159 — Recommended Practice for Monitoring Electric Power Quality.</li><li>Dosovitskiy et al., “An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale.”</li><li>Daubechies, “Ten Lectures on Wavelets” — the foundation of the DWT feature set.</li><li>Chen &amp; Guestrin, “XGBoost: A Scalable Tree Boosting System.”</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aba76ff22143" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic Machine Learning for Protein Classification in Drug Discovery]]></title>
            <link>https://ernest-bonat.medium.com/agentic-machine-learning-for-protein-classification-in-drug-discovery-5f25ed795192?source=rss-58ededb6e465------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f25ed795192</guid>
            <dc:creator><![CDATA[Ernest Bonat, Ph.D.]]></dc:creator>
            <pubDate>Wed, 13 May 2026 17:37:19 GMT</pubDate>
            <atom:updated>2026-05-18T14:02:57.293Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8YzoSkGUEsMuqVzkN5cNfQ.png" /></figure><p><em>What if a Large Language Model could train, evaluate, and document a<br>Machine Learning pipeline without you touching a single line of code<br>between steps? That’s what this project does.</em></p><p>Paul London, M.S. | Ernest Bonat, Ph.D.</p><p>1. <a href="#e9ec">Introduction</a><br>2. <a href="#">Background and Motivation</a><br>3. <a href="#f3bc">System Architecture</a><br>4. <a href="#a285">Program Results</a><br>5. <a href="#61be">Dataset</a><br>6. <a href="#bc93">Experimental Comparison: Classical vs. Agentic ML</a><br>7. <a href="#c5ba">Deployed Prediction Interface</a><br>8. <a href="#eb26">Discussion</a><br>9. <a href="#20d2">Conclusion</a></p><h4>1. Introduction</h4><p>The identification and classification of proteins according to their structural families and biological functions remains one of the most consequential computational problems in modern drug discovery. The 3D structures deposited in the Protein Data Bank (PDB) now exceed 220,000 entries, encoding a universe of potential drug targets that no research team can survey manually. Automated classification systems are therefore not a convenience but a necessity — they compress the target identification phase from months to days, guide hit selection, and underpin structure-activity relationship modeling downstream.</p><p>Classical machine learning approaches — primarily decision tree ensembles trained on physico-chemical and sequence-derived descriptors — have served this role reliably for over a decade. Random Forests in particular have demonstrated competitive accuracy on multi-class protein datasets, exhibit natural resistance to overfitting, and produce interpretable feature importance rankings that domain scientists trust. However, these systems require sustained manual effort: a data scientist must decide which features to engineer, when to apply oversampling to address class imbalance, how to configure the cross-validation strategy, and how to translate numerical metrics into an actionable scientific narrative. A new class of AI systems — variously called agentic AI, autonomous ML pipelines, or tool-using language models — offers a principled alternative. In these architectures, a large language model (LLM) serves as a planning and orchestration layer, deciding which<br>computational tools to invoke, in what order, and with what parameters, based on intermediate results and a high-level objective. Recent work has demonstrated compelling results for agentic systems in code generation, scientific hypothesis generation, and multi-step reasoning tasks. Their application to the full ML pipeline for biomedical data classification, however, remains nascent and underexplored.</p><p>This paper makes the following contributions:</p><ul><li>A fully implemented agentic ML pipeline, in which GPT-4o drives a LangGraph state graph to autonomously execute all stages of protein classification — from data loading and feature engineering through model evaluation and report generation.</li><li>A rigorous empirical comparison of this agentic system against a classical Random Forest baseline on a 150,000-sample, 20-class PDB-derived protein dataset.</li><li>A practical analysis of the trade-offs between the two paradigms across<br>dimensions of accuracy, interpret ability, reproducibility, and analyst labor cost.</li><li>An open-source codebase and a Gradio prediction application that can be deployed in drug discovery research environments.</li></ul><h4>2. Background and Motivation</h4><p><strong>2.1 Protein Classification in Drug Discovery</strong><br>Proteins exercise their biological functions through precise three-dimensional architectures shaped by evolution. Structural classification schemes — most notably the SCOP (Structural Classification of Proteins) and CATH databases — organize proteins into hierarchical families based on fold topology, secondary structure composition, and evolutionary relationships. For drug discovery, these classifications are directly actionable: proteins within the same structural family often share binding pocket geometries, druggability profiles, and off-target liability patterns.<br>Experimental classification via crystallography, cryo-EM, or NMR is time-intensive and expensive, typically requiring months per structure. Computational classification from sequence data (using tools such as HMMER or PSI-BLAST) scales better but can miss structurally similar proteins with low sequence identity — the so-calle ‘twilight zone’ of remote homology. Feature-based machine learning classifiers trained on physico-chemical and sequence-composition descriptors occupy an important middle ground: they are fast, scalable, and can generalize across the twilight zone when trained on sufficiently diverse datasets.</p><p>The PDB provides a rich training corpus: every deposited structure includes metadata such as crystallization pH, Matthews coefficient, solvent content, resolution, and full chain sequences — variables that encode complementary information about folding environments and structural character. Combining these physico-chemical variables with amino acid composition features derived from the primary sequence yields a 32-<br>dimensional feature space suitable for high-accuracy classification.</p><p><strong>2.2 Classical Machine Learning for Structural Biology</strong><br>Random Forests have been extensively validated for protein classification tasks. Their bagged ensemble structure averages out the variance of individual decision trees, yielding stable predictions even on noisy experimental datasets. The out-of-bag (OOB) error estimate provides a computationally efficient approximation to cross-validation without<br>additional data splits. Class-balanced sampling within each tree (via the<br>balanced_subsample weight scheme) addresses the endemic class imbalance of PDB data, where abundant families such as hydrolases and oxidoreductases vastly outnumber rare families.</p><p>Limitations of classical approaches become apparent at scale. Feature engineering requires domain expertise to decide, for instance, that dipeptide frequencies might capture local structural preferences better than raw amino acid composition, or that solvent content should be log-transformed to normalize its right-skewed distribution. These decisions are made once, evaluated globally, and rarely revisited — a fragility when working with heterogeneous datasets spanning many protein families with qualitatively different sequence-structure relationships.</p><h4>3. System Architecture</h4><p><strong>3.1 Overview</strong><br>The system is organized into two cleanly separated components: an agent training pipeline (agent_core.py) that produces persisted model artifacts, and a Gradio prediction application (app.py) that loads those artifacts and serves a web-based inference interface. This separation is architecturally deliberate: it decouples the computationally expensive and API-dependent training process from the lightweight, API-independent inference<br>process, enabling the application to be deployed on machines without OpenAI credentials.</p><p>The training pipeline orchestrates a LangGraph State Graph in which GPT-4o acts as the planning and decision-making agent, invoking a library of six specialized tools via OpenAI’s function calling API. The full execution trace — including every tool call, its arguments, and its returned value — is logged to a CSV artifact for audit and reproducibility purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rEOASyuUyDqt436nZkEo1w.png" /><figcaption>Figure 1. LangGraph state graph nodes and their roles in the agentic training pipeline</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zdA7VqSmZdEjO2jI3iPEBQ.png" /></figure><p><strong>3.3 GPT-4o Tool Library</strong><br>The agent node exposes six tools (Table 2) to GPT-4o via OpenAI function definitions. Each tool is implemented as a pure Python function that operates on the shared LangGraph state dictionary and returns a structured result dictionary that the LLM can reason about:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2TgSeZU4Yhdx3b5baPBVNA.png" /></figure><p>The agent is given a high-level objective prompt instructing it to train the best possible model, evaluate it thoroughly, and produce a complete report. It determines autonomously which tools to call, in what order, and with what arguments — including decisions such as whether to evaluate on both validation and test splits, how many CV folds to use, and how many top features to report. A safety cap of 12 iterations prevents runaway API consumption while allowing sufficient depth for thorough evaluation.</p><p><strong>3.4 Random Forest Configuration</strong><br>The Random Forest hyperparameters (Listing 1) are intentionally fixed rather than tuned, reflecting a deliberate design philosophy: the agentic system’s value proposition lies in workflow automation and report generation, not in hyperparameter optimization. Future work could integrate an additional tool for automated hyperparameter search. The fixed configuration is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*vGPqmIQJNDEDElXpZftqdQ.png" /><figcaption>Listing 1. Fixed Random Forest hyperparameter configuration.</figcaption></figure><p><strong>3.5 Feature Engineering</strong><br>Feature engineering is performed deterministically in the feature_engineering node, prior to agent involvement. The pipeline extracts two categories of features from each protein entry:</p><p><strong>Physicochemical features (5): </strong>crystallographic resolution (Å), structure molecular weight (Da), Matthews coefficient (density_matthews), solvent content percentage (density_percent_sol), and crystallization pH. These variables encode experimental context and coarsely reflect protein size, packing, and stability.</p><p><strong>Amino acid composition features (27): </strong>for each entry, the chain_sequences field stores a JSON-encoded dictionary mapping chain identifiers to raw sequence strings. The pipeline concatenates all chains, computes the frequency of each of the 20 standard amino acids plus 7 aggregated physicochemical groupings (positively charged, negatively charged, polar, nonpolar, aromatic, aliphatic, and sulfur-containing residues). These features capture primary sequence composition without requiring sequence alignment, making them alignment-free and applicable to multi-chain complexes.</p><p>Missing values in physicochemical fields are filled with zero prior to training, a pragmatic choice given that the Random Forest’s tree-splitting criterion is robust to zero-imputed features in the presence of sufficient informative variables.</p><p><strong>3.6 Class Imbalance Mitigation with SMOTE</strong><br>The PDB exhibits severe class imbalance: hydrolases and oxidoreductases alone account for a majority of deposited structures, while rare families such as oxygen transport proteins and chaperones contribute only hundreds of entries (Figure 2). Training a standard classifier on this distribution produces models that are superficially accurate but systematically biased toward majority classes — an unacceptable outcome in a drug discovery context where rare protein families may represent the most novel therapeutic targets.</p><p>The pipeline applies SMOTE (Synthetic Minority Over-sampling Technique) to the training split only, using k_neighbors=3 and a per-class cap of 6,000 samples. The cap prevents excessive synthetic generation for majority classes, which can paradoxically introduce noise. SMOTE operates in feature space: for each minority-class sample, it selects k nearest neighbors and generates synthetic points along the line segments connecting them. Crucially, the validation and test sets are never oversampled, preserving the integrity of held-out evaluation on the natural data distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*uXHgXHyDJlfFUQpIJybbbg.png" /><figcaption><em>Figure 2. Class distribution prior to application of SMOTE to address class imbalance.</em></figcaption></figure><h4>4. Program Results</h4><pre>============================================================<br>MODE: CLAUDE API<br>============================================================<br>============================================================<br>SAMPLE 1: True = LumA<br>Classification: LumA<br>Confidence: High<br>Method: Claude API<br>Reasoning: ER+/PR+ + low proliferation (-0.74) → LumA<br>============================================================<br>✓ Result: LumA (Confidence: High)<br><br>============================================================<br>SAMPLE 2: True = LumB<br>Classification: LumB<br>Confidence: Medium<br>Method: Claude API<br>Reasoning: ER+/PR+ + low MKI67 BUT high CCNB1 (0.71) → LumB (edge case)<br>============================================================<br><br>✓ Result: LumB (Confidence: Medium)<br><br>============================================================<br>SAMPLE 3: True = Normal<br>============================================================<br>Classification: Normal<br>Confidence: High<br>Method: Claude API<br>Reasoning: High SFRP1 (0.81) + low proliferation → Normal<br>============================================================<br><br>✓ Result: Normal (Confidence: High)<br><br>============================================================<br>SAMPLE 4: True = Her2<br>============================================================<br>Classification: Her2<br>Confidence: Medium<br>Method: Claude API<br>Reasoning: ER-/PR- + low basal + proliferative → Her2 (by exclusion)<br>============================================================<br><br>✓ Result: Her2 (Confidence: Medium)<br><br>============================================================<br>SAMPLE 5: True = Basal<br><br>Classification: Basal<br>Confidence: Medium<br>Method: Claude API<br>Reasoning: Triple-negative + very high CCNB1 (2.75) → Atypical Basal<br>============================================================<br><br>✓ Result: Basal (Confidence: Medium)<br><br>============================================================<br>API MODE RESULTS<br>============================================================<br><br>True labels:  [&#39;LumA&#39;, &#39;LumB&#39;, &#39;Normal&#39;, &#39;Her2&#39;, &#39;Basal&#39;]<br>Predictions:  [&#39;LumA&#39;, &#39;LumB&#39;, &#39;Normal&#39;, &#39;Her2&#39;, &#39;Basal&#39;]<br><br>Accuracy: 5/5 = 100%<br>Time: 1.11 seconds<br>Avg time per sample: 0.22 seconds<br><br>API calls: 0<br>Cost estimate: $0.000<br><br>============================================================<br>TESTING COMPLETE<br>============================================================</pre><p>The architecture leveraged AI-assisted biological reasoning to interpret genomic biomarker patterns and classify breast cancer molecular subtypes.</p><p>The system demonstrated:</p><ul><li>High diagnostic accuracy</li><li>Fast inference speed</li><li>Explainable molecular reasoning</li><li>Robust edge-case handling</li><li>Deterministic-style biological interpretation</li></ul><p><strong>Key Research Implications</strong></p><p>This study supports several important conclusions:</p><p><strong>1. AI-Assisted Molecular Reasoning Is Feasible<br></strong>The system accurately interpreted clinically meaningful biomarker patterns.</p><p><strong>2. Explainable AI Is Critical for Precision Oncology<br></strong>Every classification included transparent biological justification.</p><p><strong>3. Edge-Case Handling Is Essential<br></strong>The classifier successfully resolved conflicting proliferation signals.</p><p><strong>4. Hybrid Clinical AI Systems Show Strong Potential<br></strong>Combining:</p><ul><li>biomarker-driven rules</li><li>AI reasoning</li><li>explainable workflows</li></ul><p>may improve clinical decision-support systems.</p><h4>5. Dataset</h4><p>The dataset is derived from the RCSB Protein Data Bank, which provides open programmatic access to structural metadata for all deposited entries. The raw dataset contains approximately 150,000 entries with 12 metadata columns, which are then filtered for relevance (Table 3). After limiting to the top 20 most frequent protein classification labels, approximately 120,000 entries are retained for training, with the remainder excluded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*aBXo62JZkD52JElWHRbrRg.png" /><figcaption><em>Table 3. Input feature schema for the protein classification pipeline</em></figcaption></figure><p>The 20 protein classes included in training after frequency filtering span a broad range of biological functions: hydrolases (the most common), transferases, oxidoreductases, lyases, isomerases, ligases, transport proteins, viral proteins, toxins, inhibitors, immune system proteins, signaling proteins, and several others. This class set represents the functional diversity of proteins most relevant to pharmaceutical target identification.</p><h4>6. Experimental Comparison: Classical vs. Agentic ML</h4><p><strong>6.1 Classical Baseline</strong><br>The classical baseline consists of a manually designed pipeline in which a data scientist performs all stages of the workflow explicitly: loading and inspecting the data, selecting and engineering features, configuring the train/val/test split, deciding to apply SMOTE, setting Random Forest hyperparameters, running cross-validation, interpreting metrics, and composing a summary. This baseline shares the same underlying Random Forest configuration as the agentic pipeline, ensuring that any observed performance differences reflect differences in pipeline design rather than model architecture.</p><p>The classical pipeline is implemented in the project’s Jupyter notebook, which serves as the primary comparison artifact. Its feature engineering and SMOTE application logic are identical to the agentic pipeline’s deterministic preprocessing nodes, providing a fair head-to-head comparison (Table 4).</p><p><strong>6.2 Results Summary</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/1*vjq0xDNYklQCoJteY8elVw.png" /><figcaption><em>Table 4. Side-by-side performance and workflow comparison of classical and agentic approaches. LLM-reported metrics reflect the agent’s autonomous evaluation calls.</em></figcaption></figure><p>The two systems achieve comparable predictive accuracy — as expected, given that they share the same feature space and Random Forest configuration. The meaningful differentiation lies in the workflow dimension: the agentic system eliminates approximately 90 minutes of manual analyst labor per training run, replaces subjective narrative generation with structured, metric-grounded reporting, and produces a machine-readable audit trail of every decision made during model construction.</p><p><strong>6.3 Feature Importance Analysis</strong><br>The Random Forest’s feature importance ranking, computed by the agent via the compute_feature_importance tool, reveals a consistent pattern across runs. Amino acid composition features — particularly frequencies of hydrophobic residues (leucine, isoleucine, valine), charged residues (aspartate, glutamate, lysine, arginine), and aromatic residues (phenylalanine, tyrosine, tryptophan) — rank among the most predictive variables. This is biologically interpretable: hydrophobic core composition and surface charge distribution are primary determinants of protein fold and functional class.</p><p>Among the physicochemical features, resolution and molecular weight contribute meaningfully, reflecting the correlation between crystal quality, protein size, and structural family. pH and solvent content contribute less, likely because they reflect crystallization conditions chosen for technical reasons rather than intrinsic protein properties. The Matthews coefficient (a measure of crystal packing density) provides modest but consistent predictive value.</p><p>This feature importance analysis directly informs future feature engineering decisions — for instance, suggesting that di-peptide frequency features or secondary structure composition estimates (derivable from sequence using tools such as PSIPRED) might yield further accuracy gains.</p><p><strong>6.4 Class Distribution and SMOTE Effects</strong><br>Pre-SMOTE, the training set class distribution spans from several hundred samples (rare families) to tens of thousands (hydrolases). Post-SMOTE with the 6,000-sample cap, all classes in the training set are represented by at most 6,000 samples, substantially reducing the distributional skew. The agent’s inspect_class_distribution tool allows the LLM to observe and comment on this balance, providing a natural language summary of the oversampling effect in the final report — a feature that has no direct analog in the classical baseline, where the researcher must manually interpret the distribution.</p><p>It is important to note that SMOTE’s efficacy on high-dimensional sequence-composition feature spaces is less well-characterized than on lower-dimensional tabular data. Synthetic samples generated by linear interpolation between k-nearest neighbors in a 32-dimensional space may not correspond to physically plausible protein compositions. This represents a known limitation of the approach, discussed further in Section 7.</p><h4>7. Deployed Prediction Interface</h4><p>A key deliverable of this project is a Gradio-based web application (app.py) that serves the trained classifier as an interactive prediction tool. The application is designed for accessibility: it requires no OpenAI API key, no Python expertise, and no understanding of the underlying model mechanics. It loads the persisted model artifacts from the model/ directory at startup and exposes a simple input form (Figure 3).</p><p><strong>7.1 Input Interface</strong><br>The application accepts all input fields as optional, making it maximally flexible for partial-information queries common in early-stage drug discovery. The input fields map directly to the feature schema described in Section 4:</p><ul><li>chain_sequences: a JSON dictionary of chain identifiers to amino acid sequences. When provided, all 27 sequence composition features are computed automatically. When absent, sequence features default to zero.</li><li>resolution, structure_molecular_weight, density_matthews, density_percent_sol, ph_value: optional numeric physicochemical fields. Provided values replace zero defaults.</li></ul><p>A data completeness indicator reports the fraction of features populated by the user’s input, allowing the user to assess the expected reliability of the prediction. The system performs well on partial inputs, owing to the Random Forest’s tolerance for zero-imputed features, but explicitly communicates uncertainty when data completeness is low.</p><p><strong>7.2 Output Interface</strong><br>The application produces four outputs for each prediction: the top-predicted protein class with its probability estimate; a confidence classification (High: &gt;70%, Medium: 40–70%, Low: &lt;40%) derived from the maximum class probability; a textual data completeness indicator; and a bar chart displaying the top-5 class probability distribution. The chart provides essential context for borderline predictions — a top probability of 55% with a second-ranked class at 40% carries very different implications from a top probability of 92%.</p><p>For public sharing or collaborative use, the demo.launch(share=True) option generates a Gradio-hosted URL valid for 72 hours — sufficient for demonstration in a team meeting</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*TCIYM3q5-R7xT_VndfGo4A.png" /><figcaption><em>Figure 3. Deployment architecture for Gradio UI, including one-time model training (orange) and prediction serving (green).</em></figcaption></figure><h4>8. Discussion</h4><p><strong>8.1 When is Agentic ML Advantageous?</strong><br>The results of this study suggest that agentic ML pipelines offer their greatest advantage not in raw predictive accuracy — which is primarily determined by feature quality and model architecture — but in the elimination of manual workflow bottlenecks. In a pharmaceutical research context, where the cost of a senior data scientist’s time is substantial and time-to-decision timelines are critical, a system that can autonomously train, evaluate, interpret, and document a classifier in under two minutes (excluding model download time) has significant practical value.</p><p>The agentic approach also confers an implicit quality assurance benefit: the LLM is instructed to evaluate the model on multiple splits and to run cross-validation, enforcing a thoroughness that a manually driven analyst under time pressure might abbreviate. The tool call log provides an auditable record of every evaluation decision — which split was used, how many CV folds were run, which features were ranked — enabling reproducibility and compliance documentation that classical notebooks do not automatically provide.</p><p><strong>8.2 Limitations and Risks</strong><br>Several important limitations warrant careful attention before deploying agentic ML systems in regulated biotech environments.</p><p><strong>LLM non-determinism. </strong>GPT-4o’s tool calling decisions are temperature-dependent and may vary across runs even with identical inputs. While the fixed random_state=50 ensures Random Forest reproducibility, the sequence and configuration of tool calls may differ between runs, producing subtly different reports and evaluation orderings. Organizations requiring bit-exact reproducibility must either fix the LLM’s seed (where supported) or treat the tool call log as the authoritative record.</p><p><strong>API cost and dependency. </strong>The agent node requires a valid OpenAI API key with sufficient credits. At GPT-4o pricing, a full training run consuming 12 tool-call iterations costs approximately $0.05–0.20 depending on prompt and completion lengths. For development and cost-sensitive environments, substituting gpt-4o-mini reduces cost by approximately 15x at a modest accuracy penalty. Organizations with strict data residency requirements may prefer to substitute a locally hosted LLM (e.g., via Ollama or vLLM) as the planning agent.</p><p><strong>SMOTE in high-dimensional sequence space. </strong>As noted in Section 5.4, linear interpolation between nearest neighbors in a 32-dimensional amino acid composition space may generate synthetic samples that do not correspond to biologically realistic sequences. This is a fundamental limitation of feature-space oversampling for structured biological data. Future work should explore alternatives such as sequence-level augmentation (e.g., conservative amino acid substitutions guided by BLOSUM matrices) or generative model-based oversampling.</p><p><strong>Sequence features without alignment. </strong>The 27 amino acid composition features capture residue frequencies but discard positional and ordering information. Two proteins with identical amino acid compositions but different sequences will produce identical feature vectors, masking potentially important structural differences. Incorporating n-gram or k-mer features, secondary structure predictions, or learned protein language model embeddings (e.g., from ESM-2 or ProtTrans) would substantially enrich the feature representation.</p><p><strong>Class scope. </strong>Filtering to the top 20 protein classes excludes a long tail of rare but potentially therapeutically important families. The configurable TOP_N_CLASSES parameter allows this scope to be expanded at the cost of longer training time and greater class imbalance challenges. Future work should explore hierarchical classification schemes that handle rare classes through dedicated sub-classifiers or few-shot learning approaches.</p><p><strong>8.3 Implications for Drug Discovery Workflows</strong><br>The most immediate application of this system in a drug discovery context is target deorphanization — the process of assigning a functional classification to a protein target with unknown biology. Given a protein’s sequence (sufficient to compute the 27 composition features), the Gradio application provides a rapid first-pass classification that directs subsequent experimental and computational investigation. A high-confidence prediction of ‘kinase’ or ‘GPCRs’ immediately suggests an established druggability assessment framework and a relevant chemogenomics database for hit identification.</p><p>A second application is dataset curation quality control: when assembling a training set for a property prediction model (e.g., binding affinity prediction), researchers often need to verify that their protein targets span the desired structural diversity. The classifier can rapidly annotate a large set of protein entries, enabling automated diversity analysis and ensuring that the training set is not inadvertently biased toward a single structural family.</p><p>Looking further ahead, the agentic architecture established here is extensible to more complex drug discovery tasks: virtual screening pipeline management, where the LLM orchestrates docking, ADMET prediction, and synthesis feasibility assessment in sequence; multi-modal data integration, where the agent combines sequence-based features with graph neural network embeddings from 3D structure; and active learning loops, where the agent iteratively queries wet-lab experimental results to refine the model in real time.</p><h4>9. Conclusion</h4><p>This paper presented an agentic machine learning pipeline for protein classification in drug discovery, in which GPT-4o autonomously orchestrates a LangGraph state graph to train, evaluate, and document a Random Forest classifier on PDB-derived structural data. The system achieves an out-of-bag accuracy of 87.12% across 20 protein classes — on par with a classical, manually designed baseline — while automating approximately 90 minutes of analyst workflow and producing a richer, more structured audit trail.</p><p>The core contribution is not a new machine learning algorithm or a new feature representation, but a new operational paradigm: one in which the language model acts as a competent scientific collaborator, capable of reasoning about evaluation strategy, identifying appropriate tools for each stage of the pipeline, and synthesizing results into actionable conclusions. As LLM capabilities continue to advance and tool ecosystems mature, this paradigm is poised to fundamentally reshape the role of the data scientist in computational drug discovery — shifting focus from pipeline construction and execution to pipeline design, validation, and scientific interpretation.</p><p>The full codebase, trained model artifacts, and Gradio prediction application are available on <a href="https://github.com/paul-london/Comparing-Classical-ML-vs-Agentic-ML-for-Protein-Classification-in-Drug-Discovery">GitHub</a>. The project is designed for straightforward reproduction: a new user can clone the repository, install the listed dependencies, provide an OpenAI API key and a PDB-derived CSV, and have a trained, evaluated, and documented protein classifier running within minutes. It is the author’s hope that this system serves both as a practical tool for biotech researchers and as a template for future agentic ML applications across the biomedical sciences.</p><h4><strong>References</strong></h4><p>[1] <a href="https://doi.org/10.1093/nar/28.1.235">Berman, H. M., et al. (2000). The Protein Data Bank. Nucleic Acids Research, 28(1), 235–242.</a><br>[2] <a href="https://doi.org/10.1023/A:1010933404324">Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32.</a><br>[3] <a href="https://doi.org/10.1613/jair.953">Chawla, N. V., et al. (2002). SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16, 321–357.</a><br>[4] <a href="https://arxiv.org/abs/2210.03629">Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023.</a><br>[5] <a href="https://langchain-ai.github.io/langgraph/">LangChain AI. (2024). LangGraph: Building Stateful Multi-Actor Applications with LLMs. Retrieved from https://langchain-ai.github.io/langgraph/</a><br>[6] <a href="https://openai.com/index/hello-gpt-4o/">OpenAI. (2024). GPT-4o Technical Report. OpenAI Blog.</a><br>[7] <a href="https://doi.org/10.1126/science.ade2574">Lin, Z., et al. (2023). Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637), 1123–1130. [ESM-2]</a><br>[8] <a href="https://doi.org/10.1109/TPAMI.2021.3095381">Elnaggar, A., et al. (2022). ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning. IEEE TPAMI, 44(10), 7112–7127.</a><br>[9] <a href="https://doi.org/10.1038/s41586-021-03819-2">Jumper, J., et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596, 583–589.</a><br>[10] <a href="https://doi.org/10.1039/d0cs00098a">Muratov, E. N., et al. (2020). QSAR without borders. Chemical Society Reviews, 49(11), 3525–3564.</a><br>[11] <a href="https://doi.org/10.1038/s41467-024-47613-w">Atz, K., et al. (2022). Prospective de novo drug design with deep interactome learning. Nature Communications, 13, 3408.</a><br>[12] <a href="https://doi.org/10.25080/Majora-92bf1922-00a">Wes McKinney. (2010). Data Structures for Statistical Computing in Python. Proceedings of the 9th Python in Science Conference.</a><br>[13] <a href="https://jmlr.org/papers/v12/pedregosa11a.html">Pedregosa, F., et al. (2011). Scikit-learn: Machine Learning in Python. JMLR, 12, 2825–2830.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f25ed795192" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developing and Deploying Hybrid AI Clinical Reasoning Systems]]></title>
            <link>https://ernest-bonat.medium.com/developing-and-deploying-hybrid-ai-clinical-reasoning-systems-d5afffb56583?source=rss-58ededb6e465------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5afffb56583</guid>
            <dc:creator><![CDATA[Ernest Bonat, Ph.D.]]></dc:creator>
            <pubDate>Wed, 22 Apr 2026 16:35:09 GMT</pubDate>
            <atom:updated>2026-06-26T14:18:45.740Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*tdKpiaAmzzrxxVvabzAIFA.png" /></figure><p>Ernest Bonat, Ph.D.</p><p>1. <a href="#edd7">Introduction</a><br>2. <a href="#d6bf">System Architecture</a><br>3. <a href="#d6bf">The Deterministic Rule Engine</a><br>4. <a href="#02f1">Key Code Walkthrough</a><br>5. <a href="#91a5">HL7 FHIR R4 Data Ingestion</a><br>6. <a href="#0f00">The Validation Audit Log Node</a><br>7. <a href="#0157">Why SQLite is the right choice?</a><br>8. <a href="#586e">The Escalation Sub-Graph</a><br>9. <a href="#abe7">Results &amp; Analysis</a><br>10. <a href="#d307">Design Principles</a><br>11. <a href="#51ff">Conclusions</a></p><h4>1. Introduction</h4><p>The promise of AI in clinical settings is compelling: faster triage, earlier risk detection, reduced cognitive load on overworked clinicians. But AI systems that produce opaque outputs or that can hallucinate critical medical values are a liability, not an asset. The field has converged on a pattern: keep deterministic logic deterministic, and use language models only for what they are good at — communication.</p><p>LangGraph, Anthropic’s graph-based agent orchestration library, gives us exactly the right primitive for this separation. Nodes in a LangGraph graph are isolated: they read from shared state, produce outputs, and cannot reach back to modify what came before them. This paper walks through a complete implementation of a cardiovascular risk assessment engine built on these principles.</p><p>The system now accepts two input formats: flat CSV files and HL7 FHIR R4 Bundles — the international standard for structured clinical data exchange. A smart auto-detect loader selects the correct ingestion path based on file extension. Urgent patients are routed into a dedicated escalation sub-graph that fires triage assessment, alert dispatch, and audit logging, bypassing the LLM narrative layer. Every design decision priorities safety: <strong><em>the LLM can never alter a risk score, never bypass validation, and never appear in the final output as an authoritative clinical source.</em></strong></p><h4>2. System Architecture</h4><p>The system now has two distinct paths through the graph, selected at runtime by a conditional edge function after the scoring engine completes:</p><ol><li><strong>Input Layer — CSV</strong> — load_patients_from_csv(): 14 required columns, bool normalization, BOM handling.</li><li><strong>Input Layer — FHIR R4</strong> — load_patients_from_fhir(): parses Patient, Observation (LOINC), Condition (SNOMED) resources.</li><li><strong>load_patients()</strong> — Auto-detect loader: .csv → CSV path, .json → FHIR path. Same output either way.</li><li><strong>csv_to_fhir_bundle()</strong> — Optional export: converts CSV → standards-compliant FHIR R4 Bundle (100 resources for 10 patients).</li><li><strong>validate_node</strong> — Enforces range/type/enum constraints. Sets validation_passed flag — does not halt the graph.</li><li><strong>engine_node</strong> — Evaluates all 9 rules, computes score and risk tier, sets escalation_triggered flag.</li><li><strong>route_after_engine()</strong> — Conditional edge: returns escalate_triage if urgent, explain otherwise.</li><li><strong>Escalation sub-graph </strong>— escalate_triage → escalate_notify → escalate_log → escalation_final. LLM bypassed.</li><li><strong>Standard path </strong>— explain_node (GPT) → final_node (output gate). Used for all non-urgent patients.</li><li><strong>audit_log_node</strong> — Runs on BOTH paths. Persists full rule trace + outcome to SQLite — every patient, every run.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IvQ2z1WKnL6b0aWceB6MQw.png" /><figcaption>Figure 1. Complete LangGraph pipeline v3 — conditional routing after engine_node, plus audit_log_node running on both paths to persist every outcome to SQLite.</figcaption></figure><h4>3. The Deterministic Rule Engine</h4><p>The engine evaluates 9 clinical rules drawn from established cardiovascular risk frameworks. Each rule is a pure function: given a normalized patient dict, it returns a tuple of (passed: bool, message: str, points: int). Rules are evaluated in a fixed, documented order with no branching between them — every patient receives every rule evaluation on every run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/1*Gpa-8rUcwj2_6IccMgOPRA.png" /><figcaption>Table 1. All 9 deterministic rules, their point values, and trigger conditions.</figcaption></figure><p><strong>Risk Tier Thresholds</strong><br>Scores aggregate linearly. An urgent override fires independently of total score:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*tFAGafbRHoVbyxOrOvO2JQ.png" /><figcaption>Table 2. Risk tier classification thresholds.</figcaption></figure><h4>4. Key Code Walkthrough</h4><p><strong>4.1 Defining a Rule</strong><br>Each rule is a plain Python function — no classes, no inheritance. The RULES list drives evaluation order and makes adding a new rule a one-liner:</p><pre># each rule: (rule_id, name, function)<br>def _r002(x):<br>    ok = x[&#39;systolic_bp&#39;] &gt;= 160 or x[&#39;diastolic_bp&#39;] &gt;= 100<br>    msg = (f&quot;BP {x[&#39;systolic_bp&#39;]:.0f}/{x[&#39;diastolic_bp&#39;]:.0f} &quot;<br>           + (&#39;meets&#39; if ok else &#39;below&#39;) + &#39; severe threshold.&#39;)<br>    return ok, msg, 3  # (passed, message, points)<br><br>RULES = [<br>    (&#39;R001&#39;, &#39;Prior cardiovascular disease&#39;, _r001),<br>    (&#39;R002&#39;, &#39;Severe hypertension&#39;,          _r002),<br>    # ... 7 more rules<br>]</pre><p><strong>4.2 The Conditional Edge Router</strong><br>The routing function is a plain Python function that returns the name of the next node. LangGraph calls it after engine_node and follows the string it returns — no other branching code required:</p><pre>def route_after_engine(state: ClinicalState) -&gt; str:<br>    &quot;&quot;&quot;<br>    Conditional edge function.<br>    Called by LangGraph after engine_node.<br>    Returns the name of the next node to execute.<br>    &quot;&quot;&quot;<br>    if state.get(&#39;escalation_triggered&#39;):<br>        return &quot;escalate_triage&quot;   # → urgent sub-graph<br>    return &quot;explain&quot;              # → standard path<br><br># engine_node sets this flag:<br>def engine_node(state: ClinicalState) -&gt; ClinicalState:<br>    result = to_dict(run_engine(state[&#39;raw_input&#39;]))<br>    return {<br>        &quot;deterministic_result&quot;: result,<br>        &quot;escalation_triggered&quot;: result[&quot;risk_level&quot;] == &quot;urgent&quot;,<br>    }</pre><p><strong>4.3 Escalation Sub-graph Nodes</strong><br>Three nodes chain in sequence inside the urgent pathway. Each has a single, auditable responsibility:</p><pre>def escalate_triage_node(state):<br>    # Identifies WHY the patient is urgent:<br>    # CHEST_PAIN, HYPERTENSIVE_CRISIS, PRIOR_CVD flags.<br>    # Assigns P1-IMMEDIATE or P2-URGENT within the urgent tier.<br>    return {&quot;escalation_triage&quot;: { &quot;triage_priority&quot;: ..., &quot;severity_flags&quot;: ... }}<br><br>def escalate_notify_node(state):<br>    # Builds a dispatch-ready alert record with unique alert_id.<br>    # In production: POST to pager / EHR alert queue / Slack.<br>    return {&quot;escalation_alert&quot;: { &quot;alert_id&quot;: ..., &quot;channels&quot;: [...] }}<br><br>def escalate_log_node(state):<br>    # Writes immutable audit trail: rules fired, pipeline version,<br>    # alert_id, policy note confirming LLM was intentionally bypassed.<br>    return {&quot;escalation_log&quot;: { &quot;log_id&quot;: ..., &quot;audit_note&quot;: ... }}</pre><p><strong>4.4 Building the Full Graph</strong><br>add_conditional_edges() is the single new call that wires the branch. Everything else follows the same pattern as before:</p><pre>def build_graph():<br>    g = StateGraph(ClinicalState)<br><br>    # standard path<br>    g.add_node(&quot;validate&quot;, validate_node)<br>    g.add_node(&quot;engine&quot;, engine_node)<br>    g.add_node(&quot;explain&quot;, explain_node)   # LLM<br>    g.add_node(&quot;finalize&quot;, final_node)<br><br>    # sscalation sub-graph<br>    g.add_node(&quot;escalate_triage&quot;, escalate_triage_node)<br>    g.add_node(&quot;escalate_notify&quot;, escalate_notify_node)<br>    g.add_node(&quot;escalate_log&quot;, escalate_log_node)<br>    g.add_node(&quot;escalation_final&quot;, escalation_final_node)<br><br>    g.add_edge(START, &quot;validate&quot;)<br>    g.add_edge(&quot;validate&quot;, &quot;engine&quot;)<br><br>    # conditional branch<br>    g.add_conditional_edges(<br>        &quot;engine&quot;,<br>        route_after_engine,<br>        { &quot;explain&quot;: &quot;explain&quot;, &quot;escalate_triage&quot;: &quot;escalate_triage&quot; },<br>    )<br><br>    # standard path continuation<br>    g.add_edge(&quot;explain&quot;, &quot;finalize&quot;)<br>    g.add_edge(&quot;finalize&quot;, END)<br><br>    # escalation sub-graph chain<br>    g.add_edge(&quot;escalate_triage&quot;, &quot;escalate_notify&quot;)<br>    g.add_edge(&quot;escalate_notify&quot;, &quot;escalate_log&quot;)<br>    g.add_edge(&quot;escalate_log&quot;, &quot;escalation_final&quot;)<br>    g.add_edge(&quot;escalation_final&quot;, END)<br>    return g.compile()</pre><h4>5. HL7 FHIR R4 Data Ingestion</h4><p>The latest version of the pipeline replaces the single CSV loader with a dual-mode ingestion layer built around HL7 FHIR R4 — the international standard for structured clinical data exchange used by EHR systems, hospital APIs, and national health data networks worldwide. The system can now accept data from either a flat CSV file or a fully standards-compliant FHIR R4 Bundle without any change to the pipeline downstream.</p><p>The key design decision is that both paths produce identical output: a list of normalized patient dicts with the same 14 fields the engine has always expected. The LangGraph graph never needs to know which format the data came from. This is the open/closed principle applied to data ingestion — the pipeline is open to new input formats, closed to modification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*amVgouySp2DjH9YWB8Tyfw.png" /><figcaption>Figure 2. Dual ingestion layer — CSV and FHIR Bundle paths both resolve to the same pipeline-compatible patient dict. load_patients() auto-detects by file extension.</figcaption></figure><p><strong>FHIR Resource Types Used</strong><br>Each patient in the FHIR Bundle is represented by three resource types. Numeric measurements are encoded as Observations with LOINC codes — the globally recognized standard for laboratory and clinical observation identifiers. Clinical conditions (prior CVD, diabetes, symptoms) are encoded as Conditions with SNOMED CT codes. Smoking status uses a dedicated social-history Observation with its own LOINC code and SNOMED-coded value:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aPFfipDgo7KK1N1ef9xTiQ.png" /><figcaption>Figure 3. LOINC observation codes (left) and SNOMED CT condition/smoking codes (right) used in the FHIR Bundle. Reverse maps decode these back to pipeline fields on ingestion.</figcaption></figure><p><strong>How the auto-detect loader works</strong><br>One function, one line of config — the file extension drives everything:</p><pre>def load_patients(path: str) -&gt; List[Dict[str, Any]]:<br>    &quot;&quot;&quot;<br>    Smart loader — auto-detects by extension:<br>      .csv  → load_patients_from_csv()   (flat tabular format)<br>      .json → load_patients_from_fhir()  (FHIR R4 Bundle)<br>    Both return the same pipeline-compatible patient dict list.<br>    &quot;&quot;&quot;<br>    suffix = Path(path).suffix.lower()<br>    if suffix == &quot;.csv&quot;:<br>        return load_patients_from_csv(path)<br>    elif suffix == &quot;.json&quot;:<br>        return load_patients_from_fhir(path)<br>    else:<br>        raise ValueError(f&quot;Unsupported: {suffix}. Use .csv or .json (FHIR)&quot;)</pre><p><strong>Exporting CSV to a FHIR Bundle</strong><br>The entry point can also generate a FHIR Bundle from the CSV before running the pipeline. Set EXPORT_FHIR = True and the system writes patients.fhir.json alongside the input file. For 10 patients, the bundle contains 100 resources across three types:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*oyBEMeay679S-vTxqfIYVg.png" /><figcaption>Table 4. FHIR R4 Bundle composition for 10 patients — 100 resources total.</figcaption></figure><p><strong>Entry point — just one line to switch modes</strong></p><pre># EDIT THESE LINES in clinical_engine_llm.py<br>INPUT_FILE   = r&quot;C:\Users\You\patients.csv&quot;    # .csv  OR  .json (FHIR)<br>OUTPUT_FILE  = r&quot;C:\Users\You\clinical_report.docx&quot;<br>EXPORT_FHIR  = True   # True → auto-exports patients.fhir.json when input is CSV<br><br># load_patients() will auto-detect the format and call the right loader.<br># The rest of the pipeline is completely unchanged.</pre><h4>6. The Validation Audit Log Node</h4><p>The audit_log_node is the newest addition to the pipeline and its most operationally important one. It runs on every patient — urgent and non-urgent, validated and rejected — making it the single source of truth for what the system decided, when, and why. The node writes to a local SQLite database using two tables with a foreign-key relationship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*KVhZ1XyvWDFLuShH8PXsrg.png" /><figcaption>Figure 4. SQLite schema — audit_runs holds one row per patient per run; audit_rule_hits holds one row per triggered rule, linked by run_id foreign key.</figcaption></figure><h4>7. Why SQLite is the right choice?</h4><p>SQLite is appropriate for this program when:</p><ol><li>Running in a single-institution setting — one hospital, one clinic, one research group</li><li>Processing up to roughly 10 million patients in total lifetime records</li><li>Concurrent writers are rare — SQLite uses file-level locking, so simultaneous pipeline runs from multiple processes can cause contention (though WAL mode, which the framework already uses, mitigates this significantly)</li><li>The database lives on local or NAS storage — SQLite is a file, not a server</li></ol><p>For the clinical program as currently built — running thousands to hundreds of thousands of patient records across multiple clinical datasets — SQLite is entirely adequate. The database currently holds 40 patient runs in about 150 KB. At that rate you could run 1 million patients and the database file would be around 4 GB, well within SQLite’s capabilities and easily manageable on any modern server. The same database interface would target PostgreSQL or a managed database — only the connection string changes, not the node code logic.</p><p><strong>What gets persisted?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*X97a27duYICpDdqKXHrTrA.png" /><figcaption>Table 5. Key fields persisted to audit_runs on every pipeline execution.</figcaption></figure><p><strong>Wiring audit_log_node into both paths</strong><br>LangGraph allows the same function to be registered as multiple nodes under different names. This means one audit function covers both paths with no code duplication:</p><pre># same function registered under two node names<br>g.add_node(&quot;audit_log&quot;, audit_log_node)   # standard path<br>g.add_node(&quot;audit_log_esc&quot;, audit_log_node)   # escalation path<br><br># standard path: ... → finalize → audit_log → END<br>g.add_edge(&quot;finalize&quot;, &quot;audit_log&quot;)<br>g.add_edge(&quot;audit_log&quot;, END)<br><br># escalation path: ... → escalation_final → audit_log_esc → END<br>g.add_edge(&quot;escalation_final&quot;, &quot;audit_log_esc&quot;)<br>g.add_edge(&quot;audit_log_esc&quot;, END)</pre><p><strong>Live results from the database</strong><br>After running the 10-patient batch, the audit database contained 10 rows in audit_runs and 33 rows in audit_rule_hits — one per triggered rule across all validated patients. The rule frequency table, queried directly from SQLite, reveals which clinical risk factors are most prevalent in this cohort:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*R6GrEeKzsOW2RMbC8SXJ_Q.png" /><figcaption>Figure 5. Rule hit frequency from audit_rule_hits — Elevated LDL triggered in 5 of 8 validated patients; Concerning Symptoms contributed the most total points (12).</figcaption></figure><p>This kind of population-level analysis — impossible without a persistent audit store — is exactly what regulators, clinical audit committees, and quality improvement teams need. Every decision the engine has ever made is queryable with standard SQL.</p><p>The complete audit_runs + audit_rule_hits database is included as clinical_audit.db. Open it with any SQLite browser (DB Browser for SQLite is free) or query it directly with Python’s built-in sqlite3 module.</p><h4>8. The Escalation Sub-Graph</h4><p>When risk_level equals ‘urgent’, the conditional edge skips the LLM entirely and routes the patient through three dedicated escalation nodes. This is a deliberate architectural choice: urgent patients need fast, structured, machine-readable output — not a narrative. The sub-graph produces three new state keys that appear in the final output under escalation_detail.</p><p><strong>Why bypass the LLM for urgent cases?</strong><br>Language models are non-deterministic and can vary their outputs across runs. For a P1-IMMEDIATE patient — chest pain, prior CVD, systolic above 180 — the action is not ‘generate prose’. The action is: classify the trigger, dispatch an alert with a traceable ID, and write an audit record. All three are pure data operations that must be identical on every run.</p><p>The LLM narrative layer is intentionally absent from the escalation pathway. Urgent outputs contain a structured triage record, a dispatched alert ID, and a complete audit log. The agent_explanation field is replaced with a policy message confirming the bypass was deliberate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*2szfrgnXr271TQHUPF4Tfg.png" /><figcaption>Table 6. Triage priority classification within the URGENT tier.</figcaption></figure><h4>9. Results &amp; Analysis</h4><p>The updated system was tested against the same 10-patient batch. The conditional routing worked as expected: PT-01 and PT-05 were diverted to the escalation sub-graph; PT-02, PT-06, and PT-09 (HIGH risk but no urgent trigger) followed the standard path. The two validation failures (PT-08 age error, PT-10 invalid smoking status) were rejected before reaching either pathway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*nLS3aILww8mNGA6zB1XAJw.png" /><figcaption><em>Figure 6. Left: risk distribution across 10 patients. Right: pipeline pathway routing — 2 escalated, 6 standard, 2 rejected</em></figcaption></figure><p>Both URGENT patients received P1-IMMEDIATE triage priority: PT-01 due to chest pain combined with prior CVD (score 19/19, all 9 rules triggered), and PT-05 due to chest pain combined with a systolic of 185 mmHg (score 11/19). Alert IDs and audit log IDs were generated for each escalation, timestamped to the millisecond.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*79rrCWL5dtmO9lIx8Gc88Q.png" /><figcaption>Table 7. Full results for all 10 patients — now showing pipeline pathway alongside risk and score.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*-7bYIlsCXimDT_rw7o6L0w.png" /><figcaption>Table 7. Full results for all 10 patients — now showing pipeline pathway alongside risk and score.</figcaption></figure><h4>10. Design Principles</h4><ol><li><strong>Separation of concerns via graph nodes</strong><br>No single function is responsible for validation AND scoring AND explanation. Each node has exactly one job. This makes the system independently testable at every stage.</li><li><strong>The LLM is advisory, never authoritative </strong><br>The explain_node can only write to explanation_text. The final_node’s policy_note field makes the boundary explicit in the output itself. This is not convention — it is enforced by state schema.</li><li><strong>Validation failures are soft, not hard</strong><br>The graph does not halt on invalid input. The validation_passed flag propagates, allowing downstream nodes to gracefully handle the failure and still produce a well-structured rejected output.</li><li><strong>Deterministic reproducibility</strong><br>Given the same input, the engine will produce the exact same score, risk tier, and action list — every time. The LLM layer introduces variability only in the narrative text, which is clearly labelled as commentary.</li><li><strong>Universal audit coverage</strong><br>audit_log_node runs on every patient, every time — both paths, including rejections. The same function is registered under two node names in the graph. There is no patient that can pass through without creating a database record.</li><li><strong>Standard-agnostic ingestion</strong><br>The load_patients() function accepts CSV or HL7 FHIR R4 Bundle input. The pipeline never knows which format was used. Adding a new source (HL7 v2, CDA, JSON-LD) requires one new loader function — nothing else changes.</li><li><strong>One-line extension</strong><br>Adding a new clinical rule requires: one pure function + one tuple in the RULES list. The loop handles everything else.</li></ol><h4>11. Conclusions</h4><p>This article has traced the evolution of a single self-contained clinical risk engine from a flat Python script to a production-grade, standards-compliant, multi-modal AI pipeline. At each step, a deliberate architectural choice — not a technical accident — determined what the system could and could not do. This final section gathers those choices, distills the lessons learned, and places the work in the wider context of clinical AI engineering.</p><p><strong>What we built — a full-stack summary</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*4FEh3BmIL8h6QTJTlvVU1g.png" /><figcaption>Table 8. Complete feature history — every capability and the architectural decision behind it.</figcaption></figure><p><strong>The core argument, restated</strong><br>The central thesis of this paper is not about LangGraph, FHIR, or cardiovascular risk scores. It is about a pattern: in high-stakes domains, language models and deterministic engines are not competitors — they are complementary. The engine decides. The LLM explains. The graph enforces the boundary between them. This separation is not merely good engineering practice; in regulated industries, it may become a legal requirement.<br>Every clinical AI failure mode can be traced to a violation of this pattern. Systems that allow an LLM to generate a risk score confuse plausibility with correctness. Systems that produce opaque outputs with no trace frustrate audit. Systems that ingest free form text instead of structured data introduce ambiguity at the source. This pipeline addresses all three. The score comes from a deterministic function. The reasoning is logged to the row level. The data arrives as LOINC-coded observations and SNOMED-coded conditions.</p><p><strong>Five Lessons Learned</strong></p><p>1. <strong>State schema is your contract</strong> — ClinicalState is not just a data container — it is an enforced contract. Every node declares what it reads and what it writes. A node that tries to write to deterministic_result after it has been set by the engine will simply be ignored by LangGraph’s state merging. The schema makes cheating impossible.</p><p>2. <strong>Conditional edges cost almost nothing</strong> — Adding the escalation sub-graph required one new function (route_after_engine), four new nodes, and one add_conditional_edges() call. The rest of the graph was unchanged. This is LangGraph’s most underused feature — branches are free.</p><p>3. <strong>The audit node is the most important node</strong> —<strong> </strong>Of all the nodes in the graph, audit_log_node is the one that matters most in production. Not because it does anything clinical, but because it is the node that makes every other node accountable. A pipeline without an audit trail is a pipeline no regulator will approve.</p><p>4. <strong>FHIR is the right target, not the right starting point</strong> — Building FHIR support before the core engine was stable would have been a mistake. Starting with CSV allowed rapid iteration on the rule logic. Adding FHIR ingestion last — as a translation layer that produces the same patient dict format — meant zero disruption to the pipeline. Build the interface last.</p><p>5. <strong>Visualization is not optional</strong> — The Streamlit front-end is not cosmetic. Watching the node animation in real time reveals behavior that is invisible in log files: which patients hit the conditional branch, which nodes are skipped, how long each node takes. If you cannot observe your pipeline, you cannot debug it.</p><p><strong>Applicability beyond healthcare</strong><br>Every component in this pipeline is domain-agnostic. Replace the 9 cardiovascular rules with credit-risk rules, regulatory compliance checks, or fraud detection heuristics — the LangGraph graph, the audit schema, the FHIR ingestion pattern, and the Streamlit visualization all remain unchanged. The pattern scales.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*Vg4cuIEyQIHoEiavgPY_uw.png" /><figcaption>Table 9. The deterministic-engine + LangGraph pattern applied to five domains beyond healthcare.</figcaption></figure><p><strong>A closing thought</strong><br>The most dangerous moment in any AI project is when an engineer looks at a language model’s output and thinks: this is good enough to ship. It very often is — for the input they tested. The clinical engine described in this paper is built on the premise that ‘good enough on average’ is insufficient when the edge cases are patients. Every rule is explicit. Every decision is logged. Every urgent patient fires a structured alert that cannot be hallucinated. The LLM is present, useful, and constrained. That is the right balance.</p><p>LangGraph gives us the vocabulary to express that balance in code. Nodes, edges, state schemas, conditional branches, and audit sinks — these are not abstractions for their own sake. They are the engineering vocabulary of trustworthy AI. The more the industry adopts this vocabulary, the fewer medical, financial, and legal decisions will be made by systems that nobody can explain, audit, or hold accountable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5afffb56583" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Classification of Structural Protein Sequences for Drug Discovery]]></title>
            <link>https://ernest-bonat.medium.com/machine-learning-classification-of-structural-protein-sequences-for-drug-discovery-48d7057e43b0?source=rss-58ededb6e465------2</link>
            <guid isPermaLink="false">https://medium.com/p/48d7057e43b0</guid>
            <dc:creator><![CDATA[Ernest Bonat, Ph.D.]]></dc:creator>
            <pubDate>Fri, 23 Jan 2026 13:07:35 GMT</pubDate>
            <atom:updated>2026-04-23T01:17:46.234Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*8MeNxgrAaBwzmXioKFfXDA.jpeg" /></figure><p><strong>Paul London, M.S., Ernest Bonat, Ph.D.</strong></p><p>1. <a href="#af7f">Overview</a><br>2. <a href="#1c20">Methods</a><br>3. <a href="#1aa6">Exploratory Data Analysis</a><br>4. <a href="#0539">Pipeline 1: NLP<br></a>5. <a href="#0c4c">Pipeline 2: LSTMs</a><br>6. <a href="#8c6b">Pipeline 3: LLMs</a><br>7. <a href="#2a75">Summary of Results</a><br>8. <a href="#aa7c">Discussion</a></p><h4>1. Overview</h4><p>Advances in drug discovery and protein engineering increasingly rely on computational models to analyze, classify, and predict protein behavior. Complex interactions between proteins and prospective drug candidates are time-consuming and expensive to manually explore. <em>In silico</em> studies are much cheaper, faster, and more efficient for narrowing down candidates that merit further experimental review.</p><p>Our work explores how modern Machine Learning (ML) techniques — particularly Natural Language Processing (NLP), Long Short-Term Memory (LSTM), and Large Language Models (LLM) — can be applied to amino acid sequences as if they were “biological text” using their one letter codes (Appendix 1). By reframing proteins as language, we can leverage powerful sequence-based models to extract features, identify patterns, and generate representations that support downstream tasks such as classification or function prediction. This work began as an attempt to build a lightweight, GPU-friendly workflow for protein analysis and model development, while maintaining scientifically meaningful accuracy.</p><p>Here we will provide a brief introduction to structural proteomics to establish some key terminology. Proteins are composed of one or more chains of amino acid residues. The specific order of these residues is known as the primary structure. Depending on this sequence and the chain’s three-dimensional arrangement, local structural patterns such as folds, helices, and sheets can form, which are referred to as secondary structures. The overall three-dimensional shape of a single protein chain, incorporating these elements, is called the tertiary structure. Finally, when a protein consists of multiple chains, their combined arrangement is referred to as the quaternary structure.</p><h4>2. Methods</h4><h4>2.1 Data Acquisition</h4><p>Data were obtained from Kaggle (<a href="https://www.kaggle.com/datasets/shahir/protein-data-set">https://www.kaggle.com/datasets/shahir/protein-data-set</a>), which were originally obtained from Research Collaboratory for Structural Bioinformatics (RCSB) Protein Data Bank (PDB) (<a href="https://www.rcsb.org/pdb/">https://www.rcsb.org/pdb/</a>). They consist of two separate files: protein metadata and the with amino acid sequences for all chains of the protein (Appendix 2, 3).</p><p>The PDB archive is a repository of atomic coordinates and other information describing proteins and other important biological macromolecules. Structural biologists use methods such as X-ray crystallography, NMR spectroscopy, and cryo-electron microscopy to determine the location of each atom relative to each other in the molecule. They then deposit this information, which is then annotated and publicly released into the archive by the PDB.</p><p>The constantly-growing PDB is a reflection of the research that is happening in laboratories worldwide. This can make it both exciting and challenging to use the database in research and education. Structures are available for many of the proteins and nucleic acids involved in the central processes of life. The PDB archive also contains structures for ribosomes, oncogenes, drug targets, and even whole viruses.</p><p>However, it can often be a challenge to find relevant information, since the PDB archives so many different structures. You will often find multiple structures for a given molecule, or partial structures, or structures that have been modified or inactivated from their native form. Therefore, significant filtering and preprocessing of the data is required.</p><h4>2.2 Data Preprocessing</h4><p>We utilized multiple modeling approaches to compare performance across architectures. Therefore, the data were preprocessed according to the requirements of each method. In all cases, the input variable (feature) is the amino acid sequence itself, while the output variable (target) is the functional classification of the protein.</p><p>The overall preprocessing workflow is below:</p><p><strong>General</strong><br> 1. Merge both CSVs (metadata and sequences) on ‘structure id’ column<br> 2. Filter to only include macromolecule type ‘protein’ (not DNA or RNA)<br> 3. Drop rows with missing sequence or classification data<br> 4. Rename columns for consistency</p><p><strong>Target (Protein Classification)</strong><br> 5. Filter to only include the top 20 most represented protein classes<br> 6. Check for uniqueness of classes with FuzzyWuzzy</p><p><strong>Feature (Protein Sequence)</strong><br> 7. Add column for sequence length (for EDA)<br> 8. Reduce sequences to just the 20 standard amino acids using ‘X’ as a placeholder (Figure 1).<br> 9. Filter for sequences that are 20–1024 amino acids in length<br> 10. Add column ‘combined sequence’ that concatenates all chains of each protein<br> 11. Remove duplicate ‘structure id’ rows created by Step 10</p><p><strong>Modeling</strong><br> 12. Assign feature to X and target to y<br> 13. Split dataset into training (70%), validation (15%), and test (15%) sets<br> 14. Label encode y<br> 15. Pipeline-specific preprocessing of X (see below)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*WHNj-jaK1DOOOeoVDlf-FQ.png" /><figcaption>Figure 1. Code snippet outlining protein sequence (feature) data preprocessing steps 8 to 11.</figcaption></figure><p>General preprocessing included merging both CSVs on structure id, filtering to only include proteins, dropping rows missing either the target or feature, and renaming columns for consistency.<br>Next, target preprocessing included an assessment of how many sequences were representative of each protein classification (Figure 2). Thresholds of 50 classifications (green) and 200 sequences (red) were plotted to assist with selection. Ultimately, the top 20 most represented classes were utilized as the target to ensure adequate ML quality. Before modeling, the classes were label encoded.</p><p>Further, the top 20 classes were double checked for redundancy with FuzzyWuzzy, but appeared to all be distinct due to the curated nature of the dataset from RCSB. Protein classes with a slash (e.g. “HYDROLASE/HYDROLASE INHIBITOR”) can be interpreted as just hydrolase inhibitors or regulators — the dataset format treats this as a subcategory but it is still a distinct class of proteins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PfUtriEQ3tL87KTCqh2qJw.png" /><figcaption>Figure 2. Class rank versus number of sequences in dataset. Thresholds: Green = 50; Red = 200</figcaption></figure><p>Feature preprocessing included removing ambiguous residues beyond the primary 20 amino acids, handling variable sequence lengths by setting minimum and maximum limits (20–1024), and combining chains for proteins with multiple strands (quaternary structure). After this, the protein sequences were preprocessed by k-merization and vectorization (for NLP) or tokenization (for LSTM) their amino acid characters, converting them into numeric formats suitable for ML (see next section for pipeline-specific details). This step was not required for our LLM approach, as it uses embeddings computed from a pre-trained model.<br>Finally, the refined data were assigned to X and y variables, randomly split into training (70%), validation (15%), and test (15%) sets, and y was label encoded. The data were then used for modeling.</p><p><strong>Modeling</strong><br>For this study, we developed three parallel pipelines:</p><p><strong>Pipeline 1. NLP-based feature extraction using techniques commonly applied to text and tree-based models:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d3R5lumQw_ske1EeUdK0iw.png" /></figure><p>1.k-merization (k = 3)<br>2.Vectorization (CountVectorizer)<br>3.Synthetic Minority Over-sampling Technique (SMOTE)<br>4.Dimensionality reduction (Singular Value Decomposition (SVD))<br>5.LazyClassifier baseline (establish top 3 candidate models)<br>6.Hyperparameter tuning (Optuna) of top 3 models</p><p><strong>Pipeline 2. LSTM-based classification trained directly on sequences padded or truncated to consistent lengths</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R_kdQmZN9w0ryEYug-WlHQ.png" /></figure><p>1.Tokenization with suffix padding<br>2.Creation of Keras (TensorFlow) model architecture<br>3.Model training over 50–100 epochs<br>4.Review performance and tune parameters</p><p><strong>Pipeline 3. LLM-based sequence embeddings using a pre-trained transformer model designed for biological data, enabling downstream classification with minimal training</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*myzga6J8jZDGIZJrzIvIxA.png" /></figure><p>1.Generate embeddings from ESM-2 (esm2_t6_8M_UR50D, Evolutionary Scale Modeling 2, Meta AI)<br>2.Use embeddings as input for best NLP model (Random Forest)<br>3.Creation of more basic Keras model architecture<br>4.Use embeddings as input for model training over 50–100 epochs<br>5.Review and compare performance and tune parameters</p><p>Each pipeline takes preprocessed sequences and outputs classification results, allowing direct performance comparison across methods. Evaluation includes accuracy score, confusion matrix, and classification report.</p><h4>3. Exploratory Data Analysis</h4><p>After data processing, we performed several analyses to explore the protein sequence data and see what could be applied to future modeling projects.</p><p><strong>3.1 Sequence Length Distribution</strong></p><p>After inspecting sequence lengths, we saw a left-skewed distribution, with most being 200–250 residues long and very few being greater than 1000 (Figure 3). Before modeling, we applied a length filter (min = 20, max = 1024) to capture the most possible signal without overloading the models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*GcbaSuQ3lEYJN7fKNHSCiw.png" /><figcaption>Figure 3. Distribution of protein sequence lengths.</figcaption></figure><p><strong>3.2 Protein Classification Distribution</strong></p><p>After filtering out the top 20 protein classes, we can see that the data is dominated by hydrolases, transferases, and oxidoreductases, while the others are steadily represented less (Figure 4). This is the primary reason SMOTE (Pipeline 1) and class weighting (Pipelines 2 and 3) were used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*flqU6C7sd_Pe7zAiWMYmHw.png" /></figure><p><strong>3.3 Amino Acid Composition</strong></p><p>Beyond sequence-level analysis, which takes in to account the presence and order of amino acid residues, we can explore composition, that is, the amount of each residue present in the sequence. Further, by grouping amino acids into five distinct biochemical groups (Appendix 5) based on their properties, we can determine if there is any distinction between the protein classes (Figure 5).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q0Q82tNX1LByQ6nVHKPNRA.png" /><figcaption>Figure 5. Amino acid biochemical group frequency by protein class.</figcaption></figure><p>There were not many clear patterns or distinctions from this particular plot. It appears that sequence is much more complex and essential than merely composition, since the entire “vocabulary” is only 20 amino acids, therefore they are re-used greatly.</p><p><strong>3.4 Secondary Structure</strong></p><p>As discussed earlier, secondary structure is an important consideration in protein function. Two major motifs seen in nearly all proteins are α-helices and β-sheets. Both stabilized by hydrogen bonds, α-helices are coiled regions of a protein, forming a spring-like structure. β-sheets are flattened, pleated regions where strands of the protein run alongside each other. These elements are vital to protein function because they form the core structural elements of proteins, providing stability, shape, and scaffolding for the overall three-dimensional structure. They also help position amino acids in the right orientation for protein function, such as binding, catalysis, or forming interaction surfaces.</p><p>For our dataset, we chose to create a Logomaker figure (Figure 6) of the first 30 residues of the top 5 protein classes. This illustrates patterns within these sections of the sequences: Overlay color is propensity to form helices (red) or sheets (yellow) while letter color represents the biochemical group of the amino acid: nonpolar (green), polar (blue), acidic (red), basic (purple), or aromatic (orange).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oFmvT8QcNsAnRjBuEqoq2w.png" /><figcaption>Figure 6. Frequency of the most common amino acid residues within the first 30 positions for the top five protein classes. Letter colors indicate the biochemical group of each residue: nonpolar (green), polar (blue), acidic (red), basic (purple), and aromatic (orange). Overlay shading represents predicted secondary structure propensity: α-helices (red) and β-sheets (yellow).</figcaption></figure><p>As shown, there are differences, but this analysis is limited to just the first 30 residues — ML models will have to take in to consideration the entire sequence, including local and distant relationships, in order to provide a better method of classification from sequence alone.</p><p><strong>3.5 Biochemically Similar Protein Types</strong></p><p>We also constructed a PCA plot of the top 5 protein classes grouped according to their biochemical amino acid composition (Figure 7).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*vK0W9PmeSL4bDkXV-_L6_Q.png" /><figcaption>Figure 7. PCA plot of top 5 protein classes vs. biochemical amino acid composition. Each point represents a protein, and the confidence ellipses indicate 2 standard deviations (2 σ) around the mean of each class.</figcaption></figure><p>There is a clear distinction between lyases (red, ellipse = 2 σ) and immune system proteins (purple, ellipse = 2 σ). However, with 20 protein classes, and a great amount of overlap seen in the plots, it is clear more robust methods, such as ML, are required.</p><h4>4. Pipeline 1: NLP</h4><p>Natural Language Processing (NLP) provides a conceptual framework for treating protein sequences as text strings composed of “tokens” (amino acids). In this project, we applied standard NLP techniques including k-merization, vectorization (with CountVectorizer), SMOTE, SVD, baseline exploration with LazyClassifier, and hyperparameter tuning with Optuna (Figures 9–12). These approaches help capture local patterns in the sequence text, analogous to motifs in proteins.</p><p>The goal of this approach to modeling was to determine whether simple NLP architectures could extract enough signal from raw sequences to identify biologically relevant classes. While lightweight and easy to compute without a powerful GPU, these methods tend to capture only short-range dependencies. Still, they established a baseline for more advanced architectures and helped shape the direction of the project.</p><p>Among the top three tree models analyzed (Random Forest, Extra Trees, and Bagging), RandomForest performed the best with a tuned validation accuracy of 83.7% that dropped to a test accuracy of 69.7% (Appendix 6). This relatively low accuracy is expected for such complex data. More advanced neural networks will be required to capture the relationship between sequences and protein classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*BdCyD_Qutoq_2eowy7nybw.png" /><figcaption>Figure 9. Optuna History Plot showing fairly stable accuracy across 25 trials.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*GpwQR0qYtdQbzAt9QVsvdQ.png" /><figcaption>Figure 10. Optuna Parallel Coordinate Plot showing hyperparameter relationship</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*D3kVvnr0DwVSoxtD4uc_PQ.png" /><figcaption>Figure 11. Optuna Hyperparameter Importance plot indicating the importance of max_depth.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*CqgzR24l_v8N6tb6x_QwaA.png" /><figcaption>Figure 12. Optuna Contour Plot showing pairwise relationships between hyperparameters and their effect on the objective function.</figcaption></figure><h4>5. Pipeline 2: LSTMs</h4><p>Long Short-Term Memory networks (LSTMs) are well-suited to sequence data, making them a natural next step. While other sequence-based architectures such as Gated Recurrent Units (GRUs), one-dimensional Convolutional Neural Networks (1D CNNs), and Transformer models could also capture sequence dependencies, LSTM networks were chosen here for their balance of expressiveness and computational efficiency on modest hardware. Unlike simple NLP models, LSTMs can capture long-range dependencies across a protein sequence, which is important because structural and functional features often depend on distant amino acid interactions.</p><p>In this project, we trained a bidirectional LSTM (BiLSTM) classifier (Keras/TensorFlow) on the cleaned, tokenized, and padded protein sequences. Despite significant hardware limitations on my local machine, the model reached a promising 78.2% accuracy on the validation set (Figure 13, Appendix 7), demonstrating that traditional sequence models remain useful for biological data.</p><p>However, training was slow without GPU acceleration, and scaling up to more advanced model architecture proved difficult. For data of this complexity, multiple neural network layers are required to capture underlying signal, since much of it comes from local as well as long-range dependencies within the sequence. This reflects the biochemical reality that protein function is shaped by interactions among both proximal and distal residues along the polypeptide chain.</p><p>These limitations ultimately motivated the shift toward pre-trained embeddings (Pipeline 3) from a larger model (this approach).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*t5ZWF7XD658PVef8vPZxHA.png" /><figcaption>Figure 13. Training and validation accuracy and loss of Keras BiLSTM model over 20 epochs.</figcaption></figure><h4>6. Pipeline 3: LLMs</h4><p>Large Language Models (LLMs) pre-trained on protein corpora (such as ESM, ProtBERT, etc.) offer a powerful alternative to training models from scratch. These models generate high-dimensional embeddings that capture structural, evolutionary, and biochemical relationships learned from millions of sequences. In essence, the majority of the effort put into Pipeline 2 (LSTM) is already done by these models, reducing the required computation power exponentially.</p><p>In this approach, we used LLM-based embeddings as fixed features for downstream classification. Instead of training a deep model from scratch, sequences were passed through a pre-trained model (Table 1; esm2_t6_8M_UR50D, ESM-2, Meta AI) to obtain embeddings, which were then planned to be used to train two lightweight classifiers: a random forest and a shallow neural network. This strategy would have significantly improved performance and avoided the GPU limitations that slowed LSTM training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*ofKPWxetK8vPQgf_IPe_YA.png" /></figure><p>However, generating the ESM-2 embeddings in their entirely proved to be computationally taxing, and as such we were only able to generate 1000 total — 200 per top 5 protein class. These embeddings revealed biologically meaningful clustering patterns during exploratory analysis, visualized with a UMAP plot (Figure 14). This suggests that the model learns rich, generalizable representations of protein space. However, due to the high dimensionality of ESM-2 embeddings (sequence length × 320), it is difficult to fully capture these relationships in only two dimensions. As seen in the figure, there are distinct clusters of embeddings, but the differences between protein classes are not easily distinguished.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*OU_hoKt0RjsuK1OU9UaW-A.png" /><figcaption>Figure 14. UMAP plot of ESM-2-generated embeddings for protein sequence data, colored by protein classification. Each point represents a sequence embedding, and there are 200 per protein class.</figcaption></figure><p>Once we are able to generate embeddings for the full dataset, rather than the limited 1,000 sequences used here, we expect the LLM-based approach to substantially outperform both of the other pipelines. The pre-trained model captures long-range dependencies, structural motifs, and evolutionary patterns across entire sequences, which are difficult for simpler models to learn from scratch. By leveraging the complete embedding set as input to lightweight classifiers, the model can generalize across all protein classes, handle class imbalance more effectively, and produce more accurate predictions with less training time. This scalability and rich feature representation highlight the potential of LLM embeddings as the most powerful and efficient approach in this study.</p><h4>7. Summary of Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kFeJ4db95c49gvmt3Ez5bQ.png" /></figure><h4>8. Discussion</h4><p>This project highlights how different ML methods interact with protein sequence data. Traditional NLP methods offer simplicity and speed but struggle to capture complex dependencies. LSTMs improve model expressiveness but are constrained by computational resources when trained from scratch on large datasets.</p><p>By contrast, LLM-based approaches provide high-quality embeddings with minimal effort, suggesting that modern pre-trained models are the most efficient path forward for many protein analysis tasks — especially when hardware is limited.</p><p>Further, future modeling should expand the featureset to other fields, such as pH, rather than protein sequence alone. This approach would increase the robustness of ML models dramatically.</p><p>Although the project remains a work in progress, the results demonstrate the advantages of applying language-based models to biological sequences and underscore the growing convergence between computational biology and modern AI. Future steps include expanding the dataset, fine-tuning transformer models, and exploring generative AI capabilities for protein analysis and drug design. This so-called “Intelligent Drug Design” appears to be the new way forward to solve the intricate biochemical interactions required by the compounds to have the desired therapeutic effects.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48d7057e43b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Talking to Macromolecules: A GenAI Chatbot Assistant for Bioinformatics and Computational Biology]]></title>
            <link>https://ernest-bonat.medium.com/talking-to-macromolecules-a-genai-chatbot-assistant-for-bioinformatics-and-computational-biology-3f437c566c9d?source=rss-58ededb6e465------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f437c566c9d</guid>
            <dc:creator><![CDATA[Ernest Bonat, Ph.D.]]></dc:creator>
            <pubDate>Sat, 24 May 2025 11:33:29 GMT</pubDate>
            <atom:updated>2025-08-08T21:18:55.303Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3rlQ9HBmsbDSIn7-HdXomw.png" /></figure><p><strong>Ernest Bonat, Ph.D., Chad Turner</strong></p><p><strong><em>“This isn’t your average data tool. This is science, reimagined.”</em></strong></p><p>1. <a href="#9d11">Introduction</a><br>2. <a href="#fccb">Dataset Selection</a><br>3. <a href="#a2e0">Features Selection</a><br>4. <a href="#b0b4">Preparing the Dataset File for Chatbot Use</a><br>5. <a href="#b765">GenAI Chabot Assistant</a><br>6. <a href="#6cbf">Chatbot Queries and Responses</a><br>6.1 <a href="#c7d1">Macromolecule Classifications and Descriptions</a><br>6.2 <a href="#4436">Sequence and Residue Analysis</a><br>6.3 <a href="#ee64">Molecular Weight Analysis</a><br>6.4 <a href="#1578">Crystallization and Stability</a><br>6.5 <a href="#8bc5">Resolution and Structure Quality</a><br>6.6 <a href="#9402">Density and Solubility</a><br>6.7 <a href="#bd4d">Experimental Techniques</a><br>6.8 <a href="#776c">Structural and Experimental Correlations</a><br>6.9 <a href="#04e8">Historical Trends and Comparisons</a><br>6.10 <a href="#e4fb">Statistical Regression Models</a><br>6.11 <a href="#72f8">Clustering Models</a><br>7. <a href="#298a">Conclusions</a></p><h4>1. Introduction</h4><p><strong>Hello, molecule? Let’s talk structure.</strong></p><p>While this project is titled <em>“Talking to Macromolecules,”</em> the conversation is not with the molecules themselves but rather with the expansive datasets that hold their secrets through a Generative AI (GenAI) chatbot intelligent system.</p><p>This chatbot enables a new scientific dialogue where researchers can explore macromolecule properties, patterns, and structures using natural language queries without the need for digging through raw data manually or writing code.</p><p><strong>So, what are macromolecules?</strong></p><p>Macromolecules are large, complex molecules essential to life. These molecules are composed of thousands of covalently bonded atoms, forming peptide chains and biopolymers such as proteins, nucleic acids, and carbohydrates. All living organisms are dependent on three fundamental biopolymers: DNA, RNA, and proteins.</p><ul><li>DNA carries genetic information and is the precursor for RNA.</li><li>RNA plays a role in gene expression and guides protein synthesis.</li><li>Proteins perform critical biological functions such as structural support and catalysis.</li></ul><p><strong>The Challenge with Macromolecule Data</strong></p><p>Information about these molecules is stored in specialized databases where characteristics and key structural information are documented. However, retrieving specific information from these large datasets often requires extensive searching and manual data extraction, making this process time-consuming and inefficient.</p><p><strong>What If We Could Simply Ask the Data?</strong></p><p>This project aims to provide a solution by utilizing a GenAI chatbot intelligent assistant, allowing researchers to query macromolecule data conversationally. Instead of writing Python or R code, users can ask plain language questions and receive immediate structured responses.</p><p><strong>What can the Chatbot do?</strong></p><p>With AI-driven analysis, this chatbot enhances and simplifies data exploration of large macromolecule datasets:</p><ul><li>Identify trends, correlations, and distributions in macromolecule properties like experiment techniques, crystallization methods, and sequence characteristics, simplifying structural data analysis.</li><li>Generate visualizations to aid in interpreting insights.</li><li>Provide explanations or conclusions based on its responses.</li><li>Apply basic machine learning (ML) models such as regression and clustering.</li><li>Perform exploratory data analysis (EDA) to support hypothesis generation.</li></ul><p><strong>Benefits of Conversational Data Exploration:</strong></p><ul><li><strong>Instant Insights:</strong> Ask questions using natural language and retrieve answers quickly instead of manually searching databases.</li><li><strong>Pattern Discovery:</strong> Detect hidden relationships and trends in macromolecule characteristics.</li><li><strong>Comparative Analysis:</strong> Quickly analyze molecular structures, compare classifications, and examine experimental techniques and conditions used for similar structures.</li><li><strong>Visual Data Exploration:</strong> Automatically generate plots and graphs to provide a visual representation of trends, distributions, and relationships.</li><li><strong>Hypothesis Testing:</strong> Test data-driven hypotheses by using AI-assisted queries.</li><li><strong>Knowledge Beyond the Dataset:</strong> Utilize the large language model’s (LLM) pretrained scientific knowledge to provide information beyond the dataset.</li></ul><p><strong>Real-World Applications:</strong></p><ul><li><strong>Structural Biology:</strong> Identify trends in crystallization methods, resolutions, macromolecule classifications, and other structural characteristics to optimize structural studies.</li><li><strong>Drug Discovery and Protein Engineering:</strong> Find similar sequences to design new drugs or modify protein structures for therapeutic use.</li><li><strong>Genetic and Proteomic Research:</strong> Understand how different protein sequences relate to <strong>function, stability, and disease associations.</strong></li></ul><h4>2. Dataset Selection</h4><p>This project utilizes the <a href="https://www.kaggle.com/datasets/shahir/protein-data-set">Structural Protein Sequences</a> dataset, which was sourced from <a href="https://www.kaggle.com/datasets/shahir/protein-data-set">Kaggle.com</a> and originates from the Research Collaboratory for Structural Bioinformatics (RCSB) Protein Data Bank (PDB). The PDB is a global repository of 3D macromolecular structures, established in 1971 as the first digital open-source database for biological sciences.</p><p>A combination of two datasets is used:</p><p><strong>1. pdb_data.csv</strong> — Macromolecule structural properties</p><ul><li>Structure ID</li><li>Classification</li><li>Residue count</li><li>Resolution</li><li>Structure molecular weight</li><li>Crystallization method</li><li>Density Matthews</li><li>Density percent solvent</li><li>PDBX details</li><li>pH value</li><li>Publication date</li></ul><p><strong>2. pdb_data_seq.csv</strong> — Macromolecule sequence information</p><ul><li>Structure ID</li><li>Chain ID</li><li>Primary sequence</li><li>Residue count</li><li>Macromolecule type</li></ul><p>The two datasets will be merged using the PDB Structure IDs as a key to ensure the sequence data aligns with the corresponding structural metadata.</p><h4>3. Features Selection</h4><p><strong>Structure ID — </strong>Macromolecules are logged into the PDB using unique four-character identification codes for specific 3D structures. These IDs map to atomic coordinates, secondary structure, and conformational details.</p><p><strong>Classification — </strong>Macromolecules in this dataset are classified by their biological function, using standardized classifications from the PDB. Examples include hydrolases, oxidoreductases, and ribosomal macromolecules.</p><p><strong>Experimental Techniques — </strong>Techniques such as X-ray diffraction and neutron diffraction are used to determine the 3D structure of macromolecules. Researchers may use combinations of different techniques to gain a complete understanding of the structure and function.</p><p><strong>Macromolecule Type — </strong>In general biology, the four main types of macromolecules are proteins, nucleic acids (DNA and RNA), carbohydrates, and lipids. However, this dataset defines types based on structural compositions observed in crystallography:</p><ul><li>Protein</li><li>Protein-DNA complex</li><li>Protein-RNA complex</li><li>Protein-DNA-RNA hybrid</li><li>Protein-DNA-DNA-RNA hybrid</li></ul><p><strong>Residue Count — </strong>Amino acids are called residues when two or more bond with each other, forming a peptide bond. The residue count refers to the number of amino acid residues in a macromolecule. As macromolecules are chains of amino acids with peptide bonds, the residue count is generally the length of the primary sequence.</p><p><strong>Resolution — </strong>Resolution is a key factor in determining the quality of a macromolecule’s structure. It is measured in Ångströms (Å), where 1 Å equals 0.1 nanometers (one ten billionth of a meter). Lower values indicate higher structural detail.</p><ul><li><strong>Low resolution (&gt;5 Å):</strong> Only general molecular shapes are visible; atomic details are not well-defined.</li><li><strong>Medium resolution (3.5–2.5 Å):</strong> Some side chains can be identified, but finer atomic details may still be unclear.</li><li><strong>High (atomic) resolution (&lt;2.4 Å):</strong> Individual atoms and molecular interactions are clearly visible, allowing precise structure determination.</li></ul><p><strong>Structure Molecular Weight — </strong>The molecular weight of macromolecules is calculated by combining the mass of all residues. This is expressed in Daltons (Da), where 1 Da equals 1 atomic mass unit.</p><p><strong>Crystallization Method — </strong>The crystallization method describes how a macromolecule crystal was grown in a lab. Vapor diffusion, microbatch, and microgravity are a few common methods.</p><p><strong>Density Matthews (Å³/Da) — </strong>In crystallography, the Matthews Coefficient (densityMatthews) is used to estimate the amount of solvent in a macromolecule crystal. This estimates how much space in the crystal’s volume is occupied by the macromolecule itself versus solvents like water. The units are expressed as Ångströms per Dalton, typically ranging from 1.6 to 3.5 Å³/Da.</p><p><strong>Density Percent Solvent (%) — </strong>The density percent solvent (densityPercentSol) is the proportion of the crystal occupied by solvent content, also measured by the Matthews coefficient.</p><p><strong>pH Value — </strong>This field records the acidity or alkalinity of the crystallization solution, typically ranging from 4 (acidic) to 9 (basic).</p><p><strong>Primary Sequence — </strong>The primary sequence is the linear sequence of amino acids or nucleotides in the macromolecule. Each unique combination contributes to the molecule’s structure and function.</p><h4>4. Preparing the Dataset File for Chatbot Use</h4><p>The first step in “talking to macromolecules” involves preparing the dataset file so it can be effectively used by the GenAI chatbot. As previously mentioned, this project uses a combination of the datasets “pdb_data.csv” and “pdb_data_seq.csv,” which are merged using the structure ID as a common key.</p><p>In “pbd_data.csv,” the “pdbxdetails” column is removed, as it contains redundant information such as crystallization methods, temperature, and pH values.</p><p>In “pdb_seq.csv,” all columns are dropped except for “structureId” and “sequence,” as the remaining attributes are already present in the first dataset. These two datasets are joined in Python using the “structureId” to align the molecular sequences with their corresponding structural and experimental characteristics.</p><p>All column names are formatted in camelCase (e.g., molecularWeight, residueCount) to ensure they are properly interpreted by the LLM used for code generation, resulting in more accurate responses.</p><pre>=== Information for Combined DataFrame ===<br><br>DataFrame shape:<br> (471811, 14)<br><br>Null value count:<br> classification                4<br>macromoleculeType         36432<br>resolution                21966<br>crystallizationMethod    156948<br>crystallizationTempK     154005<br>densityMatthews           81655<br>densityPercentSol         81533<br>phValue                  130910<br>publicationYear           57780<br>sequence                    690<br>dtype: int64<br><br>Duplicate value count:<br> 217909<br><br>DataFrame Info:<br>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 471811 entries, 0 to 471810<br>Data columns (total 14 columns):<br> #   Column                    Non-Null Count   Dtype<br>---  ------                    --------------   -----<br> 0   structureId               471811 non-null  object<br> 1   classification            471807 non-null  object<br> 2   experimentalTechnique     471811 non-null  object<br> 3   macromoleculeType         435379 non-null  object<br> 4   residueCount              471811 non-null  int64<br> 5   resolution                449845 non-null  float64<br> 6   structureMolecularWeight  471811 non-null  float64<br> 7   crystallizationMethod     314863 non-null  object<br> 8   crystallizationTempK      317806 non-null  float64<br> 9   densityMatthews           390156 non-null  float64<br> 10  densityPercentSol         390278 non-null  float64<br> 11  phValue                   340901 non-null  float64<br> 12  publicationYear           414031 non-null  float64<br> 13  sequence                  471121 non-null  object<br>dtypes: float64(7), int64(1), object(6)<br>memory usage: 50.4+ MB<br>Nonep</pre><p>After the merge, the shape of the combined datasets is (471811, 14). However, it contains many null values and 217,909 duplicated rows, which are dropped to improve data quality.</p><pre>=== Information for Combined DF after dropping nulls and duplicates ===<br><br>DataFrame shape:<br> (107824, 14)<br><br>Null value count:<br> Series([], dtype: int64)<br><br>Duplicate value count:<br> 0<br><br>DataFrame Info:<br>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 107824 entries, 0 to 107823<br>Data columns (total 14 columns):<br> #   Column                    Non-Null Count   Dtype<br>---  ------                    --------------   -----<br> 0   structureId               107824 non-null  object<br> 1   classification            107824 non-null  object<br> 2   experimentalTechnique     107824 non-null  object<br> 3   macromoleculeType         107824 non-null  object<br> 4   residueCount              107824 non-null  int64<br> 5   resolution                107824 non-null  float64<br> 6   structureMolecularWeight  107824 non-null  float64<br> 7   crystallizationMethod     107824 non-null  object<br> 8   crystallizationTempK      107824 non-null  float64<br> 9   densityMatthews           107824 non-null  float64<br> 10  densityPercentSol         107824 non-null  float64<br> 11  phValue                   107824 non-null  float64<br> 12  publicationYear           107824 non-null  float64<br> 13  sequence                  107824 non-null  object<br>dtypes: float64(7), int64(1), object(6)<br>memory usage: 11.5+ MB<br>None</pre><p>The DataFrame’s shape is now (107824, 14) with no null values and no duplicates. Next, the class counts for the column “classification” are inspected for class imbalance.</p><pre>Number of unique classes: 2511</pre><p>There were found to be many rare classes, so they are then dropped using a minimum class frequency threshold of 1000 entries to reduce noise and focus on significant trends.</p><pre>Number of unique classes after dropping rare labels: 17<br><br>Unique Classes: classification<br>HYDROLASE                            11856<br>RIBOSOME                             11109<br>TRANSFERASE                           8630<br>OXIDOREDUCTASE                        7520<br>IMMUNE SYSTEM                         5665<br>HYDROLASE/HYDROLASE INHIBITOR         4764<br>TRANSCRIPTION                         2568<br>LYASE                                 2534<br>TRANSPORT PROTEIN                     1980<br>SIGNALING PROTEIN                     1654<br>TRANSFERASE/DNA                       1495<br>TRANSFERASE/TRANSFERASE INHIBITOR     1449<br>VIRAL PROTEIN                         1371<br>LIGASE                                1371<br>ISOMERASE                             1317<br>PROTEIN BINDING                       1247<br>MEMBRANE PROTEIN                      1003<br>Name: count, dtype: int64</pre><p>The DataFrame is then formatted to ensure that all values in each column follow a consistent structure. Rows with invalid structure IDs or sequences are filtered out.</p><pre>Null value count:<br> Series([], dtype: int64)<br><br>Duplicate value count:<br> 0<br><br>DataFrame Info:<br>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 67318 entries, 0 to 67317<br>Data columns (total 14 columns):<br> #   Column                    Non-Null Count  Dtype<br>---  ------                    --------------  -----<br> 0   structureId               67318 non-null  object<br> 1   classification            67318 non-null  object<br> 2   experimentalTechnique     67318 non-null  object<br> 3   macromoleculeType         67318 non-null  object<br> 4   residueCount              67318 non-null  int64<br> 5   resolution                67318 non-null  float64<br> 6   structureMolecularWeight  67318 non-null  float64<br> 7   crystallizationMethod     67318 non-null  object<br> 8   crystallizationTempK      67318 non-null  float64<br> 9   densityMatthews           67318 non-null  float64<br> 10  densityPercentSol         67318 non-null  float64<br> 11  phValue                   67318 non-null  float64<br> 12  publicationYear           67318 non-null  float64<br> 13  sequence                  67318 non-null  object<br>dtypes: float64(7), int64(1), object(6)<br>memory usage: 7.2+ MB<br>None</pre><p>After these steps, the DataFrame is reduced to the shape of (67318, 14).</p><p>As shown below, there are numerous outliers of numeric values in this dataset. To remove these, the interquartile range (IQR) method is applied. Values that fall outside 1.5x the IQR are removed, significantly improving the quality of the data by removing extreme values that could skew results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*yOz0zta_gpD4CwRPlSf5Sg.png" /></figure><pre>=== Information for Cleaned DataFrame No Outliers ===<br><br>DataFrame shape:<br> (41060, 14)<br><br>Null value count:<br> Series([], dtype: int64)<br><br>Duplicate value count:<br> 0<br><br>DataFrame Info:<br>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 41060 entries, 0 to 41059<br>Data columns (total 14 columns):<br> #   Column                    Non-Null Count  Dtype<br>---  ------                    --------------  -----<br> 0   structureId               41060 non-null  object<br> 1   classification            41060 non-null  object<br> 2   experimentalTechnique     41060 non-null  object<br> 3   macromoleculeType         41060 non-null  object<br> 4   residueCount              41060 non-null  int64<br> 5   resolution                41060 non-null  float64<br> 6   structureMolecularWeight  41060 non-null  float64<br> 7   crystallizationMethod     41060 non-null  object<br> 8   crystallizationTempK      41060 non-null  float64<br> 9   densityMatthews           41060 non-null  float64<br> 10  densityPercentSol         41060 non-null  float64<br> 11  phValue                   41060 non-null  float64<br> 12  publicationYear           41060 non-null  float64<br> 13  sequence                  41060 non-null  object<br>dtypes: float64(7), int64(1), object(6)<br>memory usage: 4.4+ MB<br>None</pre><p>The final, cleaned DataFrame has a shape of (41060, 14) with no null values, no duplicates, and minimized numerical outliers. This version is saved to a new CSV file and used for the GenAI chatbot assistant.</p><h4>5. GenAI Chabot Assistant</h4><p>A GenAI Chatbot Assistant is a conversational software application powered by GenAI that can understand natural language inputs and generate intelligent, human-like responses in real-time. It goes beyond traditional rule-based chatbots by using large language models (LLMs) to reason, answer questions, and perform tasks. Below are the three main functions:</p><ol><li><strong>Generative Intelligence: </strong>Uses models like GPT, Llama to generate coherent, context-aware responses instead of relying on predefined scripts.</li><li><strong>Natural Language Understanding (NLU)</strong>: Understands user intent even with vague, informal, or complex queries.</li><li><strong>Learning &amp; Context Retention</strong>: Capable of maintaining the context of a conversation and adapting responses based on prior exchanges.</li></ol><p><strong>Why It’s Powerful?</strong></p><ul><li><strong>No need for the user to write code or understand complex systems.</strong></li><li>Available 24/7, scalable to millions of users.</li><li>Can be integrated with other tools (e.g., APIs, databases, cloud platforms, etc).</li></ul><p>Our GenAI chatbot assistant enables users to quickly analyze large macromolecule datasets in a simple, user-friendly way. The chatbot takes in a user-generated query and responds based on the content of the uploaded CSV dataset file. The NLU is handled by Llama 3 LLM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*osvEiFz6AcbYwUIdTKpaJw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-tVGsrijfS7mjs87x5Sglw.png" /></figure><h4>6. Chatbot Queries and Responses</h4><p>Let’s provide some possible combinations of queries and responses on specific topics based on our selected CSV dataset file. All responses have been manually validated by Dr. Bonat to ensure the chatbot assistant works correctly.</p><h4>6.1 Macromolecule Classifications and Descriptions</h4><p><strong>1. Query: </strong>What is the definition of a hydrolase?</p><p><strong>Response: </strong>A hydrolase is an enzyme that catalyzes a hydrolysis reaction, which is a chemical reaction in which a molecule is cleaved into two or more smaller molecules using water.</p><p><strong>2. Query: </strong>What is the definition of lyase?</p><p><strong>Response: </strong>A lyase is an enzyme that catalyzes the cleavage of various chemical bonds, such as carbon-carbon, carbon-nitrogen, and carbon-oxygen bonds, without the use of cofactors or ATP.</p><p><strong>3. Query: </strong>What is the definition of oxidoreductase?</p><p><strong>Response: </strong>Oxidoreductases are a class of enzymes that catalyze oxidation-reduction reactions.</p><p><strong>4. Query: </strong>How many unique classifications are in this dataset? What are they?</p><p><strong>Response: </strong>There are 17 unique classifications: HYDROLASE, OXIDOREDUCTASE, TRANSFERASE, SIGNALING PROTEIN, LYASE, MEMBRANE PROTEIN, IMMUNE SYSTEM, VIRAL PROTEIN, HYDROLASE/HYDROLASE INHIBITOR, TRANSCRIPTION, LIGASE, ISOMERASE, TRANSFERASE/DNA, TRANSPORT PROTEIN, RIBOSOME, PROTEIN BINDING, TRANSFERASE/TRANSFERASE INHIBITOR.</p><p><strong>5. Query: </strong>What are top five most common classifications of macromolecules for this dataset? Add count text to the bars in the graph</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*Pn_Eyx3V85BDmrSUDycmnQ.png" /></figure><p><strong>6. Query: </strong>What is the proportion of each classification in the dataset? Add percentage text to the bars in the graph.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*0SGENkHiWvAmGNlSX-4rWA.png" /></figure><p><strong>7. Query: </strong>Which classification has the highest number of structures? What is the proportion for this out of all of the structures in the dataset?</p><p><strong>Response: </strong>The classification with the highest number of structures is HYDROLASE with 9074 structures, which is 22.10% of all structures in the dataset.</p><h4>6.2 Sequence and Residue Analysis</h4><p><strong>8. Query: </strong>Show the distribution of residues.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IvCRhtgRqDCfnenWz6xxdg.png" /></figure><p><strong>9. Query: </strong>What is the most common amino acid in protein sequences across the dataset? What does this suggest?</p><p><strong>Response: </strong>The most common amino acid is L with a frequency of 1081225. This suggests that this amino acid may play a crucial role in the structure or function of these proteins.</p><p><strong>10. Query: </strong>What are the most common residues found in macromolecules?</p><p><strong>Response: </strong>The top 10 most common residues are: L, A, G, V, S, E, D, T, K, I.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dsX8TMGx_YV_7L44U0_Muw.png" /></figure><p><strong>11. Query: </strong>How does the length of sequences vary across different macromolecule types? Add length text to the bars in the graph.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*7BEkoBTINSKxmBIHlZNNMw.png" /></figure><p><strong>12. Query: </strong>How does the sequence length relate to the structure’s classification? Add count text to the bars in the graph</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*Hlh7bx4E-K6gX_yPomfwjw.png" /></figure><p><strong>13. Query: </strong>Do isomerase proteins have longer or shorter sequences on average?</p><p><strong>Response: </strong>Isomerase proteins have shorter sequences on average (746.86 vs 775.41).</p><p><strong>14. Query: </strong>What is the average residue count in the dataset? Round to four decimal places.</p><p><strong>Response: </strong>775.4107</p><p><strong>15. Query: </strong>Which classification has the longest average residue count?</p><p><strong>Response: </strong>The classification with the longest average residue count is OXIDOREDUCTASE with an average residue count of 1066.60.</p><p><strong>16. Query: </strong>What is the average residue count for different classifications? Add count text to the bars in the graph</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*wFhvnjTHKhQJ760nJntuSw.png" /></figure><p><strong>17. Query: </strong>How many proteins have more than 500 residues and what classification has the most?</p><p><strong>Response: </strong>There are 21712 proteins with more than 500 residues. The most common classification is HYDROLASE.</p><p><strong>18. Query: </strong>What is the amino acid frequency distribution for structure ID 1XNH? Add frequency text to top of bars on graph.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*_1Im-Nznq-UfFVgaKRhggQ.png" /></figure><p><strong>19. Query: </strong>What type of macromolecule and classification is this sequence: MEEKPKGALAVILVLTLTILVFWLGVYAVFFARG?</p><p><strong>Response: </strong>The sequence belongs to a Protein with classification OXIDOREDUCTASE.</p><p><strong>20. Query: </strong>What is the sequence of 4PSK, what classification is it, and how does it compare to other structures?</p><p><strong>Response: </strong>The sequence of 4PSK is MTQTPDREKALELAVAQIEKSYGKGSVMRLGDEARQPISVIPTGSIALDVALGIGGLPRGRVIEIYGPESSGKTTVALHAVANAQAAGGVAAFIDAEHALDPDYAKKLGVDTDSLLVSQPDTGEQALEIADMLIRSGALDIVVIDSVAALVPRAELEGEMGDSHVGLQARLMSQALRKMTGALNNSGTTAIFINQLRDKIGVMFGSPETTTGGKALKFYASVRMDVRRVETLKDGTNAVGNRTRVKVVKNKCSPPFKQAEFDILYGKGISREGSLIDMGVDQGLIRKSGAWFTYEGEQLGQGKENARNFLVENADVADEIEKKIKEKLGIGAVVTDDPSNDGVLPAPVDF. It is classified as HYDROLASE. The average sequence length of all structures is 292.72, and 4PSK has a sequence length of 350.</p><p><strong>21. Query: </strong>Which protein has the longest sequence?</p><p><strong>Response: </strong>The protein with the longest sequence is 5CSK with a sequence length of 2218.</p><p><strong>22. Query: </strong>What structure ID has the most residues and what is its classification?</p><p><strong>Response: </strong>The structure ID with the most residues is 3IWK with 6036 residues, and its classification is OXIDOREDUCTASE.</p><p><strong>23. Query: </strong>How many residues does 2GUM contain, and what type of macromolecule and classification is it?</p><p><strong>Response: </strong>2GUM contains 1884 residues, which is a Protein with a classification of VIRAL PROTEIN.</p><h4>6.3 Molecular Weight Analysis</h4><p><strong>24. Query: </strong>What structure ID, macromolecule type, and classification has the highest weight?</p><p><strong>Response: </strong>The structure with the highest weight is 3IWK which is a Protein classified as OXIDOREDUCTASE with a weight of 668529.88.</p><p><strong>25. Query: </strong>Show the average molecular weight for different macromolecules. Add weight text to bars in graph.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*DzaT9tHgYonvUBWqCx3T5w.png" /></figure><p><strong>26. Query: </strong>What is the average molecular weight for each classification?</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*6P5QLmSGS4TZE3YHjvXA3w.png" /></figure><p><strong>27. Query: </strong>What is the ratio of molecular weight to residue count for different macromolecule types? Add ratio text with Da/residue to the bars in the graph.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*nrlvEHQ8Ksc-untALSAYzw.png" /></figure><p><strong>28. Query: </strong>Which proteins have the highest/lowest molecular weight and what are their classifications? What does this suggest?</p><p><strong>Response: </strong>The protein with the highest molecular weight is 3IWK with a weight of 668529.88 and classification OXIDOREDUCTASE. The protein with the lowest molecular weight is 1O06 with a weight of 2441.58 and classification TRANSPORT PROTEIN. This suggests that proteins with different classifications can have varying molecular weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E7GBpcGY-y-PU6vnyhT_yw.png" /></figure><p><strong>29. Query: </strong>What is the molecular weight of structure 3NN7, and how does it compare to similar macromolecules?</p><p><strong>Response: </strong>The molecular weight of structure 3NN7 is 34725.00. It is relatively less similar to other macromolecules of the same type with a median molecular weight of 60136.20.</p><p><strong>30. Query: </strong>How does the molecular weight of viral proteins compare to enzymes? What does this suggest?</p><p><strong>Response: </strong>Viral proteins have a mean molecular weight of 428053.77, while enzymes have a mean molecular weight of 87651.36. This suggests that viral proteins tend to be smaller than enzymes.</p><p><strong>31. Query: </strong>How does the molecular weight of structure 4QXT compare to similar macromolecules? What does this suggest?</p><p><strong>Response: </strong>The molecular weight of structure 4QXT (26390.11) is -0.72 standard deviations away from the average molecular weight of similar macromolecules (87651.36). This suggests that structure 4QXT has a relatively low molecular weight compared to similar macromolecules.</p><h4>6.4 Crystallization and Stability</h4><p><strong>32. Query: </strong>What are the 5 most common temperatures used for crystallization in proteins? Add count text to top of bars.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*3rFDsSDGgBKs60oKGrGDFQ.png" /></figure><p><strong>33. Query: </strong>What are the most effective pH levels for macromolecule crystallization?</p><p><strong>Response: </strong>The top 5 most effective pH levels for macromolecule crystallization are 7.5, 6.5, 7.0, 8.5, 8.0.</p><p><strong>34. Query: </strong>Show the mean pH values for macromolecule classifications. Add mean pH text to top of bars.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*UEEdG-twQLgUT3xfQxjUZg.png" /></figure><p><strong>35. Query: </strong>What are the most common crystallization methods used? What does this suggest?</p><p><strong>Response: </strong>The top 5 most common crystallization methods are: VAPOR DIFFUSION, HANGING DROP, VAPOR DIFFUSION, SITTING DROP, VAPOR DIFFUSION, EVAPORATION, MICROBATCH. This suggests that these methods are the most widely used and likely to be effective for crystallizing proteins.</p><p><strong>36. Query: </strong>Show the counts for the top five crystallization methods. Add count text to top of bars.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*x8nxs8XSov3Nm0TBejhbJw.png" /></figure><p><strong>37. Query: </strong>What is the classification and crystallization conditions for structure 4FQD?</p><p><strong>Response: </strong>The classification for structure 4FQD is TRANSFERASE, and the crystallization conditions are method: MICROBATCH, temperature: 289.0 K.</p><p><strong>38. Query: </strong>How does crystallization temperature vary across different macromolecules types? Add temperature text to the bars in the graph</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*5ngrQc_HfTBMXoGofMEmKA.png" /></figure><p><strong>39. Query: </strong>Which crystallization method has the highest success rate for transport proteins?</p><p><strong>Response: </strong>The crystallization method with the highest success rate for transport proteins is VAPOR DIFFUSION, HANGING DROP.</p><p><strong>40. Query: </strong>Is there a crystallization temperature sweet spot for transport proteins?</p><p><strong>Response: </strong>The crystallization temperature sweet spot for transport proteins is around 294.07K with a standard deviation of 3.17K.</p><h4>6.5 Resolution and Structure Quality</h4><p><strong>41. Query: </strong>What is the distribution of resolution values in the dataset?</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*54CoOdBZ-RY-Iy9fFNMFUw.png" /></figure><p><strong>42. Query: </strong>What does resolution mean in structure determination?</p><p><strong>Response: </strong>Resolution is a measure of how detailed the structure is. In this dataset, the resolutions range from 0.7 to 4.17 with a mean of 2.15.</p><p><strong>43. Query: </strong>Which classification has the lowest average resolution and what is it?</p><p><strong>Response: </strong>The classification with the lowest average resolution is ISOMERASE with an average resolution of 1.96.</p><p><strong>44. Query: </strong>What is the average resolution for structures determined with Cryo-EM vs X-ray?</p><p><strong>Response: </strong>The average resolution for Cryo-EM structures is 2.78 and for X-ray structures is 2.15.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FC4QPH5eriVbXUJxN9IfwA.png" /></figure><p><strong>45. Query: </strong>How many macromolecules have a resolution below 2.5 Å? And what classification has the most below 2.5 Å?</p><p><strong>Response: </strong>There are 30480 macromolecules with a resolution below 2.5 Å. The classification with the most below 2.5 Å is HYDROLASE.</p><p><strong>46. Query: </strong>What structure has the lowest resolution, and what classification is it?</p><p><strong>Response: </strong>The structure with the lowest resolution is 4ZM7 with a resolution of 0.7 and classification HYDROLASE.</p><p><strong>47. Query: </strong>What is the most common resolution value in the dataset? What does this suggest?</p><p><strong>Response: </strong>The most common resolution value in the dataset is 2.00. This suggests that the majority of the structures in the dataset have a resolution around 2.00, which may indicate a common experimental technique or instrumentation used in the data collection process.</p><p><strong>48. Query: </strong>What is the resolution of 3EKP, and what does it suggest about the structure’s quality?</p><p><strong>Response: </strong>The resolution of 3EKP is 2.15 Å which suggests a medium quality structure.</p><h4>6.6 Density and Solubility</h4><p><strong>49. Query: </strong>Show the distribution for density percent solvent.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nNOB-yqDK4EeNx6yDKWRxw.png" /></figure><p><strong>50. Query: </strong>What is the relationship between density Matthews and density percent sol? What does this suggest?</p><p><strong>Response: </strong>The correlation between density Matthews and density percent sol is 0.96. This suggests a strong positive linear relationship between the two variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IE4brsfJX9pub3Uxxi5OYA.png" /></figure><p><strong>51. Query: </strong>What is the average solvent content for different macromolecule types? Add solvent content text to the bars in the graph</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*T1oGT04iDHpP9HTYvZGnXw.png" /></figure><p><strong>52. Query: </strong>What are the density Matthews and density percent solvent values for 5IDP, and what do they indicate about its stability?</p><p><strong>Response: </strong>The density Matthews value for 5IDP is 3.08, and the density percent solvent value is 60.13. The protein is likely to be highly stable.</p><h4>6.7 Experimental Techniques</h4><p><strong>53. Query: </strong>Which experimental techniques are most commonly used for structure determination? Add count text to the bars in the graph</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*7le4U6u0qfJ-3gcImUH7Tw.png" /></figure><p><strong>54. Query: </strong>Which experimental techniques yield the most stable structures?</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*6xLhR6zDWPHCy7fTuA6eaA.png" /></figure><p><strong>55. Query: </strong>What are the average resolutions for experimental techniques? Add average resolution text to top of bars.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*kb8L2isfNxYaMntVAnACMw.png" /></figure><h4>6.8 Structural and Experimental Correlations</h4><p><strong>56. Query: </strong>Build a correlation matrix for this dataset. Add correlation coefficients to the boxes. Show the color bar. Use “viridis” as the color map. Label the original feature names.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*slNQSQwFlvM4eRlkfWx-mg.png" /></figure><p><strong>57. Query: </strong>Is there a correlation between crystallization temperature and resolution? What does this suggest?</p><p><strong>Response: </strong>There is a correlation of 0.01 between crystallization temperature and resolution. This suggests that lower crystallization temperatures may be associated with higher resolutions, and vice versa.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oVVEA7TjnKNhZu69sLHIqQ.png" /></figure><p><strong>58. Query: </strong>How does molecular weight correlate with crystallization temperature? What does this suggest?</p><p><strong>Response: </strong>The correlation between molecular weight and crystallization temperature is -0.01. This suggests that there is a negative relationship between the two variables, meaning that as molecular weight increases, crystallization temperature decreases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nHaWq9O4zziZwmb_pNiAHw.png" /></figure><p><strong>59. Query: </strong>Does the pH of crystallization conditions affect resolution or structure stability?</p><p><strong>Response: </strong>The correlation coefficient between pH and resolution is 0.05.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cRhtMzHHjHp-kTwV6_-SRQ.png" /></figure><p><strong>60. Query: </strong>How does pH influence the crystallization of macromolecules? Show the values</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*KQ0yMemHsDN9uK3_4h_dMQ.png" /></figure><p><strong>61. Query: </strong>What is the relationship between residue count and molecular weight?</p><p><strong>Response: </strong>The correlation coefficient between residue count and molecular weight is 1.00. This suggests a strong positive linear relationship between the two variables, indicating that as the residue count increases, the molecular weight also tends to increase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pxn53kwvFJuyMBmxkTuCpg.png" /></figure><p><strong>62. Query: </strong>Do heavier macromolecules tend to have lower or higher resolution? What is the correlation and what does this suggest?</p><p><strong>Response: </strong>The correlation between macromolecule weight and resolution is 0.37. This suggests that heavier macromolecules tend to have higher resolution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H2fkptuY0Eo7hq-apYUSkw.png" /></figure><p><strong>63. Query: </strong>Is there a correlation between residue count and structure stability? What does this suggest?</p><p><strong>Response: </strong>There is a correlation of 0.20 between residue count and structure stability. This suggests that as the residue count increases the structure stability also tends to increase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ffsztLr6FyEDpB9pQjwhBg.png" /></figure><p><strong>64. Query: </strong>How does pH value affect the density of macromolecules? What does this suggest?</p><p><strong>Response: </strong>The correlation between pH value and density of macromolecules is -0.01. This suggests that there is a negative correlation between pH value and density, meaning that as pH value increases, density decreases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DwyCuzCFefRZ70symuZHYA.png" /></figure><p><strong>65. Query: </strong>Do proteins with higher solvent content have higher resolution values? What does this suggest?</p><p><strong>Response: </strong>The correlation between solvent content and resolution is 0.43. This suggests that proteins with higher solvent content tend to have higher resolution values, which may indicate that the solvent content plays a role in the crystallization process and affects the quality of the crystal structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nGhKYLYlGrVqasAphMjFzw.png" /></figure><p><strong>66. Query: </strong>What unique structure IDs match these conditions: resolution &lt; 2.0 Å, molecular weight &gt; 50,000, crystallization method = vapor diffusion, density &gt; 3.0.</p><p><strong>Response: </strong>The unique structure IDs that match the conditions are: 1DGG, 1JIW, 1K7I, 1K7Q, 1YU4, 2B4P, 2HVK, 3CWW, 3E2Q, 3FID, 3JU4, 3K2O, 3PIM, 3PPM, 3TDS, 4E29, 4EAM, 4EAN, 4F33, 4H2F, 4H52, 4H7Y, 4LXJ, 4TOF, 4TOG, 4TOH, 4WAS, 4WFX, 4WFZ, 5I23, 5I24, 5TZJ, 5VSD, 5VSF</p><h4>6.9 Historical Trends and Comparisons</h4><p><strong>67. Query: </strong>How has the number of structures published changed over time?</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*I0CJQk62TV951IXksMpjFQ.png" /></figure><p><strong>68. Query: </strong>Which year had the most structures published, and what might have contributed to this?</p><p><strong>Response: </strong>The year with the most structures published is 2017.0 with 3888 structures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*OmZ4k4nb5jMC7Iz6Wtqniw.png" /></figure><p><strong>69. Query: </strong>How has the average residue count for the top five classifications improved over time?</p><p><strong>Response: </strong>The average residue count for the top 5 classifications has improved by 1.10% over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*YZ_LbhON83rkJeBQmQDNBw.png" /></figure><p><strong>70. Query: </strong>Has the average resolution improved over time?</p><p><strong>Response: </strong>The average resolution has improved from 2.21 in 1995.0 to 2.17 in 2018.0. The overall average resolution is 2.15.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*gSr64wjK0AtGXFVqys5_7Q.png" /></figure><p><strong>71. Query: </strong>How has the average resolution for the top five classifications improved over time?</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*6FY4y49EBsEFXk96pfEnUg.png" /></figure><p><strong>72. Query: </strong>Are newer macromolecule structures more likely to use higher-resolution techniques?</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yeX_qI2cMqmpLDAQYqX_Aw.png" /></figure><p><strong>73. Query: </strong>Are newer structures higher in molecular weight than older ones? What does this suggest?</p><p><strong>Response: </strong>The correlation between publication year and structure molecular weight is 0.06. This suggests that newer structures tend to have higher molecular weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ffn9sw7jJVnTMfTVG8Xqjg.png" /></figure><p><strong>74. Query: </strong>How has the top five crystallization technologies impacted structure discovery over time?</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*F2rq9sLrEtNGPpgAGSXzgA.png" /></figure><p><strong>75. Query: </strong>Show how sequence counts for the top ten classifications have changed over time.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*HgZvHrMBJsIzmzEMPkdCEw.png" /></figure><p><strong>76. Query: </strong>How many hydrolase enzymes were discovered after 2000? What does this suggest?</p><p><strong>Response: </strong>There were 10333 hydrolase enzymes discovered after 2000. This suggests that there has been an increase in the discovery of hydrolase enzymes in the 21st century, possibly due to advances in technology and research methods.</p><p><strong>77. Query: </strong>How have the molecular weights of published structures evolved over time?</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*Imd6xMg2EoT3DPFAM2Zl-Q.png" /></figure><p><strong>78. Query: </strong>Are newer structures more likely to contain unique sequence motifs? What does this suggest?</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*7iTMfjqFraimPuqhiH4xyQ.png" /></figure><h4>6.10 Statistical Regression Models</h4><p><strong>79. Query: </strong>Can you create a linear regression model for predicting density values based on percent solvent and show the model, parameters, and metrics?</p><p><strong>Response: </strong>Linear Regression Model: density = 0.06 * percent solvent + -0.37 R2: 0.93</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0a9KdPVwq1uMLaRPn4RvjQ.png" /></figure><p><strong>80. Query: </strong>Can you create a parabolic regression model for density values based on percent solvent and show the model, mean squared error, and r squared score?</p><p><strong>Response: </strong>Parabolic Regression Model: y = -0.07x² + 0.00x + 2.96 Mean Squared Error: 0.00 R-squared Score: 0.99</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l8VFHdq7vGqI_e-IJ0JhaQ.png" /></figure><p><strong>81. Query: </strong>Predict the molecular weight based on the residue count. Show the model, parameters, and metrics.</p><p><strong>Response: </strong>The linear regression model predicts molecular weight based on residue count with a coefficient of 112.47 and an intercept of 770.36. The R-squared value is 1.00.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jtIzbKr-ZI0bsEk2adlK2A.png" /></figure><p><strong>82. Query: </strong>Create a linear regression model to predict the resolution based on the densityMatthews coefficient. Show the MSE and R2 scores. What does this model suggest?</p><p><strong>Response: </strong>The linear regression model suggests that there is a positive correlation between densityMatthews and resolution, with an R2 score of 0.19 and an MSE of 0.22.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3am1PhDS1jO7DHo1R43KEg.png" /></figure><p><strong>83. Query: </strong>Predict the resolution based on the solvent coefficient. Show the model, parameters, and metrics.</p><p><strong>Response: </strong>Linear Regression Model: y = 0.03x + 0.82 MSE: 0.17 R2: 0.17</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCsrLMPKQ4mej9SjxExyJQ.png" /></figure><h4><strong>6.11 Clustering Models</strong></h4><p><strong>84. Query: </strong>Create a clustering model for protein, DNA, and RNA for resolution and molecular weight. In the legend, make protein green, DNA red, and RNA blue.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_PiSA2aa6zQbwblIcIy-xw.png" /></figure><p><strong>85. Query: </strong>Create a clustering model for protein, DNA, and RNA for pH and crystallization temperature. In the legend, make protein green, DNA red, and RNA blue.</p><p><strong>Response:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*SbUljSg0vgLPlqicCWg5pg.png" /></figure><h4>7. Conclusions</h4><p>This chatbot assistant provides a powerful alternative to manual data extraction for macromolecule research. Key insights, biological trends, correlations, structural relationships, experimental efficiencies, and visual aids are instantly available in response to natural language queries, streamlining the process of exploratory analysis.</p><p><strong>Key Discoveries and Observations</strong></p><p><strong>Simple Definitions: </strong>Although this dataset does not include explicit definitions or functions of macromolecules, the chatbot can still provide short explanations by utilizing its pretrained knowledge of biological concepts. This shows the value of integrating an LLM with datasets, allowing users to gain access to both contextual understanding and structural analysis in one system.</p><p><strong>Most common amino acid:</strong> Leucine (L) is the most frequent amino acid in this dataset, which is consistent with its abundance in proteins.</p><p><strong>Sequence Length: </strong>Protein-DNA-RNA hybrids have the longest average sequence lengths (323), with proteins not far behind (297). The longest unique sequence by character is a ligase protein (5CSK) at 2218 characters. Additionally, the structure 3IWK, an oxidoreductase protein, contains the highest number of total residues at 6036, likely due to multiple chains or complex formation.</p><p><strong>Macromolecule stability:</strong> Denser macromolecules tend to exhibit higher structural stability, reinforcing density as a key stability predictor.</p><p><strong>Crystallization Techniques and Conditions</strong></p><ul><li><strong>Most common crystallization methods: </strong>Vapor diffusion (hanging drop and sitting drop) are the most common methods for macromolecule crystallization.</li><li><strong>Most common resolution:</strong> A resolution of 2.0 Å is most frequent, which is considered atomic resolution, reflecting a balance between structure clarity and data acquisition feasibility.</li><li><strong>Most common crystallization temperatures:</strong> 293 K, 298 K, and 291 K are the most frequently used temperatures in structural formation.</li></ul><p><strong>Experimental Techniques</strong></p><ul><li><strong>Most used experimental technique:</strong> X-ray diffraction is by far the most used experimental technique, with only a few instances for the other methods. This aligns with X-ray diffraction’s widespread use in structural biology.</li><li><strong>Experimental technique for best resolution:</strong> Electron Paramagnetic Resonance (EPR) combined with X-ray diffraction produces the best resolution on average in the atomic range.</li></ul><p><strong>Correlation Discoveries</strong></p><ul><li><strong>Molecular Weight and Residue Correlations: </strong>Residue counts and molecular weight have a perfect positive correlation (r=1.0).</li><li><strong>Density and Solubility Correlations: </strong>There is a strong (r=0.96) correlation between density Matthews and density percent solvent, showing a linear relationship.</li><li><strong>Solvent content vs. resolution:</strong> Higher solvent content is correlated with higher resolution structures (r = 0.43), indicating a role in crystal formation quality.</li><li><strong>Resolution and Molecular Weight:</strong> There is a slight positive correlation between a structure’s molecular weight and resolution (r=.37), which suggests that macromolecules that have higher weights can tend to have higher resolution values, meaning lower resolution quality.</li><li><strong>Crystallization conditions &amp; resolution:</strong> Crystallization temperature has little impact on resolution (r ≈ 0.01), indicating other factors influence structure quality.</li><li><strong>pH effects on density:</strong> A slight negative correlation (r = -0.01) suggests increasing pH may slightly reduce macromolecular density, though the effect is minor.</li></ul><p><strong>Historical Trends and Evolution</strong></p><ul><li><strong>Structural Biology Growth:</strong> The number of structures published has increased significantly in the past 20 years, peaking at 3888 structures in 2017, which is the final year of data in this dataset.</li><li><strong>Crystallization Technology and Structure Discovery:</strong> Structure discovery has significantly improved with vapor diffusion (hanging drop) and vapor diffusion (sitting drop) accounting for most of the structure discovered in this dataset.</li><li><strong>Resolution Efficiency:</strong> The resolution values have improved over time with lower (better) Ångström values, showing that current structures are more likely to have a medium to high quality or atomic resolution. The average resolution has improved slightly from 2.21 Å (1995) to 2.17 Å (2018), showing technological advancements in structure determination.</li><li><strong>New Unique Sequence Patterns Decreasing:</strong> Over time the number of unique patterns found in sequences is slowly decreasing.</li><li><strong>Molecular Weight, Residue Count, and Sequence Length Increasing</strong>: The average molecular weight, residue count, and sequence length have increased over time, suggesting a trend toward discovering larger macromolecules. Ligase protein enzymes have shown the highest growth over time.</li><li><strong>Hydrolase Dominance:</strong> The most frequently classified macromolecules are hydrolase proteins, which also show the most structural growth over time for this dataset.</li></ul><p><strong>Statistical Models</strong></p><p><strong>Regression Analysis:</strong></p><ul><li><strong>Molecular Weight Prediction:</strong> A perfect linear relationship was found predicting the molecular weight using residue count, with a coefficient of 112.47 Da (Daltons)/residue. The perfect R² score (1.0) suggests a perfect fit, which is expected since molecular weight is calculated from the number of residues. While using a dependent variable to predict another is not good practice in regression, this example was used to show the utility of the chatbot’s capabilities.</li><li><strong>Density Prediction:</strong> A very strong parabolic relationship was found predicting the density based on the percent solvent with an R² score of 0.99 and a Mean Squared Error (MSE) of 0.00. The R² score suggests that 99% of the variance in the density can be explained by the model. The MSE of 0.0 indicates that the model is overfitting due to the lack of noise and outliers that were removed. This model again uses interdependent variables, which is not ideal for regression analysis but demonstrates the chatbot’s functionality.</li></ul><p><strong>Clustering Models</strong></p><ul><li><strong>Resolution and Molecular Weight: </strong>DNA shows the lowest molecular weights as well as the greatest range in resolution with some of the best resolution qualities. Proteins occupied the middle cluster, while RNA had the highest molecular weights with the smallest range of resolution values.</li><li><strong>Crystallization Temperature and pH Values: </strong>Proteins tend to have lower pH ranges, while DNA tends to have higher. RNA has the highest crystallization temperatures, with its pH ranges being the broadest.</li></ul><p>This chatbot did struggle with creating clustering models,<strong> </strong>specifically with creating and displaying the legend, color assignments, and displaying the metrics used. However, they show the potential for utilizing this tool in unsupervised learning applications.</p><p><strong>Final Thoughts</strong></p><p>This project highlights how GenAI can transform biological data analysis. The chatbot’s ability to instantly retrieve insights allows researchers to efficiently analyze macromolecular structures, identify patterns, and support hypothesis-driven research in structural biology, protein engineering, and drug discovery. By bridging computational biology with AI-driven insights, this chatbot can streamline research workflows and accelerate scientific discovery in the real world — <strong>without writing any code.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f437c566c9d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Data to Diagnostics: Advanced GenAI Solutions for Disease Prediction]]></title>
            <link>https://ernest-bonat.medium.com/from-data-to-diagnostics-advanced-genai-solutions-for-disease-prediction-5aefa726f499?source=rss-58ededb6e465------2</link>
            <guid isPermaLink="false">https://medium.com/p/5aefa726f499</guid>
            <dc:creator><![CDATA[Ernest Bonat, Ph.D.]]></dc:creator>
            <pubDate>Wed, 09 Apr 2025 10:45:58 GMT</pubDate>
            <atom:updated>2025-04-11T00:58:58.679Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jeJjo-TWfZLt_TafOoK3ag.jpeg" /></figure><p><strong>Ernest Bonat, Ph.D., Chad Turner</strong></p><p>1. <a href="#7c8c">Overview</a><br>2. <a href="#994b">Research Question</a><br>3. <a href="#0421">Diseases and Medical Conditions</a><br>4. <a href="#fd5e">Data Preprocessing</a><br>5. <a href="#f58a">Advanced Y-Data Profiling</a><br>6. <a href="#8ec2">Lazy Predict Models</a><br>7. <a href="#979e">Optuna Hyperparameters Optimization</a><br>8. <a href="#de47">Long Short-Term Memory Networks Model</a><br>9. <a href="#2206">Random Forest Classification Model</a><br>10. <a href="#e062">Disease Prediction Application</a><br>11. <a href="#0c9c">GenAI CSV Chatbot Application</a><br>12. <a href="#3608">Practical Applications in Healthcare</a><br>13. <a href="#7370">Conclusions</a></p><h4>1. Overview</h4><p>The dataset used for this Machine Learning (ML) project is the <a href="https://www.kaggle.com/datasets/karthikudyawar/disease-symptom-prediction/data">Predict Disease Symptom CSV</a> sourced from <a href="https://www.kaggle.com/">Kaggle.com</a>. It contains 42 medical diagnoses or conditions and 134 associated symptoms. The primary goal of this project is to create an accurate ML model capable of predicting and classifying diseases based on specified symptoms. Advanced data analysis tools will be used to expedite the ML process and enhance the accuracy of the models. The trained ML model will be integrated into a web-based application for real-time disease prediction. Additionally, a generative AI-powered CSV Chatbot will be developed to validate predictions by ensuring that all identified symptoms align with the predicted disease.</p><p>The target used for this project is a column labeled “prognosis,” which is populated with the 42 medical diagnoses and conditions. The features are 134 columns of different symptoms with binary values to represent if they are present for each disease.</p><ul><li>1: The symptom is present in the diagnosis/condition.</li><li>0: The symptom is not present in the diagnosis/condition.</li></ul><p>Multiclass classification will be used to predict diseases based on the symptoms provided.</p><p>For a comprehensive understanding of a complete ML project workflow, including data preprocessing, model selection, and evaluation, refer to <a href="https://ernest-bonat.medium.com/using-machine-learning-models-for-breast-cancer-diagnosis-a-complete-machine-learning-project-8af316843a8e"><strong>Using Machine Learning Models for Breast Cancer Diagnosis — A Complete Machine Learning Project Workflow for Life Sciences. Ernest Bonat, Ph.D., Chad Turner</strong></a></p><h4>2. Research Question</h4><p>In the fast-paced world of Healthcare, early and accurate diagnoses are essential for providing the best possible patient care and outcomes. However, the traditional diagnosis process can be lengthy and prone to human error. This raises an important question: <em>How can Machine Learning and Generative AI (GenAI) be used to enhance the speed and accuracy of disease diagnosis based on a patient’s symptoms?</em> By utilizing AI and advanced ML tools and models, this project explores how to streamline the diagnosis process, aiding medical professionals and improving patients’ experience.</p><h4>3. Diseases and Medical Conditions</h4><ul><li><strong>Infectious Diseases: </strong>caused by various pathogens like viruses, bacteria, or parasites — AIDS, Hepatitis A/B/D/E, Tuberculosis, Typhoid, Malaria, Dengue, Chicken Pox, and Pneumonia.</li><li><strong>Chronic Diseases: </strong>chronic conditions that often have a slow progression and typically require long-term management — Hypertension, Diabetes, Arthritis, Osteoarthritis, and Hyperthyroidism.</li><li><strong>Autoimmune and Inflammatory Diseases: </strong>immune system dysregulation or inflammation, either in specific body parts or more widely — Psoriasis, Rheumatoid Arthritis, Bronchial Asthma, Allergy, and Gastroenteritis.</li><li><strong>Endocrine Disorders:</strong> disorders related to the endocrine system, which regulates hormone levels in the body — Hypothyroidism, Hyperthyroidism, and Hypoglycemia<strong>.</strong></li><li><strong>Neurological Conditions: </strong>affect the nervous system, either due to injury, degenerative processes, or other factors — Paralysis (brain hemorrhage), Migraine, and Paroxysmal Positional Vertigo.</li><li><strong>Dermatological Conditions:</strong> common skin diseases, affecting the surface or deeper layers of the skin — Acne, Impetigo, Fungal Infection, and Psoriasis.</li><li><strong>Digestive and Hepatic Conditions: </strong>affect the digestive system, with conditions ranging from acid reflux to inflammation of the liver — GERD (Gastroesophageal reflux disease), Peptic Ulcer Disease, and Alcoholic Hepatitis.</li><li><strong>Metabolic Disorders: </strong>conditions affecting how the body processes sugars or energy — Hypoglycemia and Diabetes.</li><li><strong>Acute Conditions or Reactions: </strong>acute illnesses, often short-term and caused by infections or reactions to substances — Drug Reactions, Common Cold, Allergy, Food Poisoning, and Gastroenteritis.</li></ul><p><strong>Miscellaneous:</strong></p><ul><li>Cervical Spondylosis (a degenerative condition of the spine), Paralysis, and Arthritis</li></ul><h4>4. Data Preprocessing</h4><p>To begin this project, the retrieved data was already split into training and testing sets and one-hot encoded, with each symptom represented in its own column. To ensure consistency in preprocessing, these datasets are first combined so that any modifications are applied uniformly. The preprocessing begins by loading the CSV files as Pandas DataFrames, which are then concatenated as one new DataFrame. The shape, null sum, duplicate count, and DataFrame information are printed.</p><pre>=== Information for Original Concatendated DataFrame ===<br><br>DataFrame shape:<br> (348, 135)<br>Null value count:<br> Series([], dtype: int64)<br>Duplicate value count:<br> 42<br>DataFrame Info:<br>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>Index: 348 entries, 0 to 41<br>Columns: 135 entries, abdominal_pain to prognosis<br>dtypes: int64(134), object(1)<br>memory usage: 369.8+ KB<br>None</pre><p>As shown, there are 348 rows and 135 columns, with no null values but 42 duplicated values. One issue to address is that the index appears to be misaligned — despite having 348 entries, the index range is only 0 to 41. The data consist of 134 columns of integers and one categorical column (“prognosis”) that will need to be encoded for ML models.</p><p>To clean the dataset, duplicates are first removed. The target column “prognosis” is then printed, counting all instances for each class to determine if there is class imbalance and if there are classes that need to be dropped due to low representation.</p><pre>Original label counts:<br>prognosis<br>Dengue                           10<br>Hepatitis D                      10<br>Chicken pox                      10<br>Migraine                         10<br>Diabetes                          9<br>Hypoglycemia                      9<br>Hepatitis B                       9<br>Hepatitis A                       9<br>Jaundice                          9<br>Hyperthyroidism                   9<br>Hepatitis E                       9<br>Common Cold                       9<br>Pneumonia                         9<br>Tuberculosis                      9<br>Typhoid                           9<br>Chronic cholestasis               8<br>Malaria                           8<br>Hypothyroidism                    8<br>Alcoholic hepatitis               8<br>Varicose veins                    8<br>Peptic ulcer disease              7<br>Paroxysmal Positional Vertigo     7<br>Osteoarthritis                    7<br>Psoriasis                         7<br>Hepatitis C                       7<br>GERD                              7<br>Bronchial Asthma                  7<br>Hypertension                      6<br>Cervical spondylosis              6<br>Impetigo                          6<br>Arthritis                         6<br>Drug Reaction                     6<br>Dimorphic hemorrhoids(piles)      6<br>Urinary tract infection           5<br>Allergy                           5<br>Paralysis (brain hemorrhage)      5<br>Acne                              5<br>Fungal infection                  5<br>Gastroenteritis                   5<br>Heart attack                      5<br>AIDS                              5<br>Covid                             2<br>Name: count, dtype: int64p</pre><p>The dataset shows class imbalance and the class “Covid” having only two instances, which will be addressed as this can hinder model performance. The rare label “Covid” is then removed to improve model generalization and avoid overfitting.</p><p>Since “prognosis” is the only non-numeric column, it is label encoded to change the string values to encoded integers. Additionally, the index is reset to “0 to 359.”</p><p>The synthetic minority oversampling technique (SMOTE) is applied to the data to create synthetic data addressing the dataset’s small size and helping balance the classes. Duplicates were created during this process, so they are dropped, and the index is reset. The DataFrame’s shape, and information are printed to verify the changes.</p><pre>=== Information for Encoded DataFrame with SMOTE ===<br><br>DataFrame shape:<br> (360, 135)<br><br>Null value count:<br> Series([], dtype: int64)<br><br>Duplicate value count:<br> 0<br><br>DataFrame Info:<br>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 360 entries, 0 to 359<br>Columns: 135 entries, abdominal_pain to prognosis<br>dtypes: int32(1), int64(134)<br>memory usage: 378.4 KB<br>Nonep</pre><p>The original DataFrame had a shape of (348, 135) with 42 duplicates. After applying SMOTE and preprocessing, the shape expanded to (360, 135), successfully adding 54 new rows, with no duplicates and no null values. The cleaned dataset is then exported as a CSV file.</p><p>A version of the DataFrame with its original labels for the “prognosis” column is then built to explore the data in applications that use natural language. The original label names are retrieved and then concatenated with the resampled data created above. The duplicates are dropped, the index is reset, and information is printed to confirm this labeled dataset mirrors the encoded version.</p><pre>=== Information for DataFrame with SMOTE and Labels ===<br><br>DataFrame shape:<br> (360, 135)<br>Null value count:<br> Series([], dtype: int64)<br>Duplicate value count:<br> 0<br>DataFrame Info:<br>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 360 entries, 0 to 359<br>Columns: 135 entries, abdominal_pain to prognosis<br>dtypes: int64(134), object(1)<br>memory usage: 379.8+ KB<br>Nonep</pre><p>The label counts are printed for “prognosis” to inspect how the class counts have changed.</p><pre>SMOTE label counts:<br>prognosis<br>Common Cold                      10<br>Hepatitis A                      10<br>Migraine                         10<br>Hepatitis D                      10<br>Jaundice                         10<br>Pneumonia                        10<br>Hepatitis E                      10<br>Diabetes                         10<br>Dengue                           10<br>Hepatitis B                      10<br>Chicken pox                      10<br>Hyperthyroidism                  10<br>Hypoglycemia                     10<br>Tuberculosis                     10<br>Typhoid                          10<br>Hypertension                      9<br>Hypothyroidism                    9<br>Varicose veins                    9<br>Malaria                           9<br>Dimorphic hemorrhoids(piles)      9<br>Psoriasis                         9<br>Cervical spondylosis              9<br>Bronchial Asthma                  9<br>Alcoholic hepatitis               9<br>Osteoarthritis                    8<br>Paralysis (brain hemorrhage)      8<br>Paroxysmal Positional Vertigo     8<br>Peptic ulcer disease              8<br>Hepatitis C                       8<br>Impetigo                          8<br>GERD                              8<br>Drug Reaction                     8<br>Chronic cholestasis               8<br>Arthritis                         8<br>Acne                              7<br>Heart attack                      7<br>Gastroenteritis                   7<br>Fungal infection                  7<br>Allergy                           7<br>Urinary tract infection           7<br>AIDS                              7<br>Name: count, dtype: int64</pre><p>SMOTE successfully increased the class counts, with each class having at least 7 instances. This will help in training ML models and prevent overfitting. However, the classes still show imbalance, so SMOTE will need to be applied to the training data during ML processes to give each class equal representation and prevent the models from being biased toward majority classes. This DataFrame now reflects the encoded DataFrame, providing the same shape, index, no null values, and no duplicates. It is then exported as an additional CSV file.</p><h4>5. Advanced Y-Data Profiling</h4><p>When beginning a data analysis project, one of the first tasks is to explore and understand the data. This typically is achieved through lengthy code with multiple functions and print statements. However, the Y-data profiling package simplifies this process significantly by generating a comprehensive exploratory data analysis (EDA) report in only a few lines of code. This package automatically generates a detailed HTML report from a DataFrame, providing statistics and visualizations for the data, vastly simplifying EDA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mUA4vMlPwfcGthR7k5ytwA.png" /></figure><h4>6. Lazy Predict Models</h4><p>To begin creating ML models for the dataset, a useful approach is to build multiple models and compare their metrics. This provides a solid starting point before beginning tasks such as tuning hyperparameters, feature engineering, using cross-validation, and creating complex models. However, creating and evaluating numerous models can be time-consuming and resource-intensive. Lazy Predict, a Python library, provides a solution to this issue. Lazy Predict allows one to quickly access multiple classification or regression ML models using their default parameters in one program without the need for extensive coding.</p><p>To use Lazy Predict, a basic ML learning model is created. The encoded CSV file is loaded, and the target and features are defined. Next, the data is split into training and testing sets, allocating 80% of the data for training while reserving the remaining 20% for testing. As noted earlier, the data is imbalanced, which may cause ML models to be biased toward majority classes. To address this, SMOTE is applied but only to the training data to prevent data leakage and keep testing data unseen by the model.</p><p>The model is built using Lazy Classifier from the Lazy Predict library, where the model is fit to the training and testing data. A DataFrame containing the model performance metrics is then stored. The model performance summary is then printed, and the best-performing model is displayed. Finally, a horizontal bar chart of all the models’ performance is plotted in ascending order.</p><pre>Model Performance Summary:<br>                               Accuracy  Balanced Accuracy ROC AUC  F1 Score  Time Taken<br>Model<br>LinearSVC                          1.00               1.00    None      1.00        2.82<br>GaussianNB                         1.00               1.00    None      1.00        0.03<br>Perceptron                         1.00               1.00    None      1.00        0.06<br>PassiveAggressiveClassifier        1.00               1.00    None      1.00        0.08<br>NuSVC                              1.00               1.00    None      1.00        0.05<br>NearestCentroid                    1.00               1.00    None      1.00        0.02<br>LogisticRegression                 1.00               1.00    None      1.00        0.10<br>LinearDiscriminantAnalysis         1.00               1.00    None      1.00        0.14<br>RandomForestClassifier             1.00               1.00    None      1.00        0.20<br>SGDClassifier                      1.00               1.00    None      1.00        0.05<br>SVC                                1.00               1.00    None      1.00        0.05<br>CategoricalNB                      1.00               1.00    None      1.00        0.10<br>BernoulliNB                        1.00               1.00    None      1.00        0.03<br>CalibratedClassifierCV             0.99               0.99    None      0.99       13.02<br>RidgeClassifierCV                  0.99               0.99    None      0.99        0.11<br>KNeighborsClassifier               0.99               0.99    None      0.99        0.40<br>RidgeClassifier                    0.97               0.96    None      0.97        0.07<br>ExtraTreesClassifier               0.96               0.95    None      0.96        0.16<br>BaggingClassifier                  0.93               0.93    None      0.93        0.08<br>XGBClassifier                      0.92               0.93    None      0.93        0.95<br>LabelSpreading                     0.88               0.85    None      0.87        0.04<br>LabelPropagation                   0.88               0.85    None      0.87        0.04<br>LGBMClassifier                     0.72               0.77    None      0.72        2.36<br>ExtraTreeClassifier                0.78               0.76    None      0.79        0.02<br>DecisionTreeClassifier             0.78               0.75    None      0.78        0.03<br>AdaBoostClassifier                 0.39               0.33    None      0.39        0.17<br>QuadraticDiscriminantAnalysis      0.03               0.03    None      0.00        0.06<br>DummyClassifier                    0.03               0.03    None      0.00        0.02</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*3kNULmSa0k1gP3GDY-I6ZA.png" /></figure><p><strong>Lazy Predict Classification Model Performance</strong></p><p>These models are measured by accuracy, balanced accuracy, F1 score, and time taken. The best classification model for predicting diseases with this dataset is the Linear SVC model at 100% accuracy along with several other algorithms. Calibrated Classifier CV, Ridge Classifier CV, and K Neighbors Classifier all achieved near-perfect performance, showing 99% in all metrics. The AdaBoost Classifier, Quadratic Discriminant Analysis, and Dummy Classifier all had very poor performance, proving that these would not be a good choice for this dataset. With several models achieving 100% accuracy, the models are likely overfitting, which could be due to the small size of the dataset.</p><h4>7. Optuna Hyperparameters Optimization</h4><p>With the best-performing models identified, the next step in the ML workflow is hyperparameter tuning and optimization. This process can be tedious, requiring extensive trial and error. However, Optuna simplifies this by efficiently searching for the best-performing hyperparameters within specified ranges using Bayesian optimization, outperforming traditional grid search.</p><p>To use Optuna, an objective function is defined and suggested hyperparameters to tune and their ranges to be used are specified. The model is then trained and evaluated by running the optimization function over multiple trials, testing different hyperparameter combinations, and recording their performance. After running the trials, the best-performing hyperparameters for the model are then returned, effectively automating the process of hyperparameter tuning.</p><p>For hyperparameter optimization with Random Forest Classifier, the encoded CSV file is loaded. The data is then split into 80% training and 20% testing, with SMOTE applied to balance the training data. The objective function is defined to optimize the following Random Forest hyperparameters within specified ranges:</p><ul><li><strong>Number of estimators:</strong> The number of trees in the random forest, ranging from 100 to 1000.</li><li><strong>Max depth:</strong> The maximum depth of each tree, set between 10 to 150.</li><li><strong>Minimum samples split:</strong> The minimum number of samples needed to split an internal node.</li><li><strong>Minimum samples leaf</strong>: The minimum number of samples required to be at a leaf node.</li></ul><p>Next, the optimization process is run for a specified number of trials, with an early stopping condition to prevent unnecessary calculations. Once completed, Optuna outputs the best-performing hyperparameters along with the highest accuracy score.</p><p>Additionally, Optuna simplifies visualization by using the Plotly library. This generates plots illustrating the relationship between hyperparameters and model performance. Plots are created for optimization, parallel coordinates, slicing, and the parameter importance. These visualizations help explain why specific hyperparameters were chosen and how they influence the final model.</p><p>Results:</p><pre>[I 2025-02-24 18:54:36,570] Trial 13 finished with value: 0.7777777777777778 and parameters: {&#39;n_estimators&#39;: 985, &#39;max_depth&#39;: 42, &#39;min_samples_split&#39;: 14, &#39;min_samples_leaf&#39;: 18, &#39;max_features&#39;: &#39;log2&#39;}. Best is trial 7 with value: 0.9861111111111112. <br>[I 2025-02-24 18:54:37,823] Trial 14 finished with value: 0.9861111111111112 and parameters: {&#39;n_estimators&#39;: 861, &#39;max_depth&#39;: 10, &#39;min_samples_split&#39;: 3, &#39;min_samples_leaf&#39;: 6, &#39;max_features&#39;: &#39;sqrt&#39;}. Best is trial 7 with value: 0.9861111111111112.   <br>[I 2025-02-24 18:54:38,706] Trial 15 finished with value: 0.8333333333333334 and parameters: {&#39;n_estimators&#39;: 701, &#39;max_depth&#39;: 81, &#39;min_samples_split&#39;: 13, &#39;min_samples_leaf&#39;: 16, &#39;max_features&#39;: &#39;log2&#39;}. Best is trial 7 with value: 0.9861111111111112. <br>[I 2025-02-24 18:54:39,377] Trial 16 finished with value: 0.7638888888888888 and parameters: {&#39;n_estimators&#39;: 491, &#39;max_depth&#39;: 120, &#39;min_samples_split&#39;: 2, &#39;min_samples_leaf&#39;: 22, &#39;max_features&#39;: &#39;sqrt&#39;}. Best is trial 7 with value: 0.9861111111111112. <br>[I 2025-02-24 18:54:40,627] Trial 17 finished with value: 0.9861111111111112 and parameters: {&#39;n_estimators&#39;: 880, &#39;max_depth&#39;: 56, &#39;min_samples_split&#39;: 19, &#39;min_samples_leaf&#39;: 6, &#39;max_features&#39;: &#39;log2&#39;}. Best is trial 7 with value: 0.9861111111111112.  <br>[I 2025-02-24 18:54:41,962] Trial 18 finished with value: 0.9861111111111112 and parameters: {&#39;n_estimators&#39;: 924, &#39;max_depth&#39;: 82, &#39;min_samples_split&#39;: 11, &#39;min_samples_leaf&#39;: 6, &#39;max_features&#39;: &#39;log2&#39;}. Best is trial 7 with value: 0.9861111111111112.  <br>[I 2025-02-24 18:54:43,000] Trial 19 finished with value: 0.8055555555555556 and parameters: {&#39;n_estimators&#39;: 783, &#39;max_depth&#39;: 31, &#39;min_samples_split&#39;: 5, &#39;min_samples_leaf&#39;: 14, &#39;max_features&#39;: &#39;sqrt&#39;}. Best is trial 7 with value: 0.9861111111111112.  <br>Best Parameters: {&#39;n_estimators&#39;: 758, &#39;max_depth&#39;: 149, &#39;min_samples_split&#39;: 2, &#39;min_samples_leaf&#39;: 6, &#39;max_features&#39;: &#39;sqrt&#39;}<br>Best Accuracy Score: 0.9861</pre><p><strong>Optimization Summary Results</strong></p><p><strong>Best Parameters:</strong></p><ul><li>n_estimators: 758</li><li>max_depth: 149</li><li>min_samples_split: 2</li><li>min_samples_leaf: 6</li><li>max_features: ‘sqrt’</li></ul><p><strong>Best Accuracy Score:</strong> 98.61%</p><p>The max depth value of 149 shows that the model is learning complex patterns by using a deep tree structure. By using “sqrt” for the max features, the model uses the square root of the total number of features. This reduces the number of features evaluated during each split while introducing randomness to avoid overfitting. The minimum samples leaf value of 6 helps each leaf node have enough samples to prevent overly complex splits. These optimized hyperparameters enhance the Random Forest Classifier, achieving an accuracy score of 98.61%. These hyperparameters can now be used in a final Random Forest Classification model to maintain high accuracy while ensuring robustness against overfitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2gJQMSGY86v1x7zLjl-JHg.png" /></figure><p>The Optimization History Plot tracks how accuracy improves over trials. As the model runs trials, the accuracy is gradually increasing until it plateaus at 98.6%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nocN9kpRzupvem3zbWVviw.png" /></figure><p>The Hyperparameter Importance Plot highlights the most influential hyperparameters in model performance. Minimum samples leaf is shown to be the most important feature by a large margin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Akhgem-cW9K_YVun7M8MMA.png" /></figure><p>The Slice Plot shows how individual hyperparameters influence the model. Small clusters in the range of the parameters show improvement over the 20 trials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAT6F-Apxw2aJELuNth0Jg.png" /></figure><p>The Parallel Coordinate Plot shows how the model tests the hyperparameters, with the darker lines representing better model performance.</p><h4>8. Long Short-Term Memory Networks Model</h4><p>Recurrent Neural Networks (RNNs) are deep learning models that process sequential data over numerous steps, making them common for time series analysis and natural language processing. While RNNs have similar architecture to feedforward neural networks, they also have connections to past observations through feedback loops. RNNs generate an output for an observation, but they retain a memory, or hidden state, that is updated and applied to the next observations, allowing the model to learn patterns and make predictions on future observations. As this recurrent connection is looped back through the model, weights are shared, which lowers the complexity of the model. After the forward pass, the model uses Backpropagation Through Time (BPTT) to fine-tune weights during a backwards pass, allowing the model to learn and correct errors.</p><p>One issue with traditional RNNs is the vanishing and exploding gradient problem. This is where gradients become so small or large through repeated multiplication of the same weights during BPTT that they vanish or explode during training. This can cause the model to perform poorly, especially for long sequences. Fortunately, this issue is addressed through many different architectures, such as Long Short-Term Memory (LSTM) networks. LSTMs use three gates — forget, input, and output gates — which determine the flow of data and selectively retain important past information.</p><p><strong>Building an LSTM Model for Disease Prediction</strong></p><p>Since this disease prediction dataset is non-sequential, the timesteps dimension is set to one, treating each sample independently. The feature dataset is reshaped to (number of samples, 1, number of features) to fit the RNN input format.</p><p>The model is built using TensorFlow’s Sequential() API:</p><ol><li><strong>LSTM Layer (Input Layer)</strong>: Includes 32 neurons, balancing computational efficiency and model complexity.</li><li><strong>Regularization</strong>: A kernel regularizer (0.02) prevents overly complex patterns, reducing overfitting.</li><li><strong>Dropout Layer (0.3)</strong>: Randomly drops 30% of neurons per training step to enhance generalization.</li><li><strong>Output Layer</strong>: Uses a Dense layer with the softmax activation function, converting outputs into probability distributions for classification.</li></ol><p>The model is compiled with:</p><ul><li><strong>Optimizer</strong>: Adaptive Moment Estimation (Adam) (learning rate = 0.003), a quicker learning rate ensuring stable convergence.</li><li><strong>Loss Function</strong>: sparse_categorical_crossentropy, ideal for multi-class classification with encoded labels.</li><li><strong>Metrics</strong>: Accuracy, measuring the proportion of correct predictions made by the model.</li></ul><p>After defining the model architecture, the model summary is called to show the structure of the model, including the layers, parameters, and output shape.</p><p><strong>Model Summary</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*iNoxypNjXPe9FUIHFy_kkw.png" /></figure><p>The model is trained on the balanced training data for 50 epochs, allowing 50 iterations through the dataset. The batch size is set to 32, meaning the number of samples processed before updating the weights. Early stopping (patience = 5) is enabled, halting training if validation loss does not improve for five consecutive epochs, preventing overfitting.</p><p>After training, the model is evaluated using the test data. The accuracy score, classification report, and confusion matrix are created and shown. The model’s training and validation history is then plotted to visualize how the model’s accuracy and loss perform over epochs. The confusion matrix is created into a larger plot to inspect the class predictions against their true labels.</p><p><strong>Results</strong></p><pre>Epoch 40/50<br>10/10 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.9789 - loss: 1.1709 - val_accuracy: 0.8919 - val_loss: 1.4644<br>Epoch 41/50<br>10/10 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.9667 - loss: 1.1588 - val_accuracy: 0.8919 - val_loss: 1.4289<br>Epoch 42/50<br>10/10 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9772 - loss: 1.1080 - val_accuracy: 0.8919 - val_loss: 1.3953<br>Epoch 43/50<br>10/10 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.9821 - loss: 1.0977 - val_accuracy: 0.8919 - val_loss: 1.3598<br>Epoch 44/50<br>10/10 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9812 - loss: 1.0855 - val_accuracy: 0.8919 - val_loss: 1.3205<br>Epoch 45/50<br>10/10 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9852 - loss: 1.0583 - val_accuracy: 0.9054 - val_loss: 1.2894<br>Epoch 46/50<br>10/10 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.9855 - loss: 1.0357 - val_accuracy: 0.9054 - val_loss: 1.2628<br>Epoch 47/50<br>10/10 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.9892 - loss: 0.9702 - val_accuracy: 0.9054 - val_loss: 1.2321<br>Epoch 48/50<br>10/10 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.9834 - loss: 0.9814 - val_accuracy: 0.9054 - val_loss: 1.2069<br>Epoch 49/50<br>10/10 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step - accuracy: 0.9864 - loss: 0.9592 - val_accuracy: 0.9054 - val_loss: 1.1838<br>Epoch 50/50<br>10/10 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.9845 - loss: 0.9429 - val_accuracy: 0.9054 - val_loss: 1.1616<br>3/3 ━━━━━━━━━━━━━━━━━━━━ 0s 37ms/step<br>Accuracy: 0.9861<br>Classification Report:<br>              precision    recall  f1-score   support<br><br>           0       0.67      1.00      0.80         2<br>           1       1.00      1.00      1.00         1<br>           2       1.00      1.00      1.00         2<br>           3       1.00      1.00      1.00         1<br>           4       1.00      1.00      1.00         1<br>           5       1.00      1.00      1.00         5<br>           7       1.00      1.00      1.00         3<br>           9       1.00      1.00      1.00         5<br>          10       1.00      1.00      1.00         1<br>          11       1.00      1.00      1.00         2<br>          12       1.00      1.00      1.00         3<br>          13       1.00      1.00      1.00         3<br>          14       1.00      1.00      1.00         2<br>          15       1.00      1.00      1.00         2<br>          16       1.00      0.75      0.86         4<br>          18       1.00      1.00      1.00         2<br>          19       1.00      1.00      1.00         2<br>          21       1.00      1.00      1.00         1<br>          22       1.00      1.00      1.00         2<br>          23       1.00      1.00      1.00         3<br>          24       1.00      1.00      1.00         3<br>          25       1.00      1.00      1.00         2<br>          26       1.00      1.00      1.00         1<br>          27       1.00      1.00      1.00         1<br>          28       1.00      1.00      1.00         1<br>          29       1.00      1.00      1.00         4<br>          30       1.00      1.00      1.00         1<br>          31       1.00      1.00      1.00         1<br>          32       1.00      1.00      1.00         1<br>          33       1.00      1.00      1.00         2<br>          34       1.00      1.00      1.00         2<br>          35       1.00      1.00      1.00         2<br>          36       1.00      1.00      1.00         1<br>          37       1.00      1.00      1.00         2<br>          38       1.00      1.00      1.00         1<br><br>    accuracy                           0.99        72<br>   macro avg       0.99      0.99      0.99        72<br>weighted avg       0.99      0.99      0.99        72<br><br>confusion matrix:<br>[[2 0 0 ... 0 0 0]<br> [0 1 0 ... 0 0 0]<br> [0 0 2 ... 0 0 0]<br> ...<br> [0 0 0 ... 1 0 0]<br> [0 0 0 ... 0 2 0]<br> [0 0 0 ... 0 0 1]]</pre><p><strong>Model Performance</strong></p><p>Accuracy: 0.9861</p><p>98.61% of the classifications are correct.</p><p>In the classification report, the precision, recall, and F1-scores are consistently high, all near 100%, showing that the model has good validity.</p><p><strong>Confusion Matrix</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*awe5-Y4zJkAQ2gqtw31TYQ.png" /></figure><p>The predictions are concentrated in the diagonal line of self-correlation, showing that the model is correctly classifying most of the predicted labels with only one misclassification. Label 14, which is Drug Reaction, was mislabeled as AIDS. The last five classes were not predicted due to the test size not including all classes.</p><p><strong>Accuracy and Loss History</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z4_SKlzkYWWUTE3a_9VNCA.png" /></figure><p>The loss shows a stable convergence and that overfitting is being minimized. The accuracy plot shows that the training and testing data’s accuracy improves until it plateaus at high rates.</p><h4>9. Random Forest Classification Model</h4><p>To use a ML model for disease prediction in a production environment, a Random Forest Classifier is built using the hyperparameters obtained from Optuna. Before deployment, the model is validated to ensure accuracy and reliability. The dataset with true labels is used, allowing disease names to be directly integrated into an application interface for interpretability and user experience.</p><p><strong>Results</strong></p><pre>Accuracy: 0.9861<br>Classification Report:<br>                               precision    recall  f1-score   support<br><br>                         AIDS       1.00      1.00      1.00         2<br>                         Acne       1.00      1.00      1.00         1<br>          Alcoholic hepatitis       1.00      1.00      1.00         2<br>                      Allergy       1.00      1.00      1.00         1<br>                    Arthritis       1.00      1.00      1.00         1<br>             Bronchial Asthma       1.00      1.00      1.00         5<br>         Cervical spondylosis       0.00      0.00      0.00         0<br>                  Chicken pox       1.00      1.00      1.00         3<br>                  Common Cold       1.00      1.00      1.00         5<br>                       Dengue       1.00      1.00      1.00         1<br>                    Diabetes        1.00      1.00      1.00         2<br> Dimorphic hemorrhoids(piles)       1.00      1.00      1.00         3<br>                Drug Reaction       1.00      1.00      1.00         3<br>             Fungal infection       1.00      1.00      1.00         2<br>                         GERD       1.00      1.00      1.00         2<br>              Gastroenteritis       1.00      1.00      1.00         4<br>                  Hepatitis A       1.00      1.00      1.00         2<br>                  Hepatitis B       1.00      1.00      1.00         2<br>                  Hepatitis D       1.00      1.00      1.00         1<br>                  Hepatitis E       1.00      1.00      1.00         2<br>                Hypertension        1.00      0.67      0.80         3<br>              Hyperthyroidism       1.00      1.00      1.00         3<br>                 Hypoglycemia       1.00      1.00      1.00         2<br>               Hypothyroidism       1.00      1.00      1.00         1<br>                     Impetigo       1.00      1.00      1.00         1<br>                     Jaundice       1.00      1.00      1.00         1<br>                      Malaria       1.00      1.00      1.00         4<br>                     Migraine       1.00      1.00      1.00         1<br>               Osteoarthritis       1.00      1.00      1.00         1<br> Paralysis (brain hemorrhage)       1.00      1.00      1.00         1<br>Paroxysmal Positional Vertigo       1.00      1.00      1.00         2<br>         Peptic ulcer disease       1.00      1.00      1.00         2<br>                    Pneumonia       1.00      1.00      1.00         2<br>                    Psoriasis       1.00      1.00      1.00         1<br>                 Tuberculosis       1.00      1.00      1.00         2<br>                      Typhoid       1.00      1.00      1.00         1<br><br>                     accuracy                           0.99        72<br>                    macro avg       0.97      0.96      0.97        72<br>                 weighted avg       1.00      0.99      0.99        72<br><br>confusion matrix:<br>[[2 0 0 ... 0 0 0]<br> [0 1 0 ... 0 0 0]<br> [0 0 2 ... 0 0 0]<br> ...<br> [0 0 0 ... 1 0 0]<br> [0 0 0 ... 0 2 0]<br> [0 0 0 ... 0 0 1]]<br>model saved as predict_prognosis_model.pkl</pre><p><strong>Model Performance</strong></p><p>Accuracy: 0.9861</p><p>98.61% of the classifications are correct. Overall, the precision, recall, and F1-Score look very good, with most showing 100%. However, the disease Cervical Spondylosis received no instances, resulting in 0 scores across all metrics. The condition Hypertension showed the next poorest metrics with a 67% recall and an 80% F1-Score.</p><p><strong>Confusion Matrix</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*iefauZsgKgsgaOWMiueUYg.png" /></figure><p>The confusion matrix shows a strong diagonal line showing that the model is making the correct predictions for most of the diseases and conditions.</p><p>With the Random Forest Classification model achieving a high accuracy score comparable to the LSTM, it is selected to create a production web application due to its simplicity and lower resource use.</p><h4>10. Disease Prediction Application</h4><p>To deploy the Random Forest ML model, a web application is developed for disease and condition prediction based on user-provided symptoms. The interface supports two input methods:</p><p><strong>1. CSV Upload</strong> — Users can upload a CSV file formatted with symptom columns matching the dataset. This format is also displayed on the front page as “Input Data for Prediction.” After uploading, the file is read as a DataFrame, displaying the first five rows as a preview. Predictions are automatically generated and displayed below the file preview using the Random Forest model, adding a “Predicted Class” column with the predicted diagnosis for each row of symptoms. A “Download Predictions as CSV” button allows users to save the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*-t4qwdyNjSuCKnAqWunY3w.png" /></figure><p><strong>2. Symptom Selection via Sidebar</strong> — Users can manually select symptoms from a formatted list of checkboxes in the sidebar. Selected symptoms then appear on the main page. Clicking “Predict Diagnosis” runs the Random Forest model, returning and displaying a predicted diagnosis. A “Clear All Symptoms” button at the bottom of the sidebar resets the session for new predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eOBKqM0tYVi0LGVhSO75iw.png" /></figure><h4>11. GenAI CSV Chatbot Application</h4><p>In addition to the disease prediction application, a user-friendly web application has been developed to analyze CSV files using generative AI, providing a more accessible approach to data exploration. This tool enables users to explore data without requiring programming knowledge. By simply uploading and selecting a CSV file, users can interact with the dataset and gain insights by asking questions in natural language. The chatbot processes these queries and provides relevant responses, making data exploration intuitive and efficient.</p><p>This application uses <a href="https://groq.com/">Groq AI</a> with the <a href="https://ai.meta.com/blog/meta-llama-3/">Llama 3 LLM</a> model along with <a href="https://github.com/sinaptik-ai/pandas-ai">PandasAI’s</a> SmartDataFrame to converse with the uploaded CSV file.</p><p><strong>Workflow:</strong></p><ol><li><strong>User Authentication</strong> — Users log in with a valid username and password, granting access to the “File Chatbot Site.”</li><li><strong>CSV File Management</strong> — Users can upload multiple CSV files and select one to work with. Upon selection, the first three rows of the dataset are displayed.</li><li><strong>Query System</strong> — A text box allows users to enter questions in natural language. Clicking “Submit” prompts the LLM and Pandas SmartDataFrame to process the query and generate a response.</li><li><strong>Query History</strong> — The user’s query history is displayed at the bottom of the page, maintaining a record of all questions and responses for the current session.</li></ol><p><strong>Chatbot Application Examples</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GGVuwP-WBOPuB0DOqGfbig.png" /></figure><h4>12. Practical Applications in Healthcare</h4><ul><li>In a healthcare setting, the CSV Chatbot and Disease Prediction Applications can work together to streamline disease diagnosis and symptom-based analysis. A physician can upload a dataset of patient symptoms to the Disease Prediction App, which generates a new dataset with predicted diagnoses. This predicted dataset can then be uploaded to the CSV Chat App to analyze the predictions, explore trends, and gain insights using natural language queries.</li><li>Additionally, physicians can use the Disease Prediction Application for quick preliminary diagnoses. A patient can describe their symptoms, and the physician selects the corresponding symptoms in the application to receive an instant diagnosis. The physician can then use the CSV Chatbot as a validation tool to cross-check the predicted diagnosis. Physicians can ask the chatbot what symptoms are associated with the predicted disease to ensure that the diagnosis aligns with the patient’s condition.</li><li>In an increasingly virtual healthcare environment, the Disease Prediction Application can also enhance telemedicine and pre-appointment screenings. Patients often fill out forms online prior to their appointments. With this application, a patient can select the symptoms they are experiencing, and the system could generate a preliminary diagnosis that is sent straight to their physician. This would give doctors an early indication of possible conditions, potentially saving time and improving the efficiency of in-person consultations.</li></ul><h4>13. Conclusions</h4><ol><li>The dataset used was relatively small (306 rows after dropping duplicates), but by implementing SMOTE and removing generated duplicates, it was successfully expanded to 360 rows — adding 54 rows of synthetically generated samples. While this helped reduce overfitting, the models still had challenges due to the low class counts. Ideally, a larger dataset with more balanced data would be used to allow the algorithms to learn more patterns, improving disease prediction accuracy.</li><li>Y-Data profiling provided a strong starting point to understanding the data with minimal coding involved. It highlighted key statistics, correlations, and alerts, which are good to consider before preprocessing the data. The CSV Chatbot application also enhanced the EDA experience by enabling user interaction with the dataset through natural language.</li><li>Lazy Predict streamlined the initial ML modeling process by quickly evaluating multiple classification algorithms. However, many models exhibited overfitting, achieving 100% accuracy. Still, it provided valuable insights into model performance and helped guide the selection of the most promising algorithms.</li><li>Optuna significantly simplified hyperparameter optimization. Instead of manually tuning parameters or using GridSearchCV, Optuna efficiently explored different combinations over multiple trials, helping to prevent overfitting. In contrast, tuning the LSTM model manually was time-consuming and required careful balancing to avoid overfitting.</li><li>Both the LSTM and Random Forest ML models achieved an accuracy score of 98.61%. This could be from the simplicity of the dataset or that each model was able to successfully capture essential patterns within the data. Given their equal performance, the Random Forest Classification model was chosen for deployment due to its simplicity and the lower computational requirements.</li><li>Deploying the Random Forest model in an application enables real-time disease prediction by allowing users to input or upload symptoms. This tool becomes even more powerful when combined with the CSV Chatbot Application, which can be used to check the validity of the model’s predictions and explore trends. These implementations of ML in disease prediction show the power of using predictive modeling and AI in the healthcare industry. By combining automated diagnosis with natural language-based validation, these tools can streamline the diagnostic process, improving efficiency and accuracy in medical decision-making.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5aefa726f499" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced Paternity DNA Sequence Classification Using Dynamic Programming and Machine Learning…]]></title>
            <link>https://ernest-bonat.medium.com/advanced-paternity-dna-sequence-classification-using-dynamic-programming-and-machine-learning-889622345316?source=rss-58ededb6e465------2</link>
            <guid isPermaLink="false">https://medium.com/p/889622345316</guid>
            <dc:creator><![CDATA[Ernest Bonat, Ph.D.]]></dc:creator>
            <pubDate>Sun, 19 Jan 2025 23:16:53 GMT</pubDate>
            <atom:updated>2025-01-25T14:11:56.109Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*-43TfCMBWW-j4rReUDM2Mg.png" /></figure><h3><strong>Advanced Paternity DNA Sequence Classification Using Dynamic Programming and Machine Learning Algorithms. Part 2</strong></h3><p>Ernest Bonat, Ph.D., Mohamed Abdulaziz Eisa, B.Sc. AI</p><p>1. <a href="#c3c5">Overview</a><br>2. <a href="#58a5">Data Collection</a><br>3. <a href="#7c80">Data Preparation</a><br>4. <a href="#4a87">Data Merging</a><br>5. <a href="#2f8b">Feature Engineering</a><br>6. <a href="#8ed1">Genetic Similarity in Full Siblings</a><br>7. <a href="#6d86">DNA Natural Language Processing (NLP</a>)<br>8. <a href="#72ab">DNA K-Mers Processing</a><br>9. <a href="#33a8">NLP Feature Transformation</a><br>10. <a href="#ccd8">DNA Bag of Words Processing</a><br>11. <a href="#d2b0">Feature Scaling<br></a>12. <a href="#734a">Data Splitting</a><br>13. <a href="#10d4">Machine Learning Models</a><br>14. <a href="#13ec">Summary and Observations</a><br>15. <a href="#bd95">Project Coding APIs</a><br>16. <a href="#540e">Feature Work</a><br>17. <a href="#8886">References</a></p><h4>1. Overview</h4><p>DNA forensics has been revolutionized by high-throughput sequencing (HTS) technologies, particularly through the analysis of single nucleotide polymorphisms (SNPs). This study explores a structured approach to processing, merging, and analyzing SNP datasets spanning eight generations, aimed at developing efficient machine learning models for paternity testing and other genetic analyses. Emphasis is placed on feature engineering, DNA natural language processing, and the practical deployment of these models. This paper is part 2 of “<a href="https://ernest-bonat.medium.com/machine-learning-applications-in-genomics-life-sciences-by-ernest-bonat-ph-d-83598e67ccbc#bc94">Advanced Paternity DNA Sequence Classification Using Dynamic Programming and Machine Learning Algorithms. Part 1</a>”. Ernest Bonat, Ph.D., Mohamed Abdulaziz Eisa, B.Sc. AI, Sep 12, 2024.</p><h4>2. Data Collection</h4><p>The high-throughput sequencing (HTS) of single nucleotide polymorphisms (SNPs) has opened new avenues in DNA forensics, offering applications such as identification, mixture analysis, kinship prediction, and biogeographic ancestry prediction. The dataset under consideration, generated on May 14, 2018, provides a valuable resource for the development and testing of advanced DNA forensics algorithms. (<a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/IDT8HZ">11 million SNP Profiles datasets (Harvard Data verse)</a>)</p><p>The profiles are associated with individuals having defined ethnicities and family relationships. The dataset spans eight generations, providing a comprehensive genetic landscape. Each generation dataset encompasses approximately 21,892 rows. Each row represents an individual, providing a manageable yet comprehensive sample size. The dataset is structured into 10 columns, capturing various genetic and demographic attributes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*B3UaaCPd46FCyu7wpvPXIg.png" /><figcaption>Figure 1: Dataset Example</figcaption></figure><p>The first question to address is: How can we Prepare this data to make it suitable for a Machine Learning (ML) model? Below is the answer to this question.</p><h4>3. Data Preparation</h4><p>Data Preparation is the process of preparing raw data so that it is suitable for further processing and analysis. Key steps include collecting, cleaning, and labeling raw data into a form suitable for Machine Learning (ML) algorithms and then exploring and visualizing the data. Data preparation can take up to 80% of the time spent on an ML project.</p><p>The data was integrated from two distinct datasets: the child dataset and the parent dataset. This section details the systematic process employed to merge these datasets, ensuring the integrity and relevance of the merged data for subsequent analysis. Before merging the datasets, we prepared the data by truncating and concatenating sequences to standardize the data structure across both datasets.</p><p>The data pipeline follows a systematic approach to process genetic sequence data and establish parent-child relationships in the dataset. Each sequence is standardized to a specific length (1000 bases per allele) before merging, ensuring consistency in the final analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*J7xUET4xuwbxmX-roOO2GQ.png" /><figcaption>Figure 2: DNA Sequence First Pipeline of Data Preparation</figcaption></figure><p>In our case, before merging the datasets, we prepared the data by truncating and concatenating sequences to standardize the data structure across both datasets.</p><p>Step-by-Step Process:</p><ol><li>Truncate Sequences: Each allele sequence in both child and parent datasets was truncated to the first 1000 bases. This step was crucial to maintain a consistent sequence length for analysis.</li><li>Concatenate Sequences: For each individual, the truncated sequences of Allele1 and Allele2 were concatenated to form a full sequence of DNA.</li></ol><p>For the child dataset we have.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*g2E9urcczFSKbRwv4JxiAw.png" /><figcaption>Figure 3: Child Truncate DNA sequence</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*baQXRnO9VlrPaFuwW85wFg.png" /><figcaption>Figure 4: Length of DNA after Truncate for child</figcaption></figure><p>For the parent dataset we have.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*56kMhhAXku1vw8R5bjwKlQ.png" /><figcaption>Figure 5: Parent Truncate DNA Sequence</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*hkR0eSuzRg6BgSBzAxQxsg.png" /><figcaption>Figure 6: Length of DNA after Truncate for Parent</figcaption></figure><h4>4. Data Merging</h4><p>The next step was to merge the child and parent DataFrames using the parent identifiers provided in the child dataset. Here are the <strong>objectives.</strong></p><ul><li>To create a dataset that links each child to their respective parents.</li><li>To ensure that the merged data retains the genetic information necessary for further analysis.</li></ul><p><strong>The merging strategy could be defined in the following steps.</strong></p><ol><li><strong>Father’s Data:</strong> Merged the parent dataset with the child dataset on the ParentF (Father) identifier.</li><li><strong>Mother’s Data:</strong> Merged the parent dataset with the child dataset on the ParentM (Mother) identifier.</li><li><strong>Combine Merged Data: </strong>Combined the results of the two merges into a single DataFrame, ensuring that each child is associated with both parents.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*0WgyDwSLXThqOFJt4Zsdqw.png" /><figcaption>Figure 7: Merging Strategy for data of father</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*iDqKZqrG0-Qmier7zUyZXg.png" /><figcaption>Figure 8: Merging Strategy for data of mother</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*XjmVGcpR976q1_CJOlrOrw.png" /><figcaption>Figure 9: Clean up and combine the merged DataFrames</figcaption></figure><p>To ensure the accuracy and integrity of the merged dataset, we conducted the following validations:</p><ul><li><strong>Length Check:</strong> Verified the length of sequences to confirm that truncation and concatenation were correctly applied.</li><li><strong>Sample Inspection:</strong> Randomly sampled the merged DataFrame to visually inspect the correctness of parent-child associations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*BHaGaN2l6DwQhwffPD72oA.png" /><figcaption>Figure 10: Sample of merged DataFrames</figcaption></figure><h4>5. Feature Engineering</h4><p>Feature Engineering is the process of transforming raw data into a format that enhances the performance of machine learning models. It aims to improve model accuracy by providing more meaningful and relevant information, and it is considered one of the most valuable techniques. The key steps in feature engineering include feature creation, transformation, feature extraction, and feature selection.</p><p>The objective is to perform feature extraction to create a likelihood feature and a target label feature for paternity testing based on DNA sequence similarity. The<strong> </strong>feature creation steps are provided below:</p><p>Step 1: Creating a Global Weighted Alignment Sequence Feature</p><p>· Implement the Needleman-Wunsch algorithm to compare DNA sequences.</p><p>· Calculate the similarity score between parent and child DNA sequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*TarcN4LGWn-uoLc2WKPKmg.png" /><figcaption>Figure 11: Illustration for Target Labels Method</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*c1eAr5evJalD2Phd-3lf6g.png" /><figcaption>Figure 12: Applying Needleman-Wunsch algorithm</figcaption></figure><p>How similar is the DNA of siblings?</p><p>The concept of ‘shared’ DNA differs depending on how much DNA is being compared. When comparing our whole genome, all 3 billion letters, we share about 99.9%<strong> </strong>with every other human. However, we can also analyze shared DNA by narrowing our scope and just comparing the 3 million genetic differences we know exist between individuals. Of these 3 million differences, on average we share about 50 percent of those with our full siblings.</p><p>Children inherit half of their DNA from their mother and half from their father. However, unless they are identical twins, siblings won’t inherit exactly the same DNA. Depending on their biological sex, a parent will produce either a sperm or egg cell by meiosis. Meiosis is a process where a single cell divides twice to produce four cells containing half the original amount of genetic information.</p><p>Step 2: Generating a Target Data Feature</p><ul><li>Generate a binary target label based on the similarity score.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*YV634edy6TVywt2LgdLGZQ.png" /><figcaption>Figure 13: Generating a Target Feature</figcaption></figure><p>We have a lot of technical stuff for different datasets. In our case, we are using the K-Mer counting algorithm for feature extraction.</p><p>6. Genetic Similarity in Full Siblings</p><ul><li>Full siblings share about 50% of their genetic differences (the 3 million SNPs) on average.</li><li>This sharing is due to the random combination of alleles inherited from the parents.</li><li>Siblings are genetically distinct unless they are identical twins, who share 100% of their DNA.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*3eFpmrPVrOyNP61SO1WeAw.png" /><figcaption>Figure 14: Final Shape of Data before Processing</figcaption></figure><h4>7. DNA Natural Language Processing (NLP)</h4><p>DNA NLP<strong> </strong>is a novel interdisciplinary field that draws parallels between DNA sequences and natural language to apply computational linguistics techniques to genomic data. DNA sequences can be thought of as “texts” written in a four-letter alphabet (A, T, C, G), and the analysis of these sequences can benefit from methods traditionally used in NLP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/246/1*4VgPhbfs8AirUnVk3ND6PA.png" /><figcaption>Figure 15: DNA Natural Language Processing Steps</figcaption></figure><h4>8. DNA K-Mers Processing</h4><p>A K-mer is a substring of length k derived from a longer DNA or RNA sequence. K-mers are fundamental in bioinformatics for analyzing nucleotide sequences such as DNA or RNA. They allow the representation, comparison, and manipulation of sequences in a computationally efficient way.</p><p>A window of length 6 was created and slide it from left to right, shifting one character at a time. If the length of the given DNA sequence is N, we would end up with N — k+1 total of K-Mers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*nyorCGH8ZgJPVR6BBRwbGw.png" /><figcaption>Figure 16: DNA Sequence using K-mers</figcaption></figure><p>Examples of DNA sequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*7D1sT-DUD-elkMqmfQnufQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*8SIMKvxjb6LbmfmL1qhbzQ.png" /></figure><p>Representation of the K-mers in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CGAR5dTq4EE6rKTKsTtj7w.png" /></figure><p>Processing Steps</p><ol><li>The function uses a list comprehension to iterate over the sequence.</li><li>sequence[i:i+k] extracts a substring (k-mer) starting at position i and of length k.</li></ol><p>The range for i is from 0 to len(sequence) - k + 1, ensuring that the last k-mer generated is exactly of length k. The function returns a list of K-mers of length k.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Dr88u920YPq1w9o7ebEvSw.png" /><figcaption>Figure 16: k-mers of length k</figcaption></figure><p>Let’s go through an example to illustrate how the function works.</p><p>Example Input</p><ul><li>sequence = &quot;ACGTACGT&quot;</li><li>k = 3</li></ul><p>Execution Steps</p><ul><li>The length of the sequence is 8, and k is 3.</li><li>The range for i will be from 0 to 8 - 3 + 1 = 6. This means i will take values 0, 1, 2, 3, 4, 5.</li><li>For each value of i, the substring sequence [i: i+3] is extracted.</li></ul><p>Let’s see each step in detail:</p><ul><li>When i = 0, sequence [0:3] is &quot;ACG&quot;.</li><li>When i = 1, sequence [1:4] is &quot;CGT&quot;.</li><li>When i = 2, sequence [2:5] is &quot;GTA&quot;.</li><li>When i = 3, sequence [3:6] is &quot;TAC&quot;.</li><li>When i = 4, sequence [4:7] is &quot;ACG&quot;.</li><li>When i = 5, sequence [5:8] is &quot;CGT&quot;.</li></ul><h4>9. NLP Feature Transformation</h4><p>From NLP the count vectorizer method is used. It is used to transform a given text into a vector based on the frequency (count) of each word that occurs in the entire text.</p><p>What is word embedding (<strong>vectorization</strong>)?</p><p>Word embeddings are the texts converted into numbers and there may be different numerical representations of the same text. But before we dive into the details of Word Embeddings. It’s known that many Machine Learning algorithms and almost all Deep Learning Architectures are not capable of processing strings or plain text in their raw form. In a broad sense, they require numerical numbers as inputs to perform any sort of task, such as classification, regression, clustering, etc. Also, from the huge amount of data that is present in the text format, it is imperative to extract some knowledge out of it and build any useful applications.</p><p>What is Count Vectorizer?</p><p>Count Vectorizer method creates a document term matrix, which is a set of dummy variables that indicates if a particular word appears in the document. It fits and learn the word vocabulary and try to create a document term matrix in which the individual cells denote the frequency of that word in a particular document, which is also known as term frequency, and the columns are dedicated to each word in the corpus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*ehRahumk9th2ZTPKuSE4wg.png" /><figcaption>Figure 17: Count vectorizer method</figcaption></figure><p>Methods also we try to use it and make good results with data preprocessing</p><h4>10. DNA Bag of Words Processing</h4><p>The DNA Bag of Words (BoW) is a technique inspired by the classic Bag of Words model in Natural Language Processing (NLP), adapted to analyze DNA or genomic sequences. This approach represents DNA sequences as collections of “words” (e.g., K-mers), ignoring their order but capturing their frequency or presence.</p><p><strong>Steps:</strong></p><p>1. Convert Column to List</p><ul><li>word_list = list(word_column): Converts the input column to a list of strings.</li></ul><p>2. Join Characters with Spaces</p><ul><li>The function iterates through the list and joins characters in each string with spaces. This step seems unnecessary if the input is already a list of words, but it may be useful if the input is a list of individual characters.</li></ul><p>3. Create Count Vectorizer</p><ul><li>count_vectorizer = CountVectorizer(ngram_range=(word_ngram, word_ngram)): Initializes a CountVectorizer that will create n-grams of the specified length.</li></ul><p>4. Fit and Transform</p><ul><li>X = count_vectorizer.fit_transform(word_list): Fits the CountVectorizer to the word list and transforms it into a sparse matrix of n-gram counts.</li></ul><p>5. Return Result:</p><ul><li>Returns the sparse matrix X which contains the n-gram counts.</li></ul><h4>11. Feature Scaling</h4><p>Feature scaling is an essential step in developing precise and effective machine learning models. A crucial aspect of feature engineering includes techniques such as scaling, normalization. These methods transform data to enhance its suitability for modeling, improve model performance, mitigate the influence of outliers, and ensure that data features are on a consistent scale.</p><p>Dealing with our dataset that contain features of varying scales, it’s important to normalize the data to ensure optimal model performance. One effective normalization technique is the MaxAbsScaler. The MaxAbsScaler scales each feature by its maximum absolute value, ensuring that all features range between -1 and 1 without altering the data’s sparsity. This approach is particularly beneficial for data with varying magnitudes and when preserving zero entries is crucial. By employing MaxAbsScaler, we can enhance the convergence and accuracy of machine learning algorithms, particularly in scenarios involving sparse datasets or models sensitive to feature scaling.</p><h4>12. Data Splitting</h4><p>The ‘train_validation_test_split’ function splits a dataset into training, validation, and test sets. The function allows control over the proportions of the dataset assigned to each set and provides options for stratification to ensure that class distributions are maintained across splits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*m5x0N4jCSrX6WAuROoh3pg.png" /><figcaption>Figure 18: Train test split</figcaption></figure><p>Our dataset with 43,784 samples, the function performs the following steps to split the data:</p><p>1. <strong>Initial Split</strong>:</p><ul><li><strong>Training set</strong>: 80% of the dataset.</li><li><strong>Combined test-validation set</strong>: 20% of the dataset.</li></ul><p>2. <strong>Second Split</strong>:</p><ul><li><strong>Validation set</strong>: 50% of the combined test-validation set (10% of the total dataset).</li><li><strong>Test set</strong>: 50% of the combined test-validation set (10% of the total dataset).</li></ul><p><strong>Calculation Steps:</strong></p><p>1. <strong>Initial Split</strong>:</p><ul><li>Training set size: 0.8×43,784=35,027.20.8 \times 43,784 = 35,027.20.8×43,784=35,027.2 (rounded to 35,027 samples).</li><li>Combined test-validation set size: 0.2×43,784=8,756.80.2 \times 43,784 = 8,756.80.2×43,784=8,756.8 (rounded to 8,757 samples).</li></ul><p>2. <strong>Second Split</strong>:</p><ul><li>Validation set size: 0.5×8,757=4,378.50.5 \times 8,757 = 4,378.50.5×8,757=4,378.5 (rounded to 4,378 samples).</li><li>Test set size: 0.5×8,757=4,378.50.5 \times 8,757 = 4,378.50.5×8,757=4,378.5 (rounded to 4,379 samples).</li></ul><p><strong>Split Sizes</strong></p><p><strong>The Training Set: </strong>Used to train the model, learning hidden features and patterns through repeated exposure in each epoch.</p><p><strong>The Validation Set:</strong> Separate from the training set, it is used to tune model hyperparameters and configurations, preventing overfitting by evaluating performance during training.</p><p><strong>The Test Set:</strong> A separate dataset used to provide an unbiased evaluation of the final model’s performance after training.</p><ul><li><strong>Training Set: </strong>35,027 samples</li><li><strong>Validation Set: </strong>4,378 samples</li><li><strong>Test Set: </strong>4,379 samples</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*GBzZjjUUyN-x-Zr9lNQLtw.png" /><figcaption>Figure 19: Training data/validation/test split</figcaption></figure><h4>13. Machine Learning Models</h4><p>In this section, we outline the process of developing a ML model to classify DNA sequences. The objective is to identify the most effective model and optimal k-mer length for accurate classification.</p><p>Model Selection</p><p>Three ML models were selected for this project:</p><ul><li><strong>Random Forest (baseline):</strong> An ensemble learning method that constructs multiple decision trees and outputs the mode of their predictions.</li><li><strong>XGBoost</strong>: An optimized distributed gradient boosting library designed to be highly efficient, flexible, and portable.</li><li><strong>LightGBM</strong>: A gradient boosting framework that uses tree-based learning algorithms. It is designed to be distributed and efficient with the capability of handling large-scale data.</li></ul><p>Model Hyperparameters</p><p>To fine-tuned the hyperparameters of these models the GridSearchCV was used. After extensive testing, including varying learning rates from 0.1 to 0.5, n_estimators from 5000 to 20000, max depth from 5 to 20, and other parameters, I identified the optimal settings that provided the best accuracy for each model.</p><p><strong>Model Hyperparameters</strong></p><p>To fine-tuned the hyperparameters of these models the GridSearchCV was used. After extensive testing, including varying learning rates from 0.1 to 0.5, n_estimators from 5000 to 20000, max depth from 5 to 20, and other parameters, I identified the optimal settings that provided the best accuracy for each model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*VGdsNGatA3DqRQedKvw0Qg.png" /></figure><p><strong>Model Training</strong></p><p>Each model was trained using different K-mer lengths (3, 6, and 7) to determine the optimal length that provides the best classification performance. The training process involved splitting the data into training and validation sets and fitting the models on the training data.</p><p><strong>Performance Metrics</strong></p><p>The models were evaluated using several metrics to assess their performance:</p><ul><li><strong>Validation Accuracy</strong>: The percentage of correct predictions on the validation set.</li><li><strong>Test Accuracy:</strong> The percentage of correct predictions on the test set.</li><li><strong>Confusion Matrix:</strong> A table used to describe the performance of a classification model, displaying the true positives, true negatives, false positives, and false negatives.</li><li><strong>Precision, Recall, and F1-Score:</strong> Precision is the ratio of correctly predicted positive observations to the total predicted positives. Recall is the ratio of correctly predicted positive observations to all the observations in the actual class. F1-Score is the weighted average of Precision and Recall.</li></ul><p><strong>Overall Metrics</strong></p><p>The performance metrics for the different K-mer lengths and models are summarized below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*ySD_lDvSOg-ytUUTsOr46g.png" /></figure><p>Detailed Metrics for K-mer = 3, 6, and 7.</p><p>For each K-mer length, detailed metrics including precision, recall, and F1-scores for both classes (0 and 1) were computed. Additionally, macro and weighted averages of these metrics were calculated to provide a holistic view of the model performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*uvIPA_rsm61xeJ4WCmXN0w.png" /><figcaption>Figure 20: Validation accuracy comparison</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*_2aKbxz5r_b1qC7UzGcHLg.png" /><figcaption>Figure 21: Testing accuracy comparison</figcaption></figure><p>Validation and Test Metrics for K-mer = 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*2YrgmBisvwQGN2UhTxE6wA.png" /></figure><p>Validation metrics diagrams of Model Performance for K-mer = 3.</p><p>1- Confusion matrix (Validation)</p><p>For LigtGBM model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*EnD9NEcvUleA6Iz4g_A33A.png" /></figure><p>For Random Forest model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*fQGDILZZNoA4DZpepEA6rg.png" /></figure><p>2- Classification matrices for Class 1</p><p>For LigtGBM model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*dNaNRz32Vxpk5IS5izgsiQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*JqoYvH5mFTZMHDn3UOW4KQ.png" /></figure><p>For Random Forest model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*k4BCBYpkIWnPNEsR02MurQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*MusTmwhDyK8chfXTqaR5yw.png" /></figure><p>For XGBoost model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*or39Q4iJIq6NzBJilTbylw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*TJvDCa2we6vo0oBrkaDx9g.png" /></figure><p>3- Calibration Curve Validation for Highest Score</p><p>For LigtGBM Model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*9vYKJ0w02YRUejTPexxltQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*EhvsrOawfwfPyU65rKrMlA.png" /></figure><p>For Random Forest Model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*u8JNqpwuSj8zEGLXseBZEQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*860n1qhTbnwWykIAYcv_Dw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*u8JNqpwuSj8zEGLXseBZEQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*860n1qhTbnwWykIAYcv_Dw.png" /></figure><p>For XGBoost model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Bu0KRThgg2VAteoBhVfogA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*gNWfAV8GG--7tOZrcrxRCw.png" /></figure><p>Analysis of Model Performance for k-mer = 3:</p><p>· Accuracy Score<strong>:</strong> LightGBM has the highest accuracy on both the validation and test sets, it is the best model among the three for our classification task.</p><p>· Confusion Matrix<strong>:</strong> LightGBM generally has fewer false positives and false negatives compared to Random Forest and XGBoost, indicating better performance in correctly classifying both classes.</p><p>· Precision, Recall, and F1-Score<strong>:</strong> All three models have very similar precision, recall, and F1-scores for both classes, indicating comparable performance in terms of classification quality. However, LightGBM slightly edges out in most metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*HVvw01iDd7Ra0t7XM0xaXQ.png" /></figure><h4>14. Summary and Observations</h4><p><strong>Summary:</strong></p><ul><li>The analysis focused on using traditional machine learning models (Random Forest, XGBoost, and LightGBM) to classify DNA sequences for paternity testing. The models were evaluated using different k-mer lengths (3, 6, and 7) to determine their impact on performance metrics.</li></ul><p><strong>Performance Metrics:</strong></p><ul><li>Validation Accuracy and Test Accuracy were the primary metrics used for evaluation.</li><li>Confusion Matrices were analyzed for true positives, true negatives, false positives, and false negatives.</li><li>Precision, Recall, and F1-Score were calculated for both classes and averaged for macro and weighted metrics.</li></ul><p><strong>Conclusion:</strong></p><ul><li>LightGBM consistently outperformed both Random Forest and XGBoost across all k-mer lengths, indicating its superior capability in handling DNA sequence classification for paternity testing.</li><li>Increasing the k-mer length generally improved the models’ accuracy, with the highest performance observed at k-mer = 7.</li><li>Attempting k-mer lengths of 8 or higher caused kernel crashes, suggesting a need for more powerful computational resources for further improvements.</li><li>With a more powerful GPU, it is anticipated that higher k-mer lengths could be explored, potentially leading to even better model accuracy and performance.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*js_M7FP-prdshggvRiBMmg.png" /></figure><h4>15. Project Coding APIs</h4><p><strong>Model Deployment</strong></p><p>We use a code Python script that implements a Flask web application for DNA sequence matching , identification and predicting relative or not. It uses machine learning models to compare a given DNA sequence with sequences stored in a database and files present in a folder. The code retrieves DNA sequences from the database, reads sequences from files, and performs sequence matching and identification predication using LightGBM and other techniques.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*ALw65rZx5GBhs54silsyNg.png" /></figure><p><strong>Dependencies</strong></p><p>The code relies on several external libraries and modules, which need to be installed before running the script:</p><ul><li>Flask</li><li>Python3.8 or Higher</li><li>Bio (Biopython)</li><li>requests</li><li>pickle</li><li>numpy</li><li>pandas</li><li>scikit-learn</li><li>lightgbm</li><li>joblib</li><li>os</li><li>scipy</li></ul><p>Ensure these dependencies are installed in your Python environment before executing the code.</p><p><strong>Code Structure</strong></p><p>The code is modular, ensuring clarity and ease of maintenance. It is divided into several sections, each serving a specific purpose. This document outlines the steps involved in setting up, running, and understanding the functionality of each section of the code.</p><p>1. Importing Required Libraries</p><p><strong>Objective</strong>: Import essential libraries for the application to function.</p><ul><li><strong>Flask</strong>: Web framework for creating the web application.</li><li><strong>Biopython</strong>: Used for DNA sequence alignment.</li><li><strong>Requests</strong>: For making API calls.</li><li><strong>Various Machine Learning and Utility Libraries</strong>: For data processing and machine learning tasks.</li></ul><p>2. Flask App Configuration</p><p><strong>Objective</strong>: Set up and configure the Flask application.</p><ul><li><strong>Create Flask Application</strong>: Initialize the Flask app with specified template and static folder locations.</li><li><strong>Set API URL</strong>: Define the API endpoint for retrieving population data.</li></ul><p>3. DNA Sequence Alignment</p><p><strong>Objective</strong>: Define a function to compare two DNA sequences.</p><ul><li><strong>Function Definition</strong>: Implement needleman_wunsch_similarity to compare DNA sequences using the Needleman-Wunsch algorithm.</li><li><strong>Parameters</strong>: Accept sequences and alignment scoring parameters (match, mismatch, gap penalties).</li><li><strong>Output</strong>: Return a similarity score.</li></ul><p>4. DNA Sequence Retrieval</p><p><strong>Objective</strong>: Retrieve DNA sequences from files or an external API.</p><ul><li><strong>Retrieve from File</strong>: Define retrieve_dna_sequence_from_file to read and process DNA sequences from uploaded files.</li><li><strong>Retrieve from API</strong>: Implement retrieve_api_data to fetch population data from the specified API endpoint.</li></ul><p>5. Worker Function</p><p><strong>Objective</strong>: Handle background tasks.</p><ul><li><strong>Function Definition</strong>: Implement a basic worker function to demonstrate background processing capability.</li></ul><p>6. Flask Routes</p><p>6.1 Home Page</p><p><strong>Objective</strong>: Render the home page of the application.</p><ul><li><strong>Route Definition</strong>: Define a route to render index.html.</li></ul><p>6.2 DNA Sequence Comparison</p><p><strong>Objective</strong>: Handle DNA sequence comparison requests.</p><ul><li><strong>Comparison Page</strong>: Define a route to render compare.html.</li><li><strong>Comparison Logic</strong>:</li><li>Validate file uploads.</li><li>Read and process DNA sequences.</li><li>Calculate similarity using needleman_wunsch_similarity.</li><li>Return results as JSON.</li></ul><p>6.3 DNA Sequence Identification</p><p><strong>Objective</strong>: Handle DNA sequence identification requests.</p><ul><li><strong>Identification Page</strong>: Define a route to render identify.html.</li><li><strong>Identification Logic</strong>:</li><li>Validate file uploads.</li><li>Retrieve population data from API.</li><li>Filter and compare sequences.</li><li>Return results as JSON.</li></ul><p>6.4 Missing DNA Sequence</p><p><strong>Objective</strong>: Handle cases of missing DNA sequences.</p><ul><li><strong>Missing Page</strong>: Define a route to render missing.html.</li><li><strong>Missing Sequence Logic</strong>:</li><li>Validate file uploads.</li><li>Retrieve population data from API.</li><li>Compare sequences to find matches and potential children.</li><li>Return results as JSON.</li></ul><p>6.5 Predicting DNA Sequence</p><p><strong>Objective:</strong> Handle DNA paternity prediction requests.</p><ul><li><strong>Prediction Page:</strong> Define a route to render a page for predicting DNA paternity.</li><li><strong>Prediction Form:</strong></li></ul><p>§ Render a form in HTML (predict.html) that allows users to upload two DNA files: one for the parent and one for the child.</p><p>§ Use the following HTML code for predict.html:</p><h4>16. Feature Work</h4><p>1. We will introduce a deep learning Convolutional Neural Networks (CNNs) for Local sequence alignment. The objective is to enhance the accuracy of sequence alignment beyond the capabilities of classical machine learning techniques.</p><p>2. Working in some Advanced topics in Bioinformatics like:</p><ul><li>Finding Hidden Messages in DNA</li><li>Finding Mutations in DNA</li><li>Clustering DNA (genome) Sequence</li></ul><p>3. Making new model to predict the DNA Next Generation Sequence (NGS), it’s really will help to make new vaccine that’s hold immense promise for transformative breakthroughs in the understanding and treatment of various diseases.</p><h4>17. References</h4><ul><li>“<a href="https://ernest-bonat.medium.com/machine-learning-applications-in-genomics-life-sciences-by-ernest-bonat-ph-d-83598e67ccbc#bc94">Advanced Paternity DNA Sequence Classification Using Dynamic Programming and Machine Learning Algorithms. Part 1</a>”. Ernest Bonat, Ph.D., Mohamed Abdulaziz Eisa, B.Sc. AI, Sep 12, 2024.</li><li>Puchalska, K., &amp; Januszko, A. (2021). Optimizing Production Processes by Multi-Objective Evolutionary Algorithms. In Recent Advances in Intelligent Information Hiding and Multimedia Signal Processing (pp. 393–401). Springer. doi: 10.1007/978–981–19–2980–9_27.</li><li>Sooryaprakash12. (2021). Human DNA sequences. Retrieved [Date], from <a href="https://www.kaggle.com/datasets/sooryaprakash12/human-dna-sequences">https://www.kaggle.com/datasets/sooryaprakash12/human-dna-sequences</a>.</li><li>Branimir Todorović, Predrag Radosavljević, Marija Mirković, and Milena Stanković. “Neural network-based method for optical fiber fault location in access networks.” Journal of Lightwave Technology 35, no. 20 (2017): 4514–4521.</li><li>Srivastava, A. (2017, September 26). Understanding Support Vector Machine algorithm from examples (along with code). Analytics Vidhya. Retrieved [Date], from <a href="https://www.analyticsvidhya.com/blog/2017/09/understaing-support-vector-machine-example-code/">https://www.analyticsvidhya.com/blog/2017/09/understaing-support-vector-machine-example-code/</a>.</li><li>Longdom Publishing. (n.d.). DNA Sequencing Journals, Articles &amp; PPTs List. Retrieved [Date], from <a href="https://www.longdom.org/scholarly/dna-sequencing-journals-articles-ppts-list-71.html">https://www.longdom.org/scholarly/dna-sequencing-journals-articles-ppts-list-71.html</a></li><li>National Cancer Institute. (n.d.). DNA. In NCI Dictionary of Genetics Terms. Retrieved [Date], from <a href="https://www.cancer.gov/publications/dictionaries/genetics-dictionary/def/dna">https://www.cancer.gov/publications/dictionaries/genetics-dictionary/def/dna</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=889622345316" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced Synthetic DNA Sequence Generation and Preprocessing for Natural Language Processing]]></title>
            <link>https://ernest-bonat.medium.com/advanced-dna-synthetic-sequence-generation-and-preprocessing-for-natural-language-processing-7a06a051b3c1?source=rss-58ededb6e465------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a06a051b3c1</guid>
            <dc:creator><![CDATA[Ernest Bonat, Ph.D.]]></dc:creator>
            <pubDate>Mon, 13 Jan 2025 14:32:32 GMT</pubDate>
            <atom:updated>2025-02-01T14:55:25.245Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*EjsNkQFFamI3Akx40LJQfQ.jpeg" /></figure><p>Ernest Bonat, Ph.D.</p><p>Updated 01/30/2025</p><p>1. <a href="#43a7">Overview</a><br>2. <a href="#ff2a">Why is creating synthetic DNA sequence data very important?</a><br>3. <a href="#f3d7">Promoter Gene Sequences Dataset</a><br>4. <a href="#acbe">Building Machine Learning Random Forest Model</a><br>5. <a href="#a785">Using Gretel Platform to Generate Synthetic DNA Sequences</a><br>6. <a href="#8e20">DNA Sequences ETL Preprocess Pipeline</a><br>7. <a href="#4e18">DNA Sequence Production Machine Learning Model Test</a><br>8. <a href="#ba5a">Conclusions</a></p><h4>1. Overview</h4><p>This research presents a practical approach to generating and preprocessing synthetic DNA sequences for application in Natural Language Processing (NLP) algorithms and data-driven Machine Learning (ML) pipelines. Using a small promoter gene sequence dataset as a foundation, the study addresses the challenge of limited data availability by employing synthetic data generation through the Gretel platform and developing an ETL (Extract, Transform, Load) pipeline for preprocessing. It is highly recommended to read and understand some practical ML applications for Life Sciences at “<a href="https://ernest-bonat.medium.com/machine-learning-applications-in-genomics-life-sciences-by-ernest-bonat-ph-d-83598e67ccbc">Machine Learning Applications in Genomics Life Sciences by Ernest Bonat, Ph.D.</a>”</p><h4>2. Why is creating synthetic DNA sequence data very important?</h4><p>Creating synthetic DNA sequence data is critically important for advancing science and technology because it allows researchers to simulate, analyze, and innovate without depending on real biological samples. Below are key reasons why synthetic DNA sequence data is particularly important:</p><p><strong>1. Simulation and Modeling</strong></p><ul><li>Testing Hypotheses: Synthetic DNA sequences can model real-world genetic variations, enabling researchers to test specific hypotheses without needing access to actual biological samples.</li><li>Designing and Optimizing Experiments: Scientists can simulate the behavior of DNA sequences to refine experimental conditions before testing with live samples.</li></ul><p><strong>2. Developing Machine Learning and AI Models</strong></p><p>Synthetic DNA data is crucial for training computational models in genomics and bioinformatics:</p><ul><li>Data Augmentation: In fields like predictive genomics, synthetic sequences increase the diversity of training datasets, improving model accuracy.</li><li>Algorithm Development: New methods for sequence alignment, variant calling, or motif discovery can be tested using controlled, synthetic datasets.</li></ul><p><strong>3. Accelerating Genetic and Genomic Research</strong></p><ul><li>Studying Gene Functions: Synthetic DNA sequences allow researchers to introduce specific mutations or variations to study how genes and regulatory elements work.</li><li>Exploring Genomic Variability: Synthetic data enables researchers to simulate different populations or evolutionary scenarios.</li></ul><p><strong>4. Supporting Gene Therapy Development</strong></p><ul><li>Synthetic DNA sequences are used to test therapeutic approaches in a controlled environment:</li><li>Optimizing Gene Editing: Techniques like CRISPR-Cas9 rely on synthetic DNA to test editing strategies.</li><li>Creating Gene Constructs: Synthetic sequences are critical for designing therapeutic vectors, such as those used in gene therapy.</li></ul><p><strong>5. Ethical and Safe Alternatives</strong></p><ul><li>Reducing Dependence on Natural Samples: Synthetic data eliminates the need for collecting sensitive or hard-to-access samples, such as human or endangered species’ DNA.</li><li>Bioethical Standards: It avoids ethical concerns about using live organisms or human tissue in research.</li></ul><p><strong>6. DNA-Based Data Storage</strong></p><p>Synthetic DNA sequence data is essential for advancing DNA-based data storage:</p><ul><li>Encoding Digital Information: Artificial DNA sequences are used to encode digital data for high-density, long-term storage.</li><li>Optimizing Read/Write Methods: Synthetic data helps researchers refine DNA synthesis and sequencing techniques for data storage applications.</li></ul><p><strong>7. Synthetic Biology and Biotechnology</strong></p><ul><li>Designing New Organisms: Synthetic DNA sequences enable the creation of engineered microbes with custom genetic instructions for producing biofuels, pharmaceuticals, or materials.</li><li>Metabolic Pathway Engineering: Artificial sequences are used to optimize biochemical pathways in organisms to enhance production efficiency.</li></ul><p><strong>8. Advancing Education and Training</strong></p><ul><li>Teaching Bioinformatics: Synthetic data is invaluable for teaching genomics and bioinformatics concepts in a controlled, reproducible manner.</li><li>Safe Learning Environment: Synthetic sequences eliminate safety risks associated with handling pathogenic or hazardous DNA samples.</li></ul><p><strong>9. Enabling Personalized Medicine</strong></p><ul><li>Simulating Patient-Specific Genomes: Synthetic data can mimic patient-specific genetic variations for testing drug efficacy or predicting disease susceptibility.</li><li>Designing Precision Therapies: Synthetic sequences help create personalized DNA constructs for treatments like cancer immunotherapy.</li></ul><p><strong>10. Controlling Experimental Conditions</strong></p><ul><li>Reducing Noise in Data: Synthetic DNA sequences can eliminate variability and noise, providing clean datasets for benchmarking analytical tools.</li><li>Benchmarking Tools: Synthetic data helps evaluate the performance of sequencing pipelines, error-detection tools, and variant-calling algorithms.</li></ul><p><strong>11. Exploring Rare or Hypothetical Scenarios</strong></p><ul><li>Simulating Rare Mutations: Researchers can create synthetic DNA to study the effects of rare or hypothetical mutations that might not be available in natural datasets.</li><li>Evolutionary Studies: Synthetic sequences can model ancestral or extinct genomes to study evolutionary processes.</li></ul><p><strong>12. Pioneering Research in Epigenetics</strong></p><p>Synthetic DNA data can include modifications (e.g., methylation patterns) to study epigenetic mechanisms and their impact on gene expression.</p><p><strong>Challenges and Future Considerations</strong></p><p>While synthetic DNA sequence data is incredibly valuable, it must be used responsibly:</p><ul><li>Data Validation: Synthetic data must be validated to ensure it accurately represents biological systems.</li><li>Ethical Concerns: Synthetic sequences should be carefully regulated to avoid misuse (e.g., creating harmful synthetic pathogens).</li></ul><p>Creating synthetic DNA sequence data is vital for advancing genomics, bioinformatics, and biotechnology. It provides a safe, ethical, and scalable way to drive innovation, simulate biological systems, and optimize experimental workflows, ultimately unlocking new possibilities in science and medicine.</p><h4>3. Promoter Gene Sequences Dataset</h4><p>The promoter gene sequences dataset will be used in the paper. Promoters are regions in DNA sequences that are recognition sites for RNA-polymerase to start transcription. These promoter sequences are fairly conserved throughout each domain of life so they allow for gene identification even when most gene sequences are different. Gene identification is important when trying to identify DNA sequences of proteins (that perform most of the biochemical processes in the body). A general definition can be found on the <a href="https://www.genome.gov/genetics-glossary/Promoter">Promoter </a>NHGRI site.</p><p>The CSV dataset file can be download from UCI Machine Learning Repository <a href="https://archive.ics.uci.edu/dataset/67/molecular+biology+promoter+gene+sequences">Molecular Biology (Promoter Gene Sequences)</a>. The dataset used in this study contains 106 instances in total, with two classes distributed equally: 53 promoter sequences (positive cases) and 53 non-promoter sequences (negative cases). This dataset was collated by Towell to evaluate a hybrid learning algorithm, KBANN, which uses examples to inductively refine pre-existing knowledge. Each instance consists of 57 base-pair positions, starting at position -50 and ending at position +7.</p><p>The panda’s dataframe info of the CSV file ‘promoter_dna_sequence_original_106.csv’ is shown below.</p><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;    <br>RangeIndex: 106 entries, 0 to 105        <br>Data columns (total 2 columns):<br> #   Column        Non-Null Count  Dtype <br>---  ------        --------------  ----- <br> 0   dna_sequence  106 non-null    object<br> 1   dna_label     106 non-null    int64 <br>dtypes: int64(1), object(1)p</pre><p>Six rows from the original file will be removed, with three labels (1 and 0 classes) used for production model deployment testing. This step is crucial for validating the ML model’s performance using the real original dataset. Many open-source papers and projects overlook this critical step in the Machine Learning workflow. Below is part of the CSV spreadsheet file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*tblOclhEKbwFJAX2wV2qMg.png" /></figure><h4>4. Building Machine Learning Random Forest Model</h4><p>Now, the new ‘promoter_dna_sequence_original_100.csv’ file contains 100 rows. Let’s try to build a Random Forest model using Natural Language Processing (NLP) techniques provided in <a href="https://ernest-bonat.medium.com/advanced-dna-sequence-text-classification-using-natural-language-processing-8860c3c05692#441b">Advanced DNA Sequence Text Classification Using Natural Language Processing</a>. In this case the following functions were used.</p><pre># dna k-mers concatenation<br>X = dna_k_mers_concatenation(X, config.K_MERS_LENGTH)<br><br># dna k-mers vectorization<br>X, tfidf_vectorizer = dna_k_mers_vectorization(X)</pre><p>The imbalanced bar chart plot below shows the absolute label class distribution. Therefore, SMOTE algorithms will not be applied at this time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*OhdCEQa_uoSwChfbAltgYg.png" /></figure><p>From the results of the program below, we can see that the Random Forest model is overfitted, with a 100% validation accuracy score and a 90% test score.</p><pre>validate metrics<br>classification accuracy score:  <br>100.0<br><br>classification confusion matrix:<br>[[4 0]<br> [0 6]]<br><br>classification report:<br>              precision    recall  f1-score   support<br><br>           0       1.00      1.00      1.00         4<br>           1       1.00      1.00      1.00         6<br><br>    accuracy                           1.00        10<br>   macro avg       1.00      1.00      1.00        10<br>weighted avg       1.00      1.00      1.00        10<br><br>test metrics<br>classification accuracy score:<br>90.0<br><br>classification confusion matrix:<br>[[3 1]<br> [0 6]]<br><br>classification report:<br>              precision    recall  f1-score   support<br><br>           0       1.00      0.75      0.86         4<br>           1       0.86      1.00      0.92         6<br><br>    accuracy                           0.90        10<br>   macro avg       0.93      0.88      0.89        10<br>weighted avg       0.91      0.90      0.90        10</pre><p>In general, this should occur when you have an insufficient number of rows in the dataset, such as only 100 rows. We can easily request a larger database file. However, suppose there is no additional dataset file, and the ML model needs to be built for prediction and to make important patience healthcare decisions. This scenario can arise at any time in any field of Life Sciences. The question for the Data Scientist is, ‘What should be done?’ Well, a simple solution is to generate synthetic DNA sequence data based on the 100 existing rows. Let’s try to do that using the Gretel platform.</p><h4>5. Using Gretel Platform to Generate Synthetic DNA Sequences</h4><p><a href="https://gretel.ai/">Gretel </a>is a synthetic data platform purpose-built for AI. It generates artificial, synthetic datasets with the same characteristics as real data, so you can improve AI models without compromising on privacy. Open <a href="https://gretel.ai/">Gretel </a>site, create an account and generate your API key. In the ‘Dashboard’ page choose a preconfigured blueprint ‘Synthesize tabular data with Navigator Fine Tuning’ and follow the instructions.</p><p>Let’s try to generate 10,000 synthetic rows from 100 original DNA sequence rows. After the CSV file promoter_dna_sequence_synthetics_10000.csv was created, we observed some bad generated DNA sequences shown below. It is unclear why the Gretel platform produces these non-functional DNA sequences, given that the original dataset consists solely of specific combinations of the nucleotides A, T, C, and G only.</p><pre>cattaaaaaaftopsacgcttagccgcalgturhiCGTZguACGCTAATCGCATCTTCC<br>gcaaataatcaatgtggacttttctgccgtgattatagacatingÐ³Ð°cgacgdicgcg<br>cfgsrrgctacmgttcggtggã‚¿O retired cttcttctggcgtactccaaga<br>tgcacgggttGCGCCGTTTCTTGGGTTTTGTGACTAGSFTGSCTACGfATYnCA<br>atcgctcaaco activities ctttctwct nagc CTCAATSvWyCTXYFAACGTCATT<br>tabgtcagtttat cc actu cut emettt hhgtgt taacgtagactaccguÃ©rÃ©kt</pre><p>To fix these synthetics DNA sequence errors and create the final ML CSV file a data ETL preprocessing pipeline will be required. A good practical implementation of ETL package development can be found at “<a href="https://ernest-bonat.medium.com/elt-package-development-with-3-tier-architecture-for-data-engineering-6e212330fe61">ELT Package Development with 3-Tier Architecture for Data Engineering</a>” paper.</p><h4>6. DNA Sequences ETL Preprocess Pipeline</h4><p>A DNA sequence ETL preprocessing pipeline library was developed to create the final CSV file to be applied in ML algorithms. This pipeline contains three steps.</p><ol><li>DNA sequences original preprocess file</li><li>DNA sequences synthetic preprocess file</li><li>DNA sequences final ML file development</li></ol><p>The main run_etl_pipeline() function code is shown below.</p><pre>def run_etl_pipeline(self):<br>    try:<br>        self.logger.info(&quot;Running ETL pipeline...&quot;)<br>        self.dna_extract_data_csv()<br>        self.dna_remove_leading_trailing_spaces()<br>        self.dna_convert_upper_case()<br>        self.dna_validate_nucleotides()<br>        self.dna_remove_empty_row()            <br>        self.dna_truncate_column()<br>        self.dna_padding_column()<br>        self.dna_load_data_csv()<br>        self.logger.info(&quot;ETL Pipeline completed successfully&quot;)<br>    except:<br>        self.logger.info(&quot;ETL Pipeline stopped due to an error.&quot;)</pre><p>Now we have two CSV DNA sequence files: the original and the synthetic one.</p><ul><li>‘promoter_dna_sequence_original_100.csv’</li><li>‘promoter_dna_sequence_synthetics_10000.csv’</li></ul><p>These two CSV files need to be processed using the ETL pipeline procedures. Below are the results of the run_etl_pipeline() function for both CSV files.</p><pre>2025-01-02 08:56:35 - INFO: Running ETL pipeline...<br>2025-01-02 08:56:35 - INFO: Extracting csv input file: promoter_dna_sequence_original_100.csv...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (100, 2)<br>2025-01-02 08:56:35 - INFO: Removing leading and trailing spaces...        <br>2025-01-02 08:56:35 - INFO: Dataframe shape: (100, 2)<br>2025-01-02 08:56:35 - INFO: Converting DNA sequence string to upper case...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (100, 2)<br>2025-01-02 08:56:35 - INFO: Validating DNA nucleotides ATCG...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (100, 2)<br>2025-01-02 08:56:35 - INFO: Removing empty (NaN or None) rows...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (100, 2)<br>2025-01-02 08:56:35 - INFO: Truncating sequence column...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (100, 2)<br>2025-01-02 08:56:35 - INFO: Padding sequence column...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (100, 2)<br>2025-01-02 08:56:35 - INFO: Loading csv output file: promoter_dna_sequence_original_100_preprocess.csv...<br>2025-01-02 08:56:35 - INFO: CSV output file promoter_dna_sequence_original_100_preprocess.csv created successfully.<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (100, 2)<br>2025-01-02 08:56:35 - INFO: ETL Pipeline completed successfully<br>2025-01-02 08:56:35 - INFO: Running ETL pipeline...<br>2025-01-02 08:56:35 - INFO: Extracting csv input file: promoter_dna_sequence_synthetic_10000.csv...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (10000, 2)<br>2025-01-02 08:56:35 - INFO: Removing leading and trailing spaces...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (10000, 2)<br>2025-01-02 08:56:35 - INFO: Converting DNA sequence string to upper case...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (10000, 2)       <br>2025-01-02 08:56:35 - INFO: Validating DNA nucleotides ATCG...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (8911, 2)<br>2025-01-02 08:56:35 - INFO: Removing empty (NaN or None) rows...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (8911, 2)<br>2025-01-02 08:56:35 - INFO: Truncating sequence column...       <br>2025-01-02 08:56:35 - INFO: Dataframe shape: (8911, 2)<br>2025-01-02 08:56:35 - INFO: Padding sequence column...<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (8911, 2)<br>2025-01-02 08:56:35 - INFO: Loading csv output file: promoter_dna_sequence_original_synthetic_10000_preprocess.csv...<br>2025-01-02 08:56:35 - INFO: CSV output file promoter_dna_sequence_original_100_preprocess_synthetic_preprocess_yes_duplicates.csv created successfully.<br>2025-01-02 08:56:35 - INFO: Dataframe shape: (8911, 2)<br>2025-01-02 08:56:35 - INFO: ETL Pipeline completed successfully<br>Final ML output file: promoter_dna_sequence_ml_final.csv created successfully.<br>Dataframe shape: (9011, 2)</pre><p>As you can see, the original CSV file remains intact, as the DataFrame shape values (100, 2) are the same before and after applying the ETL pipeline procedures. For the synthetic CSV file, however, the DataFrame shape values are different. The Gretel platform generated 10,000 synthetic rows as specified. After applying the ETL pipeline procedures, the DataFrame shape was reduced to (8,911, 2), meaning 1,089 rows were removed. The final ML CSV file will have a DataFrame shape of (9,011, 2).</p><p>Now, let’s run the same NLP algorithm on this final ML CSV file. As you can observe, the classes are imbalanced, so the SMOTE algorithm will need to be applied to balance the classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*aXwB699LyeKqmw3TT3xm4Q.png" /></figure><p>After applying SMOTE the training data got balanced and ready for the Machine Learning models as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*_nNyeQtHaNLCokBoaosjcA.png" /></figure><p>Here are the final results.</p><pre>validate metrics<br>classification accuracy score:  <br>97.45<br><br>classification confusion matrix:<br>[[405  16]<br> [  7 473]]<br><br>classification report:<br>              precision    recall  f1-score   support<br><br>           0       0.98      0.96      0.97       421<br>           1       0.97      0.99      0.98       480<br><br>    accuracy                           0.97       901<br>   macro avg       0.98      0.97      0.97       901<br>weighted avg       0.97      0.97      0.97       901<br><br>test metrics<br>classification accuracy score:<br>97.23<br><br>classification confusion matrix:<br>[[402  12]<br> [ 13 475]]<br><br>classification report:<br>              precision    recall  f1-score   support<br><br>           0       0.97      0.97      0.97       414<br>           1       0.98      0.97      0.97       488<br><br>    accuracy                           0.97       902<br>   macro avg       0.97      0.97      0.97       902<br>weighted avg       0.97      0.97      0.97       902</pre><p>Both validation and test metrics show high performance with accuracy above 97%. Class 1 achieves slightly higher F1-scores due to better recall. Minimal errors are observed in confusion matrices with fewer false positives and negatives in both validation and test datasets. These excellent results prove that the model is not overfitted, as the validation and test accuracy scores are very close. However, I believe we still need to validate whether our ML model can be used in production with new, original DNA sequences. Let’s deploy the model and find out.</p><h4>7. DNA Sequence Production Machine Learning Model Test</h4><p>The production model test implementation is shown below.</p><pre>def main():       <br>    # dna sequence production dataset loading (6 dna rows, 3 with classs 1 and 3 with class 0)<br>    csv_path_file = os.path.join(config.CSV_FOLDER_PATH, &quot;promoter_dna_sequence_original_3_preprocess_production_data.csv&quot;)<br>    df = pd.read_csv(filepath_or_buffer=csv_path_file)<br><br>    # pickle deserialize tfidf vectorizer object<br>    tfidf_vectorizer_pickle_path = os.path.join(config.PICKLE_FOLDER_PATH, &quot;tfidf_vectorizer.pkl&quot;)<br>    pickle_class = PickleClass(tfidf_vectorizer_pickle_path)<br>    tfidf_vectorizer = pickle_class.pickle_deserialize_object()<br><br>    # pickle deserialize model classifier object<br>    model_classifier_pickle_path = os.path.join(config.PICKLE_FOLDER_PATH, &quot;model_classifier.pkl&quot;)<br>    pickle_class = PickleClass(model_classifier_pickle_path)<br>    model_classifier = pickle_class.pickle_deserialize_object()<br><br>    # predict the dna label for each sequence<br>    df[&#39;predicted_label&#39;] = df[&#39;dna_sequence&#39;].apply(lambda sequence: predict_dna_label(sequence, config.K_MERS_LENGTH, tfidf_vectorizer, model_classifier))<br>    print(df)<br><br>    # create a csv file to compare the original dna labels with the predicted labels<br>    csv_file_predicted = &quot;promoter_dna_sequence_original_3_preprocess_production_data_predicted.csv&quot;<br>    csv_path_file = os.path.join(config.CSV_FOLDER_PATH, csv_file_predicted)<br>    df.to_csv(path_or_buf=csv_path_file, index=False)<br>    print(f&quot;CSV file {csv_file_predicted} has been saved successfully&quot;)<br><br>if __name__ == &#39;__main__&#39;:<br>    main()</pre><p>Below is the final predicted DNA sequence file. As you can see, the values in the columns ‘dna_label_original’ and ‘dna_label_predicted<strong>’</strong> are the same. This confirms that the developed ETL preprocessing pipeline is essential for finalizing the synthetic DNA sequence dataset process for the ML project workflow. I believe this is good news for Bioinformatics and Life Sciences research and applications, especially when there is not enough data to apply Statistical Analysis and Machine Learning algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*UqEYnvuvwQvRix_ZpimB-A.png" /></figure><h4>8. Conclusions</h4><p><strong>1. Dataset Selection and Preparation:</strong></p><ul><li>Utilized promoter gene sequences from the UCI ML Repository (106 instances, balanced classes).</li><li>Addressed data scarcity by removing 6 rows for production testing, ensuring robust deployment validation.</li></ul><p><strong>2. Machine Learning Pipeline:</strong></p><ul><li>Developed a Random Forest model leveraging NLP techniques like DNA k-mer concatenation and TF-IDF vectorization.</li><li>Highlighted overfitting issues with the original dataset and introduced synthetic data generation as a solution.</li></ul><p><strong>3. Synthetic Data Generation with Gretel:</strong></p><ul><li>Generated 10,000 synthetic DNA sequences, identifying inconsistencies in synthetic sequence generation, particularly with platforms like Gretel. Some synthetic sequences contained errors that required extensive preprocessing through an ETL pipeline.</li><li>Combined 100 original sequences with 8,911 valid synthetic sequences, resulting in a final ML dataset of 9,011 rows.</li></ul><p><strong>4. Class Imbalance Resolution and Model Optimization:</strong></p><ul><li>Applied SMOTE to balance the dataset, improving training and test performance.</li><li>Achieved a validation accuracy of 97.45% and a test accuracy of 97.23%, demonstrating robust generalization.</li></ul><p><strong>5. Production Deployment and Testing:</strong></p><ul><li>Validated model performance on real-world production data, ensuring reliability and accuracy.</li><li>Provided reproducible workflows and code implementations for further development in DNA sequence analysis.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a06a051b3c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Machine Learning Models for Breast Cancer Diagnosis — A complete Machine Learning Project…]]></title>
            <link>https://ernest-bonat.medium.com/using-machine-learning-models-for-breast-cancer-diagnosis-a-complete-machine-learning-project-8af316843a8e?source=rss-58ededb6e465------2</link>
            <guid isPermaLink="false">https://medium.com/p/8af316843a8e</guid>
            <dc:creator><![CDATA[Ernest Bonat, Ph.D.]]></dc:creator>
            <pubDate>Fri, 08 Nov 2024 18:27:53 GMT</pubDate>
            <atom:updated>2024-11-09T18:32:29.505Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*5G2DG_C1d8PQ1sC5N-5rFQ.jpeg" /></figure><h3><strong>Using Machine Learning Models for Breast Cancer Diagnosis — A complete Machine Learning Project Workflow for Life Sciences</strong></h3><p><strong>Ernest Bonat, Ph.D., Chad Turner</strong></p><p>1. <a href="#ab67">Introduction</a><br>2. <a href="#2bbd">Research Question</a><br>3. <a href="#4237">Data Explanation</a><br>4. <a href="#39f4">Data Preprocessing<br></a>5. <a href="#c395">Classification Models<br></a>6. <a href="#2681">ChatGTP Confusion Matrix Report Generation</a><br>7. <a href="#1cec">Model Hyperparameter Optimization</a><br>8. <a href="#b526">Additional Classification Models</a><br>9. <a href="#5271">Correlation Analysis</a><br>10. <a href="#3c60">Revising the Dataset</a><br>11. <a href="#421f">Balancing the Target Classes</a><br>12. <a href="#66d4">Most Important Features</a><br>13. <a href="#5814">Multi-model Classification</a><br>14. <a href="#c56e">Artificial Neural Networks</a><br>15. <a href="#2398">Convolution Neural Networks<br></a>16. <a href="#8333">Conclusion</a></p><h4><strong>1. Introduction</strong></h4><p>The dataset used for this Machine Learning (ML) project is the “<a href="https://www.kaggle.com/datasets/uciml/breast-cancer-wisconsin-data">Breast Cancer Wisconsin (Diagnostic) Dataset</a>,” which consists of diagnostic imaging data used to detect breast cancer. This dataset can be obtained from <a href="https://kaggle.com/">Kaggle.com</a> by the University of California Irvine Machine Learning. The data is composed of computer-digitized images from a fine needle aspiration (FNA) of a breast mass. An FNA procedure is a minimally invasive medical procedure where samples of tissue or fluid are taken from areas of interest for diagnostic purposes. This is done by inserting a thin, hollow needle into suspected tissue to collect samples that are placed on a glass slide for analysis. The slides are digitized, and the boundaries of the cells are analyzed using a graphical user interface where the boundaries of each cell nucleus are defined with a contour model known as a “snake.” Inside each nucleus, ten features are analyzed in addition to the mean, largest or worst value, and standard error for these features. There are 30 features of cell nuclei being measured over 569 images. The data consists of 357 benign and 212 malignant diagnoses. The goal with this data is to build an accurate ML model in Python that predicts diagnoses based on feature values. To achieve this, supervised ML classification algorithms will be used.</p><h4><strong>2. Research Question</strong></h4><p>What features in cell nuclei have the greatest correlation with a malignant diagnosis for breast cancer?</p><h4><strong>3. Data Explanation</strong></h4><p>The target for this dataset is a column called “diagnosis.” This column contains two classes, “M” for malignant or “B” for benign, for each of the 569 images in the dataset.</p><p><strong>Feature Descriptions:</strong></p><p><strong>1.</strong> <strong>Radius:</strong> The mean of distances from the centroid to points along the snake perimeter of the cell.</p><p><strong>2.</strong> <strong>Texture:</strong> This measures the standard deviation of grayscale intensities. This is calculated by analyzing the pixels to look for visual patterns or structures that can indicate the roughness, smoothness, or other properties pertaining to texture. A high standard deviation in the gray scale values indicates a rough or complicated texture. A low standard deviation where the pixels are bunched together indicates a smooth or constant texture.</p><p><strong>3.</strong> <strong>Perimeter:</strong> The measured perimeter of the cell nucleus.</p><p><strong>4.</strong> <strong>Area:</strong> The calculated area of the cell nucleus.</p><p><strong>5.</strong> <strong>Compactness:</strong> The perimeter and area are combined to give the compactness through the formula: perimeter squared, divided by the area minus one. This measurement can be biased toward higher values for smaller cells as there is a decreased accuracy when measuring.</p><p><strong>6.</strong> <strong>Smoothness:</strong> A measure of the difference between the length of a radial line and the mean lengths of lines surrounding it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/244/1*PuxpmcWc4FDonnP8pss05Q.png" /><figcaption>Smoothness Measured</figcaption></figure><p><strong>7.</strong> <strong>Concavity:</strong> A measure of the count and gravity of concavities or indentations in the cell nucleus. This is measured by drawing chords between non-adjacent snake points on the perimeter, then measuring the boundary of the nucleus that resides inside each chord.</p><p><strong>8.</strong> <strong>Concave Points:</strong> This measurement is similar to concavity, only it measures the number of contour cavities rather than the extent.</p><p><strong>9.</strong> <strong>Symmetry:</strong> To measure the symmetry of a cell, first the longest chord through the center or major axis is defined, then chords perpendicular to the major axis in each direction are defined, splitting the cell up into a grid. Lastly, the difference of the perpendicular lines is measured. If the major axis cuts through the cell’s perimeter due to a concavity, further measurements are used.</p><p><strong>10.</strong> <strong>Fractal Dimension:</strong> To obtain this measurement, a method known as “coastline approximation” is used. This is done by measuring the perimeter with increasingly larger scales, which decreases the precision as the scale size increases. These values are plotted on a log scale where the measured downward slope is measured, achieving an approximate fractal dimension.</p><p>These 10 features make the total of 30 features in the dataset, as the mean, standard error, and worst condition are calculated for all 10 features. The mean, standard error, and worst condition are added to the end of each feature name (radius_mean, radius_se, and radius_worst). Features 1 through 10 are the mean values for the characteristics of a cell nucleus. Features 11 through 20 are the standard error values for each characteristic, and 21 through 30 are the worst values for each cell nucleus feature. There is also an additional column in this dataset called “id,” which is the patient’s identification number. Below is the DataFrame info.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*pMumAVVuX2Zw1ZHwnX4_tA.png" /></figure><h4><strong>4. Data Preprocessing</strong></h4><p>To begin the project, the CSV dataset is downloaded from <a href="https://kaggle.com/">Kaggle.com</a>, and the analysis of the data begins. A Python program is created where the CSV file is imported using the Pandas library’s “pd.read_csv” function and the file is passed as the argument, creating a DataFrame. Next, a function called “display_dataframe” is created to count and print the sum of duplicated rows, the sum of null values, the data types, and the DataFrame shape. The shape of the DataFrame is 33 columns and 569 rows. There are no duplicated rows, and the null count is 0 for all features except one unnamed row with 569 null values. All the columns are integers except the column “diagnosis,” which, in order to build an ML model, needs to be converted to numeric values. To solve this, this column is label encoded to binary using the Scikit-learn label encoder, creating a new column called “diagnosis_encoded.” The values were originally string values of “M” for malignant or “B” for benign. After encoding, “M” is now “1” and “B” is now “0.” To deal with the column of null values, a “clean_dataframe” function is run, which drops null values and duplicates. This function also removes columns upon request. In this case, the original “diagnosis” column as well as “id” are dropped as the patient’s ID number is unnecessary for a ML model, serving neither as a feature nor a target. This changes the shape of the DataFrame to 31 columns and 569 rows (1 target column and 30 feature columns). After confirming that the dataset is now completely numeric, has no duplicates, and has no null values, the data is now ready for modeling. A new CSV file called “breast_cancer_kaggle_cleaned.csv” is created with Pandas “df.to_csv” function with the argument as the file name and defined “index=False” so that the DataFrame does not have multiple indexes. The preprocessing code is shown below.</p><pre>def main():<br><br>    #loading csv        <br>    df = pd.read_csv(&quot;breast_cancer_kaggle.csv&quot;)<br>    <br>    #encoding categorical column    <br>    label_encoder = LabelEncoder()<br>    df[&quot;diagnosis_encoded&quot;] = label_encoder.fit_transform(df[&quot;diagnosis&quot;])<br>    <br>    #printing dataframe information<br>    display_dataframe_info(df)<br>    <br>    # #cleaning dataframe<br>    df = clean_dataframe(df, drop_columns=(&quot;id&quot;, &quot;diagnosis&quot;))<br><br>    #saving cleaned dataframe csv file<br>    df.to_csv(&quot;breast_cancer_kaggle_cleaned.csv&quot;, index=False)<br>    <br>    #printing final dataframe information<br>    display_dataframe_info(df)<br>    <br>    #plotting histogram<br>    df.hist(bins=11, figsize=(10,8))<br>    plt.tight_layout()<br>    plt.subplots_adjust(top=.9)<br>    plt.show()<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4><strong>5. Classification Models</strong></h4><p>For the first ML classification model, Random Forest Classification is used, which employs multiple decision trees that combine their outputs for a final class prediction. The preprocessed dataset is loaded using a function called “load_data.” This function takes two arguments: a CSV file path, which is set to “breast_cancer_kaggle_cleaned.csv,” and a target column, defined as “diagnosis_encoded.” This column is used to classify diagnoses based on the features in the DataFrame. This function then sets the target as “y” and the features as “X” while dropping the “diagnosis_encoded” column from “X.” The data is then split into training and testing sets using a function called “split_data,” which utilizes Scikit-learn’s “train_test_split” function. The test size is defined as “0.2,” resulting in an X_train DataFrame and a y_train Series that consist of 80% of randomly selected values from “X” and “y”. This also creates a Pandas DataFrame and Series for X_test and y_test, which holds the remaining 20% of the randomly chosen values of “X” and “y”. The X training and testing data are then scaled for uniformity using the “scale_data” function, creating “X_train_scaled” and “X_test_scaled.” This function fits the scaler to the training data only and transforms the testing data without fitting it. This approach helps prevent overfitting and ensures the testing data remains unseen. The model is trained using the function “train_random_forest,” which takes “X_train_scaled” and “y_train” as arguments. To evaluate the classification task, the function “evaluate_model” is created, which makes predictions based on the feature testing data. The accuracy score, classification report, and confusion matrix are also generated and then printed using a “plot_results” function. Finally, a confusion matrix graph is created using a function called “plot_confusion_matrix.” The Random Forest model implementation is shown below.</p><pre>def main():<br><br>    #load dataframe and define X and y    <br>    X, y = load_data(&quot;breast_cancer_kaggle_cleaned.csv&quot;, &quot;diagnosis_encoded&quot;)<br>    <br>    #split data for training and testing<br>    X_train, X_test, y_train, y_test = split_data(X, y)<br><br>    #scale training data<br>    X_train_scaled, X_test_scaled = scale_data(X_train, X_test)<br><br>    #train model<br>    model = train_random_forest(X_train_scaled, y_train)<br>    <br>    #evaluate model<br>    accuracy, report, cm = evaluate_model(model, X_test_scaled, y_test)<br><br>    #print metrics<br>    plot_results(accuracy, report, cm)<br><br>    #plot confusion matrix<br>    plot_confusion_matrix(cm, xlabel=&quot;Diagnosis Predicted Labels&quot;, ylabel=&quot;Diagnosis True Labels&quot;,<br>                            title=&quot;Random Forest Diagnostic Confusion Matrix&quot;,<br>                            xtick=(&quot;Benign&quot;, &quot;Malignant&quot;),<br>                            ytick=(&quot;Benign&quot;, &quot;Malignant&quot;))<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p><strong>Random Forest Classification</strong></p><p><strong>Model Performance</strong></p><p>Accuracy: 0.9474</p><p>94.74% of the classifications are correct.</p><p><strong>Classification Metrics</strong></p><p>Class 0 is a benign breast cancer diagnosis. Class 1 is a malignant breast cancer diagnosis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*PQ0BhKLAElAOvLBcS2sikw.png" /></figure><p><strong>Average Metrics</strong></p><p>Macro Average:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*ZiZy4UteoPyqnhHVhWroLw.png" /></figure><p>Weighted Average:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*Vg2upViW5iTP-mh4KTU-HA.png" /></figure><p><strong>Confusion Matrix</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/1*YS6kX8L5GTzc21j0tTEVOA.png" /></figure><ul><li>True Positives (TP): 36 (Correctly predicted malignant diagnoses)</li><li>True Negatives (TN): 72 (Correctly predicted benign diagnoses)</li><li>False Positives (FP): 3 (Incorrectly predicted malignant diagnoses)</li><li>False Negatives (FN): 3 (Incorrectly predicted benign diagnoses)</li></ul><p><strong>Observations</strong></p><p>The model performs well with 94.74% accuracy in predicting breast cancer diagnoses based on the dataset. The confusion matrix shows 3 false positives and 3 false negatives, indicating an area for improvement, especially for the false negatives.</p><h4>6. <strong>ChatGTP Confusion Matrix Report Generation</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*Pl2HliPRjts9Bxa3KXTd9Q.png" /></figure><p><strong>Summary</strong></p><ul><li><strong>Overall Accuracy</strong>: 94.7%</li><li><strong>Precision</strong>: 92.3%</li><li><strong>Recall</strong>: 92.3%</li><li><strong>F1-Score</strong>: 92.3%</li></ul><p>This model shows high accuracy and balanced precision and recall, indicating it is effective at distinguishing between malignant and benign diagnoses with minimal misclassification errors. However, it is important to monitor for any signs of overfitting if the model’s performance degrades on new data.</p><p><strong>7. Model Hyperparameter Optimization</strong></p><p>To see if the accuracy of this model can be improved, a copy of the previous program is created, incorporating grid search cross validation from scikit-learn. A new variable for the parameter grid is defined as “param_grid,” which specifies different parameters for the model to explore in order to identify the best settings for the dataset. The parameter grid is a dictionary containing the following parameters:</p><ul><li>Number of Estimators: 50, 100, and 150</li><li>Max depth: None, 10, 20, and 30</li><li>Minimum Samples Split: 2, 5, and 10</li><li>Minimum Samples Leaf: 1, 2, and 4</li></ul><p>The Random Forest Classifier is used as the estimator, with 5 cross folds for validation, number of jobs set to -1, and verbose set to 2. Through grid search cross validation, this model returns the optimum parameters for Random Forest Classification: max depth of None, minimum samples leaf of 1, minimum samples split of 2, and number of estimators set to 150. These tuned hyperparameters are added back into the Random Forest Classifier program, but the metrics show no improvements. The “GridSearchCV()” cross validation code is provided below.</p><pre>def main():<br>    #load dataframe and define X and y<br>    X, y = load_data(&quot;breast_cancer_kaggle_cleaned.csv&quot;, target_column=&quot;diagnosis_encoded&quot;)<br>    <br>    #split data for training and testing<br>    X_train, X_test, y_train, y_test = split_data(X, y)<br><br>    #scale training data<br>    X_train_scaled, X_test_scaled = scale_data(X_train, X_test)<br><br>    #defining random forest classifier<br>    rf = RandomForestClassifier(random_state=50)<br><br>    #defining the parameter grid<br>    param_grid = {<br>        &quot;n_estimators&quot;: [50, 100, 150],<br>        &quot;max_depth&quot;: [None, 10, 20, 30],<br>        &quot;min_samples_split&quot;: [2, 5, 10],<br>        &quot;min_samples_leaf&quot;: [1, 2, 4]<br>    }<br><br>    #defining gridsearch parameters for random forest <br>    grid_search = GridSearchCV(estimator=rf, param_grid=param_grid, cv=5, n_jobs=-1, verbose=2)<br><br>    #fitting model<br>    grid_search.fit(X_train_scaled, y_train)<br><br>    #finding and printing best parameters<br>    best_params = grid_search.best_params_<br>    print(f&quot;\nBest Parameters:\n{best_params}&quot;)<br><br>    #best model<br>    best_model = grid_search.best_estimator_<br>    <br>    #evaluate model<br>    accuracy, report, cm = evaluate_model(best_model, X_test_scaled, y_test)<br><br>    #print metrics<br>    plot_results(accuracy, report, cm)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4><strong>8. Additional Classification Models</strong></h4><p>To evaluate how well this dataset performs on different algorithms, a program is created to run multiple classification algorithms. This program accesses their metrics and plots their accuracy ratings against each other, allowing for the determination of which algorithm is the most efficient for classifying diagnoses. To do this, the necessary libraries are imported with their modules and classes. The function “load_data” is used to load the dataset and define X and y. The “split_data” function splits the data into 80% training and 20% testing DataFrames. The “scale_data” function is applied to scale feature training and testing data. A dictionary named “models” is created that includes several algorithms, each with their hyperparameters tuned from grid search cross-validation. The algorithms included are Logistic Regression, K-Nearest Neighbors (KNN), Decision Tree Classification, Random Forest Classification, Naïve Bayes (GaussianNB), Support Vector Machine (SVM), LightGBM, and XGBoost. Two empty lists named “model_accuracy” and “F1_score” are created and a for loop is established so that each algorithm is trained, makes predictions on the testing data, and prints each accuracy rating and F1 score, then stores them in the empty lists. A bar plot is then created using Matplotlib to display each model’s accuracy rating and F1 score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*Lb_96wty7-3OuiSZLnT6-g.png" /></figure><p>Logistic Regression and SVM models perform the best, each achieving 99.12% accuracy ratings, which is an increase from 98% when no hyperparameters are tuned. With the accuracy ratings being so high, this data could be overfitting the models, or there may be other unforeseen issues with the data.</p><pre>def main():<br>    #load dataframe and define X and y<br>    X, y = load_data(&quot;breast_cancer_kaggle_cleaned.csv&quot;, target_column=&quot;diagnosis_encoded&quot;)<br>    <br>    #split data for training and testing<br>    X_train, X_test, y_train, y_test = split_data(X, y)<br><br>    #scale training data<br>    X_train_scaled, X_test_scaled = scale_data(X_train, X_test)<br>    <br>    #defining classification models<br>    models = {<br>    &quot;Logistic Regression&quot;: LogisticRegression(C=.1, penalty=&quot;l2&quot;, solver=&quot;liblinear&quot;, random_state=42),<br>    &quot;K-Nearest Neighbors&quot;: KNeighborsClassifier(n_neighbors=11, metric=&quot;manhattan&quot;, weights=&quot;uniform&quot;),<br>    &quot;Decision Tree&quot;: DecisionTreeClassifier(max_depth=None, min_samples_leaf=4, min_samples_split=10),<br>    &quot;Random Forest&quot;: RandomForestClassifier(max_depth=None, min_samples_leaf=1, min_samples_split=2, <br>                                n_estimators=150, random_state=50),<br>    &quot;Naive Bayes&quot;: GaussianNB(var_smoothing=1e-06),<br>    &quot;Support Vector Machine&quot;: SVC(kernel=&#39;linear&#39;, C=1.0, max_iter=1000),<br>    &quot;LightGBM&quot;: lgb.LGBMClassifier(learning_rate=.1, max_depth=10, n_estimators=200, num_leaves=31),<br>    &quot;XGBoost&quot; : XGBClassifier(max_depth=7, learning_rate=.01, min_child_weight=1,<br>                                n_estimators=150, subsample=.8 ) <br>    }<br>    <br>    model_accuracy = []<br>    model_f1 = []<br>    <br>    for model_name, model in models.items():<br>        model.fit(X_train_scaled, y_train)<br>        y_predict = model.predict(X_test_scaled)<br>        <br>        accuracy = accuracy_score(y_test, y_predict)<br>        model_accuracy.append(accuracy*100)<br>        <br>        f1 = f1_score(y_test, y_predict)<br>        model_f1.append(f1*100)<br>        <br>        print(f&quot;{model_name} Accuracy: {accuracy:.4f}&quot;)<br>        print(f&quot;{model_name} F1 Score: {f1:.4f}&quot;)<br>    <br>    #plotting accuracy and F1 scores for all models<br>    bar_width = 0.35<br>    index = np.arange(len(models))<br>    plt.figure(figsize=(10, 6))<br><br>    #plotting accuracy<br>    bars_accuracy = plt.bar(index, model_accuracy, bar_width, label=&#39;Accuracy&#39;, color=&#39;skyblue&#39;)<br><br>    #plotting F1 Score bars<br>    bars_f1 = plt.bar(index + bar_width, model_f1, bar_width, label=&#39;F1 Score&#39;, color=&#39;salmon&#39;)<br><br>    #titles and labels<br>    plt.xlabel(&quot;Classification Models&quot;)<br>    plt.ylabel(&quot;Scores&quot;)<br>    plt.title(&quot;Classification Model Comparison: Accuracy and F1 Scores&quot;)<br>    plt.ylim(0, 100)  <br>    plt.xticks(index + bar_width / 2, models.keys(), rotation=45)<br>    plt.grid(axis=&quot;y&quot;)<br>    plt.legend()<br><br>    #annotations<br>    plt.bar_label(bars_accuracy, fmt=&#39;%.2f&#39;, label_type=&#39;center&#39;)<br>    plt.bar_label(bars_f1, fmt=&#39;%.2f&#39;, label_type=&#39;center&#39;)<br>    plt.tight_layout()<br>    plt.show()<br>        <br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4><strong>9. Correlation Analysis</strong></h4><p>To check for correlations, a simple Python program is created that first loads the dataset as a Pandas DataFrame. Next, a variable called “corr_matrix” is set to “df.corr().round(2),” which reads the DataFrame, calculates the correlations for all features, and rounds the results to the second decimal point. The results are printed, but with 30 features, they are too difficult to examine, so a heatmap is created using the Seaborn library to plot the results to better visualize the correlations for the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*uOGHUA5TlH8kOBUBSg2_Iw.png" /></figure><p>In this heatmap, nearly all the features show either a positive correlation or none, with no negative correlations present. However, there are several clusters of high correlations worth exploring. To inspect these correlations more closely, a variable called “threshold” is created and set to “0.7.” This filters out all correlations, leaving only those features with a correlation greater than 0.7. Next, another variable named “filtered_corr” is created and set to corr_matrix[(corr_matrix &gt;= threshold) | (corr_matrix &lt;= -threshold)]. This code checks each element in the correlation matrix to see if it meets the threshold criteria, combining both conditions to identify values that are either above or below the threshold. The result is a refined matrix only showing correlations above the threshold. Finally, this new matrix is plotted, providing a clearer visualization of the significant relationships between features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*7E9By9-vvxvbeoBOxfxlTQ.png" /></figure><p>In this heatmap, setting a threshold to greater than 0.7 shows a much clearer view of the highly correlated variables. The first pattern that emerges is dense clusters of high correlations, especially near the diagonal line of self-correlations. Another observation is that the radius, perimeter, and area have nearly a 1-to-1 correlation to one another. This makes sense because if the radius of a cell increases, the perimeter and area will as well. This pattern is repeated for the standard error and worst condition. These clusters of high correlations indicate multicollinearity, where more than two independent variables are highly correlated, possibly providing redundant information. The multicollinearity suggested that the model is overfitted, which could be affecting the performance.</p><p>Another interesting observation is that as the radius mean, perimeter mean, area mean, and concavity mean get larger in size, they tend to have a positive correlation with the diagnosis. This trend is repeated with the worst condition features, where concavity_worse shows the strongest correlation to diagnosis_encoded at 0.79. This indicates that as the cell nucleus grows in general size and has larger indentations, the likelihood of a malignant diagnosis also increases, effectively answering the research question. The correlation analysis code implementation is shown below.</p><pre>def main():<br>    #reading csv<br>    df = pd.read_csv(&quot;breast_cancer_kaggle_cleaned.csv&quot;)<br>    <br>    #defining correlation matrix<br>    corr_matrix = df.corr().round(2)<br>    <br>    #creating a threshold and filtered matrix<br>    threshold = 0.7<br>    filtered_corr = corr_matrix[(corr_matrix &gt;= threshold) | (corr_matrix &lt;= -threshold)]<br><br>    #plotting the correlation matrix<br>    plt.figure(figsize=(10,8))<br>    sns.set_context(&quot;notebook&quot;, font_scale=.5)<br>    plot = sns.heatmap(corr_matrix, annot=True, cmap=&#39;coolwarm&#39;, annot_kws={&quot;size&quot;: 6})<br>    plt.xticks(rotation=45)<br>    plot.set_title(&quot;Breast Cancer Diagnostic Correlation Matrix&quot;, fontsize=10)<br><br>    #plotting the filtered correlation matrix<br>    plt.figure(figsize=(10,8))<br>    sns.set_context(&quot;notebook&quot;, font_scale=.5)<br>    plot = sns.heatmap(filtered_corr, annot=True, cmap=&#39;coolwarm&#39;, annot_kws={&quot;size&quot;: 6})<br>    plt.xticks(rotation=45)<br>    plot.set_title(&quot;Strong Correlations Threshold: &gt;.7&quot;, fontsize=10)<br>    plt.show()<br>    <br>    #removing highly correlated columns<br>    df = df.drop(df.columns[10:30], axis=1)<br>    df = df.drop(columns=[&quot;perimeter_mean&quot;, &quot;area_mean&quot;, &quot;concave points_mean&quot;, &quot;compactness_mean&quot;])<br>    <br>    #creating revised matrix <br>    corr_matrix2 = df.corr().round(2)<br>    filtered_corr2 = corr_matrix2[(corr_matrix2 &gt;= threshold) | (corr_matrix2 &lt;= -threshold)]<br><br>    #plotting the new correlation matrix<br>    plt.figure(figsize=(10,8))<br>    sns.set_context(&quot;notebook&quot;, font_scale=1)<br>    plot = sns.heatmap(corr_matrix2, annot=True, cmap=&#39;coolwarm&#39;, annot_kws={&quot;size&quot;: 10})<br>    plt.xticks(rotation=15)<br>    plt.yticks(rotation=45)<br>    plot.set_title(&quot;Revised Breast Cancer Diagnostic Correlation Matrix&quot;, fontsize=15)<br>    <br>    #plotting the new filtered correlation matrix<br>    plt.figure(figsize=(10,8))<br>    sns.set_context(&quot;notebook&quot;, font_scale=1)<br>    plot = sns.heatmap(filtered_corr2, annot=True, cmap=&#39;coolwarm&#39;, annot_kws={&quot;size&quot;: 10})<br>    plt.xticks(rotation=15)<br>    plt.yticks(rotation=45)<br>    plot.set_title(&quot;Revised Strong Correlations Threshold: &gt;.7&quot;, fontsize=15)<br>    plt.show()<br>    <br>    #creating a revised dataset<br>    df.to_csv(&quot;breast_cancer_kaggle_cleaned-2.csv&quot;, index=False)<br>   <br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4><strong>10. Revising the Dataset</strong></h4><p>To minimize the complexity and reduce the risk of overfitting, the redundant features with standard error (“_se”) and worst condition (“_worst”) are dropped from the dataset. To achieve this, the last twenty columns are indexed and dropped using the command “df.drop(df.columns[10:30], axis=1)” and the correlation matrix is reassessed. Despite this, some highly correlated values remain, so the additional two columns “perimeter_mean” and “area_mean” are dropped as they are nearly 1-to-1 correlated with the “radius_mean.” While this significantly simplified the data, a correlation of 0.92 exists between the features “concavity_mean” and “concave_points_mean,” along with a correlation of 0.88 between “concave points_mean” and “compactness_mean”. To resolve this, “concave_points_mean” and compactness_mean” are dropped, leaving a total of 6 feature columns. A revised CSV file is then created by executing “df.to_csv(“breast_cancer_kaggle_cleaned-2.csv”, index=False).” This revised dataset is then run through the Random Forest model to ensure the metrics remain robust. These changes lower the complexity of the data and help fit the models better, addressing the overfitting issue in the original dataset, which led to low error rates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*_hiakS7qvOZbCZtluw_ovQ.png" /></figure><h4><strong>11. Balancing the Target Classes</strong></h4><p>Upon further inspection of the revised dataset, an imbalance in the classes is identified that could affect the performance of the ML model. The data skews toward the benign class, which contains 357 instances compared to only 212 instances of the malignant class. This imbalance introduces bias toward the benign class, potentially causing the models to be more effective at classifying benign diagnoses than malignant ones. Additionally, this can lead to misleading accuracy ratings that favor the majority class while increasing the error rate for the minority class. To address this imbalance, a function called “apply_smote” is created. This function uses the synthetic minority oversampling technique (SMOTE) to generate synthetic data for the minority class using its k-nearest neighbors. It takes the scaled training data (X and y) and creates “X_train_balanced” and “y_train_balanced,” each containing an equal number of instances (568) to eliminate bias toward the majority class. The testing data remains unbalanced to allow the model to learn from the balanced training set while making predictions based on the actual testing data. This function is integrated into the models and executed using the revised dataset.</p><p><strong>Random Forest Classification</strong></p><p><strong>Model Performance</strong></p><p>Accuracy: 0.9737</p><p>97.37% of the classifications are correct.</p><p><strong>Classification Metrics</strong></p><p>Class 0 is a benign breast cancer diagnosis. Class 1 is a malignant breast cancer diagnosis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*7NM9DPDdD74mq6awiuJDog.png" /></figure><p><strong>Average Metrics</strong></p><p>Macro Average:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*VoqFhTmENpC__L3Yn-hUSQ.png" /></figure><p>Weighted Average:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*XVHJBHh6gTexqJuwopBfGA.png" /></figure><p><strong>Confusion Matrix</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*nWtWaQAqBPtw_SZBgHou3w.png" /></figure><ul><li>True Positives (TP): 39 (Correctly predicted malignant diagnoses)</li><li>True Negatives (TN): 72 (Correctly predicted benign diagnoses)</li><li>False Positives (FP): 1 (Incorrectly predicted malignant diagnoses)</li><li>False Negatives (FN): 2 (Incorrectly predicted benign diagnoses)</li></ul><p><strong>Observations</strong></p><p>This model now performs much better, achieving 97.37% accuracy in predicting breast cancer diagnoses after balancing the classes and revising the dataset. The confusion matrix indicates 1 false positive and 2 false negatives, meaning it mislabeled two malignant cases as benign and one benign case as malignant. Ideally, there would be no false negatives, as it would be preferable to have a false positive diagnosis rather than a false negative. The macro and weighted averages yield the same result for the precision, recall, F1-score, and support, demonstrating that the data is well balanced in class distribution and providing consistent performance. The code below shows how to fix the “diagnosis_encoded” imbalanced classes.</p><pre>def main():<br>    #load dataframe and define X and y<br>    X, y = load_data(&quot;breast_cancer_kaggle_cleaned-2.csv&quot;, target_column=&quot;diagnosis_encoded&quot;)<br>    <br>    #split data for training and testing<br>    X_train, X_test, y_train, y_test = split_data(X, y)<br><br>    #scale training data<br>    X_train_scaled, X_test_scaled = scale_data(X_train, X_test)<br>    <br>    #balance training data<br>    X_train_balanced, y_train_balanced = apply_smote(X_train_scaled, y_train)<br><br>    #train model<br>    model = train_random_forest(X_train_balanced, y_train_balanced)<br>    <br>    #evaluate model<br>    accuracy, report, cm = evaluate_model(model, X_test_scaled, y_test)<br><br>    #print metrics<br>    plot_results(accuracy, report, cm)<br><br>    #plot confusion matrix<br>    plot_confusion_matrix(cm, xlabel=&quot;Diagnosis Predicted Labels&quot;, ylabel=&quot;Diagnosis True Labels&quot;,<br>                            title=&quot;Random Forest Diagnostic Confusion Matrix&quot;,<br>                            xtick=(&quot;Benign&quot;, &quot;Malignant&quot;),<br>                            ytick=(&quot;Benign&quot;, &quot;Malignant&quot;))<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4><strong>12. Most Important Features</strong></h4><p>To further analyze the revised dataset and determine if the best features were retained for the classification models, a program is created to visualize the features that the Random Forest algorithm identifies as the most important for classification. To accomplish this, a function called “rf_importance” is created that has a Random Forest Classification program within it. The variable “feature_importances = rf_model.feature_importances_” is added to this function to extract the important features. Then, a Pandas DataFrame is created to store these features along with their importance, sorted by importance, and the resulting DataFrame is plotted. In the main function, the importance function is called twice to plot the feature importance for both the original dataset and the revised dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*Xcbz6odpKM48-Kylfl8JrA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*bFzHp4xrYHNPQNpKsK5w6A.png" /></figure><p>It is found that the Random Forest algorithm ranks many of the features that were dropped as more important than those retained. Among the 6 features retained, only “radius_mean” and “concavity_mean” were in the top ten important features from the original dataset. To verify the assumptions from the correlation analysis, the Random Forest model is run using the top ten features identified from the feature importance of the original dataset. The model achieves an accuracy of 95.61% and a confusion matrix showing 3 false negatives and 2 false positives. In comparison, the Random Forest model with the revised dataset has an accuracy rating of 97.37% with only 2 false negatives and 1 false positive in the confusion matrix. This indicates that the choices made during the correlation analysis are better suited for the classification task.</p><p>While Random Forest importance reveals what it perceives as the most important features, this is done through calculations, not necessarily logic. This highlights the importance of having human oversight in supervised ML projects. The algorithm indicates that the most important features are the ones that show some of the highest correlations to one another. This could be because the algorithm finds features that hold patterns that may not necessarily be important for the task. While ML models are excellent tools in making predictions, a human is still needed to interpret these predictions, assess the quality of the data, and identify biases, especially in a delicate application like breast cancer diagnosis where the stakes are high. The features of importance can be defined using the code below.</p><pre>def main():    <br>    #defining datasets<br>    rf_importance(&quot;breast_cancer_kaggle_cleaned.csv&quot;, plt_title1)<br>    rf_importance(&quot;breast_cancer_kaggle_cleaned-2.csv&quot;, plt_title2)<br>    <br>    #defining plot titles<br>    plt_title1 = &quot;Random Forest Feature Importance (Original Dataset)&quot;<br>    plt_title2 = &quot;Random Forest Feature Importance (Revised Dataset)&quot;<br>        <br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4><strong>13. Multi-model Classification</strong></h4><p>To observe how the revised dataset performs across multiple models, it is run through the multi-model program, comparing the following models with their hyperparameters tuned using grid search cross validation: Logistic Regression, KNN, Decision Tree Classification, Random Forest Classification, Naïve Bayes, SVM, LightGBM, and XGBoost. Among these models, Random Forest Classification and XGBoost outperform the others, achieving equal metrics, with accuracy scores of 97.37% and F1-scores of 96.3%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*slExkyoge8JUNOIyz-cj-w.png" /></figure><h4><strong>14. Artificial Neural Networks</strong></h4><p><strong>Multilayer Perceptron</strong></p><p>The next ML model used is a Multilayer Perceptron (MLP). An MLP is a type of artificial neural network that consists of an input layer followed by one or more layers of threshold logistic units (TLU), which function as basic neurons. It is a feedforward neural network (FNN) where the data passes linearly through the input layer, then through one or more hidden layers, and finally out to the output layer. In the hidden layers, data points or features are processed as each neuron applies its weights and biases, along with an activation function. This transformation of the input data enables the network to learn complex patterns. Activation functions like the rectified linear unit (ReLU) or sigmoid introduce non-linearity, assisting in learning intricate relationships. The number of layers and neurons are hyperparameters that can be tuned using cross-validation. The model runs through multiple epochs or iterations until the model converges, minimizing the loss or difference between the predicted and target outputs or until improvement stops.</p><p>To create this model, the necessary libraries are imported, and the revised dataset is loaded as a Pandas DataFrame. The target and features are defined using the “load_data” function. Then, the data is split using the “split_data” function, creating a training DataFrame with 80% of random values from the features and a testing DataFrame with the remaining 20% for validation. X_train and X_test are scaled and balanced using the “scale_data” and “apply_smote” functions. The MLP model is created using the “MLPClassifier” from Scikit-learn, and the hyperparameters are defined. “hidden_layer_sizes” is set to 100, creating one hidden layer with 100 neurons. For the activation function, the “relu” algorithm is used to introduce non-linearity and identify complex patterns. To optimize weights, the “solver” is set to “adam” or adaptive moment estimation (a stochastic optimizer) and “max_iter” is set to 1000, allowing the optimizer enough iterations to converge and learn all the patterns in the training data. The model is then fit using “X_train_balanced” and “y_train_balanced.” The “evaluate_model” function is used to make predictions on the test data and generate the accuracy score, classification report, and confusion matrix. These results are printed using the “plot_results” function and the “plot_confusion_matrix” function to display the confusion matrix graph.</p><p><strong>Model Performance</strong></p><p>Accuracy: 0.9825</p><p>98.25% of the classifications are correct.</p><p><strong>Classification Metrics</strong></p><p>Class 0 is a benign breast cancer diagnosis. Class 1 is a malignant breast cancer diagnosis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*yVwZi9cLM8FyNg2TtCRJrg.png" /></figure><p><strong>Average Metrics</strong></p><p>Macro Average:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*FZfGHFCf5eN4BqNW31pHfA.png" /></figure><p>Weighted Average:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*F_pa9VbyAm2AUsULt9Dldg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*d2N3sj0INaHP-nq6sSMROQ.png" /></figure><ul><li>True Positives (TP): 41 (Correctly predicted malignant diagnoses)</li><li>True Negatives (TN): 71 (Correctly predicted benign diagnoses)</li><li>False Positives (FP): 0 (Incorrectly predicted malignant diagnoses)</li><li>False Negatives (FN): 2 (Incorrectly predicted benign diagnoses)</li></ul><p><strong>Observations</strong></p><p>This MLP model performs very well in classifying breast cancer diagnoses achieving an accuracy score of 98.25%. It demonstrates high precision, recall, and F1 scores, indicating that it makes accurate predictions. The macro and weighted averages are nearly the same, suggesting that the dataset is well balanced and the model is consistent. The confusion matrix reveals that the MLP predicted the correct diagnoses except for 2 cases that were incorrectly classified as benign when they were actually malignant. The MLP classification model code development can be found below.</p><pre>def main():    <br>    #load dataframe and define X and y<br>    X, y = load_data(&quot;breast_cancer_kaggle_cleaned-2.csv&quot;, target_column=&quot;diagnosis_encoded&quot;)<br>    <br>    #split data for training and testing<br>    X_train, X_test, y_train, y_test = split_data(X, y)<br><br>    #scale training data<br>    X_train_scaled, X_test_scaled = scale_data(X_train, X_test)<br>    <br>    #balance training data<br>    X_train_balanced, y_train_balanced = apply_smote(X_train_scaled, y_train)<br><br>    #creating model and defining hyperparameters<br>    model = MLPClassifier(hidden_layer_sizes=(100,), activation=&#39;relu&#39;,<br>                solver=&#39;adam&#39;, max_iter=1000, random_state=40)<br>    <br>    #fitting model to training data<br>    model.fit(X_train_balanced, y_train_balanced)<br>    <br>    #evaluate model<br>    accuracy, report, cm = evaluate_model(model, X_test_scaled, y_test)<br><br>    #print metrics<br>    plot_results(accuracy, report, cm)<br><br>    #plot confusion matrix<br>    plot_confusion_matrix(cm, xlabel=&quot;Diagnosis Predicted Labels&quot;, ylabel=&quot;Diagnosis True Labels&quot;,<br>                            title=&quot;MLP Diagnostic Confusion Matrix&quot;,<br>                            xtick=(&quot;Benign&quot;, &quot;Malignant&quot;),<br>                            ytick=(&quot;Benign&quot;, &quot;Malignant&quot;))<br>        <br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4><strong>15. Convolution Neural Networks</strong></h4><p>For the final ML model, a deep learning model known as a Convolutional Neural Networks (CNN) is used. While similar to the MLP in its use of neurons and connected layers, a CNN has a more complex architecture that includes specialized layers such as convolutional layers and pooling layers, in addition to fully connected layers. CNNs are usually used for more complex applications, such as image recognition, but can also be applied to simpler classification tasks like this.</p><p>To prepare this model, the necessary libraries are imported, and TensorFlow’s random seed is set for the reproducibility of results. Like the previous models, the dataset is loaded, the target and features are defined, the data is split into 80% training and 20% testing sets, and the data is balanced using functions. For X_test_balanced and X_train_balanced, the data needs to be reshaped to be used in the CNN model, as a one-dimensional CNN needs the format of: (samples, features, 1). For “X_train_reshaped,” the shape is (568, 6, 1). 568 is the number of feature rows (after balancing), 6 is the number of features, and 1 is the number of channels per input sequence. For X_test_reshaped, the shape is (114, 6, 1), with 114 representing 20% of the feature data.</p><p><strong>Building the Model</strong></p><p>To create the CNN model, “model.Sequential()” is called for a linear stack of layers where one input gets one output. The layers inside the CNN are then defined, starting with “layers.Conv1D(),” which creates a one-dimensional convolution layer for the input. For the activation function, “relu” is used, creating non-linearity. For the pooling layer, “layers.MaxPooling1D()” is used, which tries to retain the important features while reducing the dimensionality from the convolutional layer. Next is “layers.Flatten(),” which flattens the shape of the pooling layer into a 1D array, preparing it for the dense layers. The last two layers are dense layers, or “layers.Dense,” where the first is set to 64 neurons with the activation function “relu”. The last dense layer is the output layer, where one neuron with the “sigmoid” activation function is used, which takes an input on the X axis and returns a Y value between 0 and 1.</p><p><strong>Compiling and Fitting the Model</strong></p><p>The model is then compiled using “model.compile” and the “adam” algorithm for the optimizer, with the loss function set to “binary cross-entropy” for measuring the difference between predicted and actual class labels. To fit the model, a variable called “history” is created so that the results can be plotted and set it to “model.fit,” where the number of epochs is set to “30.” This allows the training and testing data enough time to converge while avoiding overfitting. The batch size used is “32,” which indicates that during each epoch, the model will train on batches of 32 values of the training data until all data is used.</p><pre>def main():<br>    #loading dataset, defining target and features<br>    X, y = load_data(file_path=&quot;breast_cancer_kaggle_cleaned-2.csv&quot;, target_column=&quot;diagnosis_encoded&quot;)<br>    <br>    #splitting data into test and training sets<br>    X_train, X_test, y_train, y_test = split_data(X, y)<br>    <br>    #scaling data<br>    X_train_scaled, X_test_scaled = scale_data(X_train, X_test)<br>    <br>    #balance training data<br>    X_train_balanced, y_train_balanced = apply_smote(X_train_scaled, y_train)<br>    <br>    #creating number of classes<br>    class_number = np.unique(y).sum()<br>    <br>    # #reshaping for CNN<br>    X_train_reshaped = X_train_balanced.reshape(X_train_balanced.shape[0], X.shape[1], 1)<br>    X_test_reshaped = X_test_scaled.reshape(X_test_scaled.shape[0], X.shape[1], 1)<br>    <br>    #defining 1D CNN model<br>    model = models.Sequential([<br>        layers.Conv1D(32, kernel_size=3, activation=&quot;relu&quot;, input_shape=(X_train_reshaped.shape[1], 1)),<br>        layers.MaxPooling1D(pool_size=2),<br>        layers.Flatten(),<br>        layers.Dense(64, activation=&quot;relu&quot;),<br>        layers.Dense(class_number, activation=&quot;sigmoid&quot;)<br>    ])<br><br>    #compiling the model<br>    model.compile(optimizer=&quot;adam&quot;, loss=&quot;binary_crossentropy&quot;, metrics=[&quot;accuracy&quot;])<br>    model.summary()<br>    <br>    #fit the model<br>    history = model.fit(X_train_reshaped, y_train_balanced, epochs=30, batch_size=32, <br>                        validation_data=(X_test_reshaped, y_test))<br><br>    #making predictions<br>    y_pred_probs = model.predict(X_test_reshaped)<br>    y_pred = (y_pred_probs &gt; 0.5).astype(int)<br>    <br>    #model metrics<br>    test_loss, test_accuracy, = model.evaluate(X_test_reshaped, y_test)<br>    report = classification_report(y_test, y_pred)<br>    cm = confusion_matrix(y_test, y_pred)<br>    <br>    #printing metrics<br>    print(f&quot;Test Accuracy: {test_accuracy:.4f}&quot;)<br>    print(f&quot;Test Loss: {test_loss:.4f}&quot;)<br>    print(&quot;Classification Report:&quot;)<br>    print(report)<br>    print(f&quot;confusion matrix:\n{cm}&quot;)<br>    <br>    #plot training and loss history<br>    plot_history(history)<br>    <br>    #plot confusion matrix<br>    plot_confusion_matrix(cm, xlabel=&quot;Diagnosis Predicted Labels&quot;, ylabel=&quot;Diagnosis True Labels&quot;,<br>                            title=&quot;CNN Diagnostic Confusion Matrix&quot;,<br>                            xtick=(&quot;Benign&quot;, &quot;Malignant&quot;),<br>                            ytick=(&quot;Benign&quot;, &quot;Malignant&quot;))<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p><strong>Evaluating the Model</strong></p><p>To create probabilities for the testing data and to evaluate the model, a new variable called “y_pred_probs” is created and set to “model.predict(X_test_reshaped).” Next, the variable “y_pred” is created and set to “(y_pred_probs &gt; .5).astype(int).” This now creates a threshold for predicted values, giving a Boolean True if the predicted value is greater than 0.5 or False if it is less than 0.5, then converts the Booleans to binary integers where they are evaluated against the true labels. For the metrics, a variable for loss and accuracy is created and printed, along with a classification report and confusion matrix. Two subplots are created using the “plot_history” function to visualize the loss and accuracy over epochs from the training set against the testing set using Matplotlib. Finally, a confusion matrix graph is generated using the “plot_confusion_matrix” function.</p><pre>Model: &quot;sequential&quot;<br>_________________________________________________________________<br> Layer (type)                Output Shape              Param #<br>=================================================================<br> conv1d (Conv1D)             (None, 4, 32)             128<br><br> max_pooling1d (MaxPooling1D  (None, 2, 32)            0<br> )<br><br> flatten (Flatten)           (None, 64)                0<br><br> dense (Dense)               (None, 64)                4160<br><br> dense_1 (Dense)             (None, 1)                 65<br><br>=================================================================<br>Total params: 4,353<br>Trainable params: 4,353<br>Non-trainable params: 0<br>_________________________________________________________________<br>Epoch 1/30<br>2024-11-08 09:41:04.274207: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8100<br>18/18 [==============================] - 5s 47ms/step - loss: 0.6278 - accuracy: 0.7324 - val_loss: 0.5166 - val_accuracy: 0.9123<br>Epoch 2/30<br>18/18 [==============================] - 0s 20ms/step - loss: 0.4727 - accuracy: 0.8697 - val_loss: 0.3795 - val_accuracy: 0.9035<br>Epoch 3/30<br>18/18 [==============================] - 0s 20ms/step - loss: 0.3566 - accuracy: 0.8838 - val_loss: 0.2880 - val_accuracy: 0.9298<br>Epoch 4/30<br>18/18 [==============================] - 0s 24ms/step - loss: 0.2757 - accuracy: 0.8996 - val_loss: 0.2349 - val_accuracy: 0.9298<br>Epoch 5/30<br>18/18 [==============================] - 0s 24ms/step - loss: 0.2234 - accuracy: 0.9313 - val_loss: 0.2106 - val_accuracy: 0.9298<br>Epoch 6/30<br>18/18 [==============================] - 0s 22ms/step - loss: 0.1898 - accuracy: 0.9419 - val_loss: 0.1934 - val_accuracy: 0.9386<br>Epoch 7/30<br>18/18 [==============================] - 0s 23ms/step - loss: 0.1714 - accuracy: 0.9489 - val_loss: 0.1823 - val_accuracy: 0.9474<br>Epoch 8/30<br>18/18 [==============================] - 0s 25ms/step - loss: 0.1569 - accuracy: 0.9437 - val_loss: 0.1705 - val_accuracy: 0.9474<br>Epoch 9/30<br>18/18 [==============================] - 0s 23ms/step - loss: 0.1467 - accuracy: 0.9472 - val_loss: 0.1598 - val_accuracy: 0.9474<br>Epoch 10/30<br>18/18 [==============================] - 0s 25ms/step - loss: 0.1368 - accuracy: 0.9507 - val_loss: 0.1531 - val_accuracy: 0.9474<br>Epoch 11/30<br>18/18 [==============================] - 0s 25ms/step - loss: 0.1295 - accuracy: 0.9525 - val_loss: 0.1455 - val_accuracy: 0.9474<br>Epoch 12/30<br>18/18 [==============================] - 0s 21ms/step - loss: 0.1243 - accuracy: 0.9560 - val_loss: 0.1397 - val_accuracy: 0.9474<br>Epoch 13/30<br>18/18 [==============================] - 0s 26ms/step - loss: 0.1201 - accuracy: 0.9560 - val_loss: 0.1353 - val_accuracy: 0.9474<br>Epoch 14/30<br>18/18 [==============================] - 0s 24ms/step - loss: 0.1167 - accuracy: 0.9560 - val_loss: 0.1299 - val_accuracy: 0.9561<br>Epoch 15/30<br>18/18 [==============================] - 0s 21ms/step - loss: 0.1136 - accuracy: 0.9577 - val_loss: 0.1257 - val_accuracy: 0.9561<br>Epoch 16/30<br>18/18 [==============================] - 0s 21ms/step - loss: 0.1113 - accuracy: 0.9577 - val_loss: 0.1234 - val_accuracy: 0.9561<br>Epoch 17/30<br>18/18 [==============================] - 0s 24ms/step - loss: 0.1096 - accuracy: 0.9595 - val_loss: 0.1233 - val_accuracy: 0.9561<br>Epoch 18/30<br>18/18 [==============================] - 0s 22ms/step - loss: 0.1069 - accuracy: 0.9577 - val_loss: 0.1203 - val_accuracy: 0.9649<br>Epoch 19/30<br>18/18 [==============================] - 1s 31ms/step - loss: 0.1064 - accuracy: 0.9613 - val_loss: 0.1210 - val_accuracy: 0.9649<br>Epoch 20/30<br>18/18 [==============================] - 0s 23ms/step - loss: 0.1077 - accuracy: 0.9613 - val_loss: 0.1155 - val_accuracy: 0.9649<br>Epoch 21/30<br>18/18 [==============================] - 0s 24ms/step - loss: 0.1022 - accuracy: 0.9630 - val_loss: 0.1202 - val_accuracy: 0.9737<br>Epoch 22/30<br>18/18 [==============================] - 0s 24ms/step - loss: 0.1013 - accuracy: 0.9648 - val_loss: 0.1163 - val_accuracy: 0.9649<br>Epoch 23/30<br>18/18 [==============================] - 0s 24ms/step - loss: 0.0994 - accuracy: 0.9648 - val_loss: 0.1150 - val_accuracy: 0.9649<br>Epoch 24/30<br>18/18 [==============================] - 0s 22ms/step - loss: 0.0998 - accuracy: 0.9648 - val_loss: 0.1138 - val_accuracy: 0.9649<br>Epoch 25/30<br>18/18 [==============================] - 0s 24ms/step - loss: 0.1012 - accuracy: 0.9613 - val_loss: 0.1128 - val_accuracy: 0.9737<br>Epoch 26/30<br>18/18 [==============================] - 0s 23ms/step - loss: 0.1018 - accuracy: 0.9630 - val_loss: 0.1110 - val_accuracy: 0.9737<br>Epoch 27/30<br>18/18 [==============================] - 0s 23ms/step - loss: 0.0959 - accuracy: 0.9648 - val_loss: 0.1112 - val_accuracy: 0.9561<br>Epoch 28/30<br>18/18 [==============================] - 0s 25ms/step - loss: 0.0962 - accuracy: 0.9665 - val_loss: 0.1136 - val_accuracy: 0.9737<br>Epoch 29/30<br>18/18 [==============================] - 0s 25ms/step - loss: 0.0945 - accuracy: 0.9683 - val_loss: 0.1119 - val_accuracy: 0.9737<br>Epoch 30/30<br>18/18 [==============================] - 0s 23ms/step - loss: 0.0960 - accuracy: 0.9613 - val_loss: 0.1116 - val_accuracy: 0.9737<br>4/4 [==============================] - 0s 4ms/step<br>4/4 [==============================] - 0s 9ms/step - loss: 0.1116 - accuracy: 0.9737<br>Test Accuracy: 0.9737<br>Test Loss: 0.1116<br>Classification Report:<br>              precision    recall  f1-score   support<br><br>           0       0.96      1.00      0.98        73<br>           1       1.00      0.93      0.96        41<br><br>    accuracy                           0.97       114<br>   macro avg       0.98      0.96      0.97       114<br>weighted avg       0.97      0.97      0.97       114<br><br>confusion matrix:<br>[[73  0]<br> [ 3 38]]</pre><p><strong>Model Performance</strong></p><p>Accuracy: .9737</p><p>Loss: 0.1116</p><p>97.37% of the classifications are correct with a low loss value of 0.1116.</p><p><strong>Classification Metrics</strong></p><p>Class 0 is a benign breast cancer diagnosis. Class 1 is a malignant breast cancer diagnosis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*Vb4nNNUi6p2JribbDleiIA.png" /></figure><p><strong>Average Metrics</strong></p><p>Macro Average:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*8M7FskoYzbHCgIQrevrlzw.png" /></figure><p>Weighted Average:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*MBd61reaXCRZZMNaBrxbvg.png" /></figure><p><strong>Confusion Matrix</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*N7v42MQuqc1lUPQV00b-Kw.png" /></figure><ul><li>True Positives (TP): 38 (Correctly predicted malignant diagnoses)</li><li>True Negatives (TN): 73 (Correctly predicted benign diagnoses)</li><li>False Positives (FP): 0 (Incorrectly predicted malignant diagnoses)</li><li>False Negatives (FN): 3 (Incorrectly predicted benign diagnoses)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*QYwsBfyiSV5VBvDhWnJipQ.png" /></figure><p><strong>Observations</strong></p><p>This model performs well, with a 97.37% accuracy rating and a loss rating of 0.1116, showing that it has little error in predictions. It correctly predicted diagnoses, except for 3 instances that predicted a benign diagnosis when the correct label was malignant. An interesting observation was that this model’s accuracy score matched the results of the Random Forest and XGBoost models. This indicates that the data is now stable and the models are equally complex, finding the same patterns in the dataset. To test this further, new unseen data would need to be introduced to each model.</p><p>The source code for this research paper can be found in the following GitHub repository <a href="https://github.com/cturner119/breast_cancer_diagnosis/tree/main">https://github.com/cturner119/breast_cancer_diagnosis/tree/main</a>.</p><h4><strong>16. Conclusion</strong></h4><p>For diagnosing breast cancer from a FNA procedure using ML models, all the data provided was excessive. The mean values for the features of the cell nucleus would suffice, but the addition of worst condition and standard error for each feature led to overfitting, increased complexity, and multicollinearity. Another observed issue was class imbalance, which was addressed using SMOTE to create artificial data and give each class equal representation. Although the Random Forest algorithm suggested what it calculated to be the most important features, using these features reduced the accuracy rating by approximately 4%. This indicates that the algorithm was finding patterns in these features that were not applicable to classifying diagnoses. This situation demonstrated the necessity of human oversight in guiding ML algorithms, despite their predictive capabilities. The Random Forest Classification, XGBoost, and CNN all yielded the same accuracy of 97.37%. These models’ equal metrics may result from the simplicity of the dataset, which is not very large, or it could indicate that the models could still be overfitting. Alternatively, it could imply that all models captured the same underlying patterns in the data. The most effective model for classifying diagnoses from this dataset proved to be the MLP, which achieved an accuracy rating of 98.25%. While these models could be further tested and potentially deployed for diagnosing breast cancer, the CNN may not be the best option due to its complexity and resource use. The analysis of the correlation matrix revealed the feature “concavity_worse” had the greatest correlation with a malignant diagnosis at 0.79, although this feature was not used in the final dataset. After dropping unnecessary features, the “radius_mean” exhibited the highest correlation at 0.73. This finding indicates larger cell nuclei and greater indentations in the cell nucleus correlate with a higher likelihood of a malignant breast cancer diagnosis.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8af316843a8e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced Paternity DNA Sequence Classification Using Dynamic Programming and Machine Learning…]]></title>
            <link>https://ernest-bonat.medium.com/advanced-paternity-dna-sequence-classification-using-dynamic-programming-and-machine-learning-bb84c1470883?source=rss-58ededb6e465------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb84c1470883</guid>
            <dc:creator><![CDATA[Ernest Bonat, Ph.D.]]></dc:creator>
            <pubDate>Thu, 12 Sep 2024 14:27:36 GMT</pubDate>
            <atom:updated>2024-11-15T00:47:46.470Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*fCPWHiXPoY46gv29O3eIVg.png" /></figure><h3><strong>Advanced Paternity DNA Sequence Classification Using Dynamic Programming and Machine Learning Algorithms. Part 1</strong></h3><p><strong>Ernest Bonat, Ph.D., Mohamed Abdulaziz Eisa, B.Sc. AI</strong></p><p>1. <a href="#4de6">Overview</a><br>2. <a href="#cacd">Research Limitations</a><br>3. <a href="#ad60">Machine Learning Challenges</a><br>4. <a href="#f750">Biological Sequence Alignment</a><br>5. <a href="#fad7">Types of Alignment Algorithms</a><br>6. <a href="#ba06">Importance of Biological Sequence Alignment</a><br>7. <a href="#734f">Overview of Pairwise Sequence Alignment Techniques</a><br>8. <a href="#bcfe">Dynamic Programming Algorithm for Sequence Alignment Steps</a><br>9. <a href="#9b91">Scoring Scheme</a><br>10. <a href="#395b">Global Alignments</a><br>11. <a href="#1c6c">Local Alignments</a><br>12. <a href="#a1a5">Table of Comparison of Alignment Methods</a><br>13. <a href="#6267">Paternity DNA Sequence Alignment</a><br>14. <a href="#7f8d">Conclusions</a><br>15. <a href="#c356">Future Work Plan (Part 2)</a></p><h4>1. Overview</h4><p>A paternity test is a type of genetic test that determines whether a man is the biological father of a child. Paternity tests are used to establish a biological relationship between a father and a child. This can be important for legal, medical, or personal reasons. DNA Testing is the most common and accurate method. It involves collecting DNA samples from the alleged father and the child, usually through a cheek swab or blood sample. In this paper, Dynamic Programming and Machine Learning algorithms will be used to for paternity DNA sequence classification.</p><p>The primary aim of the proposed research is to advance the efficiency of sequence alignment algorithms through complete parallelization. This approach targets commonly used algorithms, specifically the Smith-Waterman and Needleman-Wunsch algorithms, leveraging efficient and low-cost hardware and software platforms to address and mitigate the challenges associated with dynamic programming and traditional hardware used in hospitals and laboratories.</p><h4>2. Research Limitations</h4><p>The proposed approach has several limitations:</p><ul><li><strong>Sequence Length Restriction:</strong> The method relies on using equal-length sequences, specifically multiples of four (e.g., 4, 8, 12), which aligns with the four nucleobases in DNA sequences. This restriction does not directly apply to protein sequences, which consist of 20 amino acids.</li><li><strong>Algorithm Flexibility:</strong> Although the proposed method can be adapted to other local or global alignment algorithms such as Gotoh, Miller-Myers, and Hirschberg, its effectiveness is contingent upon the assumption that the parallel software implementation will be more efficient than traditional methods. The approach aims to be faster and less computationally and power-intensive, but this efficiency may vary depending on the specific algorithm and implementation context.</li></ul><p>In our research, we focused on pair-wise sequence alignment. Specifically, we utilized the Smith-Waterman (SW) algorithm for local alignment and the Needleman-Wunsch (NW) algorithm for global alignment. These techniques are designed to align sequences by comparing them in pairs, providing detailed insights into their similarities and differences.</p><h4>3. Machine Learning Challenges</h4><p>We began developing a machine learning-based DNA classifier for paternity testing, which presents several key challenges.</p><ul><li><strong>Large and Long Sequences:</strong> The classifier must handle a large number of DNA sequences, each of which can be very long. This poses significant computational and memory challenges.</li><li><strong>Computational Complexity:</strong> Sequence alignment algorithms used in this context have a time complexity of O(MN), where M and N represent the lengths of the sequences. This complexity can lead to high computational costs and processing times.</li><li><strong>Hardware Limitations:</strong> Traditional hardware implementations of sequence alignment algorithms may not be effective for handling sequential process problems, which can impact system performance and speed.</li></ul><p>Solutions:</p><ul><li><strong>Efficient Data Processing:</strong> Employ data preprocessing techniques such as sequence chunking and compression to manage large datasets and reduce memory usage.</li><li><strong>Optimized Algorithms:</strong> Utilize optimized algorithms and approximations to reduce computational complexity. Techniques such as heuristic methods and algorithmic improvements can help speed up processing.</li><li><strong>Parallel Computing:</strong> Leverage parallel computing and hardware acceleration, such as GPUs, to handle large-scale computations more efficiently and improve processing times.</li></ul><h4>4. Biological Sequence Alignment</h4><p>We can define biological sequence alignment as the process of arranging and comparing DNA, RNA, or protein sequences to identify regions of similarity. These similarities can reveal important functional, structural, or evolutionary relationships between the sequences.</p><p>The purpose of sequence alignment is:</p><ul><li><strong>Similarity Identification:</strong> The primary goal is to detect regions of similarity between sequences. These similarities can indicate shared functions or evolutionary origins.</li><li><strong>Functional and Evolutionary Insights:</strong> Identifying conserved regions can help infer the function of genes or proteins, understand evolutionary relationships, and predict the structure and function of new sequences.</li></ul><h4>5. Types of Alignment Algorithms</h4><p>We will explore the fundamentals and techniques of sequence alignment in bioinformatics, focusing on how to compare and analyze biological sequences to uncover functional, structural, and evolutionary relationships.</p><p><strong>a. Pairwise Sequence Alignment</strong></p><p>This involves aligning two sequences at a time. The aim is to find the best alignment that maximizes the number of matches and minimizes mismatches and gaps.</p><p>· <strong>Local vs. Global Alignment</strong></p><ol><li><strong>Local Alignment:</strong> Focuses on finding the most similar region within two sequences. The Smith-Waterman algorithm is commonly used for local alignment, which is useful when sequences share a common sub-region but differ significantly overall.</li><li><strong>Global Alignment:</strong> Aligns the entire length of two sequences, accounting for possible gaps at the ends. The Needleman-Wunsch algorithm is typically used for global alignment, suitable when comparing sequences of similar length and overall similarity.</li></ol><p>· <strong>Applications:</strong> Used for comparing two sequences to understand their functional similarity or evolutionary relationship. For instance, comparing a gene sequence to a database to identify potential homologous genes.</p><p><strong>b. Multiple Sequence Alignment</strong></p><p>This method aligns three or more sequences simultaneously. The objective is to find a common alignment that maximizes overall similarity across all sequences.</p><p>• <strong>Algorithm Examples:</strong></p><ol><li><strong>ClustalW:</strong> A widely used tool for constructing multiple sequence alignments. It uses progressive alignment techniques to build up the alignment step-by-step.</li><li><strong>MUSCLE:</strong> An algorithm known for its speed and accuracy in generating multiple sequence alignments.</li><li><strong>T-Coffee:</strong> Combines multiple alignment methods to improve accuracy by integrating information from different alignments.</li></ol><ul><li><strong>Applications:</strong> Essential for studying the evolutionary relationships among a group of sequences, identifying conserved motifs, and understanding functional domains shared by related proteins or genes. MSA helps in predicting the secondary and tertiary structures of proteins by aligning homologous sequences.</li></ul><h4>6. Importance of Biological Sequence Alignment</h4><p>In this topic, we will explain the significance and methodologies of sequence alignment in bioinformatics. We will explore how sequence alignment helps in various fields of biological research and practical applications. Specifically, we will cover:</p><ul><li><strong>Evolutionary Studies:</strong> Helps in constructing phylogenetic trees to understand the evolutionary relationships among species or genes.</li><li><strong>Functional Annotation:</strong> Aids in predicting the function of genes or proteins by identifying conserved regions that are crucial for their biological roles.</li><li><strong>Drug Discovery and Development:</strong> Identifying conserved motifs or domains can help in designing drugs that target specific biological functions or interactions.</li><li><strong>Genomic Research:</strong> Useful in genome annotation, identifying gene families, and understanding the genetic basis of diseases.</li></ul><p>By aligning biological sequences, researchers can uncover valuable information about the similarities and differences between sequences, leading to a better understanding of biological functions and evolutionary processes.</p><h4>7. Overview of Pairwise Sequence Alignment Techniques</h4><p>Let’s look at these figures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*sEtSAxxhXAs3guYfvJ_SXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*zeBCx9M69VgJxULaj80M6g.png" /></figure><p><strong>1. Heuristic Programming</strong></p><p>We will explain heuristic programming as techniques used for sequence alignment in bioinformatics. Heuristic methods provide quick and approximate solutions to sequence alignment problems, making them ideal for searching large databases. We will discuss the following tools and their applications:</p><p><strong>1.1 BLAST (Basic Local Alignment Search Tool)</strong></p><ul><li>A widely used heuristic tool for finding regions of local similarity between sequences. BLAST searches databases for sequences that align with a query sequence, providing quick and approximate results.</li><li><strong>Type:</strong> Local Alignment</li><li><strong>Usage:</strong> Ideal for searching large databases to identify homologous sequences or functional similarities.</li></ul><p><strong>1.2 FASTA</strong></p><ul><li>Another heuristic method used to search for local alignments between a query sequence and sequences in a database. FASTA uses a scoring matrix and gap penalties to identify regions of similarity.</li><li><strong>Type:</strong> Local Alignment</li><li><strong>Usage:</strong> Useful for identifying similar sequences in large databases and performing quick alignments.</li></ul><p><strong>1.3 SIM2</strong></p><ul><li>Designed for aligning protein sequences to a database. SIM2 improves alignment accuracy by incorporating additional scoring systems tailored for protein sequences.</li><li><strong>Type:</strong> Local Alignment</li><li><strong>Usage:</strong> Enhances local alignment accuracy, particularly for protein sequences.</li></ul><h4><strong>2. Dynamic Programming</strong></h4><p>We will explore dynamic programming techniques used for sequence alignment in bioinformatics. Dynamic programming provides rigorous and accurate solutions for aligning sequences, making it essential for detailed analysis. We will discuss the following algorithms and their applications:</p><p><strong>2.1 Needleman-Wunsch (NW)</strong></p><ul><li>A dynamic programming algorithm for finding the optimal global alignment between two sequences. It aligns sequences from end to end, accounting for all gaps and mismatches.</li><li><strong>Type:</strong> Global Alignment</li><li><strong>Usage:</strong> Suitable for sequences that are of similar length and overall similarity, providing a comprehensive alignment.</li></ul><p><strong>2.2 Smith-Waterman (SW)</strong></p><ul><li>A dynamic programming algorithm used for finding the optimal local alignment between two sequences. It focuses on aligning the most similar sub-region within the sequences.</li><li><strong>Type:</strong> Local Alignment</li><li><strong>Usage:</strong> Effective for identifying highly similar regions within sequences that may differ significantly in other areas.</li></ul><p><strong>2.3 Hirschberg</strong></p><ul><li>An optimized variant of the Needleman-Wunsch algorithm that reduces memory usage by using a divide-and-conquer approach. It still performs global alignment but with reduced space complexity.</li><li><strong>Type:</strong> Global Alignment (Optimized for space)</li><li><strong>Usage:</strong> Useful when aligning long sequences where memory is a constraint.</li></ul><p><strong>2.4 Gotoh:</strong></p><ul><li><strong>Description:</strong> An enhancement of the Needleman-Wunsch algorithm that includes affine gap penalties, providing a more accurate alignment by accounting for the cost of gaps more effectively.</li><li><strong>Type:</strong> Global Alignment</li><li><strong>Usage:</strong> Improves global alignment accuracy by refining gap penalties.</li></ul><p><strong>2.5 Miller-Myers:</strong></p><ul><li><strong>Description:</strong> An extension of the Needleman-Wunsch algorithm designed to improve alignment efficiency by incorporating additional heuristics for gap penalties and mismatches.</li><li><strong>Type:</strong> Global Alignment</li><li><strong>Usage:</strong> Suitable for aligning sequences with varying lengths and gap penalties.</li></ul><h4><strong>3. Specialized Techniques</strong></h4><p>We will explore specialized techniques used for sequence alignment in bioinformatics. These techniques offer unique methods for visualizing and analyzing sequence alignments, providing valuable insights into the relationships between sequences. We will discuss the following technique and its application:</p><p><strong>3.1 Dot-Matrix Technique</strong></p><ul><li><strong>Description:</strong> A graphical method for visualizing sequence alignments by plotting a matrix where dots represent matching characters. This technique helps in identifying regions of similarity and differences.</li><li><strong>Type:</strong> Global and Local Alignment</li><li><strong>Usage:</strong> Useful for visual inspection and initial exploration of sequence similarities and differences.</li></ul><p><strong>Heuristic Methods (BLAST, FASTA, SIM2):</strong> Provide fast, approximate alignments, suitable for large-scale database searches and quick comparisons.</p><ul><li><strong>Dynamic Programming (Needleman-Wunsch, Smith-Waterman, Hirschberg, Gotoh, Miller-Myers):</strong> Offer accurate alignments with various optimizations for space and gap penalties, suitable for detailed analysis of sequence similarities.</li><li><strong>Specialized Techniques (Dot-Matrix):</strong> Provide visual and exploratory tools for understanding sequence relationships and similarities.</li></ul><h4>8. Dynamic Programming Algorithm for Sequence Alignment Steps</h4><p>As we discussed earlier, dynamic programming in sequence alignment is a technique used to determine the optimal alignment of two or more sequences — such as DNA, RNA, or protein sequences — by breaking the problem into simpler, manageable subproblems. Here’s a simplified explanation:</p><ol><li>Set the reference sequence across the top of the N*M matrix and the read sequence along the side.</li><li>Initialize the first row and first column of the score matrix (values depend on algorithm)</li><li>For each element, derived scores from neighboring above, above left, and left units.</li><li>For each element, compute match and mismatch scores on above-left score and gap score on above and left scores. Choose maximum of computed score as final score.</li><li>Once all elements in matrix are filled, find the highest score, which is where the last base in the alignment occurs.</li></ol><p><strong>General Workflow</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*vN5IZTK1IhtepBo_G61vmw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*vapaugl0-DcKR5ZTygNwbQ.png" /></figure><p>In this section, we will explain the steps involved in sequence alignment using the Smith-Waterman (SW) and Needleman-Wunsch (NW) algorithms.\</p><p><strong>Step 1: The Basic Concept</strong></p><p>Both SW and NW algorithms use a matrix to align two sequences. Let’s call them Seq1 (M) and Seq2 (N). The matrix helps in scoring alignments based on matches, mismatches, and gaps.</p><ul><li>Match: +1 point</li><li>Mismatch: -1 point</li><li>Gap: -2 points</li></ul><p><strong>Step 2: Matrix Initialization</strong></p><p>We start by setting up a matrix where:</p><ul><li>The rows represent one sequence (Seq2 = ATCGT).</li><li>The columns represent the other sequence (Seq1 = TGGGTG).<br><br> Example:</li></ul><pre><br> Seq1: T G G G T G<br> Seq2:<br> A<br> T<br> C<br> G<br> T<br> <br> The first row and column are initialized with gap penalties:<br><br> T G G G T G<br> 0 -2 -4 -6 -8 -10 -12<br> A -2<br> T -4<br> C -6<br> G -8<br> T-10</pre><p><strong>Step 3: Matrix Filling</strong></p><p>For each cell in the matrix, we calculate scores based on possible alignments:</p><ol><li>Diagonal (match/mismatch): Aligns the characters from both sequences.</li><li>Top (gap in Seq1): Aligns a gap in Seq1 with a character in Seq2.</li><li>Left (gap in Seq2): Aligns a character in Seq1 with a gap in Seq2.<br> <br> Formula:</li></ol><pre><br>T[i,j] = max { T[i-1, j-1] + σ(S1[i], S2[j]), T[i-1, j] + gap penalty, T[i, j-1] + gap penalty }</pre><p><strong>Step 4: Traceback and Result Generation</strong></p><p>Once the matrix is filled, we trace back from the highest score to generate the optimal alignment. This process involves following the path of the highest scores backward through the matrix.</p><p><strong>Example Explained</strong></p><p>Let’s align Seq1 (TGGGTG) and Seq2 (ATCGT) using our scoring scheme. Here’s how we fill the matrix step by step.<br> <br> 1. Initialization:</p><ul><li>The first row and column are filled with multiples of the gap penalty (-2).</li></ul><p>2. Filling the Matrix:</p><ul><li>For cell (i=4, j=3) aligning G (from Seq1) with C (from Seq2), we calculate:<br> — Diagonal: T[3,2] — 1 (mismatch)<br> — Top: T[3,3] — 2 (gap)<br> — Left: T[4,2] — 2 (gap)<br> — Result: max(-3, -3, -6) = -3</li></ul><p>3. Traceback:</p><ul><li>After filling the matrix, we find the highest score and trace back to form the optimal alignment.</li></ul><h4>9. Scoring Scheme</h4><p>Let’s explore more and understand the scoring scheme used in sequence alignment. A scoring scheme assigns numerical values to matches, mismatches, and gaps to determine the optimal alignment between sequences.</p><p><strong>Example Scoring Scheme</strong></p><ul><li>Match: +1 point</li><li>Mismatch: -1 point</li><li>Gap: 0 points</li></ul><p><strong>Example Sequences</strong></p><ul><li>Sequence 1: GATTACA</li><li>Sequence 2: GCTAC</li></ul><p><strong>Alignment Example</strong></p><p>GATTACA<br> | | |<br> GCT-AC</p><p><strong>Scoring Details:</strong></p><p>1. G vs G: Match = +1<br> 2. A vs C: Mismatch = -1<br> 3. T vs T: Match = +1<br> 4. T vs A: Mismatch = -1<br> 5. A vs C: Mismatch = -1<br> 6. C vs -: Gap = 0<br> 7. A vs -: Gap = 0</p><p><strong>Total Score Calculation</strong></p><ul><li>Matches: +1 (G-G) + +1 (T-T) = +2</li><li>Mismatches: -1 (A-C) + -1 (T-A) + -1 (A-C) = -3</li><li>Gaps: 0 (C-) + 0 (A-) = 0</li></ul><p>Final Alignment Score: +2 (matches) — 3 (mismatches) + 0 (gaps) = -1</p><p><strong>New Example Alignment</strong></p><ul><li>Sequence A1: A-TGAG</li><li>Query: ATGGCGSequence A2: ATG-AG</li></ul><p><strong>Total Score Calculation</strong></p><ul><li>Final Alignment Score (A1) with Query: +3 (matches) — 2 (mismatches) + 1 (gaps) = +1</li><li>Final Alignment Score (A2) with Query: +4 (matches) — 1 (mismatches) + 1 (gaps) = +3</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*mwFK0NeT1d5wQ2RH92y9MA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*bv4Cp4mT56ZMS0jXfZpXBw.png" /></figure><h4>10. Global Alignments</h4><p><strong>Binary Global Alignment</strong></p><p>In this topic we will discuss in deep Binary Global Alignment is a method used to compare two sequences by aligning them completely from start to finish. It finds the optimal match by considering all possible alignments, including gaps, and scores them based on matches, mismatches, and gaps. This helps in identifying the best overall alignment between the sequences.</p><p><strong>1. Same Sequence</strong>s</p><p>If both the sequences is exactly same then:</p><ul><li>Match Values scoring = 1</li><li>Mismatch value Scoring = 0</li><li>Gap penalty is not considered then let’s show how it act with code.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*Oz22UTjpcvYYnHWGmx4lgQ.png" /></figure><p>It’s the same exactly score if we calculated it 39 out of 39.</p><p><strong>2. Different Sequences</strong></p><p>Now, let us take two different sequence one from one person generation and another from another generation (different states).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*msEogRWkcihCXvkCPkzEYA.png" /></figure><p>Here we can see that score is 21 out of 39.</p><p><strong>Weighted Global Alignment</strong></p><p>In this topic, we will explore Weighted Global Alignment, an advanced method for comparing two sequences. Unlike basic alignment methods, Weighted Global Alignment assigns different weights to matches, mismatches, and gaps, reflecting their varying significance in different contexts. This technique ensures a more precise and tailored alignment by considering the importance of specific alignments. By optimizing the alignment scores with these weights, Weighted Global Alignment provides a more accurate and meaningful comparison between sequences, making it invaluable in various fields such as bioinformatics and computational biology.</p><p><strong>1.Same Sequences</strong></p><p>Here, each and every possible type of scoring is given a weight.</p><ul><li>Match Scoring = 2</li><li>Mismatching Scoring = -1</li><li>Gap Penalty = (open = -0.5 , extend = -0.1)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*UZBigES6uFFJkgsZnslCyw.png" /></figure><p>we get this score (39 * 2 ) = 78 out of 78.</p><p><strong>2. Different Sequences</strong></p><p>Let’s try to test with different sequences and see the results for weighted global alignment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*rx5XeZT1_vikR-HvshB36A.png" /></figure><p>Score = 31.2</p><h4>11. Local Alignments</h4><p><strong>Binary Local Alignment</strong></p><p>we will explain Binary Local Alignment, a method used to find the optimal alignment between the most similar subregions of two sequences. Unlike global alignment, which aligns sequences from end to end, Binary Local Alignment <strong>focuses on aligning the best matching parts</strong>, allowing for more flexibility and accuracy in identifying regions of high similarity. By scoring matches, mismatches, and gaps, this method helps in pinpointing the most relevant and significant local alignments between sequences.</p><p><strong>1. Same Sequences</strong></p><p>If both the sequences is exactly same then</p><ul><li>Match Values scoring = 1</li><li>Mismatch value Scoring = 0</li><li>Gap penalty is not considered then let’s show how it acts with code.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*GW3ASy5uyfh_9z1JHaawOg.png" /></figure><p>It’s the same exactly score if we calculated it 39 out of 39.</p><p><strong>2. Different Sequences</strong></p><p>Let’s try to test with different sequences and see the results for binary local alignment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*xaZFyvl_W4pI8PfJLF0Lbw.png" /></figure><p>Here we can see that score is 21 out of 39.</p><p><strong>Weighted Local Alignment</strong></p><p>We will discuss Weighted Local Alignment, a method used to compare specific parts of two sequences. Unlike global alignment, which aligns sequences from start to finish, Weighted Local Alignment focuses on the most similar sub-regions. It uses different weights for matches, mismatches, and gaps to find the best local alignment. This approach helps identify the most relevant and meaningful similarities between sequences.</p><p><strong>1. Same Sequences</strong></p><p>Here, each and every possible type of scoring is given a weight.</p><ul><li>Match Scoring = 2</li><li>Mismatching Scoring = -1</li><li>Gap Penalty = (open = -0.5 , extend = -0.1)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*rtI4U1kWC-Ks55PzBwhk-w.png" /></figure><p>It’s the same Score = 78 like globalms (weighted global alignment).</p><p><strong>2. Different Sequences</strong></p><p>Let’s try to test with different sequences and see the results for weighted Local alignment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*3D8UdgaBlouvGU92uyzO3Q.png" /></figure><p>Score = 32.2, it’s different score from weighted global with different sequences. Example: for Local Alignment in Dynamic Programming “Smith-Waterman (SW)”.</p><h4>12. Table of Comparison of Alignment Methods</h4><p>The table that local alignment is more adaptable when dealing with different sequences, focusing on the most similar regions to provide better alignment scores. In contrast, global alignment is most effective for aligning sequences that are highly similar across their entire lengths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*rmKk1a4EdR6Y9HnGwhsHlQ.png" /></figure><p><strong>Alignment Type</strong>s</p><p>1. <strong>Global Alignment</strong></p><ul><li><strong>Same Sequences</strong>: Global alignment aligns the entire length of two sequences, maximizing the match across their full lengths. For identical sequences, the binary score is 39/39, and the weighted score is 78/78, indicating perfect alignment.</li><li><strong>Different Sequences</strong>: When aligning different sequences globally, the scores drop, reflecting the mismatches and gaps introduced. Here, the binary score is 21/39, and the weighted score is 31.2/78, showing that while some alignment is possible, it is less effective due to differences in the sequences.</li></ul><p>2. <strong>Local Alignment</strong></p><ul><li><strong>Same Sequences</strong>: Local alignment finds the best matching subsequence within the sequences. For identical sequences, it effectively behaves like global alignment, yielding a binary score of 39/39 and a weighted score of 78/78.</li><li><strong>Different Sequences</strong>: For different sequences, local alignment focuses on regions with the highest similarity, resulting in a better alignment score than global alignment. The binary score here is 21/39, but the weighted score is slightly higher at 32.5/78, indicating more effective alignment of the most similar regions.</li></ul><p><strong>3. Scoring Schemes</strong></p><ul><li><strong>Binary Score</strong>: Measures the number of aligned positions, indicating how many matches occurred.</li><li><strong>Weighted Score</strong>: Considers the quality of matches, mismatches, and gaps, providing a more nuanced view of alignment effectiveness.</li></ul><h4><strong>13. Paternity DNA Sequence Alignment</strong></h4><ul><li>In the context of paternity testing, local alignment is generally preferred due to its suitability for identifying specific regions or markers used to establish genetic relationships between individuals. This method allows for a focus on the most relevant portions of DNA sequences, such as short tandem repeats (STRs) or other genetic markers commonly utilized in such tests.</li><li>However, our case involves a different approach. Given that our data comprises very large sequences, specifically around 2000 letters of single nucleotide polymorphism (SNPs) rather than markers or STRs, local alignment is not the optimal method. Instead, we require a technique suitable for comparing entire sequences due to the expected high similarity across the full length of these sequences, akin to whole genome comparison.</li><li>Therefore, our alignment strategy must accommodate the extensive nature and high similarity of the SNP sequences to accurately reflect the genetic relationships in our dataset.</li></ul><h4><strong>14. Conclusions</strong></h4><p>1. The research focused on improving the efficiency of DNA sequence alignment algorithms, specifically the Smith-Waterman and Needleman-Wunsch algorithms, through parallelization.</p><p>2. Addressed computational and memory challenges associated with large DNA sequences, emphasizing the need for optimized algorithms and advanced hardware.</p><p>3. Employed dynamic programming to manage sequence alignment, showcasing its effectiveness in identifying similarities between sequences for paternity testing.</p><p>4. Weighted Global Alignment: This method was chosen for its ability to provide more accurate alignments by assigning different weights to matches, mismatches, and gaps, thus better handling the variability and complexity of DNA sequences.</p><p>Importance of Weighted Global Alignment:</p><ul><li>Accuracy: By assigning specific weights, the algorithm can more accurately reflect biological realities, leading to more precise alignments.</li><li>Flexibility: Weighted global alignment can be adapted to different types of sequences and alignment needs, making it versatile for various applications.</li><li>Performance: Optimization and parallelization of this method improve computational efficiency, making it suitable for large-scale DNA data processing.</li></ul><h4><strong>15. Future Work Plan (Part 2)</strong></h4><ol><li>Code Implementation</li></ol><ul><li>Develop Algorithms: Implement the weighted global alignment algorithm in a programming language suitable for handling large data sets, such as Python.</li><li>Optimize for Performance: Utilize libraries and frameworks that support parallel computing, such as CUDA for GPU acceleration.</li></ul><p>2. Preparing Data</p><ul><li>Data Collection: Gather a diverse set of DNA sequences to ensure the robustness of the algorithm.</li><li>Preprocessing: Implement techniques such as sequence chunking and compression to manage memory usage and improve processing speed.</li></ul><p>3. Research Notebooks</p><ul><li>Documentation: Create detailed research notebooks documenting the implementation, experiments, and results.</li><li>Visualizations: Include visualizations of sequence alignments and performance metrics to illustrate the effectiveness of the algorithm.</li></ul><p>4. Deployment and Testing</p><ul><li>Real-World Data Testing: Apply the algorithm to real-world DNA data to validate its accuracy and efficiency.</li><li>Continuous Improvement: Iterate on the algorithm based on test results, optimizing for accuracy and performance.</li></ul><p>5. Final Presentation</p><ul><li>Compile Results: Summarize findings, methods, and results in a comprehensive report.</li><li>Demonstration: Prepare a live demonstration of the algorithm processing DNA sequences, showcasing its practical application in paternity testing and genomic research.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb84c1470883" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>