<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Youssefshebl on Medium]]></title>
        <description><![CDATA[Stories by Youssefshebl on Medium]]></description>
        <link>https://medium.com/@youssefshibl000?source=rss-ba2081e78814------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*02B-4X2KPI-lnT_P</url>
            <title>Stories by Youssefshebl on Medium</title>
            <link>https://medium.com/@youssefshibl000?source=rss-ba2081e78814------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:40:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@youssefshibl000/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ PostgreSQL Transaction IDs (XIDs) Explained]]></title>
            <link>https://medium.com/@youssefshibl000/postgresql-transaction-ids-xids-explained-ca655f0e79f9?source=rss-ba2081e78814------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca655f0e79f9</guid>
            <category><![CDATA[concurency]]></category>
            <category><![CDATA[mvcc]]></category>
            <category><![CDATA[transactions]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Youssefshebl]]></dc:creator>
            <pubDate>Sat, 20 Sep 2025 23:01:38 GMT</pubDate>
            <atom:updated>2025-09-20T23:52:03.376Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SzlNukTeu2KMnNLzNZYHJA.png" /></figure><h3>📌 Introduction</h3><p>PostgreSQL uses <strong>Multi-Version Concurrency Control (MVCC)</strong> to handle multiple transactions simultaneously without locking conflicts. At the heart of MVCC is the <strong>Transaction ID (XID)</strong>.</p><p>Each transaction is assigned a <strong>unique integer ID (XID)</strong>. These IDs are stored in every row to determine <strong>whether a transaction can see that row version or not</strong>.</p><h4>🔹 What is an XID?</h4><ul><li><strong>XID</strong> = <strong>Transaction ID</strong></li><li>A sequential number assigned to each transaction</li><li>Works like a <strong>logical timestamp</strong> for visibility decisions</li><li>Stored in hidden system columns inside every row:</li><li>xmin → XID of the transaction that created the row</li><li>xmax → XID of the transaction that deleted (or updated) the row</li></ul><h4>🔹 Step-by-Step Example Walkthrough</h4><h4>1. Create a table</h4><pre>CREATE TABLE accounts (<br>    id SERIAL PRIMARY KEY,<br>    balance INT<br>);</pre><h4>2. Insert a row</h4><pre>BEGIN;<br>INSERT INTO accounts (balance) VALUES (100);<br>SELECT txid_current();  -- suppose result = 1001<br>COMMIT;</pre><p><strong>Row metadata after insert:</strong></p><pre>xmin = 1001<br>xmax = 0  (still valid, not deleted)</pre><h4>3. Update the row</h4><pre>BEGIN;<br>UPDATE accounts SET balance = 200 WHERE id = 1;<br>SELECT txid_current();  -- suppose result = 1002<br>COMMIT;</pre><p>Now PostgreSQL creates <strong>two versions</strong> of the row:</p><ul><li><strong>Old version:</strong> balance = 100, xmin = 1001, xmax = 1002</li><li><strong>New version:</strong> balance = 200, xmin = 1002, xmax = 0</li></ul><h4>4. Inspect tuple XIDs</h4><p>You can see row metadata with:</p><pre>SELECT xmin, xmax, ctid, * FROM accounts;</pre><p><strong>Example output:</strong></p><pre>xmin | xmax | ctid  | id | balance<br>------+------+-------+----+---------<br> 1002 |    0 | (0,2) |  1 |     200</pre><h4>🔹 How PostgreSQL Uses XIDs for Visibility</h4><p>When a query runs, PostgreSQL takes a <strong>snapshot</strong> of active and committed XIDs:</p><ul><li>Rows with xmin <strong>greater than current XID</strong> → <strong>invisible</strong> (not created yet)</li><li>Rows with xmax set by a <strong>committed</strong> transaction → <strong>invisible</strong> (already deleted/updated)</li><li>Rows with xmax from an <strong>uncommitted</strong> transaction → <strong>visible to some transactions, invisible to others</strong></li></ul><p>This ensures <strong>readers don’t block writers</strong> and vice versa.</p><h4>Visibility Rules Example</h4><pre>-- Transaction 1003 starts<br>BEGIN; -- XID = 1003<br>SELECT * FROM accounts; -- sees balance = 200 (xmin=1002 &lt; 1003)<br><br>-- Meanwhile, Transaction 1004 starts<br>BEGIN; -- XID = 1004<br>UPDATE accounts SET balance = 300 WHERE id = 1;<br>-- Creates: balance = 300, xmin = 1004, xmax = 0<br>-- Updates old: balance = 200, xmin = 1002, xmax = 1004<br><br>-- Transaction 1003 still sees balance = 200<br>-- because xmax = 1004 is from an uncommitted transaction<br>SELECT * FROM accounts; -- still sees balance = 200<br><br>COMMIT; -- Transaction 1004 commits<br><br>-- Now Transaction 1003 sees balance = 300<br>-- because xmax = 1004 is now from a committed transaction<br>SELECT * FROM accounts; -- now sees balance = 300</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4oZ739GIdameBL2bRRvO_Q.png" /></figure><h4>🔹 XID Wraparound Challenge</h4><ul><li>XIDs are <strong>32-bit integers</strong> (≈ 4.2 billion values)</li><li>They wrap around after reaching the limit</li><li>To prevent data corruption, PostgreSQL uses <strong>autovacuum</strong> to “freeze” old tuples</li></ul><h4>Freezing Process</h4><p>The <strong>Freezing Process</strong> is PostgreSQL’s mechanism to prevent transaction ID wraparound by permanently marking old tuples as “<strong>always visible.</strong>”</p><blockquote>Wraparound in general means when a fixed-size counter reaches its maximum value and starts again from zero.<br> This happens in many systems like clocks, timers, or sequence numbers.<br> In PostgreSQL, every transaction gets a 32-bit transaction ID (XID).<br> After about 4 billion transactions, the XID counter overflows and restarts at zero , If old rows aren’t “frozen” by VACUUM, they may appear new after wraparound, which risks data corruption.</blockquote><h3>📌 How PostgreSQL XID Wraparound Happens?</h3><h4>The Problem: Running Out of Numbers</h4><p>PostgreSQL uses 32-bit integers for transaction IDs (XIDs), giving us about 4.3 billion possible values. But here’s the catch: PostgreSQL treats XIDs as a circular timeline, and you can only safely use about 2 billion before hitting the wraparound danger zone.</p><p>Simple Example: The Wraparound Process</p><p>Step 1: Normal Operations</p><pre>-- Your database after 3 years of heavy use<br>Current XID: 2,000,000,000<br>Old data XID: 500,000,000<br>-- Visibility check (normal):<br>Is 500,000,000 &lt; 2,000,000,000? → YES<br>Result: Old data is visible ✅</pre><p>Step 2: Approaching the Limit</p><pre>-- Getting close to 32-bit integer limit<br>Current XID: 2,147,483,647  -- Maximum 32-bit signed integer<br>Old data XID: 500,000,000<br>-- Still works:<br>Is 500,000,000 &lt; 2,147,483,647? → YES<br>Result: Old data is visible ✅</pre><p>Step 3: The Wraparound Moment</p><pre>-- Next transaction wraps around<br>Current XID: 2,147,483,647 + 1 = -2,147,483,648<br>-- PostgreSQL converts this to: 0, then 1, 2, 3...<br>Current XID: 50 (after wraparound)<br>Old data XID: 500,000,000<br>-- The problem:<br>Is 500,000,000 &lt; 50? → NO!<br>Result: Old data appears &quot;future&quot; → INVISIBLE ❌</pre><h4>Real-World Impact</h4><p>What Happens to Your Data</p><pre>-- Before wraparound - Customer orders visible<br>SELECT * FROM orders WHERE customer_id = 123;<br>-- Returns: 25 orders<br>-- After wraparound - Same query<br>SELECT * FROM orders WHERE customer_id = 123;<br>-- Returns: 0 orders (data still exists but invisible!)</pre><p>The Data Timeline</p><pre>Past ←------------------------→ Future<br>     Old XIDs    Current XID<br>BEFORE: [500M] ←----------→ [2.1B] ✅ Normal<br>AFTER:  [500M] ←----------→ [50]   ❌ Old data appears &quot;future&quot;</pre><p>PostgreSQL’s Protection</p><p>When your database reaches ~2 billion transactions, PostgreSQL protects you by shutting down:</p><pre>ERROR: database is not accepting commands to avoid wraparound data loss<br>HINT: Stop the postmaster and vacuum that database in single-user mode.</pre><p>The Fix: VACUUM FREEZE</p><p>VACUUM FREEZE solves wraparound by “freezing” old transaction IDs:</p><p>Before VACUUM FREEZE</p><pre>Order 1: XID = 500,000,000  -- Appears &quot;future&quot; after wraparound<br>Order 2: XID = 750,000,000  -- Appears &quot;future&quot; after wraparound<br>Order 3: XID = 45           -- Recent, visible</pre><p>After VACUUM FREEZE</p><pre>Order 1: XID = 2 (FROZEN)   -- Always visible<br>Order 2: XID = 2 (FROZEN)   -- Always visible  <br>Order 3: XID = 45           -- Still visible</pre><p>Prevention: Monitor Your XID Age</p><p>Check Your Database Age</p><pre>SELECT datname, age(datfrozenxid) as xid_age<br>FROM pg_database <br>WHERE datname = current_database();</pre><p>Interpret the Results</p><ul><li><strong>&lt; 100M</strong>: Healthy ✅</li><li><strong>100M — 500M</strong>: Monitor 👀</li><li><strong>500M — 1B</strong>: Schedule vacuum 📅</li><li><strong>1B — 1.5B</strong>: Urgent vacuum needed ⚠️</li><li><strong>&gt; 1.5B</strong>: Critical — immediate action 🚨</li></ul><p>Regular Maintenance</p><pre>-- Run this monthly to prevent wraparound<br>VACUUM FREEZE;<br>-- Or for specific tables<br>VACUUM FREEZE your_busy_table;</pre><h4>Key Takeaway</h4><p>XID wraparound doesn’t delete your data — it makes old data temporarily invisible because PostgreSQL thinks it’s “from the future.” Regular VACUUM FREEZE operations prevent this by resetting old transaction IDs to a special “frozen” state that’s always visible.</p><h3>📌 When Freezing Occurs ?</h3><p>PostgreSQL’s transaction ID freezing mechanism operates through a two-stage evaluation process that determines when tables require maintenance and which individual tuples within those tables should be frozen.</p><h4>Stage 1: Table-Level Age Assessment</h4><h4>The age(relfrozenxid) Calculation</h4><p>The freezing process begins with PostgreSQL calculating the table’s transaction age using the formula:</p><pre>age(relfrozenxid) = currentXID - relfrozenxid</pre><h4>Understanding relfrozenxid</h4><p>The relfrozenxid is a metadata field stored in the pg_class system catalog that represents the <strong>oldest unfrozen transaction ID remaining in the table</strong>. This value serves as a checkpoint, indicating that all transactions with XIDs older than relfrozenxid have already been frozen (converted to FrozenTransactionId = 2).</p><p><strong>Key Properties of relfrozenxid:</strong></p><ul><li>Initially set to the table’s creation XID</li><li>Updated after each successful freezing operation</li><li>Points to the oldest transaction ID that still requires evaluation</li><li>Acts as an optimization boundary for vacuum operations</li></ul><h4>Table-Level Threshold Evaluation</h4><p>When the calculated age exceeds specific thresholds, PostgreSQL triggers different levels of vacuum operations:</p><pre>IF age(relfrozenxid) &gt; autovacuum_freeze_max_age/2 (typically 100,000,000) THEN<br>    -- Schedule routine autovacuum<br>    initiate_autovacuum()<br>END IF<br><br>IF age(relfrozenxid) &gt;= autovacuum_freeze_max_age (typically 200,000,000) THEN<br>    -- Force emergency autovacuum<br>    initiate_emergency_autovacuum()<br>END IF</pre><h4>Stage 2: Tuple-Level Evaluation</h4><p>Once a vacuum operation begins, PostgreSQL examines each individual tuple within the table to determine freezing eligibility.</p><h4>Individual Tuple Age Calculation</h4><p>For each tuple encountered during vacuum, the system calculates:</p><pre>tuple_age = currentXID - tuple_xmin</pre><p>Where tuple_xmin is the transaction ID that created the specific tuple.</p><h4>Tuple Freezing Threshold</h4><p>The tuple becomes eligible for freezing when:</p><pre>IF tuple_age &gt; vacuum_freeze_min_age (default: 50,000,000) THEN<br>    -- Convert tuple&#39;s XID to FrozenTransactionId (2)<br>    freeze_tuple()<br>END IF</pre><h4>Comprehensive Example</h4><h4>Initial Database State</h4><p>Consider the following scenario:</p><pre>Current Database Transaction ID: 150,000,000<br>Table: customer_orders<br>relfrozenxid: 80,000,000<br>vacuum_freeze_min_age: 50,000,000<br>autovacuum_freeze_max_age: 200,000,000</pre><h4>Stage 1: Table Age Assessment</h4><pre>age(relfrozenxid) = 150,000,000 - 80,000,000 = 70,000,000<br>Evaluation:<br>- Is 70,000,000 &gt; 100,000,000 (autovacuum threshold)? → NO<br>- Is 70,000,000 &gt; 200,000,000 (emergency threshold)? → NO<br>Result: No automatic vacuum triggered yet</pre><h4>Scenario: Time Progression</h4><p>After additional database activity:</p><pre>Current Database Transaction ID: 185,000,000<br>relfrozenxid: 80,000,000 (unchanged)<br>age(relfrozenxid) = 185,000,000 - 80,000,000 = 105,000,000<br>Evaluation:<br>- Is 105,000,000 &gt; 100,000,000? → YES<br>- Trigger: Routine autovacuum scheduled<br>Result: Autovacuum process initiated</pre><h4>Stage 2: Tuple-by-Tuple Evaluation</h4><p>During the autovacuum process, each tuple is examined:</p><pre>Table Contents:<br>Tuple A: xmin = 75,000,000, Data = &quot;Order #1001&quot;<br>Tuple B: xmin = 90,000,000, Data = &quot;Order #1002&quot;  <br>Tuple C: xmin = 140,000,000, Data = &quot;Order #1003&quot;<br>Tuple D: xmin = 175,000,000, Data = &quot;Order #1004&quot;<br>Current XID: 185,000,000<br>vacuum_freeze_min_age: 50,000,000</pre><p>Tuple A Evaluation:</p><pre>tuple_age = 185,000,000 - 75,000,000 = 110,000,000<br>Is 110,000,000 &gt; 50,000,000? → YES<br>Action: FREEZE (convert xmin from 75,000,000 to 2)</pre><p>Tuple B Evaluation:</p><pre>tuple_age = 185,000,000 - 90,000,000 = 95,000,000  <br>Is 95,000,000 &gt; 50,000,000? → YES<br>Action: FREEZE (convert xmin from 90,000,000 to 2)</pre><p>Tuple C Evaluation:</p><pre>tuple_age = 185,000,000 - 140,000,000 = 45,000,000<br>Is 45,000,000 &gt; 50,000,000? → NO<br>Action: RETAIN (keep xmin = 140,000,000)</pre><p>Tuple D Evaluation:</p><pre>tuple_age = 185,000,000 - 175,000,000 = 10,000,000<br>Is 10,000,000 &gt; 50,000,000? → NO  <br>Action: RETAIN (keep xmin = 175,000,000)</pre><h4>Post-Vacuum State</h4><p>After the vacuum operation completes:</p><pre>Table Contents:<br>Tuple A: xmin = 2 (FROZEN), Data = &quot;Order #1001&quot;<br>Tuple B: xmin = 2 (FROZEN), Data = &quot;Order #1002&quot;<br>Tuple C: xmin = 140,000,000, Data = &quot;Order #1003&quot; ← Oldest unfrozen<br>Tuple D: xmin = 175,000,000, Data = &quot;Order #1004&quot;<br>Updated Metadata:<br>relfrozenxid: 140,000,000 ← Updated to oldest unfrozen XID<br>age(relfrozenxid): 185,000,000 - 140,000,000 = 45,000,000<br>Result: Table age significantly reduced, vacuum cycle complete</pre><h4>Critical Thresholds Summary</h4><p>Table-Level Thresholds (based on age(relfrozenxid)):</p><ul><li><strong>100,000,000</strong>: Routine autovacuum scheduling threshold</li><li><strong>200,000,000</strong>: Emergency autovacuum forced execution</li></ul><p>Tuple-Level Threshold (based on individual tuple age):</p><ul><li><strong>50,000,000</strong>: Individual tuple freezing threshold (vacuum_freeze_min_age)</li></ul><h4>Process Flow Diagram</h4><pre>Database Activity → XID Counter Increases<br>                          ↓<br>Calculate: age(relfrozenxid) = currentXID - relfrozenxid<br>                          ↓<br>        age(relfrozenxid) &gt; threshold?<br>                    ↓               ↓<br>                  YES              NO<br>                    ↓               ↓<br>            Start VACUUM    Continue Normal Operations<br>                    ↓<br>        For Each Tuple in Table:<br>        Calculate: tuple_age = currentXID - tuple_xmin<br>                    ↓<br>        tuple_age &gt; vacuum_freeze_min_age?<br>                    ↓               ↓<br>                  YES              NO<br>                    ↓               ↓<br>            FREEZE Tuple        RETAIN Tuple<br>                    ↓               ↓<br>            xmin → 2        Keep original xmin<br>                    ↓<br>        Update relfrozenxid to oldest unfrozen XID<br>                    ↓<br>                Complete</pre><h4>🔹 Monitoring XIDs</h4><p>Useful Queries with Example Outputs</p><p>1. Current Transaction ID</p><pre>SELECT txid_current();</pre><p>Example Output:</p><pre>txid_current <br>--------------<br>      1847293</pre><p>2. Current Snapshot Info</p><pre>SELECT txid_current_snapshot();</pre><p>Example Output:</p><pre>txid_current_snapshot <br>-----------------------<br> 1000:1003:1001,1002</pre><p><em>Format: </em><em>xmin:xmax:xip_list where:</em></p><ul><li>xmin = 1000 (earliest XID still active when snapshot was taken)</li><li>xmax = 1003 (first unassigned XID at snapshot time)</li><li>xip_list = 1001,1002 (active XIDs between xmin and xmax)</li></ul><p>This means:</p><ul><li>Transactions before 1000 are committed and visible</li><li>Transaction 1003 is the next ID to be assigned</li><li>Transactions 1001 and 1002 are currently running</li></ul><p>3. Check the Table’s XID Age</p><pre>SELECT schemaname, tablename, <br>       age(relfrozenxid) as xid_age,<br>       n_tup_ins, n_tup_upd, n_tup_del<br>FROM pg_stat_user_tables <br>JOIN pg_class ON pg_stat_user_tables.relid = pg_class.oid;</pre><p>Example Output:</p><pre>schemaname |   tablename    | xid_age  | n_tup_ins | n_tup_upd | n_tup_del<br>------------+----------------+----------+-----------+-----------+-----------<br> public     | accounts       |   847293 |      1250 |       890 |        45<br> public     | transactions   |   234567 |     15600 |      2340 |       120<br> public     | users          |  1500000 |      5000 |       340 |        23<br> public     | audit_log      | 45000000 |   2500000 |         0 |         0</pre><p>4. Monitor XID Wraparound Distance</p><pre>SELECT datname, age(datfrozenxid), <br>       2^31 - age(datfrozenxid) AS xids_until_wraparound<br>FROM pg_database</pre><p>Example Output:</p><pre>datname   |    age     | xids_until_wraparound<br>-------------+------------+----------------------<br> postgres    |   45000000 |            2102447648<br> myapp_prod  |  150000000 |            1997447648<br> myapp_test  |   12000000 |            2135447648<br> template1   |        500 |            2147483148<br> template0   |        500 |            2147483148</pre><h4>Critical XID Thresholds (with Examples)</h4><pre>| XID Age     | Status     | Action Required        | Example Database        |<br>|-------------|------------|------------------------|-------------------------|<br>| &lt; 200M      | ✅ Safe    | Normal operation       | `myapp_test` (12M age)  |<br>| 200M - 1B   | ⚠️ Caution | Monitor autovacuum     | `myapp_prod` (150M age) |<br>| 1B - 2B     | 🚨 Warning | Force vacuum freeze    | Database at 1.2B age    |<br>| &gt; 2B        | 💀 Critical| Database shutdown risk | Only ~100M XIDs left    |</pre><p><strong>Reading the Examples Above:</strong></p><ul><li><strong>myapp_test</strong>: 12M XID age → 2.1B XIDs until wraparound = Very safe</li><li><strong>myapp_prod</strong>: 150M XID age → 1.9B XIDs until wraparound = Caution zone</li><li><strong>postgres</strong>: 45M XID age → 2.1B XIDs until wraparound = Safe</li></ul><p><strong>When to Act:</strong></p><pre>-- Find databases needing attention<br>SELECT datname, age(datfrozenxid),<br>       CASE <br>         WHEN age(datfrozenxid) &gt; 2000000000 THEN &#39;💀 CRITICAL - Shutdown Risk&#39;<br>         WHEN age(datfrozenxid) &gt; 1000000000 THEN &#39;🚨 WARNING - Force Vacuum&#39;<br>         WHEN age(datfrozenxid) &gt; 200000000  THEN &#39;⚠️  CAUTION - Monitor&#39;<br>         ELSE &#39;✅ SAFE&#39;<br>       END as status<br>FROM pg_database<br>WHERE datname NOT IN (&#39;template0&#39;, &#39;template1&#39;)<br>ORDER BY age(datfrozenxid) DESC;</pre><p><strong>Example Output:</strong></p><pre>datname   |    age     |           status<br>-------------+------------+---------------------------<br> old_archive | 1200000000 | 🚨 WARNING - Force Vacuum<br> myapp_prod  |  150000000 | ⚠️  CAUTION - Monitor  <br> myapp_test  |   12000000 | ✅ SAFE</pre><p>XID Age Status Action Required &lt; 200M Safe Normal operation 200M — 1B Caution Monitor autovacuum 1B — 2B Warning Force vacuum freeze &gt; 2B Critical Database shutdown risk</p><h4>🔹 Best Practices</h4><h4>DO ✅</h4><ul><li>Keep autovacuum enabled (default)</li><li>Monitor XID age regularly</li><li>Use appropriate autovacuum_freeze_max_age settings</li><li>Run manual VACUUM FREEZE on heavily updated tables</li></ul><h4>DON’T ❌</h4><ul><li>Disable autovacuum without proper monitoring</li><li>Run extremely long transactions (blocks XID advancement)</li><li>Ignore XID wraparound warnings in logs</li></ul><h4>🔹 Real-World Example: Bank Transfer</h4><pre>-- Account balances table<br>CREATE TABLE account_balances (<br>    account_id INT PRIMARY KEY,<br>    balance DECIMAL(10,2),<br>    last_updated TIMESTAMP DEFAULT NOW()<br>);<br>INSERT INTO account_balances VALUES (1, 1000.00), (2, 500.00);<br>-- Transaction 1: Transfer $200 from account 1 to account 2<br>BEGIN; -- XID = 2001<br>UPDATE account_balances SET balance = balance - 200 WHERE account_id = 1;<br>UPDATE account_balances SET balance = balance + 200 WHERE account_id = 2;<br>-- Check XIDs during transaction<br>SELECT xmin, xmax, ctid, * FROM account_balances;<br>/*<br> xmin | xmax | ctid  | account_id | balance<br>------+------+-------+------------+---------<br> 2001 |    0 | (0,2) |          1 |  800.00<br> 2001 |    0 | (0,3) |          2 |  700.00<br>*/<br>COMMIT;<br>-- Other concurrent transactions see consistent data<br>-- based on their snapshot and XID visibility rules</pre><h4>DeepDive Diagram</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cRau8xvxQhZmz445hENIvQ.png" /></figure><h4>✅ Summary</h4><ul><li><strong>Every transaction gets a unique XID</strong></li><li><strong>Rows track </strong><strong>xmin and </strong><strong>xmax</strong> to record which transaction created or deleted them</li><li><strong>Snapshots + XIDs</strong> determine row visibility → this is how MVCC works in PostgreSQL</li><li><strong>Autovacuum prevents wraparound issues</strong> by freezing old XIDs</li><li><strong>Monitor XID age</strong> to prevent wraparound problems</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca655f0e79f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MongoDB to ClickHouse Data Pipeline]]></title>
            <link>https://medium.com/@youssefshibl000/mongodb-to-clickhouse-data-pipeline-30652d83ef80?source=rss-ba2081e78814------2</link>
            <guid isPermaLink="false">https://medium.com/p/30652d83ef80</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[clickhouse]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Youssefshebl]]></dc:creator>
            <pubDate>Sun, 02 Feb 2025 13:14:14 GMT</pubDate>
            <atom:updated>2025-02-02T13:14:14.938Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5MmWUsalwrqnn-U1PfRQng.png" /></figure><p>Hi Folk, in this article I will discuss how we solved some problems in a side project in my company <a href="https://storexweb.com/">StorexWeb</a> by creating a data pipeline between all MongoDB instances for all microservices and ClickHouse</p><p>In our side project which is based on Microservice architecture, every service has its own MongoDB database instance and we require some metrics like statistics or smaller of it, the main problem is that this data should use aggregating &amp; joining between collections (tables) and DBMS which support OLAP query like ClickHouse.</p><p>So the regular solution is making this join in microservice level between all services for Example if we want to join two collections or two types of data in two different services (A, B), service A should CAll service B to get all data related to joining and aggregating and this has many issues</p><h4>1. Performance Issues</h4><ul><li><strong>Latency</strong>: Service A must wait for Service B to respond with potentially large datasets. This can introduce significant latency, especially if the volume of data is high.</li><li><strong>Network Overhead</strong>: Transferring large amounts of data between services over the network can cause bottlenecks and degrade overall system performance.</li></ul><h4>2. Scalability Problems</h4><ul><li><strong>Service Dependency</strong>: Service A’s performance becomes tightly coupled to Service B’s ability to handle large queries or data processing tasks.</li><li><strong>Increased Load</strong>: Service B might become overloaded with requests for data, impacting its ability to serve other clients or processes.</li><li><strong>Limited Horizontal Scalability</strong>: As data grows, the approach may not scale well, since each service is handling part of the join/aggregation logic.</li></ul><h4>3. Complexity in Implementation</h4><ul><li><strong>Code Duplication</strong>: Logic for joining and aggregating data might need to be duplicated or maintained across services, increasing the maintenance burden.</li><li><strong>Error Handling</strong>: Handling partial failures (e.g., Service B is down) adds complexity to Service A, which needs fallback mechanisms or retries.</li></ul><h4>4. Operational Problems</h4><ul><li><strong>Monitoring and Debugging</strong>: Identifying performance issues or bugs across multiple services can be challenging and requires comprehensive monitoring tools.</li><li><strong>Tight Coupling</strong>: Service A becomes tightly coupled with Service B, violating microservices’ independence and autonomy principles.</li></ul><p>So we can solve that by using “<strong>Centralized Data Warehousing</strong>” with “<strong>Event-Driven Architecture</strong>” &amp; “<strong>Stream Processing</strong>”</p><ul><li><strong>Centralized Data Warehousing</strong>: For analytical queries, use a centralized OLAP system (like ClickHouse, BigQuery, or Snowflake), where data from multiple services is periodically aggregated and stored.</li><li><strong>Event-Driven Architecture</strong>: Use an event-driven approach to update a central store with pre-aggregated data whenever Service A or B changes occur.</li><li><strong>Stream Processing</strong>: Use stream-processing tools (e.g., Apache Kafka, Apache Flink) to handle real-time data joining and aggregation outside the services.</li></ul><p>So, let us talk about stage by stage in a pipeline</p><h4>1- from Mongodb To Kafka (1)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zkXR5erXsxJW0zf-yOth5A.png" /><figcaption>(1)</figcaption></figure><p>To transfer data from MongoDB to ClickHouse, we should apply an <strong>Event-Driven Architecture</strong>. To achieve this, we can use a stream-processing tool like <strong>Kafka</strong>. Specifically, we need to listen to MongoDB’s operations logs, which are available when MongoDB is running in replica set mode, to achieve that we need CDC (Change Data Capture) which listens to any change in the MongoDB database and sends this change event to Kafka or make any other action to this data change event so we choose “Debezium” as CDC tool (2)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MuL69tU_kvbR0-9C.png" /><figcaption>(2)</figcaption></figure><p>So, after we add the CDC tool the pipeline will be there (3)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JOLL-f6G-7mAgXCaRaPntQ.png" /><figcaption>(3)</figcaption></figure><p>Let&#39;s break this into three pieces, configure MongoDB in replica set mode, configure debezium connector, configure Kafka</p><h4>MongoDB replica set mode</h4><p>to get all changes that are happening in the database like ( insert, update, and delete), we should enable <strong>Oplog (Operations Log)</strong></p><blockquote>In MongoDB, the <strong>oplog</strong> (operations log) is a special capped collection (local.oplog.rs) that records all the write operations (inserts, updates, deletes) that occur on a primary node in a <strong>replica set</strong>.</blockquote><blockquote>The oplog is used to ensure data consistency and replication within the replica set. Secondary nodes continuously read and apply the operations from the oplog to synchronize with the primary node.</blockquote><p>Here’s an example of an <strong>insert operation</strong> in the oplog for a document being added to the user collection in the test database:</p><pre>{<br>  &quot;ts&quot;: Timestamp(1672531200, 1),  // Operation timestamp<br>  &quot;t&quot;: NumberLong(2),             // Term (replica set election information)<br>  &quot;h&quot;: NumberLong(&quot;91872391012398291&quot;),  // Hash of the operation<br>  &quot;v&quot;: 2,                         // Oplog version<br>  &quot;op&quot;: &quot;i&quot;,                      // Type of operation (insert)<br>  &quot;ns&quot;: &quot;test.user&quot;,              // Namespace (test database, user collection)<br>  &quot;o&quot;: {                          // Document being inserted<br>    &quot;_id&quot;: ObjectId(&quot;64f5e7dc8b0c4d0e8424eae8&quot;),<br>    &quot;username&quot;: &quot;john_doe&quot;,<br>    &quot;email&quot;: &quot;john.doe@example.com&quot;,<br>    &quot;age&quot;: 30<br>  }<br>}</pre><p>to enable the replica set mode in MongoDB we should start with the config file “mongo/mongod.conf”</p><pre>security:<br>  authorization: enabled<br>  keyFile: /opt/keyfile/mongo-keyfile<br><br># Replica Set configuration<br>replication:<br>  replSetName: rs0</pre><blockquote>A <strong>keyfile</strong> is a special file that contains a shared secret used by the members of a replica set for internal authentication, it ensures that only trusted nodes can join the replica set and replicate data,the keyfile is a plain-text file with the shared secret (usually random alphanumeric characters). The file must have strict permissions (read/write only by the MongoDB user).</blockquote><p>we can make a small bash script to generate key file</p><pre>#!/bin/bash<br>openssl rand -base64 756 &gt; /opt/keyfile/mongo-keyfile<br>chmod 400 /opt/keyfile/mongo-keyfile<br>chown mongodb:mongodb /opt/keyfile/mongo-keyfile</pre><p>after the MongoDB instance runs successfully we should open the Mongo shell and initialize the replica set</p><pre>rs.initiate({<br>        _id: &#39;rs0&#39;,<br>        members: [<br>            { _id: 0, host: &#39;localhost:27017&#39; }<br>        ]<br>    });</pre><p>This command initializes a MongoDB replica set named rs0 with a single member. The member is the MongoDB instance running on localhost:27017 and is assigned an ID of 0. The replica set enables replication features but requires additional nodes for full fault tolerance. It’s commonly used for testing or <strong>enabling features like Change Streams</strong></p><h4><strong>Debezium Connector</strong></h4><p>Now, the MongoDB instance is ready to listen in <strong>oplog</strong> of it, let us now make a debezium connector that gets data events and pushes them into Kafka topics</p><blockquote>The <strong>Debezium Connector</strong> is a set of Kafka Connect connectors that capture and stream changes in real-time from databases (such as MySQL, PostgreSQL, MongoDB, etc.) to Kafka topics. It uses <strong>Change Data Capture (CDC)</strong> to capture insert, update, and delete operations. This allows systems to stay synchronized by publishing real-time database changes to Kafka. Debezium simplifies event-driven architectures by providing a reliable stream of data changes from databases.</blockquote><p>After we start an instance of debezium successfully we can make the connector through its API</p><pre>{<br>  &quot;name&quot;: &quot;debezium-mongo-connector&quot;,  <br>  // Unique name for this connector instance. This name is used to reference the connector in the Kafka Connect API.<br><br>  &quot;config&quot;: {<br>    // Main configuration section for the Debezium MongoDB connector.<br>    <br>    &quot;connector.class&quot;: &quot;io.debezium.connector.mongodb.MongoDbConnector&quot;,  <br>    // Specifies the connector class to be used. Here, it uses the MongoDB Debezium connector to capture changes from MongoDB.<br><br>    &quot;tasks.max&quot;: &quot;1&quot;,  <br>    // Defines the maximum number of tasks the connector can use. Here, it’s set to 1, meaning only one task will run to capture changes.<br><br>    &quot;mongodb.hosts&quot;: &quot;localhost:27017&quot;,  <br>    // The host and port of the MongoDB instance to connect to. In this case, it connects to MongoDB running on `localhost:27017`.<br><br>    &quot;mongodb.name&quot;: &quot;dbserver1&quot;,  <br>    // Logical name for the MongoDB instance in the Kafka topics. This prefix will be added to Kafka topic names.<br><br>    &quot;mongodb.user&quot;: &quot;yourMongoUser&quot;,  <br>    // MongoDB user credentials used for authentication. Replace with your MongoDB username.<br><br>    &quot;mongodb.password&quot;: &quot;yourMongoPassword&quot;,  <br>    // MongoDB password corresponding to the user for authentication. Replace with your MongoDB password.<br><br>    &quot;database.whitelist&quot;: &quot;test&quot;,  <br>    // Specifies the MongoDB database to monitor for changes. Here, it&#39;s set to only monitor the `test` database.<br><br>    &quot;collection.whitelist&quot;: &quot;test.user&quot;,  <br>    // Specifies the collection to capture changes from within the `test` database. Here, it is set to the `user` collection.<br><br>    &quot;topic.prefix&quot;: &quot;dbserver1.&quot;,  <br>    // Prefix added to Kafka topic names. For example, changes to `test.user` would go to the Kafka topic `dbserver1.test.user`.<br><br>    &quot;database.history.kafka.bootstrap.servers&quot;: &quot;kafka:9092&quot;,<br>    // set kafka host<br><br>    &quot;key.converter&quot;: &quot;org.apache.kafka.connect.json.JsonConverter&quot;,  <br>    // Specifies the converter to use for serializing the key in Kafka. Here, it’s set to JSON format.<br><br>    &quot;key.converter.schemas.enable&quot;: &quot;false&quot;,  <br>    // Disables schema inclusion for the key. Since MongoDB data does not require a schema for the key, it’s set to `false`.<br><br>    &quot;value.converter&quot;: &quot;org.apache.kafka.connect.json.JsonConverter&quot;,  <br>    // Specifies the converter to use for serializing the value in Kafka. It is set to JSON format for the value as well.<br><br>    &quot;value.converter.schemas.enable&quot;: &quot;false&quot;,  <br>    // Disables schema inclusion for the value. The MongoDB data in the Kafka value does not require schema information, so it’s set to `false`.<br><br>    // Unwrap transformation settings for processing MongoDB change events<br>    &quot;transforms&quot;: &quot;unwrap&quot;,  <br>    // Defines the transformation that will be applied to the change event. In this case, it&#39;s the `unwrap` transformation to extract the new document state.<br><br>    &quot;transforms.unwrap.type&quot;: &quot;io.debezium.connector.mongodb.transforms.ExtractNewDocumentState&quot;,  <br>    // Specifies the transformation class to use for unwrapping the change event. Here, it extracts the &quot;new state&quot; of the document after an update or insert.<br><br>    &quot;transforms.unwrap.add.fields&quot;: &quot;operation,lsn&quot;,  <br>    // Optional setting to add extra fields to the output record. Here, it adds `operation` (to track insert/update/delete events) and `lsn` (log sequence number).<br><br>    &quot;transforms.unwrap.drop.tombstones&quot;: &quot;false&quot;  <br>    // Determines whether tombstone records (delete events) should be included in the Kafka topic. Set to `false` to keep delete events.<br>  }<br>}</pre><p>JSON configuration above to make a new connector, if we save this JSON in file “config.json” we create a new connector throw API using curl</p><pre>curl -X POST -H &quot;Content-Type: application/json&quot; \<br>     --data @config.json \<br>     http://localhost:8083/connectors</pre><blockquote><strong>http://localhost:8083/connectors</strong>: This is the URL of the debizium Connect REST API endpoint that manages connectors. If your debizium Connect service is running on a different host or port, update this URL accordingly.</blockquote><p>After the Kafka connector is created it will get all records in this collection and set them in Kafka you can see them in <a href="https://docs.kafka-ui.provectus.io/">Kafka-ui</a> (4)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CY1ckqn-nW6hGmEX.png" /><figcaption>(4)</figcaption></figure><p>At this point everything is good but does not remain as it is, the first problem is that we don&#39;t need all data in every record, for example in the user collection we want only the email &amp; age of the user don’t care about other data, to solve this problem we use <a href="https://docs.confluent.io/platform/current/connect/transforms/overview.html"><strong>Single Message Transformations (SMTs)</strong></a></p><p>So we use</p><ul><li>ReplaceField SMT: we use it for Filter or rename fields within a Struct or Map.</li></ul><p>Another problem is that in the delete operation id of the record is not set in the Value of the message but is set only in the key we need it in value to be accessible by ClickHouse, we searched for SMTs to make that but did unfortunately not find, so we decided to write our custom SMT using java and help with documentation and other <a href="https://www.confluent.io/blog/kafka-connect-single-message-transformation-tutorial-with-examples/"><strong>articles</strong></a>, as well as we write other one for extract nested value of MongoDB schema and add it to flat level, now the pipeline will be like that (5)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JzR1qy_H_4N460gukLhVXQ.png" /><figcaption>(5)</figcaption></figure><h4>Stream Data To ClickHouse</h4><p>now we can get data from Kafka topics to clickhouse, after we run the clickhouse instance successfully, we open the clickhouse-client shell</p><pre>$ clickhouse-client<br>ClickHouse client version 20.13.1.5273 (official build).<br>Connecting to localhost:9000 as user default.<br>Connected to ClickHouse server version 20.13.1.<br><br>:)<br>:)<br># ------------------ make kafka consumer -----------<br>:)CREATE TABLE users_kafka (<br>    user String,<br>    email String,<br>    age UInt8,<br>    phone String,<br>    role String,<br>    __deleted UInt8,<br>    createdAt Int64<br>) ENGINE = Kafka <br>SETTINGS <br>    kafka_broker_list = &#39;kafka:9092&#39;,<br>    kafka_topic_list = &#39;kafka.topic.users&#39;,<br>    kafka_group_name = &#39;test&#39;,<br>    kafka_format = &#39;JSONEachRow&#39;,<br>    kafka_num_consumers = 1;<br>#-------------------- make users_view -----------<br>:)CREATE TABLE users_view (<br>    id String,<br>    user String,<br>    email String,<br>    age UInt8,<br>    phone String,<br>    role String,<br>    deleted UInt8,<br>    createdAt Int64<br>) ENGINE = ReplacingMergeTree<br>ORDER BY id;<br># ------------------ make materialized view ------------------<br>:)CREATE MATERIALIZED VIEW users_view_mv TO users_view AS<br>SELECT<br>    id,<br>    user,<br>    email,<br>    age,<br>    phone,<br>    role,<br>    __deleted as deleted,<br>    createdAt<br>FROM users_kafka;</pre><blockquote>Materialized views allow users to shift the cost of computation from query time to insert time, resulting in faster <strong>SELECT</strong> queries (6), for <a href="https://clickhouse.com/docs/en/materialized-view">more</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B1cZASx11voMHnTU.png" /><figcaption>(6)</figcaption></figure><p>In ClickHouse, an <strong>engine</strong> defines how data is stored, processed, and retrieved in a table. ClickHouse provides a variety of table engines, each designed for specific use cases, Here’s a general overview of the most commonly used engines:</p><h4>MergeTree Family</h4><p>The <strong>MergeTree</strong> family of engines is designed for high-performance, large-scale storage and queries. This family includes several variations with different optimizations for specific use cases.</p><ul><li><strong>MergeTree</strong>: The base engine for large tables, supporting indexing, partitioning, and data sorting.</li><li><strong>ReplacingMergeTree</strong>: Similar to <strong>MergeTree</strong>, it replaces old records with new records based on a specified version column.</li><li><strong>SummingMergeTree</strong>: Used when you want to automatically aggregate data by summing values for rows with the same primary key.</li><li><strong>AggregatingMergeTree</strong>: Optimized for tables that store pre-aggregated data (for example, for OLAP workloads).</li><li><strong>VersionedCollapsingMergeTree</strong>: Used for tables where rows may change, and the latest version is desired (supports “soft delete”).</li></ul><p>Now clickhouse has many tables every table mapped for collection in different microservices, now all required data is in one place we can join and aggregate between them and run OLAP queries without any problem , now pipeline will be like that (7)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yqWblP2zDzrtLzkeOdTalg.png" /><figcaption>(7)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30652d83ef80" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to get Flame graphs for PHP App Using Xdebug]]></title>
            <link>https://medium.com/@youssefshibl000/how-to-get-flame-graphs-for-php-app-using-xdebug-5e4c397e47dc?source=rss-ba2081e78814------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e4c397e47dc</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[php]]></category>
            <category><![CDATA[profiling]]></category>
            <category><![CDATA[tracking]]></category>
            <category><![CDATA[xdebug]]></category>
            <dc:creator><![CDATA[Youssefshebl]]></dc:creator>
            <pubDate>Sat, 07 Sep 2024 14:29:09 GMT</pubDate>
            <atom:updated>2024-09-09T08:20:31.014Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*wzn7paX5yf5vHh0tUYKU4w.jpeg" /></figure><h3><strong>what are Flame graphs?</strong></h3><blockquote>Flame graphs are a visualization of hierarchical data, created to visualize stack traces of profiled software so that the most frequent code-paths to be identified quickly and accurately</blockquote><blockquote>With flame graphs you can visualize hierarchical data, created to visualize stack traces of profiled software so that the most frequent code-paths to be identified quickly and accurately.</blockquote><p>So we can use flame graph trace execution of functions as well as profiling period of them, so let us generate it.</p><p>First, we should install Xdebu with a version equal to or greater than v3.3, because the flame graph starts supported with v3.3 and above, in you Linux you can run</p><pre>sudo apt install php-xdebug</pre><p>but after you install it you see that the last version of xdebug is available in Ubuntu is “php8.3-xdebug (3.2.0)” so in xdebug documentation they say</p><blockquote>Linux distributions might be providing an old and/or outdated version. If the package manager installs a version that is no longer supported (see <a href="https://xdebug.org/docs/compat#supported-versions"><strong>Supported Versions</strong></a>), please install Xdebug with <a href="https://xdebug.org/docs/install#pecl"><strong>PECL</strong></a>, or from <a href="https://xdebug.org/docs/install#source"><strong>source</strong></a> instead.</blockquote><p>So let us install xdebug last version using Pecl</p><blockquote><strong>PECL</strong> is a repository for PHP Extensions, providing a directory of all known extensions and hosting facilities for downloading and development of PHP extensions.</blockquote><pre>sudo apt-get update<br>sudo apt-get install php-pear<br>sudo apt install php-dev<br>sudo apt-get install libcurl3-openssl-dev<br>sudo pecl install pecl_http<br>sudo pecl install xdebug</pre><p>After installation to ensure that our version is v3.3 or greater, run</p><pre>&gt;php  -v               <br>PHP 8.3.6 (cli) (built: Jun 13 2024 15:23:20) (NTS)<br>Copyright (c) The PHP Group<br>Zend Engine v4.3.6, Copyright (c) Zend Technologies<br>    with Zend OPcache v8.3.6, Copyright (c), by Zend Technologies<br>    with Xdebug v3.3.2, Copyright (c) 2002-2024, by Derick Rethans<br><br>&gt; php -m -c<br>[PHP Modules]<br>.....<br>xdebug  # &lt;- xdebug in php modules <br>......<br><br>[Zend Modules]<br>Xdebug<br>Zend OPcache</pre><p>after that, we want to configure the xdebug extension in the php config, in the high php version make configuration of every extension in the separation file ( optional ) so we should get to the config file or xdebug and set this setting</p><pre>&gt; sudo vim /etc/php/8.3/cli/conf.d/20-xdebug.ini<br>  zend_extension=xdebug.so<br>  xdebug.mode=debug,trace<br>  xdebug.trace_format=3<br>  xdebug.trace_output_name=trace.%R<br>  xdebug.client_host=127.0.0.1<br>  xdebug.client_port=9003<br>  xdebug.start_with_request=yes</pre><p>After that you can get an old PHP project or create a new PHP project, I choose to create a new <strong>Laravel</strong> project</p><pre>&gt; composer create-project laravel/laravel test-xdebug-flame-app<br>&gt; cd ./test-xdebug-flame-app<br>&gt; php artisan serve</pre><p>After you run the HTTP server in the project by <em>php artisan serve, </em>you can access the app using the browser or curl cli tool, so every request to the application xdebug will generate a file in <em>/tmp/trace.* like /tmp/trace._.xt.gz</em></p><blockquote>Last step to convert this .gz file to interactive SVG which contains flame graph?</blockquote><p>We use <a href="https://github.com/brendangregg/FlameGraph">FlameGraph</a> tool to convert data from .gz file to SVG so let us make this</p><pre>cd /tmp/<br>ls -li /tmp/trace.*   # ensure that file exist <br>git clone https://github.com/brendangregg/FlameGraph.git<br>zcat /tmp/trace._.xt.gz | ./FlameGraph/flamegraph.pl &gt; ./flame-docs.svg<br>google-chrome ./flame-docs.svg</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LnsiD6wiECHgFSU1moIU1Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lNnetpHcAwemTa7O1Yn0zA.png" /></figure><blockquote>Click a box to zoom the Flame Graph to this stack frame only. To search and highlight all stack frames matching a regular expression, click the <em>search</em> button in the upper right corner or press Ctrl-F. By default, search is case sensitive, but this can be toggled by pressing Ctrl-I or by clicking the <em>ic</em> button in the upper right corner.</blockquote><p>when you hover over any box you will see this</p><blockquote>Illuminate\Http\Middleware\TrustProxies-&gt;handle (5,586,456 samples , 90%)</blockquote><p><strong>Illuminate\Http\Middleware\TrustProxies-&gt;handle</strong>: This indicates that the flame graph is showing performance data for the handle method of the TrustProxies middleware class within the Laravel framework. This middleware is responsible for handling the trust proxy settings, which can affect how requests are processed when your application is behind a proxy.</p><p><strong>5,586,456 samples</strong>: This number represents the total number of samples collected by Xdebug for this particular method during the profiling session. In Xdebug, sampling involves periodically capturing the call stack and profiling the application&#39;s execution. A higher number of samples typically indicates that this method is frequently executed or that it takes up a significant portion of the application&#39;s runtime.</p><p><strong>90%</strong>: This percentage shows the proportion of the total execution time or the number of samples that this method represents relative to the entire application’s profiling data. In this case, 90% suggests that TrustProxies-&gt;handle is consuming a substantial amount of the application&#39;s execution time or is involved in a significant percentage of the sampled calls.</p><p>For more , show <a href="https://www.youtube.com/watch?v=6uKZXIwd6M0&amp;t=1082s&amp;ab_channel=SPACE"><strong>Miha Rekar — What Are Flame Graphs and How to Read Them, RubyConfBY 2017</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e4c397e47dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Database Internals Chapter 4 Implementing B-Trees]]></title>
            <link>https://medium.com/@youssefshibl000/database-internals-chapter-4-implementing-b-trees-3b78dd076b67?source=rss-ba2081e78814------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b78dd076b67</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[sqlite]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[b-tree]]></category>
            <dc:creator><![CDATA[Youssefshebl]]></dc:creator>
            <pubDate>Thu, 16 May 2024 20:50:35 GMT</pubDate>
            <atom:updated>2024-05-16T20:50:35.237Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*Jp9-gOk8QwtVbal9zwH4nA.jpeg" /></figure><p>In the previous chapter, we discussed the general principles of binary format composition and learned how to create cells, build hierarchies, and connect them to pages using pointers. These concepts apply to both in-place updates and append-only storage structures. In this chapter, we discuss some concepts specific to B-Trees, the first concept we discuss is page header.</p><h4>Page Header</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RpRi2mKtyZtwBM0HOXrC-A.png" /><figcaption>p1</figcaption></figure><p>The page header holds information about the page that can be used for navigation, maintenance, and optimizations, It usually contains flags that describe page contents and layout, the number of cells on the page, lower and upper offsets marking the space ( p1 ), For example, PostgreSQL stores the page size and layout version in the header, and in SQLite, page header stores the number of cells and a rightmost pointer (we will take about later), let us take about some of page content in deep.</p><h4>Magic Numbers</h4><p>it’s a multibyte block, containing a constant value that can be used to explain kind or identify its version of the page, we can also use it for check validation of the page by setting some random offset byte in the header like “50 41 47 45 (hex for PAGE)” and During the read, we validate the page by comparing the four bytes from the read header with the expected byte sequence</p><blockquote>For some file types, especially binary files, there are “magic numbers” at the beginning of the file. These are specific sequences of bytes that help identify the file type. For example, executable files often start with specific bytes that indicate they are executable:ELF executables start with 0x7F 0x45 0x4C 0x46 (representing the characters .ELF).</blockquote><h4>Sibling Links</h4><p>Just store forward and backward links, pointing to the left and right sibling pages, this helps us to locate neighboring nodes without having to ascend back to the parent, but this approach adds some complexity to split and merge operations, as the sibling offsets have to be updated. ( p2 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*txCpcj5uLtzihdK13Zd2ag.png" /><figcaption>p2- Locating a sibling by following parent links (a) versus sibling links (b)</figcaption></figure><h4>Rightmost Pointers</h4><p>As we said before every node has “Separator Keys” Let&#39;s say the number of this key is N so we have the child pointer as ( N+1 ), In many implementations, nodes look more like the ones displayed in P3: each separator key has a child pointer, while the last pointer is stored separately,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*SXHFWlXTmS9FoNVKCOq_JA.png" /><figcaption>p3</figcaption></figure><p>In SQLite in store in the page header, If the rightmost child is split and the new cell is appended to its parent, the rightmost child pointer has to be updated ( p4 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*0MhUxJ2sgDVASadZ6D0Xmw.png" /><figcaption>p4</figcaption></figure><h4>Node High Keys</h4><p>Beside the rightmost pointer, we can store a value that will be useful as in the feature this value is high key, high key is the highest possible key that<br>can be present in the subtree under the current node ( p5 ), if the high key is 15 we can say that the highest number in the current node and his childer is 15</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*EaRgLl47Rm-norK9-JB7nw.png" /><figcaption>p5 — B-Trees without (a) and with (b) a high key</figcaption></figure><p>Now, the number of the key will equal the number of child pointers so we can store a rightmost pointer in the high key ( p6 ) k3 is a high key</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*aMyXDkccJQW2s1LAU6Z80Q.png" /><figcaption>p6 — Using +∞ as a virtual key (a) versus storing the high key (b)</figcaption></figure><h4>Overflow Pages</h4><p>We said previously that we split the file into header and footer and fixed size pages for every node stored in the page, but there are problems here what if there are nodes not have enough page size, someone can say we should extend the fixed size of the page but this will wisted some free space because if the node is tiny the resume of free page space will wisted, the pest solution that we can make “linked pages”.</p><p>For example, the default page size is 4K, and after inserting a few values, its data size has grown over 4 K. Instead of allowing arbitrary sizes, nodes are allowed to grow in 4 K increments, so we allocate a 4 K extension page and link it from the original one. These linked page extensions are called overflow pages. For clarity, we call the original page the primary page in the scope of this section.</p><p>Most B-Tree implementations allow storing only up to a fixed number of payload bytes in the B-Tree node directly and spilling the rest to the overflow page, This value is calculated by dividing the node size by fanout. Using this approach ( p7 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*2AgtvEqZBnLcqiCXUTPWoQ.png" /><figcaption>p7</figcaption></figure><p>When the first overflow page is allocated, its page ID is stored in the header of the primary page. If a single overflow page is not enough, multiple overflow pages are linked together by storing the next overflow page ID in the previous one’s header. Several pages may have to be traversed to locate the overflow part for the given payload, now let us talk about other concepts away from the page header.</p><h4>Binary Search</h4><p>We’ve already discussed the B-Tree lookup algorithm and mentioned that we locate a searched key within the node using the binary search algorithm. Binary search works only for sorted data, If keys are not ordered, they can’t be binary searched.</p><p>The binary search algorithm receives an array of sorted items and a searched key and returns a number. If the returned number is positive, we know that the searched key was found and the number specifies its position in the input array. A negative return value indicates that the searched key is not in the input array and gives us an insertion point. The insertion point is the index of the first element greater than the given key.</p><h4>Binary Search with Indirection Pointers</h4><p>Cells in the B-Tree page are stored in the insertion order, and only cell offsets preserve the logical element order, To perform a binary search through page cells, we pick the middle cell offset, follow its pointer to locate the cell, and compare the key from this cell with the searched key to decide whether the search should continue left or right, and continue this process recursively until the searched element or the insertion point is found as ( p8 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*uCNtF1oehpU5_G8-N5aSlA.png" /><figcaption>p8</figcaption></figure><h4>Propagating Splits and Merges</h4><p>As we said before when inserting or deleting a new record we may do split or merge operations to make this action we should able to traverse from the current node to a parent or root node we can achieve this by saving the parent pointer in the node ( p9 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jem7pJUEsaa0t7y4Hqv6-Q.png" /><figcaption>p9</figcaption></figure><p>But like sibling pointers, parent pointers have to be updated whenever the parent changes, This happens in all the cases when the separator key with the page identifier is transferred from one node to another during the parent node splits, merges, or rebalancing of the parent node.</p><p>Some implementations (for example, WiredTiger) use parent pointers for leaf traversal to avoid deadlocks, which may happen when using sibling pointers, but how does this make deadlocks</p><blockquote>WiredTiger is the default storage engine for mongodb</blockquote><p>Consider two leaf nodes, A and B, which are siblings:</p><p><strong>Thread 1</strong>, Acquires a lock on A, attempts to acquire a lock on B (to traverse to the next leaf)</p><p><strong>Thread 2</strong>, Acquires a lock on B , attempts to acquire a lock on A</p><p><strong><em>Deadlock Avoidance</em></strong></p><p>Consider leaf nodes A and B with parent node P:</p><p><strong>Thread </strong>1: Acquires a lock on A, Releases the lock on A, Acquires a lock on P, Acquires a lock on B from the parent P</p><p><strong>Thread 2</strong>: Acquires a lock on B, Releases the lock on B, Acquires a lock on P, Acquires a lock on A from the parent P</p><p>Since the traversal involves releasing locks before moving to the next node, and typically the parent nodes are locked and unlocked in a controlled manner, the risk of deadlocks is minimized.</p><p>To address and locate a sibling, we can follow a pointer from the parent node and recursively descend back to the lower level. Whenever we reach the end of the parent node after traversing all the siblings sharing the parent, the search recursively upward, eventually reaching up to the root and continuing back down to the leaf level.</p><h4>Breadcrumbs</h4><p>We can use other techniques to access sliding nodes or parent nodes by saving the pointer of every node during the traverse tree, Breadcrumbs contain references to the nodes followed from the root and are used to backtrack them in reverse when propagating splits or merges, PostgreSQL stores breadcrumbs in a stack, internally referenced as BTStack ( p10 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QpzvlJ4Zc98Rpwjskg8C-Q.png" /><figcaption>p10</figcaption></figure><p><em>Breadcrumbs collected during lookup, containing traversed nodes and cell indices. Dotted lines represent logical links to visited nodes. Numbers in the breadcrumbs table represent indices of the following child pointers</em></p><h4>Rebalancing</h4><p>Some B-Tree implementations attempt to reduce split and merge operations to reduce their costs by rebalancing elements within the level, or moving elements from more occupied nodes to less occupied ones for as long as possible before finally performing a split or merge. This helps to improve node occupancy and may reduce the number of levels within the tree at a potentially higher maintenance cost of rebalancing ( p11 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XvIV4idiSMSYhktvC940WQ.png" /><figcaption>p11</figcaption></figure><p>B*-Trees keep distributing data between the neighboring nodes until both siblings are full, Then, instead of splitting a single node into two half-empty ones the algorithm splits two nodes into three nodes, each of which is two-thirds full. SQLite uses this variant in the implementation.</p><h4>Right-Only Appends</h4><p>Many database systems use auto-incremented monotonically increasing values as primary index keys. This case opens up an opportunity for optimization since all the insertions are happening toward the end of the index (in the rightmost leaf), so most of the splits occur on the rightmost node on each level. Moreover, since the keys are incremented, because of the ratio of insertion greater updating and deleting so when adding a new record we should care about ording in cell pointers ( p12 ) For example auto-incremented by 5 in the first and other is not auto-incremented, we should make an extra effort to resort pointers to can use binary search, in PostgreSQL is calling this case a <strong><em>fastpath</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZJSJfsNYAExW0rO23suHjw.png" /><figcaption>p12</figcaption></figure><h4>Bulk Loading</h4><blockquote>bulk loading is a process whereby one can load large amounts of data into a database in a relatively short period of time</blockquote><p>But I have to rebuild the tree (for example, for defragmentation), so we can take the idea with right-only appends even further. Since the data required for tree creation is already <strong>sorted</strong>, during bulk loading we only need to append the items at the rightmost location in the tree</p><p>In this case, we can avoid splits and merges altogether and compose the tree from the bottom up, writing it out level by level, or writing out higher-level nodes as soon as we have enough pointers to already written lower-level nodes like (p13) from left to right</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*tA9TMh9z6rSOlRxMOojbxQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*0-PWFh49iiQzW6vLQKM7Bg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*BQcR_Ne6nCl0pX_M7igG5Q.png" /><figcaption>p13</figcaption></figure><p>We can simplify the idea by copying the leaf node once and trying to build a tree from the pointer of leaf nodes instead of inserting one by one a record</p><h4>Compression</h4><p>Storing the raw, uncompressed data can induce significant overhead, and many databases offer ways to compress it to save space. The apparent trade-off here is between access speed and compression ratio: larger compression ratios can improve data size, allowing you to fetch more data in a single access, but might require more RAM and CPU cycles to compress and decompress it.</p><p>We can compress all data files or index files or pages, but our page compression will fit our discussion, when we compress a page, the size of compression may be smaller than the block size (smallest size I can read or write ) so we compress two different size page like “A” and “B” can be like this ( p14 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IWUSi0bMllnWNHJNxiQmEQ.png" /><figcaption>p14</figcaption></figure><p>you can see a compressed page (a) taking less space than the disk block. When we load this page, we load additional bytes that belong to the other page. With pages that span multiple disk blocks, like (b) in the same image, we have to read an additional block</p><h4>Vacuum and Maintenance</h4><p>other processes happen in parallel with queries that maintain storage integrity, reclaim space, reduce overhead, and keep pages in order. Performing these operations in the background allows us to save some time and avoid paying the price of cleanup during inserts, updates, and deletes.</p><p>Splits and merges in internal nodes or inserts, updates, and deletes on the leaf level can result in a page that has enough logical space but does not have enough contiguous space, since it is fragmented. an example of such a situation: the page still has some logical space available, but it’s fragmented and is split between the two deleted (garbage) records and some remaining free space between the header/cell pointers and cells (p15)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rvGVBhrPiErfuUj6LOCF5g.png" /><figcaption>p15</figcaption></figure><p>Data records that can be reached by following pointers down from the root node are live (addressable). Nonaddressable data records are said to be garbage: these records are not referenced anywhere and cannot be read or interpreted, so their contents are as good as nullified.</p><h4>Fragmentation Caused by Updates and Deletes</h4><p>When you delete any record it deletes from the page this is done by removing offset only from cell pointers, which makes a deleted record not addressable.</p><p>When the page is split, only offsets are trimmed and, since the rest of the page is not addressable, cells whose offsets were truncated are not reachable, so they will be over-written whenever the new data arrives, or garbage-collected when the vacuum process kicks in.</p><p>when many recorders are deleted that makes parts of free space since this page fragmented, so we should make defragmentation this means rewriting all pages this make addressable records Besides that save free space by merging all free space in one free space block</p><p>Updates are mostly applicable to the leaf level: internal page keys are used for guided navigation and only define subtree boundaries so we don’t write a page just reorder cell pointers or offset if key is updated in the tree (p16)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C6OtDgdw_Sf6KbJAJ5JbJQ.png" /><figcaption>p16</figcaption></figure><h4>Page Defragmentation</h4><p>The process that takes care of space reclamation and page rewrites is called compaction, vacuum, or maintenance. Page rewrites can be done synchronously on write if the page does not have enough free physical space <strong><em>(to avoid creating unnecessary overflow pages). Still, compaction</em></strong> is mostly referred to as a distinct, asynchronous process of walking through pages, performing garbage collection, and rewriting their contents.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b78dd076b67" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Database Internals Chapter 3 File Formats]]></title>
            <link>https://medium.com/@youssefshibl000/database-internals-chapter-3-file-formats-ec05752c79c3?source=rss-ba2081e78814------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec05752c79c3</guid>
            <category><![CDATA[desk]]></category>
            <category><![CDATA[b-tree]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Youssefshebl]]></dc:creator>
            <pubDate>Tue, 07 May 2024 07:27:17 GMT</pubDate>
            <atom:updated>2024-05-07T11:46:24.096Z</atom:updated>
            <content:encoded><![CDATA[<p>Database Internals, A Deep Dive into How Distributed Data Systems Work, chapter 3, File Formats</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*Jp9-gOk8QwtVbal9zwH4nA.jpeg" /></figure><h4>Binary Encoding</h4><p>As we mentioned in the previous chapter when we say the best data structure used in a database is B-Tree, we say that it is a disk-storage type because of the huge keys of data in a tree that are oversized to memory storage so we should optimize the disk storing process.</p><p>First, Let us mention the data type stored according to the B-Tree data structure.</p><blockquote><strong>Primitive data structure</strong> : is the data structure that allows you to store only single data type values , like integer, boolean, character, float, etc</blockquote><blockquote><strong>Non-Primitive data structure</strong> : is a data structure that allows you to store multiple data type values, like array , object , tree , linkedlist ,etc</blockquote><p>The B-Tree is a non-primitive data structure but components of this tree are keys and values which are primitive values, Keys and values have a type, such as an integer, date, or string, let us discover a method to store this value.</p><p>A single byte can only contain 256 values, from 0 to 255. Therefore, today&#39;s data types contain multiple bytes, such as floating-point numbers and long integers. The problem arises when a computer attempts to read more than one byte at a time, Where should the biggest byte appear when multibyte data is to be read? , there are two method to read or write .</p><blockquote>Big-endian : The order starts from the most-significant byte (MSB), followed by the bytes in decreasing significance order. In other words, MSB has the lowest address.<br>Little-endian : The order starts from the least-significant byte (LSB), followed by the bytes in increasing significance order</blockquote><p>The hexadecimal 32-bit integer 0xAABBCCDD, where AA is the MSB, is shown using both big- and little-endian byte order ( p1 ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*2nFj8RyQ4LJobHatLUqcoQ.png" /><figcaption>p1</figcaption></figure><blockquote>We know the importance of this classification in the next section in this chapter.</blockquote><p>Records consist of primitives like numbers, strings, booleans, and their combinations. However, we can only use byte sequences when transferring data over the network or storing it on disk. This means that to send or write the record, we have to serialize it (convert it to an interpretable sequence of bytes) and, before we can use it after receiving or reading, we have to deserialize it (translate the sequence of bytes back to the original record).</p><p>In binary data formats, we always start with primitives that serve as building blocks for more complex structures. Different numeric types may vary in size. byte value is 8 bits, short is 2 bytes (16 bits), int is 4 bytes (32 bits), and long is 8 bytes (64 bits).</p><p>Floating-point numbers (such as float and double) are represented by their sign, fraction, and exponent, For example, a floating-point number 0.15652 has a binary representation ( p2 ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*ecvR06YOY07q6MJqDB3qmw.png" /><figcaption>p2</figcaption></figure><p>To understand how this convert done read this <a href="https://www.log2base2.com/storage/how-float-values-are-stored-in-memory.html">article</a>.</p><h4>Strings and Variable-Size Data</h4><p>All primitive numeric types have a fixed size, but when setting complex values together, You can combine primitive values into structures and use fixed-size arrays or pointers to other memory regions, you have two methods to make these Pascal strings or null-terminated strings</p><ul><li>A Pascal-style string has one leading byte ( length ), followed by length bytes of character data ( p3 )</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*hkILBD1XJTlOUxyWpIMsOA.png" /><figcaption>p3</figcaption></figure><ul><li>Null-terminated strings are character arrays where the end of the string is marked by a null character (&#39;\0&#39;). This null character serves as a sentinel value to indicate the end of the string, In C programming, a null-terminated string &quot;Hello&quot; is stored {&#39;H&#39;, &#39;e&#39;, &#39;l&#39;, &#39;l&#39;, &#39;o&#39;, &#39;\0&#39;} in memory</li></ul><h4>Bit-Packed Data: Booleans, Enums, and Flags</h4><blockquote>Booleans can be represented either by using a single byte, or encoding true and false as 1 and 0 values. Since a boolean has only two values, using an entire byte for its representation is wasteful, and developers often batch boolean values together in groups of eight, each boolean occupying just one bit. We say that every 1 bit is set and every 0 bit is unset or empty</blockquote><blockquote>Enums, short for enumerated types, can be represented as integers and are often used in binary formats and communication protocols. Enums are used to represent often repeated low-cardinality values. For example, we can encode a B-Tree node type using an enum</blockquote><pre>enum NodeType {<br>ROOT, // 0x00h<br>INTERNAL, // 0x01h<br>LEAF // 0x02h<br>}</pre><blockquote>Flags are kind of a combination of packed booleans and enums. Flags can represent non-mutually exclusive named boolean parameters. For example, we can use flags to denote whether or not the page holds value cells</blockquote><h4>General Principles</h4><p>After we mentioned a little about how to store data we assumed that we would store our B-Tree in a <strong>File </strong>so we split the file into pages every page has a disk block or multiple contiguous blocks, the file usually starts with a fixed-size header and may end with a fixed-size trailer, which holds auxiliary information that should be accessed quickly or is required for decoding the rest of the file. The rest of the file is split into pages ( p4 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*MCFoHejxAlXIhecz-8fsQQ.png" /><figcaption>p4</figcaption></figure><p>After splitting the file let us thinking how to store data, In many cases, the stored data has a schema which has the number, order, and type of fields the table can hold to reduce the amount of data stored on disk, for example, if we have company directory and want store data for each employee so record in page will be like (p5), first name and last name is variable-size attribute so we should write size of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BJP3dmKvP99_Fpflgxn1_w.png" /><figcaption>p5</figcaption></figure><p>Now, to access first_name, we can slice first_name_length bytes after the fixed-size area, to access last_name, we can locate its starting position by checking the sizes of the variable-size fields that precede it, Database files often consist of multiple parts</p><h4>Page Structure</h4><p>Database systems store data records in data and index files. These files are partitioned into fixed-size units called pages, which often have a size of multiple filesystem blocks., Page sizes usually range from 4 to 16 KB.</p><p>Let us try to store nodes of B-Tree in pages, If I have B-tree have kyes (k) and every key has value ( V ) and between keys, there are pointers of child pages ( P ) the tree be like ( p6 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlNGJ38NC-vy9oAEc4WLcg.png" /></figure><p>Store root node in page be like (p7 — Page organization for xed-size records)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*rD8J5qZkh4Dj8X0me0jbGg.png" /><figcaption>p7</figcaption></figure><p>But there are many problems</p><ul><li>Appending a key anywhere except the right side requires relocating elements</li><li>It doesn’t allow variable-size records efficiently only for fixed-size data</li></ul><p>To solve this problem when should think in another method to store tree</p><h4>Slotted Pages</h4><p>When storing variable-size records, the main problem is <strong>free space management</strong>: reclaiming the space occupied by removed records. If we attempt to put a record of size n into the space previously occupied by the record of size m, unless m == n or we can find another record that has a size exactly m — n, this space will remain unused (p8)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qaGu_XN9AZPmDkGvCpTy1A.png" /><figcaption>p8</figcaption></figure><p>To simplify space management for variable-size records, we can split the page into fixed-size segments. However, For example, if we use a segment size of 64 bytes unless the record size is a multiple of 64, we waste <strong>64 — (n modulo 64)</strong> bytes, where n is the size of the inserted record. In other<br>words, unless the record is a multiple of 64, one of the blocks will be only partially filled, so I solve the variable-size record problem, Space reclamation can be done by simply rewriting the page and moving the records around</p><p>But this makes a big problem, assume that other pages have pointers to the current page to a specific record when I make a reclamation this pointer will not be valid because I move the location of the record during reclamation (p9)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NDooagI4BCuZoYioxMBUnQ.png" /><figcaption>p9</figcaption></figure><p>After reclamation pointer 2 refers to an invalid record or data so I should preserve record offsets, To summarize, we need a page format that allows us to:<br>• Store variable-size records with a minimal overhead.<br>• Reclaim space occupied by the removed records.<br>• Reference records on the page without regard to their exact locations.</p><p>To efficiently store variable-size records such as strings, binary large objects (BLOBs), etc., we can use an organization technique called slotted page</p><blockquote>We organize the page into a collection of slots or cells and split out pointers and cells in two independent memory regions residing on different sides of the page. This means that we only need to reorganize pointers addressing the cells to preserve the order, and deleting a record can be done either by nullifying its pointer or removing it (p10).</blockquote><p>A slotted page has a fixed-size header that holds important information about the page and cells. Cells may differ in size and can hold arbitrary data: keys, pointers, data records, etc. a slotted page organization, where every page has a maintenance region (header), cells, and pointers to them</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*Z9tiwh95yLBtcUXmgCQm-w.png" /><figcaption>p10</figcaption></figure><p>by Slott&#39;s page, we solve many problems, the only overhead incurred by slotted pages is a pointer holding offsets to the exact positions where the records are stored, space can be reclaimed by defragmenting and rewriting the page and changing pointers array, the pointer of recorders be inside page and not reference from outside, outside reference only to page id not records so when changing the location of records in defragmenting not affected to out-page reference.</p><h4>Cell Layout</h4><p>The next step is to store data in a cell there are two types (key cell and key-value cell )</p><p>To compose a key cell ( p11 ), we need to know:<br>• Cell type (can be inferred from the page metadata)<br>• Key size<br>• ID of the child page this cell is pointing to<br>• Key bytes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*VcY2oSa3F2_cBPqmczsJpw.png" /><figcaption>p11</figcaption></figure><p>The key-value cells hold data records instead of the child page IDs ( p12 ). Otherwise, their structure is similar:<br>• Cell type (can be inferred from page metadata)<br>• Key size<br>• Value size<br>• Key bytes<br>• Data record bytes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*kU0bgHY5NXfteRk_YYOMAQ.png" /><figcaption>p12</figcaption></figure><p>Before going to the next section we should distinguish between page ID and offset, the page ID is a reference to the location of the page which is later translated to the actual offset in the file using the lookup table,osets are page-local and are relative to the page start offset.</p><h4>Combining Cells into Slotted Pages</h4><p>To organize cells into pages, we can use the slotted page technique that we discussed (p13)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*EFAAQurqwwUIfvPSphKtig.png" /><figcaption>p13</figcaption></figure><p>We append cells to the right side of the page (toward its end) and keep cell offsets/pointers on the left side of the page, we take about this in page structure <em>(Little-endian)</em>, and when we insert a new cell we insert it in first free space in left by little-endian after this add a pointer to it in pointer array (offsets) but before reordering the pointer array to new indexing, for example, Let’s consider an example of a page that holds names. Two names are added to the page, and their insertion order is: Tom and Leslie ( p14 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*kXpXYeBIwyJ9C5ntVQnOWA.png" /><figcaption>p14</figcaption></figure><p>Now, we’d like to add one more name to this page: Ron. New data is appended to the upper boundary of the free space of the page, but cell offsets have to preserve the lexicographical key order: Leslie, Ron, Tom, be ( p15 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*IC-e1RS-u1FMEFjFDcPMlQ.png" /><figcaption>p15</figcaption></figure><h4>Managing Variable-Size Data</h4><p>Removing an item from the page does not have to remove the actual cell and shift other cells to reoccupy the freed space. Instead, the cell can be marked as deleted and an in-memory availability list can be updated with the amount of freed memory and a pointer to the freed value. The availability list stores offsets of freed segments and their sizes. When inserting a new cell, we first check the availability list to find if there’s a segment where it may fit. You can see an example of a fragmented page with available segments on ( p16 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*ohavFPinDCpY6mynwNpJMA.png" /><figcaption>p16</figcaption></figure><p>SQLite calls unoccupied segments free blocks and stores a pointer to the first free block in the page header. Additionally, it stores the total number of available bytes within the page to quickly check whether or not we can fit a new element into the page after defragmenting it</p><p><em>First, fit<br></em>This might cause a larger overhead since the space remaining after reusing the first suitable segment might be too small to fit any other cell, so it will be effectively wasted</p><p><em>Best fit<br></em>For best fit, we try to find a segment for which insertion leaves the smallest remainder</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*grnkbyqYQkn8u6E95zxuUA.png" /><figcaption>p17</figcaption></figure><p>( p17 ) explain the difference between first fit and best fit</p><h4>Versioning</h4><p>In processing of updating in the database by adding features or fixing bugs the binary file format changes into different versions from the database engine so should write a version of the file somewhere, For example, Apache Cassandra uses version prefixes in filenames, PostgreSQL<br>stores the version in the PG_VERSION file, The version can also be stored directly in the index file header.</p><h4>Checksumming</h4><p>Files on disk may get damaged or corrupted by software bugs and hardware failures. To identify these problems preemptively and avoid propagating corrupt data to other subsystems or even nodes, we can use checksums and cyclic redundancy checks (CRCs)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*rGGZS07lM-MlJf05KBe9eA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec05752c79c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Database Internals Chapter 2 B-Tree Basics]]></title>
            <link>https://medium.com/@youssefshibl000/database-internals-chapter-2-b-tree-basics-ffa4983afa8c?source=rss-ba2081e78814------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffa4983afa8c</guid>
            <category><![CDATA[b-tree]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[nosql]]></category>
            <dc:creator><![CDATA[Youssefshebl]]></dc:creator>
            <pubDate>Sat, 27 Apr 2024 19:22:34 GMT</pubDate>
            <atom:updated>2024-04-27T20:35:08.429Z</atom:updated>
            <content:encoded><![CDATA[<p>Database Internals, A Deep Dive into How Distributed Data Systems Work, chapter 2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*Jp9-gOk8QwtVbal9zwH4nA.jpeg" /></figure><h4>Binary Search Trees</h4><p>To explain why we use Binary Search Tree, let us first get our scenario, let say that we have a table and each row in this table is stored in a desk, The index table should have a pointer to get a specific row of the table and key based on column with a table like id ( p1 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7LS3P2MQuc4t9OmN5A9EfA.png" /><figcaption>p1</figcaption></figure><p>In the first look, we save the index table that has pointers and an indexing column ( id ) as an array, so, if we need some rows we use “Liner Search” to get the row pointer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*m6Ni_Ry4iRIPMHY44uy49A.gif" /></figure><p>But this method or data structure is Bad, why?</p><p>because the complexity of searching is O(n), inserting is O(n) ( you should shift all keys after inserting in the middle), deleting is O(n) ( you should shift all keys after inserting in the middle), So we should use other data structure for get data pointer faster than a traditional array? from here we use a binary search tree ( p2 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*qXNNjLZ5Q8WoOawvI8PBKg.png" /><figcaption>p2</figcaption></figure><blockquote>A binary search tree (BST) is a sorted in-memory data structure, used for efficient key-value lookups. BSTs consist of multiple nodes. Each tree node is represented by a key, a value associated with this key, and two child pointers (hence the name binary). BSTs start from a single node, called a root node. There can be only one root in the tree , Each node splits the search space into left and right subtrees, a node key is greater than any key stored in its left subtree and less than any key stored in its right subtree ( p3 )</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*rx6e6eArWHTgfEHI90iC_A.png" /><figcaption>p3</figcaption></figure><p>In this DS the complexity of insert is O(log(n)) where n is the number of nodes, updating O(log(n)), deleting O(log(n)), but it is O(h) where h is the height of tree we can say h = log(n) in balance tree, but what if tree not balanced ( p4 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*waOJTHRvZMXRYhnCGniAuw.png" /><figcaption>p4</figcaption></figure><p>We lose the performance benefits of the binary search tree structure and allow insertions and deletions to determine tree shape, In the balanced tree, following the left or right node pointer reduces the search space in half on average, so lookup complexity is logarithmic: O(log2 N). If the tree is not balanced, worst-case complexity goes up to O(N), since we might end up in a situation where all elements end up on one side of the tree.</p><p>One of the ways to keep the tree balanced is to perform a rotation step after nodes are added or removed. If the insert operation leaves a branch unbalanced (two consecutive nodes have only one child), we can rotate nodes around the middle one. In the example shown, during rotation, the middle node (3), known as a rotation pivot, is promoted one level higher, and its parent becomes its right child ( p5 ), From here the term “B-Tree” appears</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*oqf40isnNS_qUPEvmjnk5A.png" /><figcaption>p5</figcaption></figure><p>When data is huge it is hard to set all trees in memory so we store trees in the disk “ on-disk data structures” When building this tree by inserting data or nodes we do not insert them at the same time but at different times and this makes data write in desk in different locations blocks or pages, this led to a long time to search in a tree this because disk seeks ( p6 ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*vqRQ_1UwPFMEketV_A1VyQ.jpeg" /><figcaption>p6</figcaption></figure><p>To solve this problem we have many solutions</p><p>1 — Increase fanout, this makes many nodes in the same blocked and reduces the number of seeks during traversal , from left to right in ( p7 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/1*AxeeIRcROHDfIn-ZE6CsiQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*yGPRiy5SY2YL6vqnWqoTEg.png" /><figcaption>p7</figcaption></figure><blockquote>fanout: fanout is the maximum allowed number of children per node ( Separator Keys — 1 )</blockquote><blockquote>Fanout and height are inversely correlated: the higher the fanout,<br>the lower the height. If fanout is high, each node can hold more<br>children, reducing the number of nodes and, subsequently, reduc‐<br>ing height</blockquote><p>2 — Paged Binary Trees, Laying out a binary tree by grouping nodes into pages ( p8 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*j_aeKTMhoNTPmZsEpGZnmA.png" /><figcaption>p8</figcaption></figure><h4>Disk-Based Structures</h4><blockquote>Database systems store data in memory and on disk. In-memory database manage‐ment systems (sometimes called main memory DBMS) store data primarily in memory and use the disk for recovery and logging. Disk-based DBMS hold most of the data on disk and use memory for caching disk contents or as a temporary storage. Both types of systems use the disk to a certain extent, but main memory databases store their contents almost exclusively in RAM</blockquote><p>In Disk-based structure, there are two main problems, the first problem is the number of seeks during traversal because data is stored in different pages or blocks, but this problem is almost solved in SSD Disk Unlike HDD Disk this solution will be expensive (p9)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*lnhGt492AVbnkdnX6bGrOQ.jpeg" /><figcaption>p9</figcaption></figure><p>The second problem is that the minimum block size that can read or write is 512 bytes to 4 KB so if we have a block with 4 rows and want to read or write one attribute you should read the complete block and update it or insert it again so to make any action we wast time to read or write not necessary data</p><h4>B-Trees</h4><blockquote>A B-tree is a sort of self-balancing search tree whereby each node could have more than two children and hold multiple keys</blockquote><p>There are two types of B- — Trees, B+ — Trees</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hTXHck8HpLL7u3dlBjmHnA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_X4ktj-WKfRmQw-SB__0Q.png" /><figcaption>p10</figcaption></figure><p>B- — Tree in left, in this type each node has a value and pointer of data and can’t make range query only point query, in right B+ — tree it is the default type of B-Tree, each node has only key (Separator Keys) and it down to leaf node which has value ( pointer ) (p10) .</p><p>What is the difference between a point query and a range query?</p><p>Point Query:</p><ul><li>A point query retrieves a specific record or row from a database based on exact match criteria.</li><li>It typically involves specifying one or more conditions in the query’s WHERE clause that precisely identify the record you want to retrieve.</li><li>For example, a point query might look like: SELECT * FROM users WHERE id = 123;</li><li>In this query, id = 123 is the condition that uniquely identifies a single record with the ID 123.</li></ul><p>Range Query:</p><ul><li>A range query retrieves multiple records or rows from a database based on a range of values.</li><li>It involves specifying a range condition in the WHERE clause, such as using comparison operators (&lt;, &lt;=, &gt;, &gt;=) or between two values.</li><li>For example, a range query might look like: SELECT * FROM products WHERE price BETWEEN 10 AND 50;</li><li>In this query, price BETWEEN 10 AND 50 retrieves all products with prices ranging from $10 to $50.</li><li>B- — Tree can’t make a range query because it requires to back in the tree and taking another path to get the next value, there are some types of the B- — tree with bi-direction to range query</li></ul><h4>B-Tree Hierarchy</h4><blockquote>Root node: This has no parents and is the top of the tree.<br>Leaf nodes: These are the bottom layer nodes that have no child nodes.<br>Internal nodes: These are all other nodes, connecting root with leaves. There is usually more than one level of internal nodes.</blockquote><p>Before we make an Example to build sample B+ — Tree let us calculate the complexity of searching in Tree.</p><p>Assume that we have k as the order of trees or the most number of children from the node and M is the total number of items in the B-Tree, so the complexity of the search, insert, delete is O(logk(M)), so if we take a special case from B-Tree (binary search tree ) with order 2 ( k=2 ) the complexity will be O(log(M))</p><h4>How Build B-Tree</h4><p>We try to build a tree with random numbers from 0 to 100 values with 4-order we use <a href="https://www.cs.usfca.edu/~galles/visualization/BPlusTree.html">B+ — Tree</a> to visualize the tree, we first add 5,10,15, keys in one sort node, and when set to 20, the size of the node is 4 so should split because 4-order mean max number of key in the node is 3 and child is 4, we split node to [5,10,15,20] to [[5,10], [15,20]] then get smallest value from right and set it in root as ( p11 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*siHMOW13tbuwM_V8XAfokg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*Xd-pYDXlq_QnjhGRKaMeqA.png" /><figcaption>p11</figcaption></figure><p>Next, we add 25 and 30 by the same method we add 25 to the right node, when we add 30 the right node will be full so we split it from [15,20,25,30] to [15,20] &amp; [25,30] and get the smallest value in the right node and set it parent node (p12)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*f_K8mfghDkNuX8JniKzzuA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*0amf0GhaELsDSxbK31-gog.png" /><figcaption>p12</figcaption></figure><p>we set [35,40,45] as the previous method (p13)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*_Xp9OCQAfuKvhUvHWARc2w.png" /><figcaption>p13</figcaption></figure><p>After this we add 50 when we add 50, the right split from [35,40,45,50] to [[35,40],[45,50]] so we get the smallest value in the right node and add it to the parent so the parent will be [15,25,35,40] but it oversize so we should split parent to [[15,25],[35,40]] and we get smallest value in right node and make it new root node and remove it from current node, the file flow will be like this (p14)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*W0xj8qthtJ5T4NL5yeB3Zg.png" /><figcaption>p14</figcaption></figure><p>So, we should see differences between adding a key in the leaf node ( left ) and the non-leaf node ( right ) (p15)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*Eb6tPOC1Hhk5RZt_0RV_bg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*yMQ8hC0s6Rqy2a_UUtBWbQ.png" /><figcaption>p15</figcaption></figure><p>Next, we add 16, 17, and 18, when we add 17 the 2nd node will split from [15,16,17,20] to [[15,16],[17,20]] so we move 17 to the parent node like this (p16)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*JjOSNLusFcJW4cmv8zLr-w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*RuRF-WPm2pcJMbOu8Z4eFw.png" /><figcaption>p16</figcaption></figure><p>Then add 18 (p17)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*JxkV5bBHIaQHeYoTa-lZmQ.png" /><figcaption>p17</figcaption></figure><p>what if we add 19?</p><p>[17,18,19,20] split to [[17,18],[19,20]] add 19 to parent, [15,17,19,25] will split to [[15,17],[19,25]] so we will move 19 to root, final will be like this (p18)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*N9oCAOFdIo9qOzM9PSSa8Q.png" /><figcaption>p18</figcaption></figure><h4>B-Tree Node Merges</h4><p>When we delete the key in the node we should check if there are merges or not because some merges may reduce the height of the tree, For faster search, we can classify node merges into 2 types, leaf node, non-leaf node</p><blockquote>N (key-Value) = O(tree order) -1</blockquote><ul><li>leaf node</li></ul><p>if a node can hold up to N key-value pairs, and a combined num‐<br>ber of key-value pairs in two neighboring nodes is less than or equal to N, in (p19) we delete 16 from left node</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*72pgN0uWGapp6WmBD8m-Mg.png" /><figcaption>p19</figcaption></figure><ul><li>non-leaf nodes:</li></ul><p>if a node can hold up to N + 1 pointers, and a combined<br>number of pointers in two neighboring nodes is less than or equal to N + 1 , in (p20) we delete 10 from left node</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*aMcfyp1DelLMj-NcZUewew.png" /><figcaption>p20</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffa4983afa8c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build your Event-Loop]]></title>
            <link>https://medium.com/@youssefshibl000/build-your-event-loop-2b341dcf7c67?source=rss-ba2081e78814------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b341dcf7c67</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[coroutine]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[event-loop]]></category>
            <category><![CDATA[asynchronous]]></category>
            <dc:creator><![CDATA[Youssefshebl]]></dc:creator>
            <pubDate>Sat, 20 Apr 2024 18:48:41 GMT</pubDate>
            <atom:updated>2024-04-20T20:14:05.298Z</atom:updated>
            <content:encoded><![CDATA[<h4>In this article, we build a sample event loop and how event-loop works behind the scene</h4><h3>Introduction</h3><h4>1- How Computer Work?</h4><figure><img alt="1" src="https://cdn-images-1.medium.com/max/436/1*Uhw93ss-aVsFmsGYbZQFPg.png" /><figcaption>p1</figcaption></figure><p>As you show ( p1 ) this is the main component in the computer, a group of instructions that will pass from RAM to the CPU first from the “ control unit “ cu which has all instructions set of this CPU architecture ( e.g. “AMD” or “ARM”), then go to ALU which make all arithmetic and bitwise operations on integer binary numbers then save all calculations in the cache then RAM, but this will be hard to write a program in assembly language (CPU instructions set ) so we need some extra layer which compiles readable/writeable language to the assembly which run in CPU &amp; controlling hardware resources and memory of my program, so we make “Operating System”, at the end flow of your program will be like this (p2)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*tHRHQFS5jxSo5Qo7ArQ3_A.png" /><figcaption>p2</figcaption></figure><p>this continues for a long time until some problem, the problem that you can run only one program in CPU This means stopping all other programs until run current program, Hence the term “Multicore Processor” appeared</p><h4>2- Multicore Processor</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*X21mXviaHzYoRFyS6nkMRg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/339/1*FJMqqBOtPOPdTTEyOF3G9A.png" /><figcaption>P3</figcaption></figure><p>This makes the user able to run multiple at the same time ( p3 ), Hence the term “parallelism” appeared, also there are important terms you should know “Concurrency”, but what is the difference between Concurrency and parallelism</p><h4>3 — Concurrency vs. parallelism</h4><blockquote>parallelism: as we see before run multi task in same time using multi core</blockquote><blockquote>Concurrency: run multi tasks in same core without mulit core by switching excutin between tasks</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oJiJpFMonDpOpuiswt6zEg.png" /><figcaption>P4</figcaption></figure><p>Let us give an example of a task with a single core and make it with a multi-core “parallelism”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*Ml4scivbAYfSqKkEmHYH_g.png" /><figcaption>p5</figcaption></figure><blockquote>Imagine that you are working at the FBI IT department, and for the next mission, you must implement a program to crack a password (number combination of a particular length) and access a system that can destroy the whole world (P5)</blockquote><pre>#!/usr/bin/env python3.9<br><br>&quot;&quot;&quot;Program for cracking the password consisting with only numbers using brute<br>force approach sequentially&quot;&quot;&quot;<br><br>import time<br>import math<br>import hashlib<br>import typing as T<br><br><br>def get_combinations(*, length: int, min_number: int = 0,<br>                     max_number: T.Optional[int] = None) -&gt; T.List[str]:<br>    &quot;&quot;&quot;Generate all possible password combinations&quot;&quot;&quot;<br>    combinations = []<br>    if not max_number:<br>        # calculating maximum number based on the length<br>        max_number = int(math.pow(10, length) - 1)<br><br>    # go through all possible combinations in a given range<br>    for i in range(min_number, max_number + 1):<br>        str_num = str(i)<br>        # fill in the missing numbers with zeros<br>        zeros = &quot;0&quot; * (length - len(str_num))<br>        combinations.append(&quot;&quot;.join((zeros, str_num)))<br>    return combinations<br><br><br>def get_crypto_hash(password: str) -&gt; str:<br>    &quot;&quot;&quot;&quot;Calculating cryptographic hash of the password&quot;&quot;&quot;<br>    return hashlib.sha256(password.encode()).hexdigest()<br><br><br>def check_password(expected_crypto_hash: str,<br>                   possible_password: str) -&gt; bool:<br>    actual_crypto_hash = get_crypto_hash(possible_password)<br>    # compare the resulted cryptographic hash with the one stored in the system<br>    return expected_crypto_hash == actual_crypto_hash<br><br><br>def crack_password(crypto_hash: str, length: int) -&gt; None:<br>    &quot;&quot;&quot;Brute force the password combinations&quot;&quot;&quot;<br>    print(&quot;Processing number combinations sequentially&quot;)<br>    start_time = time.perf_counter()<br>    combinations = get_combinations(length=length)<br>    for combination in combinations:<br>        if check_password(crypto_hash, combination):<br>            print(f&quot;PASSWORD CRACKED: {combination}&quot;)<br>            break<br><br>    process_time = time.perf_counter() - start_time<br>    print(f&quot;PROCESS TIME: {process_time}&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    crypto_hash = \<br>        &quot;e24df920078c3dd4e7e8d2442f00e5c9ab2a231bb3918d65cc50906e49ecaef4&quot;<br>    length = 8<br>    crack_password(crypto_hash, length)</pre><p>when running this task output will be like this</p><pre>Processing number combinations sequentially<br>PASSWORD CRACKED: 87654321<br>PROCESS TIME: 64.60886170799999</pre><p>In another case, we use multi-process divide combinations checks in many cores, this is called “data decomposition”</p><pre>#!/usr/bin/env python3.9<br><br>&quot;&quot;&quot;Program for cracking the password consisting of only numbers using multi<br>cores in parallel&quot;&quot;&quot;<br><br>import os<br>import math<br>import time<br>import typing as T<br>import hashlib<br>from multiprocessing import Pool<br><br>ChunkRange = T.Tuple[int, int]<br><br><br>def get_combinations(*, length: int, min_number: int = 0, max_number: int = None) -&gt; T.List[str]:<br>    &quot;&quot;&quot;Generate all possible password combinations&quot;&quot;&quot;<br>    combinations = []<br>    if not max_number:<br>        # calculating maximum number based on the length<br>        max_number = int(math.pow(10, length) - 1)<br><br>    # go through all possible combinations in a given range<br>    for i in range(min_number, max_number + 1):<br>        str_num = str(i)<br>        # fill in the missing numbers with zeros<br>        zeros = &quot;0&quot; * (length - len(str_num))<br>        combinations.append(&quot;&quot;.join((zeros, str_num)))<br>    return combinations<br><br><br>def get_crypto_hash(password: str) -&gt; str:<br>    &quot;&quot;&quot;&quot;Calculating cryptographic hash of the password&quot;&quot;&quot;<br>    return hashlib.sha256(password.encode()).hexdigest()<br><br><br>def check_password(expected_crypto_hash: str,<br>                   possible_password: str) -&gt; bool:<br>    actual_crypto_hash = get_crypto_hash(possible_password)<br>    # compare the resulted cryptographic hash with the one stored on the system<br>    return expected_crypto_hash == actual_crypto_hash<br><br><br>def get_chunks(num_ranges: int,<br>               length: int) -&gt; T.Iterator[ChunkRange]:<br>    &quot;&quot;&quot;Splitting the passwords into chunks using break points&quot;&quot;&quot;<br>    max_number = int(math.pow(10, length) - 1)<br>    chunk_starts = [int(max_number / num_ranges * i)<br>                    for i in range(num_ranges)]<br>    chunk_ends = [start_point - 1<br>                  for start_point in<br>                  chunk_starts[1:]] + [max_number]<br>    return zip(chunk_starts, chunk_ends)<br><br><br>def crack_chunk(crypto_hash: str, length: int, chunk_start: int,<br>                chunk_end: int) -&gt; T.Union[str, None]:<br>    &quot;&quot;&quot;Brute force the password combinations&quot;&quot;&quot;<br>    print(f&quot;Processing {chunk_start} to {chunk_end}&quot;)<br>    combinations = get_combinations(length=length, min_number=chunk_start,<br>                                    max_number=chunk_end)<br>    for combination in combinations:<br>        if check_password(crypto_hash, combination):<br>            return combination  # found it<br>    return  # not found<br><br><br>def crack_password_parallel(crypto_hash: str, length: int) -&gt; None:<br>    &quot;&quot;&quot;Orchestrate cracking the password between different processes&quot;&quot;&quot;<br>    # getting number of available processors<br>    num_cores = os.cpu_count()<br>    print(&quot;Processing number combinations concurrently&quot;)<br>    start_time = time.perf_counter()<br><br>    # processing each chunk in a separate process concurrently<br>    with Pool() as pool:<br>        arguments = ((crypto_hash, length, chunk_start, chunk_end) for<br>                     chunk_start, chunk_end in<br>                     get_chunks(num_cores, length))<br>        results = pool.starmap(crack_chunk, arguments)<br>        print(&quot;Waiting for chunks to finish&quot;)<br>        pool.close()<br>        pool.join()<br><br>    result = [res for res in results if res]<br>    print(f&quot;PASSWORD CRACKED: {result[0]}&quot;)<br>    process_time = time.perf_counter() - start_time<br>    print(f&quot;PROCESS TIME: {process_time}&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    crypto_hash = \<br>        &quot;e24df920078c3dd4e7e8d2442f00e5c9ab2a231bb3918d65cc50906e49ecaef4&quot;<br>    length = 8<br>    crack_password_parallel(crypto_hash, length)</pre><pre>Processing number combinations concurrently<br>Chunk submitted checking 0 to 12499998<br>Chunk submitted checking 12499999 to 24999998<br>Chunk submitted checking 24999999 to 37499998<br>Chunk submitted checking 37499999 to 49999998<br>Chunk submitted checking 49999999 to 62499998<br>Chunk submitted checking 62499999 to 74999998<br>Chunk submitted checking 74999999 to 87499998<br>Chunk submitted checking 87499999 to 99999999<br>Waiting for chunks to finish<br>PASSWORD CRACKED: 87654321<br>PROCESS TIME: 17.183910416</pre><p>But does this mean that the number of tasks should be an equal number of CPU cores? No, we can run a number of tasks greater than the number of CPU cores, OS solves this problem by “Context Switching”</p><blockquote>Context switching in an operating system involves saving the context or state of a running process so that it can be restored later, and then loading the context or state of another. process and run it.</blockquote><p>To simplify the idea, we can say that OS will run process1 for some time and save the current state of process1 and switch to process2 for the other time then save the current state and return to process1 ( p6 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*vHOfbS33cBmfs9dthz5-3g.png" /><figcaption>P6</figcaption></figure><p>Problems end now?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*i-AUQk8EXNMkEBZ3VfvqkA.gif" /></figure><p>Context switching incurs overhead due to the time and resources required to save and restore the state of a process. This overhead includes saving and reloading CPU registers, memory management data, and other process-specific information, Modern CPUs use caches (e.g., L1, L2, L3 caches) to store frequently accessed data and instructions for faster access. Context switching can lead to cache thrashing, where the CPU cache is constantly invalidated and refilled with data from different processes. Cache thrashing reduces cache efficiency and can negatively impact performance, So many in many cases preferred to use “Threads”</p><h4>4- Threads</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*z2JxMR1I9F1sGKTp9WqP3A.png" /><figcaption>P7</figcaption></figure><blockquote>A thread is a single sequence stream within a process. Threads are also called lightweight processes as they possess some of the properties of processes. Each thread belongs to exactly one process. In an operating system that supports multithreading, the process can consist of many threads. But threads can be effective only if the CPU is more than 1 otherwise two threads have to context switch for that single CPU</blockquote><p>All threads in the same Process have the same heap and many things ( p7 ) so when making a switching context between threads OS only switches stacks &amp; registers and small other things, so the complete flow of execution will be ( p8 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CzH2idpzSpJTGuwxZjTJGw.jpeg" /><figcaption>P8</figcaption></figure><p><em>Where event-loop?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/1*RtF74_waM-ICFcQyQKb1Nw.gif" /></figure><p>before we start making the <em>event-loop</em> we should mention two reasons that make us go to <em>event-loop</em></p><p>1- I/O-bound</p><p>when any of the running threads make i/o operations the thread will blocked until i/o finishes but this wastes time in execution because the scheduler will switch to it but it freezes, we can solve this by OS optimization</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*pfh2OlZ3GBOjA69QrreuKg.png" /><figcaption>P9</figcaption></figure><p>If a program has a lot of I/O-bound operations, context switching is an advantage. As soon as a task goes into a Blocked state, another task in a Ready state takes its place. This allows the processor to stay busy if work (tasks in the Ready state) needs to be done ( p9 )</p><p>2- C10k problem</p><p>If we make the web server make a new thread for every request to handle this request concurrently, Context-switching threads take precious CPU time. The more threads we have, the more time we spend switching instead of doing actual work. Thus, the overhead of starting and stopping threads can become quite high, With a high level of concurrency (say, 10,000 threads, if you can configure the OS to create that many threads), having many threads can affect throughput due to the frequent context-switching overhead ( p10 ). This is a scalability problem</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*lKlID5loGuoB56kSn5Q8Vw.png" /><figcaption>P10</figcaption></figure><p>To solve this we should back some steps, why do we use multi-thread/multi-process, because a single thread or process runs sequentially so if your task has any i/o bound it will freeze until the i/o operation is done, so if we solve this i/o blocking we can solve the problem, Hence the term “i/o multiplexing ” appeared</p><h4>5- i/o multiplexing</h4><blockquote>Modern OSs typically include event notification subsystems, commonly called I/O multiplexing. These subsystems collect and queue I/O events from monitored resources and block them until the user application is available to process them. This allows the user application to perform a simple check for incoming I/O events that require attention</blockquote><p>To simplify the idea of this we say to OS don’t block my code for groups of i/o but trigger me and run this callback when any of this i/o response group is ready ( p11 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jn-9quFm4CSQaaHGLMNnVQ.jpeg" /><figcaption>P11</figcaption></figure><p>We try to make two tasks one without i/o multiplexing and the other with it</p><pre>import socket<br><br>HOST = &#39;127.0.0.1&#39;  # Loopback address for local testing<br>PORT = 12345        # Port number (can be any available port)<br>server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)<br>server_socket.bind((HOST, PORT))<br>server_socket.listen(1)<br>print(f&#39;Server listening on {HOST}:{PORT}&#39;)<br>client_socket, client_address = server_socket.accept()<br>print(f&#39;Connected to client: {client_address}&#39;)<br>try:<br>    while True:<br>        data = client_socket.recv(1024)  # Maximum data size to receive<br>        if not data:<br>            break<br>        client_socket.sendall(data)<br>except Exception as e:<br>    print(f&#39;Error: {e}&#39;)<br>finally:<br>    client_socket.close()<br>server_socket.close()</pre><p>when the client starts to connect to my server, the server waits to receive data from the client and then sends it again, in this case, if another client tries to connect to my server I can’t because my server is blocked from receiving data from the first client, so to handle many clients we should using i/o multiplexing by using select interface In python</p><pre>import socket<br>import select<br><br># Define the server host and port<br>HOST = &#39;127.0.0.1&#39;  # Loopback address for local testing<br>PORT = 12345        # Port number (can be any available port)<br><br># Create a socket object<br>server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)<br><br># Set socket options to allow reusing the address<br>server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)<br><br># Bind the socket to the host and port<br>server_socket.bind((HOST, PORT))<br><br># Listen for incoming connections<br>server_socket.listen()<br><br>print(f&#39;Server listening on {HOST}:{PORT}&#39;)<br><br># List to hold connected client sockets<br>client_sockets = []<br><br>while True:<br>    # Use select to multiplex I/O events (readable sockets)<br>    readable, _, _ = select.select([server_socket] + client_sockets, [], [])<br><br>    for sock in readable:<br>        if sock is server_socket:<br>            # Accept incoming connection from a client<br>            client_socket, client_address = server_socket.accept()<br>            print(f&#39;Connected to client: {client_address}&#39;)<br><br>            # Add the client socket to the list<br>            client_sockets.append(client_socket)<br>        else:<br>            # Receive data from a connected client<br>            data = sock.recv(1024)  # Maximum data size to receive<br><br>            if not data:<br>                # If no data received, close the client socket<br>                print(f&#39;Client disconnected: {sock.getpeername()}&#39;)<br>                client_sockets.remove(sock)<br>                sock.close()<br>            else:<br>                # Send the received data back to the client<br>                sock.sendall(data)</pre><p>We make this set all list of descriptors in select method,</p><pre><br>select.select(readList, writeList, exceptionList[, timeOut])<br># realist: List of descriptors waiting for a condition when data is ready for reading<br># whitelist: List of descriptors waiting for a condition when data is ready for write<br># exceptionalism: List of descriptors waiting for an exception condition<br># Timeout: Number of seconds to wait before coming out of blocking</pre><p>In the above example, the select method will block code until a new client is connected or an existing client sends a message to serve, but we can set timeout to this blocking by Timeout, so at any time i/o is ready callback will run without i/o block “event-based concurrency” ( p12 )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*QN9CQhZGuBUYO9F0srJmXA.png" /><figcaption>P12</figcaption></figure><p>Now we can make a sample <em>event-loop</em></p><h3>event-loop</h3><pre>import select<br>class EventLoop:<br>    def __init__(self) -&gt; None:<br>        self._running = False<br>        self._io = []<br>        self._timeouts = []<br>        self._queuecallback = []<br>        self.readables = {}<br>        self.writables = {}<br><br>    def AddIo(self, io):<br>        io.eventloop = self<br>        self._io.append(io)<br><br>    def RunForever(self):<br>        self._running = True<br>        while self._running:<br>            # print(&quot;running&quot;)<br>            # self.readables = self._io[0].objs()<br>            self.readables = {}<br>            for io in self._io:<br>                self.readables.update(io.objs())<br><br><br>            try:<br>                # block until there is something to read <br>                # if there is nothing to read, it will block<br>                # to disable blocking, pass 0 as the last argument<br>                readables, writables, _ = select.select(<br>                    self.readables,<br>                    self.writables,<br>                    [],<br>                    None,<br>                )<br>            except KeyboardInterrupt:<br>                print(&quot;KeyboardInterrupt&quot;)<br>                self._running = False<br>                continue<br>            if readables:<br>                for readable in readables:<br>                    obj, callback, *args = self.readables[readable]<br>                    callback(obj, *args)</pre><p>We will make Class for i/o operation</p><pre>import socket<br>from EventLoop1 import EventLoop<br><br><br>class IO:<br>    def __init__(self) -&gt; None:<br>        self.eventloop: EventLoop = None<br>        self.objects = {}<br><br>    def add_socket(<br>        self,<br>        sock,<br>        callback,<br>    ):<br>        # check if sock is a socket<br>        if isinstance(sock, socket.socket):<br>            sock.setblocking(False)<br>            key = sock.fileno()<br>            self.objects[key] = (sock, self.HandleSocketConnection, callback)<br><br>    def objs(self):<br>        # return list(self.objects.values())<br>        return self.objects<br><br>    def HandleSocketConnection(self, obj, callback):<br>        conn, addr = obj.accept()<br>        conn.setblocking(False)<br>        key = conn.fileno()<br>        self.objects[key] = (conn, self.HandleSocketData, callback)<br><br>    def HandleSocketData(self, obj, callback):<br>        data = obj.recv(1024)<br>        if data:<br>            callback(obj, data)<br><br>    def Add_file(self, file, callback):<br>        key = file.fileno()<br>        self.objects[key] = (file, self.HandleFile, callback)<br><br>    def HandleFile(self, obj, callback):<br>        data = obj.read(1024)<br>        if data:<br>            callback(obj, data)<br>        else:<br>            del self.objects[obj.fileno()]<br>            obj.close()</pre><pre>from socket import create_server<br>from EventLoop1 import EventLoop<br>from IO import IO<br><br>server = create_server((&quot;&quot;, 8081))<br>server2 = create_server((&quot;&quot;, 8082))<br><br>def callback1(conn, data):<br>    print(&quot;-----------8081----------&quot;)<br>    print(data)<br>    conn.sendall(data)<br>    print(&quot;---------------------&quot;)<br>def callback2(conn, data):<br>    print(&quot;-----------8082----------&quot;)<br>    print(data)<br>    conn.sendall(data)<br>    print(&quot;---------------------&quot;)<br>def readfilecallback(file, data):<br>    print(&quot;-----------file----------&quot;)<br>    print(data)<br>    print(&quot;---------------------&quot;)<br><br>io = IO()<br>io.add_socket(server, callback1)<br>io.add_socket(server2, callback2)<br>io.Add_file(open(&quot;test.txt&quot;), readfilecallback)<br>eventloop = EventLoop()<br>eventloop.AddIo(io)<br><br>if __name__ == &quot;__main__&quot;:<br>    eventloop.RunForever()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nT5oVbE04kDWHsltfsSXVw.png" /></figure><p>At this point, we make a sample <em>event-loop</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*OnzvSkOs7qVelhWXiIDkJQ.gif" /></figure><p>There is one thing that remains. this thing is “coroutines”</p><p>When the event is triggered the callback of this event will run, but what if this callback is CPU bound this means if any event is pushed during the first callback this new event callback not run until the first callback finishes, so we need something to stop execution callback 1 and run part of callback 2 then return to callback 1, Coroutin e can make this</p><h4>Coroutine</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*s1kKfTp_kedgeeWSXDMq1g.png" /></figure><p>We can make this using generators</p><blockquote>In computer science, a generator is a routine that can be used to control the iteration behaviour of a loop</blockquote><pre>#!/usr/bin/env python3.9<br><br>&quot;&quot;&quot;Coroutine implementation and usage&quot;&quot;&quot;<br><br>from collections import deque<br>import typing as T<br><br># there is a similar type in collections.abc<br>Coroutine = T.Generator[None, None, int]<br><br><br>class EventLoop:<br>    def __init__(self) -&gt; None:<br>        self.tasks: T.Deque[Coroutine] = deque()<br><br>    def add_coroutine(self, task: Coroutine) -&gt; None:<br>        &quot;&quot;&quot; Add a task to the queue. &quot;&quot;&quot;<br>        self.tasks.append(task)<br><br>    def run_coroutine(self, task: Coroutine) -&gt; None:<br>        try:<br>            task.send(None)<br>            self.add_coroutine(task)<br>        except StopIteration:<br>            print(&quot;Task completed&quot;)<br><br>    def run_forever(self) -&gt; None:<br>        &quot;&quot;&quot; Run the loop until no more tasks exist in the queue. &quot;&quot;&quot;<br>        while self.tasks:<br>            print(&quot;Event loop cycle.&quot;)<br>            self.run_coroutine(self.tasks.popleft())<br><br><br>def fibonacci(n: int) -&gt; Coroutine:<br>    &quot;&quot;&quot; Generating fibonacci sequence as a non-recursive<br>    awaitable task (coroutine) &quot;&quot;&quot;<br>    a, b = 0, 1<br>    for i in range(n):<br>        a, b = b, a + b<br>        print(f&quot;Fibonacci({i}): {a}&quot;)<br>        # yield control back to the event loop<br>        yield<br>    return a<br><br><br>if __name__ == &quot;__main__&quot;:<br>    event_loop = EventLoop()<br>    event_loop.add_coroutine(fibonacci(5))<br>    event_loop.run_forever()</pre><pre>Event loop cycle.<br>Fibonacci(0): 1<br>Event loop cycle.<br>Fibonacci(1): 1<br>Event loop cycle.<br>Fibonacci(2): 2<br>Event loop cycle.<br>Fibonacci(3): 3<br>Event loop cycle.<br>Fibonacci(4): 5<br>Event loop cycle.<br>Task completed</pre><p>So if we make event callback be Coroutine object to can switch between running callbacks</p><p>goodbye</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*_7Gx9M-mQO-IN5LGm-eQJQ.gif" /></figure><p>references</p><p>1 — Kirill Bobrov — Grokking Concurrency-Manning book</p><p>2- <a href="https://www.youtube.com/watch?v=CRPnkTv1phs&amp;ab_channel=PyConCZ">Building an async event loop</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b341dcf7c67" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Expose to share your local sites and applications with others on the internet]]></title>
            <link>https://medium.com/@youssefshibl000/expose-to-share-your-local-sites-and-applications-with-others-on-the-internet-aacca7f2d6f4?source=rss-ba2081e78814------2</link>
            <guid isPermaLink="false">https://medium.com/p/aacca7f2d6f4</guid>
            <category><![CDATA[php]]></category>
            <dc:creator><![CDATA[Youssefshebl]]></dc:creator>
            <pubDate>Mon, 05 Dec 2022 22:46:21 GMT</pubDate>
            <atom:updated>2022-12-05T22:50:06.690Z</atom:updated>
            <content:encoded><![CDATA[<h3>Expose by Beyond code to share your local sites and applications with others on the internet</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L_BYwIBiv4CqGSGXN9EK1A.png" /></figure><h3>what is expose ?</h3><p>Expose is a tunnel application that allows you to share your local sites and applications with others on the internet. It is open source, written in PHP and the best alternative to ngrok for PHP developers.</p><h4>How it work ?</h4><p>when you run client server in your local machine you open tunnel between your local machine and expose host with specific subdomain like</p><p><em>uniquesubdomain.sharedwithexpose.com</em></p><p>when user visit this subdomain in internet the “shardwithexpose” server get the file from your local machine</p><h3>how install it ?</h3><p>1- install export in local machine as global by composer</p><pre>composer global require beyondcode/exposec</pre><p>2-Setup your authentication token</p><p>you should take token to authentication by creating an account from <a href="https://expose.dev/register">here</a></p><p>when you make account you get page like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*nYw7Vt17YvtXlt7Ubrjv6Q.png" /></figure><p>to enter your token make</p><pre>expose token &quot;your_token&quot;</pre><p>now you make all installation of expose in local machine now share file</p><p>3- share file to internet</p><pre>expose share http://localhost:3000</pre><p>this will share your local server on port 3000 to internet so you must run local server before share your file so if you in php you can make this first</p><pre>php -S localhost:8000</pre><p>after you run “expose share” you get screen like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GI2lJoHp1Iuieo2Jx34M_w.png" /></figure><p><strong>local </strong>URL: can access your local app by this a url</p><p><strong>Dashboard </strong>URL: URL of dashboard which display all request to your app like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qxfp8e4ulIbVOHjIz8E7oA.png" /></figure><p><strong>expose URL</strong> : public URL in internet</p><h3>options</h3><p>you can add option during sharing your files like name of subdomain in internet --subdomain=my-site</p><pre>expose share localhost:8000 --subdomain=my-site</pre><p>you can also Sharing sites with basic authentication by</p><pre>expose share localhost:8000 --auth=&quot;admin:secret&quot;</pre><p>also can add Global Server Infrastructure</p><pre>+------+---------------------------+------+<br>| Key  | Region                    | Type |<br>+------+---------------------------+------+<br>| eu-1 | EU (Frankfurt)            | Pro  |<br>| us-1 | US (New York)             | Pro  |<br>| us-2 | US (San Francisco)        | Pro  |<br>| ap-1 | Asia Pacific (Singapore)  | Pro  |<br>| in-1 | India (Bangalore)         | Pro  |<br>| sa-1 | South America (São Paulo) | Pro  |<br>| au-1 | Australia (Sydney)        | Pro  |<br>+------+---------------------------+------+</pre><pre>expose share localhost:8000 --server=eu-1</pre><p>can set default</p><pre>expose default-server us-2</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aacca7f2d6f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>