<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by John Sener on Medium]]></title>
        <description><![CDATA[Stories by John Sener on Medium]]></description>
        <link>https://medium.com/@mcansener?source=rss-332f518a15da------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hDhCe8YRoedhbqjcgF7KhQ.png</url>
            <title>Stories by John Sener on Medium</title>
            <link>https://medium.com/@mcansener?source=rss-332f518a15da------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:10:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mcansener/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Detecting Broken Image URLs Efficiently in .NET]]></title>
            <link>https://medium.com/@mcansener/detecting-broken-image-urls-efficiently-in-net-059a070765e2?source=rss-332f518a15da------2</link>
            <guid isPermaLink="false">https://medium.com/p/059a070765e2</guid>
            <category><![CDATA[url]]></category>
            <category><![CDATA[image]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[John Sener]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 19:37:22 GMT</pubDate>
            <atom:updated>2026-04-01T19:43:28.337Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/313/0*PeVZJf17vpidxYlv" /></figure><p>Broken image URLs are easy to ignore when the dataset is small.</p><p>A single GET request, a quick status-code check, and the problem looks solved.</p><p>That approach does not survive contact with production volume.</p><p>Once validation moves into media audits, catalog ingestion, migration checks, or scheduled content hygiene jobs, the cost profile changes. Slow endpoints, missing assets, and rate-limited hosts stop being edge cases and start dominating total runtime.</p><p>At that point, the validator itself becomes part of the operational problem.</p><h3>What This Project Demonstrates</h3><p>This project shows two implementations side by side:</p><ul><li>NaiveValidator: sequential GET requests with straightforward logging</li><li>OptimizedValidator:</li><li>bounded concurrency</li><li>singleton HttpClient</li><li>HEAD first, GET fallback</li><li>per-request timeout control</li></ul><p>The goal is not to build the cleverest validator.</p><p>The goal is to build one that behaves <strong>predictably under load</strong>.</p><h3>The Failure Mode of the Naive Approach</h3><p>The baseline implementation is intentionally simple:</p><pre>public async Task&lt;IReadOnlyList&lt;ValidationResult&gt;&gt; ValidateAsync(<br>    IReadOnlyList&lt;string&gt; urls,<br>    CancellationToken cancellationToken = default)<br>{<br>    var results = new List&lt;ValidationResult&gt;(urls.Count);<br><br>    foreach (var url in urls)<br>    {<br>        var result = await ValidateSingleAsync(url, cancellationToken);<br>        results.Add(result);<br>        LogResult(result);<br>    }<br><br>    return results;<br>}</pre><p>There is nothing wrong with this code from a correctness perspective.</p><p>The issue is <strong>throughput</strong>.</p><p>With sequential execution:</p><p><em>Total runtime = sum of all request latencies</em></p><p>If ten URLs each take three seconds to time out, you just spent ~30 seconds waiting — even if everything else is fast.</p><h3>Sample workload</h3><ul><li>20 valid responses</li><li>10 404</li><li>10 403</li><li>10 delayed (timeouts)</li></ul><p><strong>Naive result: </strong>Processed 50 URLs in 38.1s</p><h3>Why “Just Make It Parallel” Is Not Enough</h3><p>The next instinct is:</p><p><em>“Let’s run everything in parallel”</em></p><p>That works — until it breaks.</p><p>Unbounded concurrency introduces new problems:</p><ul><li>socket exhaustion</li><li>connection churn</li><li>bursty timeouts</li><li>remote throttling</li><li>memory pressure</li></ul><p>In short: Unlimited parallelism is not optimization — it’s loss of control.</p><h3>The Optimized Design</h3><p>The optimized version makes four deliberate choices.</p><h3>1. Reuse HttpClient</h3><pre>private static readonly HttpClient SharedHttpClient = CreateClient();<br><br>private static HttpClient CreateClient()<br>{<br>    var handler = new SocketsHttpHandler<br>    {<br>        AutomaticDecompression = DecompressionMethods.All,<br>        PooledConnectionLifetime = TimeSpan.FromMinutes(5)<br>    };<br><br>    var client = new HttpClient(handler)<br>    {<br>        Timeout = Timeout.InfiniteTimeSpan<br>    };<br><br>    client.DefaultRequestHeaders.UserAgent.ParseAdd(&quot;ImageUrlValidator/1.0&quot;);<br><br>    return client;<br>}</pre><p>Why it matters:</p><ul><li>avoids socket exhaustion</li><li>enables connection pooling</li><li>reduces overhead per request</li></ul><h3>2. Cap Concurrency with SemaphoreSlim</h3><pre>using var semaphore = new SemaphoreSlim(_maxConcurrency, _maxConcurrency);<br><br>var tasks = urls<br>    .Select(url =&gt; ValidateWithConcurrencyAsync(url, semaphore, cancellationToken))<br>    .ToArray();<br><br>return await Task.WhenAll(tasks);</pre><p>Key idea:</p><p><em>Controlled concurrency &gt; maximum concurrency</em></p><p>The value 10 is not special — it is <strong>intentional</strong>.</p><h3>3. Try HEAD Before GET</h3><pre>var headResult = await SendAsync(url, HttpMethod.Head, cancellationToken);<br><br>if (!ShouldFallbackToGet(headResult))<br>{<br>    return headResult;<br>}</pre><p>Why:</p><ul><li>avoids downloading full image bodies</li><li>reduces bandwidth + latency</li></ul><p>Fallback handles:</p><ul><li>405 Method Not Allowed</li><li>501 Not Implemented</li><li>unreliable servers</li></ul><h3>4. Apply Per-Request Timeout</h3><pre>using var timeoutCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);<br>timeoutCts.CancelAfter(_requestTimeout);<br><br>using var response = await SharedHttpClient.SendAsync(<br>    request,<br>    HttpCompletionOption.ResponseHeadersRead,<br>    timeoutCts.Token);</pre><p>Why it matters:</p><ul><li>prevents slow endpoints from dominating</li><li>enables explicit timeout classification</li></ul><h3>Classification Matters</h3><p>This validator does not just return true/false.</p><p>It produces meaningful categories:</p><ul><li>Success</li><li>NotFound (404)</li><li>Forbidden (403)</li><li>Timeout</li><li>Other</li></ul><p>This matters because:</p><ul><li>404 → asset is gone</li><li>403 → asset exists but access is blocked (WAF, hotlink protection, etc.)</li></ul><h3>Result model</h3><pre>internal sealed record ValidationResult(<br>    string Url,<br>    ValidationStatus Status,<br>    HttpStatusCode? StatusCode,<br>    string Method,<br>    TimeSpan Duration,<br>    string? Detail = null);</pre><p>Small, but operationally useful.</p><h3>Running Both Approaches</h3><pre>var naiveSummary = await RunValidatorAsync(<br>    &quot;NAIVE&quot;,<br>    urls.Count,<br>    token =&gt; naiveValidator.ValidateAsync(urls, token));<br><br>var optimizedSummary = await RunValidatorAsync(<br>    &quot;OPTIMIZED&quot;,<br>    urls.Count,<br>    token =&gt; optimizedValidator.ValidateAsync(urls, token));</pre><h3>Output</h3><pre>[NAIVE]<br>Processed 50 urls in 38.1s<br>Success: 20 | NotFound: 10 | Forbidden: 10 | Timeout: 10 | Other: 0<br><br>[OPTIMIZED]<br>Processed 50 urls in 5.8s<br>Success: 20 | NotFound: 10 | Forbidden: 10 | Timeout: 10 | Other: 0</pre><p>Same results.<br>Very different cost.</p><h3>Production Takeaways</h3><p>The principles here are simple but critical:</p><ul><li>bound concurrency</li><li>reuse network resources</li><li>keep the fast path cheap</li><li>fail slow endpoints quickly</li><li>produce useful diagnostics</li></ul><h3>What Comes Next (If This Were Production)</h3><ul><li>host-level rate limiting</li><li>retry strategies (with backoff)</li><li>structured logging</li><li>latency metrics (p95, p99)</li><li>configurable limits</li></ul><p>These are <strong>extensions</strong>, not redesigns.</p><h3>Closing Thought</h3><p>Broken image validation looks trivial — until scale exposes the hidden costs.</p><p>The naive version works.</p><p>The optimized version <strong>keeps working</strong>.</p><p>And in distributed systems, that difference is everything.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=059a070765e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why My .NET HttpClient Kept Getting 403 Forbidden Until I Forced HTTP/1.1]]></title>
            <link>https://medium.com/@mcansener/why-my-net-httpclient-kept-getting-403-forbidden-until-i-forced-http-1-1-997a6e0d03be?source=rss-332f518a15da------2</link>
            <guid isPermaLink="false">https://medium.com/p/997a6e0d03be</guid>
            <category><![CDATA[http-client]]></category>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[unauthorized]]></category>
            <dc:creator><![CDATA[John Sener]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 08:15:07 GMT</pubDate>
            <atom:updated>2026-03-31T08:15:07.934Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*FxnekrUKffJdoyZb7-u4pA.png" /></figure><p>Browsers worked. The API endpoint worked in Postman. A basic .NET client did not.</p><p>From the infrastructure side, it was not simple at all.</p><p>The actual problem was not authorization.<br> It was <strong>request shape</strong>.</p><p>That distinction matters when the target system sits behind services like Cloudflare, Akamai, a reverse proxy, or a custom WAF. In those environments, the request can be rejected long before application authorization runs.</p><p>Things like:</p><ul><li>HTTP version</li><li>headers</li><li>compression negotiation</li><li>connection behavior</li><li>TLS fingerprinting</li></ul><p>…can all influence whether the edge accepts or blocks the call.</p><h3>Repo</h3><p>I put together a minimal .NET 8 repro for this pattern:</p><p>👉 <a href="https://github.com/mcansener/httpclient-403-demo">https://github.com/mcansener/httpclient-403-demo</a></p><p>This repo is intentionally <strong>not a tutorial toy</strong>. It behaves like an engineering investigation:</p><ul><li>a plain HttpClient request gets rejected with 403</li><li>a hardened request using SocketsHttpHandler succeeds with 200</li><li>console output shows protocol, headers, and block reason</li></ul><h3>The Symptom</h3><p>The simplest possible client looked harmless:</p><pre>internal sealed class BadHttpClientExample<br>{<br>    public async Task ExecuteAsync(Uri targetUri, CancellationToken cancellationToken = default)<br>    {<br>        using var client = new HttpClient<br>        {<br>            Timeout = TimeSpan.FromSeconds(15)<br>        };<br><br>        using var response = await client.GetAsync(<br>            targetUri,<br>            HttpCompletionOption.ResponseHeadersRead,<br>            cancellationToken);<br><br>        await Program.WriteResponseAsync(&quot;BAD REQUEST&quot;, targetUri, response, cancellationToken);<br>    }<br>}</pre><p>Nothing looks obviously wrong.<br> That is exactly what makes this class of issue frustrating in production.</p><ul><li>The request is valid</li><li>DNS works</li><li>TLS may succeed</li></ul><p>But the request <strong>does not look trustworthy to the edge</strong>.</p><p>Example output:</p><pre>=== BAD REQUEST ===<br>Status: 403<br>Failure: request was rejected by the edge policy simulator.</pre><h3>What the Request Was Missing</h3><p>The rejected request lacked signals that modern edge systems often expect:</p><ul><li>no meaningful User-Agent</li><li>no explicit Accept</li><li>no compression negotiation</li><li>no explicit keep-alive</li><li>no enforced HTTP version</li></ul><p>Individually, these might pass.</p><p>Together, they often do not.</p><h3>Why 403 Does Not Always Mean “Auth Failed”</h3><p>When developers see 403, the instinct is:</p><p><em>“Token is wrong” or “permission issue”</em></p><p>Sometimes true. Often wrong.</p><p>With CDNs or WAFs, requests can be rejected <strong>before your application even sees them</strong>.</p><p>That decision can depend on:</p><ul><li>HTTP/2 vs HTTP/1.1</li><li>header presence</li><li>Accept-Encoding</li><li>connection reuse patterns</li><li>TLS / ALPN negotiation</li><li>bot detection rules</li></ul><p>So the request may be blocked as <strong>suspicious</strong>, not unauthorized.</p><h3>Making the Failure Reproducible</h3><p>To avoid relying on a random external service, the sample spins up a <strong>local edge-policy simulator</strong>.</p><pre>app.MapGet(&quot;/edge-protected/resource&quot;, (HttpContext context) =&gt;<br>{<br>    var evaluation = EdgePolicyEvaluation.From(context);<br><br>    context.Response.Headers[&quot;Cache-Control&quot;] = &quot;no-store&quot;;<br>    context.Response.Headers[&quot;X-Edge-Decision&quot;] = evaluation.Decision;<br>    context.Response.Headers[&quot;X-Observed-Protocol&quot;] = evaluation.ObservedProtocol;<br><br>    if (evaluation.BlockReasons.Count &gt; 0)<br>    {<br>        context.Response.Headers[&quot;X-Block-Reason&quot;] = &quot;request-fingerprint-mismatch&quot;;<br>        return Results.Json(evaluation, statusCode: StatusCodes.Status403Forbidden);<br>    }<br><br>    return Results.Json(evaluation, statusCode: StatusCodes.Status200OK);<br>});</pre><h3>Policy Rules</h3><p>The simulated edge rejects requests that:</p><ul><li>lack a browser-like User-Agent</li><li>lack a meaningful Accept</li><li>do not negotiate compression</li><li>do not match expected protocol behavior</li></ul><h3>The Fix</h3><p>The solution is not “just add a User-Agent.”</p><p>It’s a <strong>coherent transport profile</strong>:</p><pre>internal sealed class FixedHttpClientExample<br>{<br>    public async Task ExecuteAsync(Uri targetUri, CancellationToken cancellationToken = default)<br>    {<br>        using var handler = new SocketsHttpHandler<br>        {<br>            AutomaticDecompression = DecompressionMethods.GZip |<br>                                     DecompressionMethods.Deflate |<br>                                     DecompressionMethods.Brotli,<br>            PooledConnectionLifetime = TimeSpan.FromMinutes(2)<br>        };<br><br>        using var client = new HttpClient(handler)<br>        {<br>            Timeout = TimeSpan.FromSeconds(15),<br>            DefaultRequestVersion = HttpVersion.Version11,<br>            DefaultVersionPolicy = HttpVersionPolicy.RequestVersionExact<br>        };<br><br>        using var request = new HttpRequestMessage(HttpMethod.Get, targetUri)<br>        {<br>            Version = HttpVersion.Version11,<br>            VersionPolicy = HttpVersionPolicy.RequestVersionExact<br>        };<br><br>        request.Headers.UserAgent.ParseAdd(<br>            &quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 &quot; +<br>            &quot;(KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36&quot;);<br><br>        request.Headers.Accept.ParseAdd(<br>            &quot;text/html,application/xhtml+xml,application/xml;q=0.9,&quot; +<br>            &quot;application/json;q=0.8,*/*;q=0.7&quot;);<br><br>        request.Headers.Connection.Add(&quot;keep-alive&quot;);<br><br>        using var response = await client.SendAsync(<br>            request,<br>            HttpCompletionOption.ResponseHeadersRead,<br>            cancellationToken);<br><br>        await Program.WriteResponseAsync(&quot;FIXED REQUEST&quot;, targetUri, response, cancellationToken);<br>    }<br>}</pre><h3>Why Forcing HTTP/1.1 Matters</h3><p>Normally, downgrading HTTP is not recommended.</p><p>But this is not about preference — it’s about <strong>diagnostics</strong>.</p><p>If:</p><ul><li>browser works</li><li>service client fails</li></ul><p>Then the real question is:</p><p><em>“What makes this request look different to the edge?”</em></p><p>Sometimes:</p><ul><li>headers are the issue</li><li>sometimes compression</li><li>sometimes HTTP/2 behavior</li></ul><p>Forcing HTTP/1.1 gives you a <strong>controlled baseline</strong>.</p><h3>Before vs After</h3><pre>=== BAD REQUEST ===<br>Status: 403<br>=== FIXED REQUEST ===<br>Status: 200</pre><p>The response also includes server-side evaluation explaining <strong>why</strong> it was blocked.</p><h3>The Real Engineering Takeaway</h3><p>When HttpClient returns 403, don’t assume business logic.</p><p>Start with transport:</p><ol><li>Compare browser vs service requests</li><li>Inspect HTTP version</li><li>Check Accept-Encoding</li><li>Add realistic headers</li><li>Review connection reuse</li><li>Test HTTP/1.1 vs HTTP/2</li></ol><p>If the target is behind a CDN or WAF, this can be the entire issue.</p><h3>Final Thought</h3><p>This wasn’t an auth bug.</p><p>It was a <strong>protocol-level mismatch</strong>.</p><p>And those are the ones that waste the most time — unless you know where to look.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=997a6e0d03be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Powerful Search Index in MongoDB Atlas]]></title>
            <link>https://medium.com/@mcansener/building-a-powerful-search-index-in-mongodb-atlas-9ef4859e4360?source=rss-332f518a15da------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ef4859e4360</guid>
            <category><![CDATA[index]]></category>
            <category><![CDATA[mongo]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[John Sener]]></dc:creator>
            <pubDate>Tue, 17 Jun 2025 13:00:22 GMT</pubDate>
            <atom:updated>2025-06-17T13:00:22.586Z</atom:updated>
            <content:encoded><![CDATA[<p>Whether you’re building a real estate platform, an e-commerce search, or a location-based listing system, <strong>MongoDB Atlas Search</strong> can significantly streamline your querying capabilities. With its native integration of Lucene, it brings full-text search, geo-search, filtering, and ranking — without requiring a separate search engine like Elasticsearch.</p><p>In this article, I’ll walk through how I structured a MongoDB Atlas Search index for a listings platform, using mocked field names to keep it general. The focus is on designing an index that supports common real-world queries: filtering by location, price, bedrooms, status, and more — while keeping performance high.</p><p><strong>🧱 Index Definition Overview</strong></p><p>Here’s a sample of the search index definition I used (simplified and anonymized):</p><pre>{<br>  &quot;mappings&quot;: {<br>    &quot;dynamic&quot;: false,<br>    &quot;fields&quot;: {<br>      &quot;location&quot;: {<br>        &quot;type&quot;: &quot;geo&quot;<br>      },<br>      &quot;lat&quot;: {<br>        &quot;type&quot;: &quot;number&quot;<br>      },<br>      &quot;lng&quot;: {<br>        &quot;type&quot;: &quot;number&quot;<br>      },<br>      &quot;price&quot;: {<br>        &quot;type&quot;: &quot;number&quot;<br>      },<br>      &quot;address&quot;: {<br>        &quot;type&quot;: &quot;string&quot;,<br>        &quot;analyzer&quot;: &quot;lucene.keyword&quot;<br>      },<br>      &quot;bedrooms&quot;: {<br>        &quot;type&quot;: &quot;number&quot;<br>      },<br>      &quot;updatedAt&quot;: {<br>        &quot;type&quot;: &quot;date&quot;<br>      },<br>      &quot;createdAt&quot;: {<br>        &quot;type&quot;: &quot;date&quot;<br>      },<br>      &quot;city&quot;: {<br>        &quot;type&quot;: &quot;string&quot;,<br>        &quot;analyzer&quot;: &quot;lucene.keyword&quot;<br>      },<br>      &quot;region&quot;: {<br>        &quot;type&quot;: &quot;string&quot;,<br>        &quot;analyzer&quot;: &quot;lucene.keyword&quot;<br>      },<br>      &quot;state&quot;: {<br>        &quot;type&quot;: &quot;string&quot;,<br>        &quot;analyzer&quot;: &quot;lucene.keyword&quot;<br>      },<br>      &quot;postal&quot;: {<br>        &quot;type&quot;: &quot;string&quot;,<br>        &quot;analyzer&quot;: &quot;lucene.keyword&quot;<br>      },<br>      &quot;calculations&quot;: {<br>        &quot;type&quot;: &quot;document&quot;,<br>        &quot;dynamic&quot;: true<br>      }<br>    }<br>  }<br>}</pre><h4>🔍 Field-by-Field Breakdown</h4><p>Let’s explore the design decisions behind this schema.</p><h4>Location — Geo Point Search</h4><pre>&quot;location&quot;: {<br>  &quot;type&quot;: &quot;geo&quot;<br>}</pre><p>This field is critical for location-based filtering or sorting (e.g., “properties within 5 miles of X”). It uses MongoDB’s geospatial capabilities to power queries like $geoWithin or $near.</p><h4>🏷️ Keyword Fields</h4><p>Fields like id, status, source, city, state, and postal use:</p><pre>&quot;type&quot;: &quot;string&quot;,<br>&quot;analyzer&quot;: &quot;lucene.keyword&quot;</pre><p>Using the lucene.keyword analyzer ensures <strong>exact matching</strong>. This is important for filters (e.g., status = &quot;active&quot;, or city = &quot;London&quot;). It disables full-text tokenization and ensures that the search matches the entire value.</p><h4>Numeric Fields</h4><p>Fields like price, bedrooms, bathrooms, area, lat, and lng are defined as:</p><pre>&quot;type&quot;: &quot;number&quot;</pre><p>This allows you to build <strong>range-based filters</strong> (e.g., “price between $300K and $500K” or “at least 2 bathrooms”). You can also use these fields for sorting results.</p><h4>📆 Date Fields</h4><pre>&quot;type&quot;: &quot;date&quot;</pre><p>Useful for filtering or sorting listings based on recency (updatedAt, createdAt), or applying business logic like &quot;show only listings added in the last 30 days&quot;.</p><h4>🌐 Address Fields</h4><p>Fields such as address, region, and postal are keyword strings, which allows filtering without fuzzy matching. These could be enhanced later with full-text analyzers for free-form searches.</p><h3>🧮 Embedded Object</h3><pre>&quot;calculations&quot;: {<br>  &quot;type&quot;: &quot;document&quot;,<br>  &quot;dynamic&quot;: true<br>}</pre><p>This section is reserved for calculated metrics or dynamic fields that don’t follow a fixed schema. Setting dynamic: true allows MongoDB to index subfields as they appear, which is helpful when structure varies per document.</p><h4>🚨 Why dynamic: false at the Root?</h4><p>We set the root dynamic to false to <strong>avoid accidentally indexing fields</strong> we don’t intend to search on. This helps reduce index bloat and keeps search performance optimal.</p><p>Only explicitly defined fields are indexed. You can allow dynamic behavior selectively (like we did in calculations) when flexibility is necessary.</p><h4>✅ Final Tips</h4><ul><li>Use <strong>keyword analyzers</strong> for exact filters.</li><li>Use <strong>geo fields</strong> for spatial queries.</li><li>Index only what’s needed. Let the rest live outside of the search index.</li><li>When necessary, combine Atlas Search with MongoDB’s native filters for hybrid queries.</li></ul><h4>🔚 Conclusion</h4><p>MongoDB Atlas Search provides a powerful toolkit for building efficient and scalable search experiences directly on top of your MongoDB collections. With well-structured index definitions, you can support fast geo queries, range filtering, and complex search logic — all without leaving the MongoDB ecosystem.</p><p>If you’re planning to implement real-time search on user-facing platforms, this is a technology worth exploring. The key is to <strong>know your queries first</strong>, and build your index to serve them efficiently.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ef4859e4360" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficiently Inserting Large Datasets into SQL Server]]></title>
            <link>https://medium.com/@mcansener/efficiently-inserting-large-datasets-into-sql-server-cc944d167589?source=rss-332f518a15da------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc944d167589</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[insert]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[John Sener]]></dc:creator>
            <pubDate>Tue, 29 Oct 2024 07:12:48 GMT</pubDate>
            <atom:updated>2024-10-29T07:12:48.739Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k2LOuH02OIPZkX20" /></figure><p>When working with massive datasets, such as millions of rows, inserting data efficiently into SQL Server can be challenging. Inserting a large dataset requires careful planning to avoid performance bottlenecks, transaction log issues, and excessive memory usage. This article covers the best approaches to efficiently insert large datasets into SQL Server, using strategies such as batch insertion, the BULK INSERT command, and other optimization techniques.</p><h4>Why Efficient Data Insertion Matters</h4><p>Large data insertions can strain system resources and may lead to timeouts, transaction log overflow, and even table locking issues that impact overall server performance. Optimizing large data inserts helps ensure that your server remains responsive and data operations are completed without unnecessary overhead.</p><h3>Key Techniques for Efficient Data Insertion</h3><h4>1. Batch Insertion</h4><p>Batch insertion involves breaking down a large dataset into smaller chunks, or batches, and inserting each batch sequentially. This approach reduces the risk of timeouts, excessive transaction log usage, and overall memory strain.</p><p><strong>Example in SQL Server:</strong> Using a loop to insert data in batches:</p><pre>DECLARE @BatchSize INT = 10000; -- Set batch size<br>DECLARE @Start INT = 1;<br>DECLARE @End INT = @BatchSize;<br><br>WHILE @Start &lt;= 16000000<br>BEGIN<br>    -- Insert data in batches<br>    INSERT INTO YourTable (Column1, Column2, ...)<br>    SELECT Column1, Column2, ...<br>    FROM YourSourceTable<br>    WHERE Id BETWEEN @Start AND @End;<br><br>    -- Move to the next batch<br>    SET @Start = @End + 1;<br>    SET @End = @End + @BatchSize;<br>END</pre><p>This method avoids loading all 16 million rows in one operation and reduces transaction size, making it easier to handle in memory.</p><h4>2. Using BULK INSERT for High-Performance Data Loading</h4><p>BULK INSERT is one of the fastest methods to load large datasets into SQL Server. This approach requires exporting your data to a file (such as CSV or TSV) and then using the BULK INSERT command to load the data directly into your SQL Server table.</p><p><strong>Steps for BULK INSERT:</strong></p><ol><li><strong>Prepare the Data File</strong><br>Export your data to a CSV file that matches your SQL Server table’s structure.</li><li><strong>Place the Data File on the SQL Server Machine</strong><br>Copy the file to a directory that SQL Server can access (e.g., C:\Data\YourFile.csv).</li><li><strong>Run BULK INSERT Command</strong><br>Use the following BULK INSERT command to load the data:</li></ol><pre>BULK INSERT YourTable<br>FROM &#39;C:\Data\YourFile.csv&#39; -- Specify the file path<br>WITH (<br>    FIELDTERMINATOR = &#39;,&#39;, -- Column delimiter (change if needed)<br>    ROWTERMINATOR = &#39;\n&#39;, -- Row delimiter (use newline)<br>    FIRSTROW = 2, -- Skip header row if present<br>    BATCHSIZE = 50000 -- Batch size to manage memory use<br>);</pre><ul><li><strong>FIELDTERMINATOR</strong> and <strong>ROWTERMINATOR</strong> allow customization based on your file format.</li><li><strong>BATCHSIZE</strong> controls the number of rows processed at once. Adjust it based on your server’s memory capacity.</li></ul><h4>3. Using SSIS for Large Data Loads</h4><p>SQL Server Integration Services (SSIS) is another efficient method for loading large datasets. SSIS provides advanced data transformation and load options, handling errors and retries automatically, and managing resource usage.</p><ol><li><strong>Create an SSIS Package</strong><br>Use the SSIS designer to create a data flow that connects to your source file or database and loads data into the SQL Server table.</li><li><strong>Configure Batch Size</strong><br>In SSIS, set the Rows per Batch and Maximum Insert Commit Size properties to control the size of each batch.</li><li><strong>Execute the Package</strong><br>SSIS optimizes performance with parallel processing, logging, and error handling options, making it suitable for high-volume data loads.</li></ol><h4>4. Optimize SQL Server Settings</h4><p>SQL Server has configuration options that help improve performance for large data loads. These options are beneficial when performing a one-time or infrequent large insert, but use them cautiously on production databases.</p><ul><li><strong>Disable Indexes Temporarily</strong>: Disable indexes on the target table before inserting data and then rebuild them after the insert completes. This reduces the overhead of maintaining indexes during insertion.</li></ul><pre>ALTER INDEX ALL ON YourTable DISABLE;<br>-- Perform data insert<br>ALTER INDEX ALL ON YourTable REBUILD;</pre><ul><li><strong>Disable Constraints</strong>: If foreign keys or other constraints are present, consider disabling them temporarily and re-enabling them after data insertion to improve speed.</li><li><strong>Set Recovery Model to Bulk-Logged</strong>: Switch the recovery model to <strong>Bulk-Logged</strong> for the duration of the data load. This reduces logging requirements and can improve performance. Remember to set it back to <strong>Full</strong> afterward if you require full recovery.</li></ul><pre>ALTER DATABASE YourDatabase SET RECOVERY BULK_LOGGED;<br>-- Perform data load<br>ALTER DATABASE YourDatabase SET RECOVERY FULL;</pre><h3>Practical Example: Loading 16 Million Rows</h3><p>Here’s how a complete example might look when using both BULK INSERT and batch processing for maximum efficiency:</p><ol><li>Export the dataset into a CSV file, then move it to the SQL Server machine.</li><li>Use BULK INSERT with batch processing to load the data, making use of temporary table disabling and bulk-logged recovery mode.</li></ol><pre>-- Switch to bulk-logged recovery mode<br>ALTER DATABASE YourDatabase SET RECOVERY BULK_LOGGED;<br><br>-- Disable indexes to speed up insert<br>ALTER INDEX ALL ON YourTable DISABLE;<br><br>-- Perform BULK INSERT<br>BULK INSERT YourTable<br>FROM &#39;C:\Data\LargeDataFile.csv&#39;<br>WITH (<br>    FIELDTERMINATOR = &#39;,&#39;,<br>    ROWTERMINATOR = &#39;\n&#39;,<br>    FIRSTROW = 2,<br>    BATCHSIZE = 100000 -- Adjust based on server capacity<br>);<br>-- Rebuild indexes and revert recovery mode<br>ALTER INDEX ALL ON YourTable REBUILD;<br>ALTER DATABASE YourDatabase SET RECOVERY FULL;</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc944d167589" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Quickly Become an Expert in Any Programming Language]]></title>
            <link>https://medium.com/@mcansener/how-to-quickly-become-an-expert-in-any-programming-language-88eb13425fc7?source=rss-332f518a15da------2</link>
            <guid isPermaLink="false">https://medium.com/p/88eb13425fc7</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[money]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[how-to]]></category>
            <dc:creator><![CDATA[John Sener]]></dc:creator>
            <pubDate>Tue, 17 Sep 2024 06:18:37 GMT</pubDate>
            <atom:updated>2024-09-17T06:25:20.435Z</atom:updated>
            <content:encoded><![CDATA[<p>TOP 1 RULE</p><p>There is no quick way. You had to work long hours, days and nights. If someone is telling you that there is a simple way, it is bullshit.</p><p>Now, go back to work, you lazy ass!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88eb13425fc7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Draw on Google Maps with JavaScript]]></title>
            <link>https://blog.devgenius.io/how-to-draw-on-google-maps-with-javascript-791efea35ef6?source=rss-332f518a15da------2</link>
            <guid isPermaLink="false">https://medium.com/p/791efea35ef6</guid>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[drawing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[John Sener]]></dc:creator>
            <pubDate>Thu, 08 Aug 2024 22:24:00 GMT</pubDate>
            <atom:updated>2024-08-09T19:05:45.906Z</atom:updated>
            <content:encoded><![CDATA[<p>Although Google Maps is a strong tool in itself, it can be further enhanced with custom features coded in JavaScript. Drawing on the map is one such feature. This article will explain how users could code in JavaScript to enable themselves to draw polylines on a Google Map.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fdFJUcj2zP9G5dMwpOCeuA.png" /></figure><p><strong>Setting Up HTML and CSS</strong><br>The first part of our code sets up HTML and CSS. This sets up an environment where the Google Map is embedded and provides a few buttons for user interaction.</p><pre>@{<br>    ViewData[&quot;Title&quot;] = &quot;Drawing on Google Maps&quot;;<br>}<br>&lt;style&gt;<br>    #map {<br>        height: 100%;<br>    }<br><br>    html, body {<br>        height: 100%;<br>        margin: 0;<br>        padding: 0;<br>    }<br>&lt;/style&gt;<br><br>&lt;div id=&quot;map&quot; style=&quot;height: 500px;&quot;&gt;&lt;/div&gt;<br>&lt;button id=&quot;draw-button&quot; onclick=&quot;enableDrawing()&quot;&gt;Enable Drawing&lt;/button&gt;<br>&lt;button id=&quot;disable-button&quot; onclick=&quot;disableDrawing()&quot;&gt;Disable Drawing&lt;/button&gt;<br>&lt;button onclick=&quot;deleteAllShapes()&quot;&gt;Delete All Shapes&lt;/button&gt;<br>&lt;button onclick=&quot;getPolylineCoordinates()&quot;&gt;Get Line Coordinates&lt;/button&gt;</pre><ul><li>Full-Screen Map CSS The following &lt;style&gt; section ensures the map occupies the full height of the screen by setting the height of the html, body and #map elements to 100%. Here, I have put the actual height of the #map div to 500px, but you can change it as per your need.</li><li><strong>Buttons to Interact:</strong> Below the map div, we have a few buttons. These buttons call different JavaScript functions that would enable or disable drawing, delete all shapes, and get the coordinates for drawn shapes.</li></ul><p><strong>Integrating the Google Maps API</strong><br>Next, we add in the Google Maps JavaScript API along with the proper libraries to be used for drawing and geometry:</p><pre>&lt;script src=&quot;https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&amp;libraries=drawing,geometry&quot;&gt;&lt;/script&gt;</pre><ul><li><strong>API Key</strong>: The literarily explained part is to replace YOUR_API_KEY with your actual Google Maps API key.</li><li><strong>Libraries</strong>: The libraries=drawing,geometry attribute includes the Drawing and Geometry libraries, which our functionality depends on.</li></ul><p><strong>Initialize the Map and Drawing Functionality</strong><br>Now let’s do the main part of our code — the map initialization and drawing logic.</p><pre>&lt;script&gt;<br>    let map;<br>    let drawingManager;<br>    let polylines = []; // To store all drawn polylines<br>    let isDrawing = false;<br>    let isMouseDown = false;<br>    let currentPolyline;<br>    let currentPath;<br><br>    function initMap() {<br>        map = new google.maps.Map(document.getElementById(&quot;map&quot;), {<br>            center: { lat: 37.7749, lng: -122.4194 },<br>            zoom: 13,<br>        });<br><br>        drawingManager = new google.maps.drawing.DrawingManager({<br>            drawingMode: null,<br>            drawingControl: false,<br>            polylineOptions: {<br>                editable: false,<br>                draggable: false,<br>                strokeColor: &#39;#FF0000&#39;,<br>                strokeOpacity: 0.8,<br>                strokeWeight: 2,<br>            },<br>        });<br><br>        map.addListener(&#39;mousedown&#39;, function (event) {<br>            if (isDrawing) {<br>                isMouseDown = true;<br>                if (!currentPolyline) {<br>                    currentPolyline = new google.maps.Polyline({<br>                        map: map,<br>                        path: [event.latLng],<br>                        strokeColor: &#39;#FF0000&#39;,<br>                        strokeOpacity: 0.8,<br>                        strokeWeight: 2,<br>                        editable: false,<br>                        draggable: false,<br>                    });<br>                    currentPath = currentPolyline.getPath();<br>                } else {<br>                    currentPath.push(event.latLng);<br>                }<br>                map.setOptions({ draggable: false });<br>            }<br>        });<br><br>        map.addListener(&#39;mousemove&#39;, function (event) {<br>            if (isDrawing &amp;&amp; isMouseDown &amp;&amp; currentPolyline) {<br>                currentPath.push(event.latLng);<br>            }<br>        });<br><br>        map.addListener(&#39;mouseup&#39;, function (event) {<br>            if (isDrawing) {<br>                isMouseDown = false;<br>                map.setOptions({ draggable: true });<br>                if (currentPath &amp;&amp; currentPath.getLength() &gt; 1) {<br>                    currentPath.push(currentPath.getAt(0));<br>                }<br>                if (currentPolyline) {<br>                    polylines.push(currentPolyline);<br>                }<br>                currentPolyline = null;<br>                currentPath = null;<br>            }<br>        });<br><br>        drawingManager.setMap(map);<br>    }<br><br>    function enableDrawing() {<br>        isDrawing = true;<br>        currentPolyline = null;<br>        currentPath = null;<br>    }<br><br>    function disableDrawing() {<br>        isDrawing = false;<br>        isMouseDown = false;<br>        map.setOptions({ draggable: true });<br>    }<br><br>    function deleteAllShapes() {<br>        for (let i = 0; i &lt; polylines.length; i++) {<br>            polylines[i].setMap(null);<br>        }<br>        polylines = [];<br>        currentPolyline = null;<br>        currentPath = null;<br>    }<br><br>    function getPolylineCoordinates() {<br>        const allCoordinates = polylines.map(polyline =&gt; {<br>            return polyline.getPath().getArray().map(coord =&gt; {<br>                return { lat: coord.lat(), lng: coord.lng() };<br>            });<br>        });<br>        console.log(JSON.stringify(allCoordinates));<br>        alert(JSON.stringify(allCoordinates));<br>    }<br><br>    initMap();<br>&lt;/script&gt;</pre><ul><li><strong>Variables</strong>: A few variables are declared that will hold the map instance, drawing manager, drawn polylines and flags to keep track of the drawing state.</li><li><strong>initMap function</strong>: This function initializes a Google Map and sets event listeners for mouse events. Upon mouse button press create a Polyline, on mouse move add points to the Polyline, on mouse up finalize the drawing. Close the polyline by connecting first and last point.</li><li><strong>Enable Drawing/Disable Drawing</strong>: The enableDrawing function simply sets the isDrawing flag to true to allow drawing. Conversely, the disableDrawing function sets this flag to false, effectively stopping drawing.</li><li><strong>Clearing All Shapes</strong>: The deleteAllShapes function iterates through the polylines array to remove each Polyline from the map.</li><li><strong>Get Polyline Coordinates</strong>: The getPolylineCoordinates function will retrieve the coordinates of all drawn polylines and alert them.</li></ul><p><strong>Conclusion</strong><br>This implementation provides a simple way for drawing shapes on a Google Map, managing those shapes, and retrieving their coordinates. In case you understand the key parts of this code, further customization and extending of such functionality is most easily done.</p><p>Whether for a mapping application or merely to have a tinker with the Google Maps API, this code forms a very good base for adding interactive drawing features to your project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=791efea35ef6" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/how-to-draw-on-google-maps-with-javascript-791efea35ef6">How to Draw on Google Maps with JavaScript</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing SQL Queries for Better Performance : 1 million records]]></title>
            <link>https://blog.stackademic.com/optimizing-sql-queries-for-better-performance-1-million-records-e9bbd0bf7ee1?source=rss-332f518a15da------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9bbd0bf7ee1</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[John Sener]]></dc:creator>
            <pubDate>Tue, 06 Aug 2024 05:36:46 GMT</pubDate>
            <atom:updated>2024-08-06T05:43:11.153Z</atom:updated>
            <content:encoded><![CDATA[<h3>Optimizing SQL Queries for Better Performance: 1 million records</h3><p>We have already created a scenario to optimize our query in this article : <a href="https://blog.devgenius.io/optimizing-sql-queries-for-better-performance-ad9ccefdb0c0">https://blog.devgenius.io/optimizing-sql-queries-for-better-performance-ad9ccefdb0c0</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/0*xSkiF6MmyKdnS4MJ.png" /></figure><p>Following the advice in the article, we chose to try with 1 million-record dataset. Below is what we got from our first tests.</p><p>Now, to simulate a problem situation, we should create 1 million records in our database. Here is the procedure:</p><pre>-- Create the Products table<br>CREATE TABLE Products (<br>    ProductID INT IDENTITY(1,1) PRIMARY KEY,<br>    ProductName NVARCHAR(100),<br>    Price DECIMAL(10, 2)<br>);<br><br>-- Insert 1 million data to increase the table size<br>DECLARE @j INT = 1;<br>WHILE @j &lt;= 1000000<br>BEGIN<br>    INSERT INTO Products (ProductName, Price)<br>    VALUES (&#39;Product &#39; + CAST(@j + 1026 AS NVARCHAR(10)), 10.00 + (RAND() * 1000));<br>    SET @j = @j + 1;<br>END;</pre><h4>Performance Metrics Gathering</h4><pre>-- Enable statistics<br>SET STATISTICS TIME ON;<br>SET STATISTICS IO ON;<br><br>-- Query 1: Suboptimal Query<br>PRINT &#39;Query 1: Suboptimal Query&#39;;<br>GO<br>SELECT TOP 10 * FROM Products ORDER BY Price DESC;<br>GO<br><br>-- Query 2: Optimized Query with ROW_NUMBER()<br>PRINT &#39;Query 2: Optimized Query&#39;;<br>GO<br>SELECT * <br>FROM (<br>    SELECT *, ROW_NUMBER() OVER (ORDER BY Price DESC) AS RowNum<br>    FROM Products<br>) AS RankedProducts<br>WHERE RowNum &lt;= 10;<br>GO<br><br>-- Disable statistics<br>SET STATISTICS TIME OFF;<br>SET STATISTICS IO OFF;</pre><p>And the results are :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g4GifLZq8LOSmhUpbPHyVg.png" /></figure><h3>Stackademic 🎓</h3><p>Thank you for reading until the end. Before you go:</p><ul><li>Please consider <strong>clapping</strong> and <strong>following</strong> the writer! 👏</li><li>Follow us <a href="https://twitter.com/stackademichq"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/stackademic"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/c/stackademic"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a></li><li>Visit our other platforms: <a href="https://plainenglish.io/"><strong>In Plain English</strong></a> | <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://stackademic.com/"><strong>Stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9bbd0bf7ee1" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/optimizing-sql-queries-for-better-performance-1-million-records-e9bbd0bf7ee1">Optimizing SQL Queries for Better Performance : 1 million records</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing SQL Queries for Better Performance]]></title>
            <link>https://blog.devgenius.io/optimizing-sql-queries-for-better-performance-ad9ccefdb0c0?source=rss-332f518a15da------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad9ccefdb0c0</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[John Sener]]></dc:creator>
            <pubDate>Sun, 23 Jun 2024 22:16:43 GMT</pubDate>
            <atom:updated>2024-08-06T05:40:20.686Z</atom:updated>
            <content:encoded><![CDATA[<p>SQL queries are crucial to making databases perform efficiently and quickly. In this post, we will take a look into the benefits of query optimization with one specific example that was done in our existing system and show how much optimized query helps us to get performance gain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*LgEHP7KVIp1KYbwe4Cly5g.png" /></figure><h4>The Scenario</h4><p>Let’s say that we want to fetch 10 most expensive products from a table called Products. At the beginning wrong query can be :</p><pre>SELECT TOP 10 * FROM Products ORDER BY Price DESC;</pre><p>Although simple, this query might not be very efficient if you have a large dataset. No, we can use the ROW_VALUE() to improve its performance:</p><pre>SELECT * <br>FROM (<br>    SELECT *, ROW_NUMBER() OVER (ORDER BY Price DESC) AS RowNum<br>    FROM Products<br>) AS RankedProducts<br>WHERE RowNum &lt;= 10;</pre><p>By optimising the query to enable ranking of products by price, there is no need for a full sort on the result set.</p><h4>Create the Products table, Insert data into it</h4><p>We will now create the Products table and insert some dummy data to mimic real-world use cases.</p><pre>-- Drop the table if it already exists<br>DROP TABLE IF EXISTS Products;<br><br>-- Create the Products table<br>CREATE TABLE Products (<br>    ProductID INT IDENTITY(1,1) PRIMARY KEY,<br>    ProductName NVARCHAR(100),<br>    Price DECIMAL(10, 2)<br>);<br><br>-- Insert sample data into the Products table<br>INSERT INTO Products (ProductName, Price)<br>VALUES<br>(&#39;Product A&#39;, 10.00),<br>(&#39;Product B&#39;, 20.00),<br>(&#39;Product C&#39;, 30.00),<br>(&#39;Product D&#39;, 40.00),<br>(&#39;Product E&#39;, 50.00),<br>(&#39;Product F&#39;, 60.00),<br>(&#39;Product G&#39;, 70.00),<br>(&#39;Product H&#39;, 80.00),<br>(&#39;Product I&#39;, 90.00),<br>(&#39;Product J&#39;, 100.00),<br>(&#39;Product K&#39;, 110.00),<br>(&#39;Product L&#39;, 120.00),<br>(&#39;Product M&#39;, 130.00),<br>(&#39;Product N&#39;, 140.00),<br>(&#39;Product O&#39;, 150.00),<br>(&#39;Product P&#39;, 160.00),<br>(&#39;Product Q&#39;, 170.00),<br>(&#39;Product R&#39;, 180.00),<br>(&#39;Product S&#39;, 190.00),<br>(&#39;Product T&#39;, 200.00);<br><br>-- Insert more sample data to increase the table size<br>DECLARE @j INT = 1;<br>WHILE @j &lt;= 100000<br>BEGIN<br>    INSERT INTO Products (ProductName, Price)<br>    VALUES (&#39;Product &#39; + CAST(@j + 1026 AS NVARCHAR(10)), 10.00 + (RAND() * 1000));<br>    SET @j = @j + 1;<br>END;</pre><h4>Performance Metrics Gathering</h4><p>By comparing the performance of these suboptimal and optimized queries, we can gather execution time metrics, logical reads consumed by a query or CPU time utilized.</p><pre>-- Enable statistics<br>SET STATISTICS TIME ON;<br>SET STATISTICS IO ON;<br><br>-- Query 1: Suboptimal Query<br>PRINT &#39;Query 1: Suboptimal Query&#39;;<br>GO<br>SELECT TOP 10 * FROM Products ORDER BY Price DESC;<br>GO<br><br>-- Query 2: Optimized Query with ROW_NUMBER()<br>PRINT &#39;Query 2: Optimized Query&#39;;<br>GO<br>SELECT * <br>FROM (<br>    SELECT *, ROW_NUMBER() OVER (ORDER BY Price DESC) AS RowNum<br>    FROM Products<br>) AS RankedProducts<br>WHERE RowNum &lt;= 10;<br>GO<br><br>-- Disable statistics<br>SET STATISTICS TIME OFF;<br>SET STATISTICS IO OFF;</pre><h3>Analyzing the Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K7uVhSG2XBS0LatHdmp03Q.png" /></figure><p>Query 1: Suboptimal Query SQL Server Execution TimesCPU time = 0 ms, elapsed time = 41 ms.Table ‘Products’. Scan count 12, logical reads 15202, physical reads 6., page server read-6039(Also reduced due to Buffer Pool),read-ahead sequential7992 lobtree_width3192.</p><p><strong>Query 2: Optimized Query SQL Server Execution Times:</strong> CPU time = 109 ms, elapsed time = <strong>11 ms</strong>. Table ‘Worktable’. Scan count 0, logical reads 0, physical reads 0. Table ‘Products’. Scan count 9, logical reads 861, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.</p><h4>Edit:</h4><p>Some of our readers suggested that we should try this on 1 million records. I did and the results are in this article : <a href="https://medium.com/@mcansener/optimizing-sql-queries-for-better-performance-1-million-records-e9bbd0bf7ee1">https://medium.com/@mcansener/optimizing-sql-queries-for-better-performance-1-million-records-e9bbd0bf7ee1</a></p><h4>Conclusion</h4><p>From the results, we observe that:</p><ul><li><strong>Elapsed Time:</strong> The optimized query (11 ms) is significantly faster than the suboptimal query (41 ms).</li><li><strong>Logical Reads:</strong> Both queries have the same number of logical reads (861), but the optimized query eliminates read-ahead reads.</li><li><strong>CPU Time:</strong> The optimized query has a higher CPU time due to the computation involved in ROW_NUMBER(), but the trade-off is justified by the reduced elapsed time.</li></ul><h4>Summary of Benefits</h4><ul><li><strong>Reduced Elapsed Time:</strong> The optimized query returns results faster, enhancing user experience.</li><li><strong>Efficient Data Handling:</strong> The optimized query avoids unnecessary read-ahead reads, beneficial for high-volume data scenarios.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad9ccefdb0c0" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/optimizing-sql-queries-for-better-performance-ad9ccefdb0c0">Optimizing SQL Queries for Better Performance</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Publishing Messages to RabbitMQ Using the HTTP API]]></title>
            <link>https://blog.devgenius.io/publishing-messages-to-rabbitmq-using-the-http-api-73c9d6ae8522?source=rss-332f518a15da------2</link>
            <guid isPermaLink="false">https://medium.com/p/73c9d6ae8522</guid>
            <category><![CDATA[rabbitmq]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[postman]]></category>
            <category><![CDATA[post]]></category>
            <dc:creator><![CDATA[John Sener]]></dc:creator>
            <pubDate>Fri, 21 Jun 2024 14:53:37 GMT</pubDate>
            <atom:updated>2024-08-01T12:05:21.060Z</atom:updated>
            <content:encoded><![CDATA[<p>RabbitMQ is an open-source message broker that is amazingly popular in enabling applications to talk to each other. Despite RabbitMQ using the Advanced Message Queuing Protocol as its means of internal communication, it is exposed to the outside world as an HTTP API to perform operations like management and message publishing. This article will walk you through publishing a message onto a RabbitMQ queue using the HTTP API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*r_P7vmRMqMpn0axB.png" /></figure><h4>Prerequisites</h4><ul><li>RabbitMQ installed and running with the Management Plugin turned on.</li><li>Basic knowledge about RabbitMQ concepts such as Exchanges, Queues and Bindings</li><li>Postman or any other tool which can send HTTP requests</li></ul><p>1. <strong>Set Up RabbitMQ</strong></p><p>Make sure RabbitMQ is running and that the Management Plugin is enabled. This plugin enables an HTTP-based API for RabbitMQ that by default, is available at port 15672.</p><p>2. <strong>Create an Exchange and Queue</strong></p><p>Suppose that you have an exchange named AppExchange, and a queue named OrderQueue.Processing in the virtual host AppVHost.</p><p>3. <strong>Using Postman to Publish a Message</strong></p><h4>Create a New Request</h4><ul><li>Open Postman and click on the “New” button.</li><li>Select “Request” to create a new HTTP request.</li></ul><h4>Configure the Request</h4><ul><li>The type of request to be used is a POST.</li><li>You can use the following URL:</li></ul><pre>http://localhost:15672/api/exchanges/AppVHost/AppExchange/publish</pre><h4>Set Authorization</h4><ul><li>Under the “Authorization” tab</li><li>Select “Basic Auth” as type</li><li>Add your RabbitMQ username and password — defaults are `guest` for both</li></ul><h4>Set Headers</h4><ul><li>Under the “Headers” tab</li><li>Add a new header with key `Content-Type` and value `application/json`</li></ul><h4>Set Body</h4><ul><li>Under the “Body” tab</li><li>Select “raw” and make sure it’s set to JSON format</li><li>Add the following JSON payload:</li></ul><pre>{<br>  &quot;properties&quot;: {},<br>  &quot;routing_key&quot;: &quot;OrderQueue.Processing&quot;,<br>  &quot;payload&quot;: &quot;Hello, RabbitMQ!&quot;,<br>  &quot;payload_encoding&quot;: &quot;string&quot;<br>}</pre><ul><li>Here, “routing_key”: “OrderQueue.Processing” indicates routing key, that is, your queue name.</li><li>“payload”: “Hello, RabbitMQ!” is the message you are sending.</li><li>“payload_encoding”: “string” This defines the payload as a plain string.</li><li>Lastly, hit the send button.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73c9d6ae8522" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/publishing-messages-to-rabbitmq-using-the-http-api-73c9d6ae8522">Publishing Messages to RabbitMQ Using the HTTP API</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Open MySQL for External Connections on Windows Server]]></title>
            <link>https://blog.devgenius.io/how-to-open-mysql-for-external-connections-on-windows-server-7c603a0c482b?source=rss-332f518a15da------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c603a0c482b</guid>
            <category><![CDATA[remote]]></category>
            <category><![CDATA[connection]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[mysql]]></category>
            <dc:creator><![CDATA[John Sener]]></dc:creator>
            <pubDate>Tue, 11 Jun 2024 10:35:52 GMT</pubDate>
            <atom:updated>2024-08-01T12:12:18.638Z</atom:updated>
            <content:encoded><![CDATA[<p>The following steps show how to open your MySQL Server to external connections on a Windows Server. This is the step-by-step process, from the configuration of MySQL itself to the configuration of firewall settings for secure and reliable access from other machines on the network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*Q0gaO_pW4-8sLqTK.png" /></figure><h4>Step 1: Configure MySQL to Listen on All IP Addresses</h4><p><strong>1.</strong>Find the MySQL Configuration File:</p><ul><li>By default, the MySQL configuration file, usually named my.ini, is located at C:\ProgramData\MySQL\MySQL Server X.Y\. (version of mysql like 8.0)</li></ul><p><strong>2.Edit the Configuration File:</strong></p><ul><li>Open my.ini in Notepad or any other text editor.</li><li>The [mysqld] section needs to be found.</li><li>The bind-address directive is located in this section and will be, by default, 127.0.0.1 for allowing only localhost connections.</li><li>Modify bind-address to 0.0.0.0 so that MySQL can bind on all available interfaces.</li></ul><pre>[mysqld]<br>bind-address = 0.0.0.0</pre><ul><li>Save and Exit the File.</li></ul><p><strong>3.Restart MySQL Service:</strong></p><ul><li>Using Win + R, open the Services application by typing services.msc and Enter.</li><li>In the opened list, find MySQL, right-click and select Restart.</li></ul><p><strong>NOTE: While using notepad to update the .ini file, you may get an error while trying to start again. This is because notepad adds hex characters while saving the file. You can use </strong><a href="https://hexed.it/"><strong>https://hexed.it/</strong></a><strong> to edit a file.<br>You can see the 3 chars at the beginning of the file, remove them and replace the file with correct one.</strong></p><h4>Step 2: Allow MySQL Through Windows Firewall</h4><p><strong>1.Open Windows Firewall Settings:</strong></p><ul><li>Go to Control Panel &gt; System and Security &gt; Windows Defender Firewall.</li></ul><p><strong>2.Create a New Inbound Rule:</strong></p><ul><li>Click on Advanced settings on left side.</li><li>In the Windows Defender Firewall with Advanced Security window, click Inbound Rules.</li><li>Click New Rule. on right side.</li></ul><p><strong>3.</strong>Create the Rule:</p><ul><li>Select Port and click Next.</li><li>Choose TCP and specify port 3306 (the default MySQL port).</li><li>Click Next.</li></ul><p><strong>4.Allow the Connection:</strong></p><ul><li>Select Allow the connection and click Next.</li></ul><p><strong>5.Select Profiles:</strong></p><ul><li>Choose the profiles this rule applies to (Domain, Private, Public) based on your network environment.</li><li>Click Next.</li></ul><p><strong>6.Name the Rule:</strong></p><ul><li>Give the rule a name, such as “MySQL”.</li><li>Click Finish.</li></ul><h4>Step 3: Create a MySQL User for Remote Access</h4><p><strong>1.Log in to MySQL:</strong></p><ul><li>Open up Command Prompt as an Administrator.</li><li>Go to the MySQL bin directory. It usually looks something like: C:\Program Files\MySQL\MySQL Server X.Y\bin</li><li>Open command prompt.</li><li>Log into MySQL with the root user:</li></ul><pre>mysql -u root -p</pre><p><strong>2.Create a Remote User:</strong></p><ul><li>Replace remote_user with your desired username and remote_password with a strong password.</li></ul><pre>CREATE USER &#39;remote_user&#39;@&#39;%&#39; IDENTIFIED BY &#39;remote_password&#39;;<br>GRANT ALL PRIVILEGES ON *.* TO &#39;remote_user&#39;@&#39;%&#39; WITH GRANT OPTION;<br>FLUSH PRIVILEGES;</pre><h4>Step 4: Verify and Test Remote Access</h4><p><strong>Test from Another Machine:</strong></p><ul><li>Use a MySQL client or command line to test the connection:</li></ul><pre>mysql -h your_server_ip -u remote_user -p</pre><ul><li>Replace your_server_ip with the IP address of your MySQL server.</li><li>OR you can use simple mysql workbench application to connect to the server.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c603a0c482b" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/how-to-open-mysql-for-external-connections-on-windows-server-7c603a0c482b">How to Open MySQL for External Connections on Windows Server</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>