<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maksim Lysenko on Medium]]></title>
        <description><![CDATA[Stories by Maksim Lysenko on Medium]]></description>
        <link>https://medium.com/@maksimlysenko?source=rss-2ad4d0e63bf8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*tQ6Op3162WsL5qqT</url>
            <title>Stories by Maksim Lysenko on Medium</title>
            <link>https://medium.com/@maksimlysenko?source=rss-2ad4d0e63bf8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 20:15:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maksimlysenko/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Built a Steam Game Recommender. The Collab Filter Kept Telling Everyone to Play GTA V.]]></title>
            <link>https://medium.com/@maksimlysenko/i-built-a-steam-game-recommender-the-collab-filter-kept-telling-everyone-to-play-gta-v-c070ae900e6e?source=rss-2ad4d0e63bf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c070ae900e6e</guid>
            <category><![CDATA[games]]></category>
            <category><![CDATA[recommendations]]></category>
            <category><![CDATA[steam]]></category>
            <dc:creator><![CDATA[Maksim Lysenko]]></dc:creator>
            <pubDate>Sun, 17 May 2026 15:56:26 GMT</pubDate>
            <atom:updated>2026-05-17T15:56:26.193Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why collaborative filtering inside hybrid recommenders mostly emits a popularity leaderboard, and the one-line fix that actually changes the output.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jKCb35ffaNFn925BD7zzLw.png" /></figure><p><strong>TL;DR.</strong> Most hybrid recommenders blend collaborative filtering and content similarity through a weighted sum. That blend keeps popularity bias in the output even after you bolt on the standard debiasing knobs. Switching the combination from sum to product — content * collab — plus a small logarithmic popularity penalty on top of that fixed seed-mode similarity in my project: <strong>+11% NDCG@10</strong>, <strong>+12.5% P@10</strong>, <strong>-28% average review count in the top 10</strong>, on a 113-seed labeled benchmark with three labelers. Live at <a href="https://imho.run/"><strong>imho.run</strong></a><strong> </strong>— no login needed for the part this article is about.</p><p>I gave my recommender three games: <em>Bannerlord</em>, <em>Total War: Warhammer 3</em>, and <em>Crusader Kings 3</em>. It came back with <em>Rising Storm 2: Vietnam</em>, <em>HELLDIVERS 2</em>, and <em>Counter-Strike 2</em>.</p><p>These are not similar games. These are popular games. The recommender didn’t know the difference.</p><p>The thing I’m optimizing isn’t “most similar.” A perfect <em>Bannerlord</em> clone wouldn’t be a useful recommendation — you’ve already played <em>Bannerlord</em>, that’s why you typed it in. It also isn’t “most popular,” because you’ve heard of those games already. What I actually want is a game you’d enjoy playing, which is a fuzzier third thing in the middle. Figuring out how to even measure that turned out to be most of the work.</p><p>A lot of naive hybrid recommenders drift toward popularity when you skip the calibration steps that production stacks usually wire in (popularity debiasing, two-stage retrieval, score calibration, multi-objective ranking — Spotify, Netflix, YouTube and similar shops keep a whole shelf of these techniques on standby). The single most impactful fix in my case turned out to be one line of code, gated_collab = collab * content, and once it was in, the <em>Bannerlord</em> seed returned <em>Mount &amp; Blade: Warband</em> — much closer to what a human curator would pick.</p><p>If you’ve ever built a content-similarity feature and watched it default to whatever’s trending this week, this is for you.</p><h2>The recommender that kept recommending GTA V</h2><p>The project is <a href="https://imho.run/">imho.run</a>, a Steam game recommender I’ve been building solo on a home VPS. It indexes about 96,000 Steam games and analyzes a couple of million player libraries. Architecture-wise it’s textbook: a hybrid recommender blending collaborative filtering with content-based scoring over genres, categories, Steam tags, developers, and publishers. The collab side started as a per-request bake-off between ALS, iALS, EASE, and LightGCN. EASE won by a clear margin on the labeled benchmark, so the abstraction layer letting me swap models is now scaffolding I should probably delete. On top of that there’s a quality signal, a novelty bonus, AI-extracted gameplay vibes from review text, and a “vibe graph” mined from reviewers’ name-drops of similar titles.</p><p>Library mode worked fine. Paste a public SteamID or sign in with Steam, point the recommender at everything you own, get a personalized ranking. Library mode has thousands of signal points; the recommender gets the answer right by brute force of evidence.</p><p>The problem started with <em>seed mode</em>. Seed mode is what 90% of visitors actually want: type two or three games you love, get a list of games like them. No library at all. Just “I liked these three, what else?”</p><p>This is also the head-term SEO query. Type “games like <em>Hades</em>” into Google and you get a million listicle results. “Games like <em>Project Zomboid</em>”, same. The query has volume and clear intent. A recommender that can’t ace it is a library-mode toy.</p><p>And mine was failing it.</p><p>Here are the actual outputs that broke me. Inputs in <strong>bold</strong>, outputs after the arrow:</p><ul><li><strong>Bannerlord</strong> → <em>Rising Storm 2: Vietnam</em>, <em>Squad</em>, <em>HELLDIVERS 2</em></li><li><strong>Project Zomboid + 7 Days to Die</strong> → <em>Counter-Strike 2</em>, <em>Rust</em>, <em>PUBG</em></li><li><strong>Stardew Valley</strong> → <em>Counter-Strike 2</em>, <em>Apex Legends</em>, <em>Rocket League</em></li><li><strong>Crusader Kings 3</strong> → <em>GTA V</em>, <em>Cyberpunk 2077</em>, <em>Elden Ring</em></li></ul><p>Every list collapses to the same six or seven games. The recommender is just outputting a popularity leaderboard with the seed games removed. The “similarity” signal is a placebo.</p><p>This happens regardless of which of the four collab models I have wired up is active — ALS, iALS, LightGCN, or EASE with its ridge regularizer. The structural problem lives in how the signals are combined downstream, not in which CF model you plug in.</p><p>Here’s why.</p><h2>What collaborative filtering actually measures</h2><p>There’s a piece of recommender-systems folklore that says “collaborative filtering captures latent taste signals beyond what explicit metadata can express.” That’s true in the same loose sense in which “human language captures latent meaning beyond what dictionary definitions express” is true — accurate, but it’s leaving out the part where the words still mean the dictionary thing first, and the latent stuff comes second.</p><p>In an item-item collaborative filter, the similarity between game A and game B reduces, ultimately, to how often A and B appear in the same library. The functional form varies. ALS factors a sparse user-item matrix into low-rank embeddings (typically with confidence weighting on observed entries). EASE solves a closed-form item-item ridge regression. Conditional probability counts co(A,B) / (popularity(A) + smoothing). The substrate is co-ownership.</p><p>Here’s what’s wrong with co-ownership as a similarity proxy: <strong>popular games co-own with everything.</strong></p><p>Suppose, hypothetically, <em>Counter-Strike 2</em> sits in a large fraction of Steam libraries — say one in three. Then no matter what your seed game is, there’s roughly a baseline one-in-three chance the next library you sample also owns <em>Counter-Strike 2</em>. Co-ownership with <em>CS2</em> tells you almost nothing about your seed; it tells you about <em>CS2</em>.</p><p>The naive collaborative filter, asked “what’s most similar to <em>Bannerlord</em>?”, picks the games whose embeddings have the highest dot product with <em>Bannerlord</em>’s. Those are the games most often co-owned with <em>Bannerlord</em>. And those are: popular games.</p><p>This isn’t a bug; it’s the loss function doing what you trained it to do. The signal you fed in was co-ownership, and co-ownership is dominated by popularity, so the output is too.</p><p>Smoothing helps a little. Many production CF systems include a popularity penalty in the form of an inverse-frequency term: co(A,B) / (popularity(A)^α * popularity(B)^β) with α, β somewhere around 0.5. EASE has an L2 ridge regularizer on its closed-form B matrix, which keeps any single item-item relationship from blowing up. Some implicit-ALS variants add explicit popularity reweighting on top of confidence weighting. These help at the margin; they don&#39;t solve the problem.</p><p>They don’t fully solve it because popularity isn’t really a separable <em>bias</em> you can subtract out — it’s a big chunk of the underlying signal you fed in. The reason <em>CS2</em> gets recommended for <em>Bannerlord</em> is genuinely “lots of people who own <em>Bannerlord</em> also own <em>CS2</em>”, which is correct but not actionable for the user who asked.</p><h2>The content-only counterpoint</h2><p>You can build a recommender that doesn’t have this problem. Use only content features. Genre, Steam tags, developer, publisher. <em>Bannerlord</em> carries tags like “Medieval”, “Open World”, “RPG”, “Strategy”, “Sandbox” plus a long tail of others; <em>Mount &amp; Blade: Warband</em> overlaps on almost all of them. Cosine similarity over tag-IDF vectors, done. <em>CS2</em> doesn’t appear in the output.</p><p>This is what every content-similarity feature on the internet does. And it has its own failure mode.</p><p>Steam tags are noisy. Tags like “Singleplayer”, “Steam Cloud”, “Steam Achievements” sit on tens of thousands of games and contribute nothing to similarity. Even after stripping ubiquitous noise, content similarity rewards generic-tag-overlap over creative kinship. <em>Bannerlord</em> and <em>Total War: Warhammer 3</em> share “Strategy”, “Medieval”, “Singleplayer”, “Multiplayer”, but the experiences are wildly different. <em>Bannerlord</em> puts you on the field with a single character through real-time combat and persistent progression. <em>Warhammer 3</em> is a turn-based campaign map with real-time tactical battles where you command armies, not a person. A reader who liked one isn’t a sure bet for the other, but cosine similarity over tags doesn’t see the gap.</p><p>Content also can’t see the thing that collaborative filtering <em>does</em> see: the patterns of co-engagement that aren’t captured by any tag. Why do people who like <em>Slay the Spire</em> also like <em>Hades</em>? Both are “Roguelike”, sure, but lots of roguelikes don’t co-own. The bond between those two specifically has something to do with run length, with the feel of incremental power, with the kind of player attention they reward. Steam tags don’t have words for that. Collaborative filtering does, encoded as a learned correlation in the embedding space.</p><p>So content-only loses information. Collab-only is dominated by popularity. The standard advice (<em>blend them, that’s what hybrid recommenders are for!</em>) is correct in spirit and disastrous in practice. A weighted sum like 0.5 * content + 0.5 * collab produces a recommender whose top-1 is <em>Bannerlord</em>&#39;s actual nearest neighbor in tag-space, whose top-2 is the second-nearest, and whose top-3 onward collapses to <em>CS2</em> and friends because the collab term saturates. The collab term, normalized across candidates, is uniformly high for popular games. Anywhere the content term is even slightly ambiguous, and content is <em>always</em> slightly ambiguous, popularity wins.</p><p>You need something stronger than a sum.</p><h2>The fix: multiplicative fusion instead of a weighted sum</h2><p>Each signal is good for a different thing.</p><p>Content tells you whether a game is <em>plausibly similar</em>. Collab tells you whether a game is <em>desirable conditional on similarity</em>. Those are different questions. A weighted sum treats them as the same question and lets each side compensate for the other. That’s the root failure.</p><p>Instead: use content as a gate.</p><pre>if explicit_seeds and has_content:<br>    # Collab scores are uniformly high for popular games (Rising Storm 2<br>    # collab=0.95 on Bannerlord seed) regardless of genre relevance.<br>    # Collab measures co-ownership patterns, not game similarity.<br>    # Multiply collab by content^1.0 so collab only contributes when<br>    # content already validates the candidate as plausibly similar.<br>    gated_collab = score.collaborative * score.content<br>else:<br>    gated_collab = score.collaborative</pre><p>This is a form of multiplicative late fusion: instead of summing the two signals, content acts as a soft gate and only candidates that clear it get a meaningful collab contribution. content * collab instead of 0.5 * content + 0.5 * collab. Popular-and-irrelevant games (high collab, low content) get crushed; niche-and-relevant games (low collab, high content) stay in the running; popular-and-relevant games stay on top, as they should.</p><p>The effect on the <em>Bannerlord</em> example: <em>Rising Storm 2: Vietnam</em> has near-zero content overlap with <em>Bannerlord</em> (modern shooter vs. medieval strategy), so its gated_collab drops to near zero and it falls out of the top 50. <em>Mount &amp; Blade: Warband</em> has near-perfect content overlap and meaningful collab (lots of <em>Bannerlord</em> players own it), so its gated_collab is high and it rises to #1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3NMt3xFfi8uj6-caMcCUWQ.png" /></figure><p>The before-and-after is jarring once you see it.</p><p>Bannerlord seeds:</p><p><strong>Before gate:</strong></p><ol><li>Rising Storm 2: Vietnam</li><li>Squad</li><li>HELLDIVERS 2</li><li>War Thunder</li><li>PUBG</li></ol><p><strong>After gate</strong> (actual current top-5, screenshot below):</p><ol><li>Mount &amp; Blade: Warband</li><li>Mount &amp; Blade</li><li>Han Xiongnu Wars</li><li>Arma 3</li><li>Battle Brothers</li></ol><p>The second list feels much closer to what a human curator would produce: three Mount &amp; Blade-family games on top, then medieval/tactical-RPG titles. Arma 3 at #4 is the edge case the gate doesn’t fully catch — Arma shares “Action” and “Open World” tags with <em>Bannerlord</em> plus enough military-historical co-ownership to keep its gated_collab non-trivial. That&#39;s the kind of residual the popularity penalty is supposed to grind down further; the result here is &quot;much better than the unblended baseline&quot;, not &quot;perfectly curated&quot;.</p><p>I went looking later for what the literature calls this. The closest name is “multiplicative late fusion” — scoring candidates by combining signals through a product rather than a sum. It isn’t a new idea — Burke’s 2002 hybrid-recommender survey already discusses weighted vs. switching vs. mixed designs, and production recommenders at Spotify, Netflix, and similar shops use multiplicative score combinations alongside two-stage retrieval, calibration, and explicit popularity debiasing. The frustrating bit is how rarely the gate-as-product framing shows up in the <em>intro</em> material I went through — most beginner-hybrids stop at the weighted sum and leave you to discover the structural fix on your own.</p><h2>Why a gate and not a hard filter</h2><p>The obvious follow-up: if content is the gate, why not just <em>filter</em> candidates by content similarity first and only rank the survivors? Set a content threshold, drop everything below it, then sort by collab.</p><p>I tried this. It’s worse.</p><p>The problem with a hard threshold is the threshold itself. Content similarity scores vary in shape depending on the seed. <em>Bannerlord</em>’s top content-similar games cluster around 0.7-0.9 cosine. <em>Stardew Valley</em>&#39;s cluster around 0.4-0.6 (the cozy-farming niche is small and Steam tags don&#39;t capture it well). A threshold tuned for <em>Bannerlord</em> over-filters <em>Stardew Valley</em>; a threshold tuned for <em>Stardew Valley</em> under-filters <em>Bannerlord</em>.</p><p>A multiplicative gate is <em>adaptive</em>. Candidates with content similarity 0.9 keep almost all of their collab score; candidates at 0.3 keep 30% of theirs; candidates at 0.05 keep almost none. The threshold isn’t a number you tune. It’s an implicit, per-seed function of where the content distribution happens to land. The gate stays calibrated across seeds.</p><p>It also preserves a useful property: a niche game with very high content similarity but mediocre collab (because there isn’t much co-ownership data, the game is small) still scores well. A hard threshold would let it through but it’d be ranked low because its collab term is small. The product 0.85 * 0.3 = 0.255 is small in absolute terms but still beats the popular-irrelevant noise that the gate kills outright.</p><h2>The anti-popularity penalty: catching what the gate misses</h2><p>The gate fixes the worst of it but leaves a residual: among the games that <em>do</em> pass the content bar, popular ones still beat less-popular ones, all else equal. <em>Crusader Kings 3</em> recommends both <em>Europa Universalis IV</em> and <em>Imperator: Rome</em>, both content-similar (Paradox grand strategy), but <em>EU IV</em> has 90,000 reviews to <em>Imperator</em>’s 7,000. <em>EU IV</em> wins the collab tie-break every time.</p><p>That’s fine, mostly. <em>EU IV</em> is the bigger, better-known game and it’s a defensible recommendation. But the residual stack-ranks all 6 Paradox grand-strategy titles in popularity order, which means <em>EU IV</em> is always #1, <em>Hearts of Iron IV</em> always #2, and so on for every grand-strategy seed. The recommender loses its ability to surface anything <em>except</em> the top of the Paradox catalog.</p><p>The second fix is a popularity penalty. After scoring, apply a small multiplicative drag proportional to log(reviews):</p><pre>pop_norm = math.log(reviews + 1) / _SEED_ANTI_POP_LOG_MAX  # ~14.0<br>score.final *= max(1.0 - seed_anti_pop_alpha * pop_norm, 0.0)</pre><p>With seed_anti_pop_alpha = 0.20, the most-popular games take roughly a 20% hit. <em>EU IV</em> (90k reviews, log≈11.4) loses about 16% of its score; <em>Imperator: Rome</em> (7k reviews, log≈8.9) loses about 13%; <em>Victoria 3</em> (35k reviews, log≈10.5) loses about 15%. The relative damping pushes the less-popular but equally-relevant titles up the list, without flipping their order. <em>EU IV</em> still ranks #1 most of the time, as it deserves. But now <em>Imperator</em> and <em>Victoria 3</em> sometimes rank ahead of <em>Hearts of Iron IV</em> when the seed is <em>CK3</em>, which is the right answer.</p><p>I calibrated alpha against a labeled benchmark of 113 seeds and three labelers, with NDCG@10 and P@10 as the targets. Two important caveats up front: the gate alone is what does the qualitative shift you see in the <em>Bannerlord</em> lists above — <em>Mount &amp; Blade: Warband</em> doesn&#39;t rise to #1 without it — but the benchmark gain I&#39;m about to quote is for <strong>gate + popularity-penalty stacked</strong> vs. a naive weighted-sum baseline, not for either in isolation. On this benchmark the two interventions are evaluated together because in practice they&#39;re always shipped together.</p><p>With both in place at alpha=0.20, NDCG@10 moved from 0.419 to 0.466 (+11%), P@10 moved by +12.5% in the same configuration, and the average review count of the top 10 recommendations dropped from ~142k to ~102k (-28%). Less mainstream and more accurate in the same shipped config.</p><p>The interesting bit is that alpha=0.20 is a hyperparameter, but it&#39;s not a sensitive one inside its useful range. The benchmark NDCG@10 is stable between 0.15 and 0.30. Below 0.15 the penalty gets too soft and popular titles leak back through; above 0.30 it starts eating recommendations the labelers liked. The implication is that the popularity penalty is doing real work in a narrow band, and the gate is doing the heavy lifting elsewhere.</p><h2>Things that didn’t work</h2><p>A few experiments that didn’t work, because the literature is full of them and I’d like to spare somebody else the time.</p><p><strong>MMR (Maximal Marginal Relevance) diversity reranking.</strong> The idea is to penalize candidates that are too similar to candidates already chosen. In practice MMR is great when the candidate pool is genuinely diverse and you want to surface variety. In a top-50 list of <em>Bannerlord</em>-similar games, the candidates are <em>already</em> clustered: they’re all medieval-or-fantasy strategy/RPG titles. MMR mostly pushes around the order of equally-relevant picks without solving the popularity problem, and introduces a new failure mode where the #2 recommendation is something genuinely weird because it’s “diverse from #1.” MMR is the wrong tool when the candidate pool is already topically clustered.</p><p><strong>Learning-to-rank with a separate model.</strong> I tried training an LTR model (LambdaMART on Steam-tag features + collab features + popularity) on a small labeled set. It generalized poorly. With ~100 seeds the LTR model had no chance to learn what “similarity” means across the full Steam catalog. To make this work you’d need 10x-100x the labels and probably a more structured feature space than I had.</p><p><strong>LLM reranking.</strong> Pipe the top 50 candidates through an LLM and ask it to pick the best 10. This works — the picks are good. Cost isn’t the blocker either: at my current volume (call it $0.0001 per request × roughly 100k requests/month) it’d run on the order of ten dollars a month. The actual blocker is latency. Adding several hundred milliseconds to every recommendation request is a different product. The gate is O(1) math on numbers I already have, sub-millisecond. I do still use an LLM in the pipeline, but only offline, to extract structured features from review text — there the latency budget is &quot;overnight&quot;, not &quot;before the page paints&quot;. That&#39;s a Phase 2 / Phase 3 story for another post.</p><h2>The takeaways</h2><p>If you’re building a similarity feature, for games, movies, music, articles, products, <em>anything</em> with sparse co-engagement data and rich content metadata, there’s a high chance your recommender is doing what mine was doing for a long time. Outputting popularity. Calling it similarity.</p><p>Three things to try, in order:</p><p><strong>1. Audit your collaborative filter for popularity bias.</strong> Pick a niche seed (something with under 10k reviews, distinctive tags). Run your similarity. If the top 10 includes anything from the top 100 most-popular items in your catalog, the bias is real. If the top 5 <em>is mostly</em> top-100 popular items, the bias is dominant.</p><p><strong>2. Switch from sum to product for hybrid scoring.</strong> Multiply content × collab instead of weighting them additively. Run the audit again. The list should change. If it doesn’t, your content signal is too weak to gate. Fix the content features first.</p><p><strong>3. Add a logarithmic popularity penalty.</strong> Calibrate it against a labeled benchmark; don’t trust your gut for the magnitude. In my case the stable plateau was a narrow band (alpha 0.15–0.30); outside it the benchmark fell off fast.</p><p>The whole change is maybe 30 lines of code. It took longer to find than I’d like to admit, because every blog post and tutorial in the space pushed me toward bigger swings (more layers! Graph neural nets! Diversity rerankers!) that are easier to write up but harder to make work.</p><p>This is the first writeup in what I want to make a series. A few other parts of the system that deserve their own posts: the LLM pipeline that extracts gameplay vibes from review text, the eval methodology and the labeled benchmark behind these numbers, the production stack on a home VPS with CD through Cloudflare Tunnel, and an open API I’m planning to expose so other people can build on the rankings. Follow along if any of that sounds interesting.</p><p><a href="https://imho.run/"><em>imho.run</em></a><em> is a Steam game recommender. Login is optional everywhere: seed mode runs anonymously, library mode accepts a pasted public SteamID, and signing in with Steam only adds personalization and feedback memory on top. The </em><a href="https://imho.run/methodology"><em>methodology page</em></a><em> covers the rest of the stack: the gated collab, the anti-popularity penalty, the EASE collab model, and the LLM extractor that mines gameplay vibes from reviews.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c070ae900e6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Beat LinkedIn’s Broken Job Search with AI + Telegram (My own SaaS)]]></title>
            <link>https://medium.com/@maksimlysenko/how-i-beat-linkedins-broken-job-search-with-ai-telegram-my-own-saas-6282781e788a?source=rss-2ad4d0e63bf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6282781e788a</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[job-search]]></category>
            <category><![CDATA[jobs]]></category>
            <dc:creator><![CDATA[Maksim Lysenko]]></dc:creator>
            <pubDate>Tue, 20 May 2025 22:19:30 GMT</pubDate>
            <atom:updated>2025-05-20T22:19:30.022Z</atom:updated>
            <content:encoded><![CDATA[<h3>Quick Navigation</h3><ul><li><a href="#a692">Intro</a></li><li><a href="#0605">The Problem</a></li><li><a href="#386b">The Platform</a></li><li><a href="#3079">Collecting Job Data</a></li><li><a href="#7a97">Filtering the jobs</a></li><li><a href="#2401">User Feedback &amp; Iteration</a></li><li><a href="#e08c">Try it</a></li></ul><h3>Introduction</h3><p>When I started looking for a job, the first thing I ran into was how painfully clunky LinkedIn’s filters are. If you’ve used them even once, you probably know what I mean.</p><p>Want to filter jobs posted in the last 24 hours? Be ready to click through dozens of pages full of spammy or irrelevant job posts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*L6V4HEVb62sVMRyp8Jgfjw.png" /></figure><p>Looking for remote work? You’ll need to open each job post individually to check if they actually hire from your location — and don’t forget to scroll all the way down to make sure there’s no sneaky line saying you’ll need to pop into the office once a week.</p><p>Want jobs with salary info or specific benefits? Sorry, there’s no filter for that.</p><p>And let’s not even talk about the recommended jobs LinkedIn sends to your inbox. When I was looking for Python developer roles, I had to wade through tons of listings for QA testers, software architects, and — no joke — translators of ancient extraterrestrial languages. Also, by the time the job lands in your inbox, it’s often hours old, which means your resume has a good chance of never being seen.</p><p><strong>Sounds familiar? Then you’re in the right place.</strong></p><h3>The Problem</h3><p>I broke the problem down into two main parts:</p><ol><li>Get notified about new jobs <em>on time</em></li><li>Filter <em>only</em> the jobs that actually fit <em>me</em>.</li></ol><h3>1. The Platform</h3><p>I chose Telegram. Why? Because it’s exploding in popularity, has a great <a href="https://core.telegram.org/bots/api">Bot API</a>, and I’m not a frontend guy. Telegram gave me an easy way to create a user-friendly interface without building a full app. Plus, it supports push notifications and handles auth painlessly. And perhaps most importantly — Telegram is always at hand. Alerts come in instantly, and I never miss a new match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*9A4CnI_4gStHr1bgtpCaDA.png" /></figure><h3>2. Collecting Job Data</h3><p>There are many third-party services that offer access to LinkedIn job listings via APIs. I won’t advertise specific ones here, but a quick search for “LinkedIn API jobs” will give you plenty of options. I chose the cheapest one — about $1 for 10,000 job posts.</p><h3>3. Filtering the Jobs</h3><p>This was the hardest part. I started with basic keyword matching (e.g., if the job description contains “Python,” it’s a match). Surprise — it sucked. Totally unreliable.</p><p>So I brought out the big guns: <strong>LLMs</strong>.</p><h4>3.0 Model Experiments</h4><p>I tested many self-hosted models first:</p><ul><li>Qwen</li><li>LLaMA</li><li>PHO</li><li>and others</li></ul><p>None of them worked well. They struggled with long inputs (some job descriptions are 3000+ tokens), hallucinated a lot, and were too optimistic about matches.</p><p>Then I tried paid APIs:</p><ul><li><strong>ChatGPT 4o</strong> — Struggled with structured JSON outputs on long prompts.</li><li><strong>ChatGPT o1</strong> — Excellent results, best overall, but expensive.</li><li><strong>Gemini 2.5</strong> — Also very good, also pricey.</li></ul><p>Eventually, I settled on:</p><ul><li><strong>Gemini 2.0 Flash</strong> — Very solid, followed prompts well, low cost.</li><li><strong>Gemini 2.0 Flash Lite</strong> — Almost as good, <em>and</em> has a generous free tier (30 requests/minute). Ideal for testing.</li><li><strong>DeepSeek R1</strong> — Surprisingly accurate! Almost 100% match rate with good prompting. But pricey — it cost $5 in one week just for me.</li></ul><p>Here’s a quick price comparison table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*sHd0nAl4q5ezdcTuVzPGng.png" /></figure><p>So my stack: <strong>Gemini 2.0 Flash Lite</strong> for daily use, <strong>DeepSeek R1</strong> as backup.</p><h4>3.1 Tuning the Filtering</h4><p>Instead of just asking the model “Does this job match the user’s request?”, I added a preprocessing step:</p><ul><li>Each job post is parsed into structured JSON: location, required skills, tech stack, salary, language, benefits, etc.</li><li>Then the AI evaluates <em>only</em> relevance.</li></ul><p>This 2-step system boosted accuracy by ~20%.</p><h3>3.2 The Location Struggle</h3><p>One surprising challenge: location filtering on LinkedIn.</p><p>Turns out LinkedIn uses <strong>geoIds</strong> for locations, and they’re not easy to guess. You have to query a separate endpoint to get the correct geoId.</p><p>Even subtle differences matter: Prague, Czechia and Prague, Prague, Czechia are treated as entirely separate locations with different job postings.</p><p>I ended up building a location selector that mirrors LinkedIn’s own UI and fetches geoIds dynamically, making location-based filtering as accurate as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/145/1*X8Ot0RcsmsXfywWi2NmEYQ.png" /></figure><h3>4. User Feedback &amp; Iteration</h3><p>I gave the bot to friends and observed how they used it. That’s when I realized: <em>people write way too much fluff in their prompts.</em></p><p>One friend wrote:</p><blockquote><em>“Hi, I’m Bob. I’m looking for a cool job as a social media designer. I’ve worked at XXX, did YYY, improved ZZZ by 300%, love using Figma and Photoshop, have 17 years of experience, want full-time and on-site only.”</em></blockquote><p>Which is great — for a recruiter. But terrible for AI filtering.</p><p>So I added another pre-step: user prompts now get parsed and cleaned <em>just like job descriptions</em>. The above becomes:</p><pre>Adobe Photoshop, Figma, only on-site and full-time, mid-senior</pre><p>This dramatically improved match quality. Some fields (like job type or experience level) also become filters <em>before</em> the AI even gets involved.</p><h3>5. Going Public</h3><p>I posted the bot to a small subreddit. It gained traction quickly, with hundreds of users in the first week.</p><p>My goal is to keep the bot <em>as free as possible</em>.</p><ul><li><strong>Job notifications</strong> will always be free.</li><li><strong>Smart filtering</strong> might become a paid feature to cover LLM costs.</li><li>Early adopters? They’ll get full access forever.</li></ul><h3>What’s Next</h3><p>I plan to publish a more technical series soon (lots of cool tricks for working with large data and low latency), and open-source the code so you can build your own or fork mine.</p><p>During testing, I also built up a dataset of over 500,000 job listings. I’m planning to release insights and analysis based on that data in a future post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*g88mx6leqGSapuT4-AtU7w.png" /><figcaption>Notification from bot about job posting</figcaption></figure><p><strong>Curious? Try the bot here:</strong> <a href="https://t.me/linkedin_alertbot">https://t.me/linkedin_alertbot</a></p><p>I’d love to hear your thoughts — whether it’s feedback, feature ideas, or just to say hi!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6282781e788a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Profiling FastApi using pyroscope & Docker compose]]></title>
            <link>https://medium.com/@maksimlysenko/profiling-fastapi-using-pyroscope-docker-compose-f35801b8a859?source=rss-2ad4d0e63bf8------2</link>
            <guid isPermaLink="false">https://medium.com/p/f35801b8a859</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[pyroscope]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[profiling]]></category>
            <dc:creator><![CDATA[Maksim Lysenko]]></dc:creator>
            <pubDate>Thu, 08 May 2025 21:42:14 GMT</pubDate>
            <atom:updated>2025-05-08T21:42:14.418Z</atom:updated>
            <content:encoded><![CDATA[<h3>Profiling a FastAPI App with Pyroscope</h3><p>Recently, I was working on a pet project and decided to use the Grafana stack. That’s when Pyroscope caught my eye — an open-source tool for analyzing your application’s performance.</p><h3><strong>Why bother with profiling?</strong></h3><p>Performance problems tend to hide in places you don’t expect — and in production, you’re often blind until real traffic hits your app.</p><h3>Why Pyroscope?</h3><p>What stood out to me was its smooth integration with Grafana and its low overhead. In my case, profiling added around 1–2% — which is more than acceptable.</p><h3>Let’s dive in.</h3><p>I’ll be using Docker Compose to run Grafana, Pyroscope, and simple FastAPI app. You can use a different approach — for example, by downloading the binary directly from <a href="https://grafana.com/docs/pyroscope/latest/get-started/">here</a>.</p><p>We’ll start with a simple docker-compose.yml file that includes services for Grafana and Pyroscope.</p><pre>services:<br>  grafana:<br>    image: grafana/grafana:latest<br>    ports:<br>      # open port to access grafana<br>      - &quot;3000:3000&quot;<br>    environment:<br>      # enable anonymous access | DO NOT USE IN PRODUCTION<br>      - GF_AUTH_ANONYMOUS_ENABLED=true<br>      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin<br>      - GF_AUTH_DISABLE_LOGIN_FORM=true<br>      - GF_INSTALL_PLUGINS=grafana-pyroscope-app<br>    volumes:<br>      # persist grafana data<br>      - grafana-data:/var/lib/grafana<br>      # folders to store grafana provisioning (data sources)<br>      - ./grafana/provisioning:/etc/grafana/provisioning<br>    restart: unless-stopped<br><br>  pyroscope:<br>    image: grafana/pyroscope:latest<br>    ports:<br>      # open port to access frontend<br>      - &quot;4040:4040&quot;<br>    command:<br>      # run pyroscope in server mode<br>      - &quot;server&quot;<br>    restart: unless-stopped<br><br>volumes:<br>  grafana-data:<br><br></pre><p>Next, create the following directory next to docker-compose.yml:</p><p>grafana/provisioning/datasources</p><p>Inside, create a file named pyroscope.yml with the following content:</p><pre>apiVersion: 1<br>datasources:<br>  - uid: local-pyroscope<br>    type: grafana-pyroscope-datasource<br>    name: Pyroscope<br>    url: http://pyroscope:4040<br>    jsonData:<br>      keepCookies: [pyroscope_git_session]</pre><p>This setup will automatically add Pyroscope as a data source in Grafana when it starts up.<br><strong>Note</strong>: the url is set to the container name (pyroscope) because, within Docker&#39;s internal network, services communicate using these names. If you&#39;re running Pyroscope outside of Docker, use localhost, 127.0.0.1, or host.docker.internal (when Grafana is in Docker and Pyroscope is not).</p><p>Now let’s run everything:</p><pre>docker compose up</pre><p>Once it’s up, visit <a href="http://localhost:3000/">http://localhost:3000/</a> to make sure Grafana is running.</p><p>Check under <strong>Connections → Data Sources</strong> — you should see Pyroscope listed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_abkIWEE9KytdKbjyy8Sww.png" /></figure><p>Also, open <a href="http://localhost:4040/">http://localhost:4040/</a> to verify that the Pyroscope UI is available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ohedMtcNP4Yk08dxC7Adw.png" /></figure><p>All good — the setup is ready, and we can move on to the FastAPI app.</p><h3>Creating the FastAPI App</h3><p>In the backend folder, create a basic Dockerfile:</p><pre>FROM python:3.12-slim<br><br>WORKDIR /app<br><br>COPY requirements.txt .<br><br>RUN pip install --no-cache-dir -r requirements.txt<br><br>COPY . .<br><br>CMD [&quot;python&quot;, &quot;main.py&quot;]</pre><p>Now let’s add this service to docker-compose.yml:</p><pre>  backend:<br>    build:<br>      context: ./backend<br>      dockerfile: Dockerfile<br>    ports:<br>      - &quot;5000:5000&quot;<br>    restart: unless-stopped</pre><p>Inside the backend folder, create requirements.txt:</p><pre>fastapi==0.115.2<br>uvicorn==0.34.2<br>pyroscope-io==0.8.11</pre><p>and main.py</p><pre>from fastapi import FastAPI<br>import uvicorn<br><br>app = FastAPI()<br><br>@app.get(&quot;/&quot;)<br>def read_root():<br>    return {&quot;message&quot;: &quot;Hello, World!&quot;}<br><br>if __name__ == &quot;__main__&quot;:<br>    uvicorn.run(app, host=&quot;0.0.0.0&quot;, port=5000)</pre><p>At this point, your project should look like this:</p><pre>.<br>├── backend<br>│   ├── Dockerfile<br>│   ├── main.py<br>│   └── requirements.txt<br>├── grafana<br>│   └── provisioning<br>│       └── datasources<br>│           └── pyroscope.yml<br>└── docker-compose.yml</pre><p>Start the project with:</p><pre>docker compose up --build</pre><p>And open <a href="http://127.0.0.1:5000/">http://127.0.0.1:5000/</a> — you should see a simple JSON response.</p><h3>Integrating Pyroscope</h3><p>Here’s a minimal setup to get Pyroscope working with FastAPI:</p><pre>import uvicorn<br>from fastapi import FastAPI<br>import pyroscope<br><br>app = FastAPI()<br><br>pyroscope.configure(<br>  application_name = &quot;backend&quot;,<br>  server_address   = &quot;http://pyroscope:4040&quot;,<br>)<br><br>@app.get(&quot;/&quot;)<br>def read_root():<br>    return {&quot;message&quot;: &quot;Hello, World!&quot;}<br><br>if __name__ == &quot;__main__&quot;:<br>    uvicorn.run(app, host=&quot;0.0.0.0&quot;, port=5000)</pre><p>To profile a specific endpoint, we can wrap it with tags:</p><pre>@app.get(&quot;/&quot;)<br>def read_root():<br>    with pyroscope.tag_wrapper({&quot;endpoint&quot;: &quot;slow_controller_i_want_to_profile&quot;}):<br>        time.sleep(1)<br>        return {&quot;message&quot;: &quot;Hello, World!&quot;}</pre><p>⚠️ default sample_rate is 100 — that might not be enough to catch very fast endpoints. Just something to keep in mind.</p><h3>Automatically Profiling All Endpoints</h3><p>If we want to apply profiling across the board — without wrapping every single route — we can use a custom APIRoute class:</p><pre>import pyroscope<br>from fastapi import Request, Response<br>from fastapi.routing import APIRoute<br><br>class PyroscopeRoute(APIRoute):<br>    def get_route_handler(self):<br>        original_handler = super().get_route_handler()<br>        async def custom_handler(request: Request) -&gt; Response:<br>            route = request.scope.get(&quot;route&quot;)<br>            template = getattr(route, &quot;path_format&quot;, getattr(route, &quot;path&quot;, request.url.path))<br>            method = request.method<br>            tag = f&quot;{method}:{template}&quot;<br>            with pyroscope.tag_wrapper({&quot;endpoint&quot;: tag}):<br>                return await original_handler(request)<br>        return custom_handler</pre><p>Then we just create a router using this route class:</p><pre>api_router = APIRouter(prefix=&quot;/api&quot;, route_class=PyroscopeRoute)</pre><p>And here’s the full example:</p><pre>import asyncio<br>import uvicorn<br>from fastapi.routing import APIRoute<br>from fastapi import APIRouter, FastAPI, Request, Response<br>import pyroscope<br><br>app = FastAPI()<br><br>pyroscope.configure(<br>  application_name = &quot;backend&quot;,<br>  server_address   = &quot;http://pyroscope:4040&quot;,<br>)<br><br>class PyroscopeRoute(APIRoute):<br>    def get_route_handler(self):<br>        original_handler = super().get_route_handler()<br>        async def custom_handler(request: Request) -&gt; Response:<br>            route = request.scope.get(&quot;route&quot;)<br>            template = getattr(route, &quot;path_format&quot;, getattr(route, &quot;path&quot;, request.url.path))<br>            method = request.method<br>            tag = f&quot;{method}:{template}&quot;<br>            with pyroscope.tag_wrapper({&quot;endpoint&quot;: tag}):<br>                return await original_handler(request)<br>        return custom_handler<br><br>api_router = APIRouter(prefix=&quot;/api&quot;, route_class=PyroscopeRoute)<br><br>@api_router.get(&quot;/slow&quot;)<br>async def slow_controller():<br>    for _ in range(100_000_00):<br>        1 + 1<br>    return {&quot;message&quot;: &quot;Hello, World!&quot;}<br><br>app.include_router(api_router)<br><br>if __name__ == &quot;__main__&quot;:<br>    uvicorn.run(app, host=&quot;0.0.0.0&quot;, port=5000)</pre><p>Make a few requests to <a href="http://127.0.0.1:5000/api/slow">http://127.0.0.1:5000/api/slow</a>, then check Grafana under <strong>Drilldown → Profiles</strong> — you should see your endpoint in the flamegraph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6OWXqr9bYM_m5a92VQTjCA.png" /><figcaption><strong>Drilldown → Profiles</strong></figcaption></figure><h3>Wrapping up</h3><p>You’ve now seen how to integrate basic profiling into a FastAPI project. For production environments, don’t forget to:</p><ul><li>Adjust settings like sampling rate</li><li>Avoid exposing ports to the internet</li><li>Limit profiling to specific areas to avoid wasting resources</li></ul><p>📦 Full source code: <a href="https://github.com/0x216/medium-article-profiling-fastapi">https://github.com/0x216/medium-article-profiling-fastapi</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f35801b8a859" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>