<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sergey Kolchenko on Medium]]></title>
        <description><![CDATA[Stories by Sergey Kolchenko on Medium]]></description>
        <link>https://medium.com/@sergeykolchenko?source=rss-e626cc224497------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*gs8lA4S8Hs-BqrrX.</url>
            <title>Stories by Sergey Kolchenko on Medium</title>
            <link>https://medium.com/@sergeykolchenko?source=rss-e626cc224497------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 21:40:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sergeykolchenko/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Wisdom Of The Crowd: Is Reddit Worth Listening To?]]></title>
            <link>https://sergeykolchenko.medium.com/reddit-and-stock-returns-is-the-crowd-worth-listening-to-25b5b2dc86a3?source=rss-e626cc224497------2</link>
            <guid isPermaLink="false">https://medium.com/p/25b5b2dc86a3</guid>
            <category><![CDATA[reddit]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Sergey Kolchenko]]></dc:creator>
            <pubDate>Wed, 16 Jul 2025 22:11:35 GMT</pubDate>
            <atom:updated>2025-07-18T01:11:55.731Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TL;DR:</strong> After parsing 18 million Reddit comments spanning 8 years with an LLM-based sentiment pipeline, I found that crowd chatter <strong>does </strong>contain a faint signal for forecasting stock returns (tested on 323 tickers). However, the signal is intermittent and far too weak to trade on independently.</p><p>Code 8 years of processed data: <a href="https://github.com/azkalot1/WisdomOfReddit">azkalot1/WisdomOfReddit</a></p><h3>Introduction</h3><p>I’ve always been fascinated by Reddit, a platform where you can discuss almost anything. One area I particularly enjoy is investment subreddits. While many have heard about WallStreetBets, thanks largely to the infamous GME and AMC runs, there are numerous other interesting communities like ValueInvesting, Trading, StockInvest, and various combinations of Trading/Stock/Invest/Meme.</p><p>Although it’s enjoyable to read these occasionally, I don’t have the bandwidth to cover them comprehensively. So, as any sensible machine learning scientist might do, I built a system to parse these subreddits, read all comments, and extract actionable signals — what to buy or avoid. I’m sure hundreds of Reddit bots attempt similar tasks; this is essentially a classic sentiment analysis problem.</p><p>I’m not an expert in sentiment systems specifically, but certain truths apply universally in ML projects: you need good data, proper splitting methods, useful features, and reliable performance metrics.</p><h3>Part 1: Getting the data</h3><p>Where to source data? Conveniently,<a href="https://www.reddit.com/r/pushshift/comments/1itme1k/separate_dump_files_for_the_top_40k_subreddits/"> Reddit archives exist for most subreddits, available up to the end of 2024</a>. This solves part of the historical data issue. After downloading and unpacking selected subreddits, each subreddit yields two files: %name%_comments.csv and %name%_submission.csv. Submission files provide details like author, title, score, date, URL, and text. Comment files include author, score, date, and text. However, I wanted a slightly different structure—each submission as a JSON file containing submission ID, title, date, comments (as a list of strings), and metadata (author, link). Thus, I created the RedditCSVConverter class to reshape data into my desired format.</p><p>Next, continuous updates of new daily comments were managed using PRAW, a Python wrapper for Reddit. One limitation of PRAW (to my knowledge) is it can only fetch recent submissions, not arbitrary historical data — hence the initial historical data step. I set up a daily scheduled job to regularly update submissions and comments.</p><pre>python parse.py --subreddits wallstreetbets StockMarket TheRaceTo10Million TheRaceTo1Million pennystocks OptionsMillionaire ValueInvesting investing Trading stocks options RobinHoodPennyStocks StocksAndTrading smallstreetbets StockInvest stockstobuytoday MemeStockMarket \<br>--time=96h --limit=450 --force-update --output-dir ../reddit_comments</pre><p>After processing data from January 2017 to July 2025 across 21 subreddits, I collected <strong>808,417 submissions and 18,194,606 comments</strong>.</p><p>Additionally, financial data (daily Open, Close, and Volume) was obtained using yfinance and a ticker list from financialmodelingprep.com (requires an API key).</p><pre>python .\scripts\save_financial_data.py --start-date 2017-01-01 \<br> --output .\data\financial_data.parquet</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d8MEsIurGTBwhYjV8eRnkA.png" /><figcaption>WallStreeBets is the most active subreddit, just as expected</figcaption></figure><p>Here is an example of obtained submission and comments, as a single json file:</p><pre>{<br>  &quot;submission_id&quot;: &quot;REDACTED&quot;,<br>  &quot;title&quot;: &quot;Collective+Artificial Intelligence Fund&quot;,<br>  &quot;date&quot;: &quot;2017-01-11T03:27:00&quot;,<br>  &quot;subreddit&quot;: &quot;StockMarket&quot;,<br>  &quot;comments&quot;: [<br>    &quot;Submission body: Interesting concept. Your thoughts? http://fund.cindicator.com (score 5)&quot;,<br> &quot;conversion_source&quot;: &quot;csv_dump&quot;, # indicates thos comes from the dump parser<br>    &quot;url&quot;: &quot;https://www.reddit.com/r/StockMarket/comments/REDACTED/collectiveartificial_intelligence_fund/&quot;,<br>    &quot;cleaning_applied&quot;: true,<br>    &quot;year_filtering_applied&quot;: true,<br>    &quot;comment_breakdown&quot;: {<br>      &quot;top_level_comments&quot;: 1,<br>      &quot;total_replies&quot;: 0,<br>      &quot;empty_comments&quot;: 0,<br>      &quot;threads_with_replies&quot;: 0<br>    },<br>    &quot;data_quality&quot;: {<br>      &quot;comments_with_scores&quot;: 1,<br>      &quot;deleted_comments&quot;: 0,<br>      &quot;processing_success_rate&quot;: 100.0<br>    }<br>  }<br>}</pre><h3>Part 2: Finding relevant submissions</h3><p>Although I had ~18 million comments, not all were relevant to stock sentiment. To filter this effectively, I employed a binary classifier: submissions labeled 0 (irrelevant) or 1 (relevant — containing stock sentiment).</p><p>Examples of relevant and irrelevant submissions include:</p><p>Irrelevant:</p><pre>Is Mare Barak legit?...<br>Submission body: Thinking of buying his trading course, wondering if this guy is legit? (score 0)...<br>Top level comment: All trading courses that cost something are scams. Ask yourself this: Why would t...<br>Top level comment: You can learn to trade for free, you don’t need to pay for a course. Sure, some i...<br>Top level comment: What are good YouTube people to watch? thanks (score 1)...</pre><pre>Welp here it goes. Don’t miss out...<br>Top level comment: We&#39;re all traders here, but this sub is not your personal trading journal. If you...</pre><p>Relevant:</p><pre>$TTOO DD with technical analysis...<br>Submission body: Hey all, Wondering if $TTOO might be a good buy after a strong bleed today ? I go t...<br>Top level comment: This submission is flaired as &quot;DD.&quot; If you do not know what DD is, or how to comp...<br>Top level comment: I bought 2.5c 7/17 Let’s hope I don’t get FUCKED boys (score 5)...<br>Top level comment: I&#39;m holding out on this one, I averaged up today @ 1.75. This is one of the best ...<br>Top level comment: Looks like the graph that was posted about selling before the bull trap. No thank...</pre><pre>HTBX jumped to 1$ today, here is why...<br>Submission body: Very important Upcoming Dates &amp; Catalysts** * 2nd Quarter results is expected on Au...<br>Top level comment: This submission is flaired as &quot;DD.&quot; If you do not know what DD is, or how to comp...<br>Top level comment: The big catalyst for me on this is that if it closes over $1 and can hold it for ...<br>Top level comment: I’m in at .834 looking to hold for a while ive seen some nice price targets on th...<br>Top level comment: in with 40% of my portfolio at .88 I want another home run after making slow and ...<br>Top level comment: I been in this one forever. Looking to sell my initial investment at 1.20 and let...<br>Top level comment: I&#39;ve swung trade this a few times to secure some profits, now in at $1 after star...<br>Top level comment: Will rocket next week (score 6)...</pre><p>Why not process all submissions with an LLM? Cost and efficiency! Each LLM call costs money and adds processing time. Thus, for historical data, pre-filtering relevant submissions makes practical sense.</p><p>I created labels via two methods:</p><ol><li><strong>Manual Labeling</strong>: I manually checked random submissions, assigning 1 or 0 labels in labels.csv.</li><li><strong>Automatic Labeling</strong>: Using AutomaticRelevanceScorer (built with LangChain and OpenAI API), I auto-labeled additional submissions to supplement manual efforts. The prompt looked like this:</li></ol><pre># Reddit Investment Sentiment Analyzer - Prescoring<br><br>You are an equity-sentiment analyst.  <br>Your job is to decide whether a **Reddit thread (submission + supplied comments + metadata)**<br> contains at least one **clear, actionable view on a specific stock or ETF** focused on the future.<br><br>---<br><br>## 1  Evidence to consider<br>You will receive, in this order:<br><br>1. Submission title  <br>2. Submission body  <br>3. All comments (concatenated after the body)<br>4. Submission metadata<br><br>Treat the entire concatenated text as one document.<br><br>---<br><br>## 2  Label definitions<br><br><br>**Class 1 - Relevant** <br> Any sentence anywhere in the thread satisfies **both**<br>  a) Mentions an explicit ticker/ETF (e.g., TSLA, NVDA, SPY)<br>  b) Expresses a directional or valuation stance (buy, sell, hold, long, short, <br>     price target, option strategy, “undervalued/overvalued”, etc.). <br>  c) Focuses on the future performance of the stock or ETF.<br>**Class 0 - Irrelevant** <br> Everything else, including: <br>  Macro or political talk without single-name calls.<br>  Asking for advice (“What should I buy?”).<br>  Bragging, venting, or pure memes.<br>  Screenshots / option chains with **no** accompanying analysis.<br>  Personal-finance questions (401k, budgeting, FIRE, etc.).<br>  Results of previous trades, unless they are followed by a clear, <br>   actionable view or prediction on a specific stock or ETF <br><br>&gt; **Be strict:** If the signal is weak, contradictory, or ambiguous, <br>  default to Class 0.<br><br>---<br><br>## 3  Decision rule<br>1. Scan the whole thread.  <br>2. **If at least one qualifying sentence meets the Class 1 criteria, <br>   label the entire thread Class 1.**  <br>3. Otherwise, label Class 0.<br><br>Conflicting opinions across comments do **not** invalidate relevance; <br>  the presence of a single actionable view is enough.<br><br>---<br><br>## 4  Edge-case guidance<br>- **Index ETFs** (SPY, QQQ, IWM) with a trade thesis → Class 1.  <br>- “Diamond-hands GME 🚀🚀” **without** rationale → Class 0.  <br>- “Bought 100 TSLA @ $180, thoughts?” → Class 0 (asking advice) unless followed by ideas/analysis on the future performance of the stock or ETF.<br>- Position screenshot **plus** a comment “Holding until $250” → Class 1.  <br>- Meme tickers/wordplay (“STONK”) are acceptable **only if** the true ticker is clear and a stance is present.<br><br>---<br><br>## 5  Output schema<br>Call the `RelevanceResults` class, which contains the following fields:<br>- `is_relevant`: True if the discussion contains valid discussions of investment opportunities, False otherwise<br>- `reasoning`: Explanation of the relevance decision<br>- `confidence`: Confidence score for the relevance decision</pre><p>In total, I labeled 1,245 submissions (685 relevant) with an additional 103 manually labeled submissions (54 relevant) reserved for testing.</p><p>Next was classifier training. Although few-shot learning with LLMs could work, my goal was a fast, local model. Features included naive ticker mentions (TickerFeaturizer) and embeddings from FinBert for generalizable representations. Combining embeddings with handcrafted ticker/key-word features, I trained the RelevanceTrainer class using a RandomForest classifier.</p><pre>class TickerFeaturizer(BaseFeaturizer):<br>    &quot;&quot;&quot;Extract ticker-related features&quot;&quot;&quot;<br>    <br>    def __init__(self, ticker_file: Optional[Path] = None):<br>        self.known_tickers = self._load_tickers_yfinance(ticker_file)<br>        self.ticker_patterns = [<br>            r&#39;\$[A-Z]{1,5}\b&#39;,           # $AAPL<br>            r&#39;\b[A-Z]{1,5}(?=\s|$)&#39;,     # AAPL (standalone)<br>            r&#39;\b[A-Z]{2,5}(?=\s+stock)&#39;, # AAPL stock<br>        ]<br>    <br>    def get_feature_names(self) -&gt; List[str]:<br>        return [<br>            &#39;ticker_count&#39;,<br>            &#39;explicit_ticker_count&#39;,<br>            &#39;ticker_in_title&#39;,<br>            &#39;ticker_density&#39;,<br>            &#39;unique_tickers&#39;,<br>            &#39;ticker_title_ratio&#39;<br>        ]<br>    <br>    def extract_features(self, submission_data: Dict[str, Any]) -&gt; Dict[str, float]:<br>        title = submission_data.get(&#39;title&#39;, &#39;&#39;)<br>        comments = submission_data.get(&#39;comments&#39;, [])<br>        all_text = f&quot;{title} &quot; + &quot; &quot;.join(comments)<br>        <br>        # Find all tickers<br>        all_tickers = set()<br>        explicit_tickers = set()<br>        title_tickers = set()<br>        <br>        for pattern in self.ticker_patterns:<br>            # All text<br>            matches = re.findall(pattern, all_text, re.IGNORECASE)<br>            for match in matches:<br>                clean_ticker = match.replace(&#39;$&#39;, &#39;&#39;).upper()<br>                if clean_ticker in self.known_tickers:<br>                    all_tickers.add(clean_ticker)<br>                    if &#39;$&#39; in match:<br>                        explicit_tickers.add(clean_ticker)<br>            <br>            # Title only<br>            title_matches = re.findall(pattern, title, re.IGNORECASE)<br>            for match in title_matches:<br>                clean_ticker = match.replace(&#39;$&#39;, &#39;&#39;).upper()<br>                if clean_ticker in self.known_tickers:<br>                    title_tickers.add(clean_ticker)<br>        <br>        # Calculate features<br>        word_count = len(all_text.split())<br>        title_word_count = len(title.split())<br>        <br>        return {<br>            &#39;ticker_count&#39;: float(len(all_tickers)),<br>            &#39;explicit_ticker_count&#39;: float(len(explicit_tickers)),<br>            &#39;ticker_in_title&#39;: float(len(title_tickers) &gt; 0),<br>            &#39;ticker_density&#39;: (len(all_tickers) / max(word_count, 1)) * 100,<br>            &#39;unique_tickers&#39;: float(len(all_tickers)),<br>            &#39;ticker_title_ratio&#39;: len(title_tickers) / max(title_word_count, 1)<br>        }<br></pre><pre>class SlidingWindowEmbeddingFeaturizer(BaseFeaturizer):<br>    def __init__(self,<br>                 model_name=&quot;ProsusAI/finbert&quot;,<br>                 window_tokens=256,<br>                 stride_tokens=64,<br>                 max_chars=1000,<br>                 device=None):<br>        self.tok = AutoTokenizer.from_pretrained(model_name)<br>        self.mdl = AutoModel.from_pretrained(model_name).eval()<br>        self.device = device or (&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>        self.mdl.to(self.device)<br><br>        self.W = window_tokens<br>        self.S = stride_tokens<br>        self.max_chars = max_chars<br>        self.dim = self.mdl.config.hidden_size<br>        self._names = [f&quot;embed_{i}&quot; for i in range(self.dim)]<br><br>    # ---------------------------------------------------------------------<br>    def get_feature_names(self) -&gt; List[str]:<br>        return self._names<br><br>    @torch.inference_mode()<br>    def extract_features(self, submission_data: Dict[str, Any]) -&gt; Dict[str, float]:<br>        &quot;&quot;&quot;<br>        Convert the entire submission (title + comments) into one 768-d vector by:<br>        1. Tokenising once (no specials)                        → token_ids<br>        2. Slicing into overlapping windows  ≤ 510 tokens long → windows<br>        3. Adding CLS/SEP, padding, attention mask             → input_ids, attn<br>        4. Forward pass, take CLS of each window               → [N, 768]<br>        5. Mean-pool across windows                            → [768]<br>        Returns { &quot;embed_0&quot;: float, … &quot;embed_767&quot;: float }.<br>        &quot;&quot;&quot;<br>        # -------- 1. flatten + early-exit<br>        title = submission_data.get(&quot;title&quot;, &quot;&quot;)<br>        comments = submission_data.get(&quot;comments&quot;, [])<br>        text = (title + &quot; &quot; + &quot; &quot;.join(comments))[: self.max_chars]<br><br>        if not text.strip():<br>            return {name: 0.0 for name in self._names}<br><br>        token_ids = self.tok(text, add_special_tokens=False)[&quot;input_ids&quot;]<br>        if not token_ids:<br>            return {name: 0.0 for name in self._names}<br><br>        # -------- 2. build windows<br>        cls_id, sep_id = self.tok.cls_token_id, self.tok.sep_token_id<br>        pad_id = self.tok.pad_token_id<br>        max_model_len = self.tok.model_max_length          # 512 for FinBERT<br>        win_len = min(self.W, max_model_len - 2)           # reserve CLS+SEP<br>        stride = min(self.S, win_len)<br><br>        windows = [<br>            [cls_id] + token_ids[i : i + win_len] + [sep_id]<br>            for i in range(0, len(token_ids), stride)<br>        ] or [[cls_id, sep_id]]                            # guarantee ≥1 window<br><br>        # -------- 3. pad batch + attention mask<br>        max_len = max(len(w) for w in windows)<br>        assert max_len &lt;= max_model_len, f&quot;window={max_len} &gt; {max_model_len}&quot;<br><br>        input_ids = torch.full(<br>            (len(windows), max_len), pad_id, dtype=torch.long, device=self.device<br>        )<br>        attn = torch.zeros_like(input_ids)<br><br>        for i, w in enumerate(windows):<br>            input_ids[i, : len(w)] = torch.tensor(w, dtype=torch.long, device=self.device)<br>            attn[i, : len(w)] = 1<br><br>        # -------- 4. model forward → CLS per window<br>        cls_vecs = self.mdl(input_ids=input_ids, attention_mask=attn).last_hidden_state[:, 0]<br><br>        # -------- 5. mean-pool windows → 1 vector<br>        doc_vec = cls_vecs.mean(dim=0).cpu().numpy()       # [768]<br><br>        return {name: float(v) for name, v in zip(self._names, doc_vec)}</pre><pre>+-------------------------------------------------------------------+<br>|                     RELEVANCE CLASSIFICATION PIPELINE             |<br>+-------------------------------------------------------------------+<br>|                                                                   |<br>|  +---------------------+                                          |<br>|  | Raw Submission Data |                                          |<br>|  +---------------------+                                          |<br>|             |                                                     |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | SubmissionLookup    |                                          |<br>|  | (Finds Submissions) |                                          |<br>|  +---------------------+                                          |<br>|             | Loads data for:                                     |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | AutomaticRelevanceS.|                                          |<br>|  | (LLM-based Scorer)  |                                          |<br>|  | - RelevanceResults  |                                          |<br>|  | - ScoringResult     |                                          |<br>|  | - BatchScoringResults|                                          |<br>|  +---------------------+                                          |<br>|             | Auto-labels submissions                             |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | LabeledSubmission   |                                          |<br>|  | (Human/Auto Labeled)|                                          |<br>|  +---------------------+                                          |<br>|             | Added to:                                           |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | TrainingDataset     |                                          |<br>|  | (Manages LabeledData)|                                          |<br>|  | - TrainingDatasetMeta|                                          |<br>|  +---------------------+                                          |<br>|             | Uses:                                               |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | SubmissionFeaturizer|                                          |<br>|  | (Extracts Features) |                                          |<br>|  | - BaseFeaturizer    |                                          |<br>|  |   (Ticker, Keyword,  |                                          |<br>|  |    Psychology, etc.)|                                          |<br>|  +---------------------+                                          |<br>|             | Produces:                                           |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | FeaturizedSample    |                                          |<br>|  | (Raw Features)      |                                          |<br>|  | LabeledFeaturizedS. |                                          |<br>|  | (Features + Label)  |                                          |<br>|  +---------------------+                                          |<br>|             | Used by:                                            |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | RelevanceTrainer    |                                          |<br>|  | (Trains Classifier) |                                          |<br>|  | - TrainingConfig    |                                          |<br>|  | - TrainingResults   |                                          |<br>|  +---------------------+                                          |<br>|             | Produces trained model for:                         |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | RelevancePredictor  |                                          |<br>|  | (Single Prediction) |                                          |<br>|  +---------------------+                                          |<br>|             | Used by:                                            |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | SubmissionBatchPred.|                                          |<br>|  | (Batch Prediction)  |                                          |<br>|  | - PredictionResult  |                                          |<br>|  | - BatchPredictionRes.|                                          |<br>|  +---------------------+                                          |<br>|             | Analyzed by:                                        |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | PredictionAnalyzer  |                                          |<br>|  | (Reports on Results)|                                          |<br>|  +---------------------+                                          |<br>|                                                                   |<br>+-------------------------------------------------------------------+</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l4F4dGFtyrX6zgELxfABSQ.png" /><figcaption>Sentiment pre-scorer results</figcaption></figure><p>Is the performance perfect? No, but it clearly shows a signal, with room for improvement.</p><h3>Part 3: Extracting sentiment</h3><p>Having filtered submissions (~35% relevance rate), roughly 6 million comments remained for LLM processing. The sentiment extraction required capturing sentiment direction (bullish, bearish, neutral), numerical sentiment strength, confidence, intensity, and explanations. For structured outputs, I defined:</p><pre>class Sentiment(BaseModel):<br>    confidence: int = Field(<br>        description=&#39;Confidence score (1-10) for the extraction.&#39;,<br>        default=5<br>    )<br>    ticker: Optional[str] = Field(<br>        description=&#39;Ticker symbol of the stock being discussed.&#39;,<br>        default=None<br>    )<br>    is_implicit: bool = Field(<br>        description=&#39;Whether ticker was mentioned explicitly or inferred from context.&#39;,<br>        default=False<br>    )<br>    sentiment: Literal[&#39;bullish&#39;, &#39;bearish&#39;, &#39;neutral&#39;, &#39;unclear&#39;] = Field(<br>        description=&#39;Overall sentiment direction.&#39;<br>    )<br>    sentiment_intensity: int = Field(<br>        description=&#39;Intensity of the sentiment (1-10).&#39;,<br>        default=5<br>    )<br>    position: Literal[&#39;shares&#39;, &#39;calls&#39;, &#39;puts&#39;, &#39;spreads&#39;, &#39;watching&#39;, &#39;unclear&#39;] = Field(<br>        description=&#39;Type of position discussed.&#39;,<br>        default=&#39;watching&#39;<br>    )<br>    conviction: Literal[&#39;yolo&#39;, &#39;high&#39;, &#39;moderate&#39;, &#39;low&#39;, &#39;hedge&#39;, &#39;unclear&#39;] = Field(<br>        description=&#39;Conviction level about the sentiment.&#39;,<br>        default=&#39;unclear&#39;<br>    )<br>    time_horizon: Literal[&#39;day&#39;, &#39;swing&#39;, &#39;long&#39;, &#39;unclear&#39;] = Field(<br>        description=&#39;Time horizon for the position.&#39;,<br>        default=&#39;unclear&#39;<br>    )<br>    source: Literal[&#39;submission&#39;, &#39;comments&#39;, &#39;both&#39;] = Field(<br>        description=&#39;Where this sentiment was extracted from.&#39;,<br>        default=&#39;both&#39;<br>    )<br>    explanation: str = Field(<br>        description=&#39;Explanation of the sentiment extraction.&#39;,<br>        default=&#39;&#39;<br>    )<br>    extraction_refusal: bool = Field(<br>        description=&#39;Whether the sentiment extraction was refused.&#39;,<br>        default=False<br>    )</pre><p>Since multiple stocks could be discussed in one submission/comment chain, I included a list structure for multiple sentiments:</p><pre>class SentimentResults(BaseModel):<br>    primary_ticker: Optional[str] = Field(<br>        description=&#39;Primary ticker discussed in the submission, if identifiable.&#39;,<br>        default=None<br>    )<br>    sentiments: List[Sentiment] = Field(<br>        description=&#39;List of extracted sentiments from the discussion.&#39;,<br>        default_factory=list<br>    )<br>    relationships: Optional[List[TickerRelationship]] = Field(<br>        description=&#39;Relationships between different tickers mentioned.&#39;,<br>        default_factory=list<br>    )</pre><p>And this is the system prompt:</p><pre>extractor_system_prompt = &quot;&quot;&quot;<br># Reddit Investment Sentiment Analyzer<br><br>As a financial sentiment analyst, analyze this Reddit investment discussion and extract structured sentiment data<br>Your goal is to provide accurate, structured insights for investment analysis while maintaining neutrality and analytical rigor.<br>You only want to extract existing, valid tickers and relationships focusing of the future of the stock.<br>We are not interested in the discussions of users past performance, stock past\historical performance, unless <br>in is mentioned in the context of the future\outlook of the stock.<br>## Extraction Requirements<br><br><br>for the submision:<br>    1. Primary ticker from submission (if any)<br>For the submission and provided comments, identify:<br>    1. Tickers mentioned (explicit or implicit) - valid stock tickers only<br>    2. Sentiment per ticker (One of bullish/bearish/neutral/unclear + intensity 1-10)<br>    3. Position type (One of shares/calls/puts/spreads/watching)<br>    4. Time horizon (day/swing/long/unclear)<br>    5. Conviction (yolo/high/moderate/low/hedge/unclear)<br>    6. Is implicit: true if ticker inferred from context<br>    7. Source: submission/comments/both<br>    8. Explanation: Explanation of the sentiment extraction.<br>    9. Extraction refusal: Whether the sentiment extraction was refused.<br>Relationships between tickers (e.g., &quot;NVDA up means AMD down&quot;)<br><br>## Handling Complexities<br>- Comments may discuss multiple tickers with different sentiments<br>- Comments may reference the submission ticker without naming it (flag these as &quot;implicit reference&quot;)<br>- Inter-ticker relationships (e.g., &quot;NVDA up means AMD down&quot;)<br>- Sarcasm and WSB culture (e.g., &quot;literally can&#39;t go tits up&quot; = very risky, bearish)<br>- Only extract tickers with sufficient textual evidence (avoid hallucinating tickers)<br>- Assign lower confidence scores (1-4) for uncertain or ambiguous extractions<br>- Flag potentially inconsistent extractions (e.g., bullish sentiment with put position)<br>- For low confident comments, set sentiment, position, time, conviction as unclear<br>- For relationships, always try to identify both tickers involved. If the relationship is with a non-ticker entity (e.g., &#39;interest rates&#39;), specify this in the description.&quot;<br>- Only extract sentiment for which you can clearly articulate the reasoning.<br>- If you cannot extract sentiment, set extraction_refusal to true.<br>- Only extract tickers for existing stocks, ETFs, companies, etc.<br><br><br>## Expected Output Format<br>Call the `SentimentResults` <br>    - list of `Sentiment`, each consists of:<br>    confidence: Confidence score for the sentiment evaluation, if available.<br>    ticker: Ticker symbol of the stock being discussed.<br>    sentiment: Overall sentiment direction based on the comments.<br>    sentiment_intensity: Intensity of the sentiment expressed, on a scale from 1 to 10.<br>    position: Type of position discussed in the comments.<br>    conviction: How convinced the authors are about the sentiment expressed in the comments section.<br>    time_horizon: Time horizon for the position discussed in the comments.<br>    is_implicit: Whether the ticker was mentioned explicitly or inferred from context.<br>    - primary_ticker: Primary ticker discussed in the submission, if identifiable.<br>    - relationships: List of `TickerRelationship`, each consists of:<br>        ticker1: First ticker in the relationship<br>        ticker2: Second ticker in the relationship, if exists<br>        relationship_type: Type of relationship (inverse/correlated/causal)<br>        description: Brief description of the relationship, how it was inferred<br>        confidence: Confidence score (1-10) for the relationship extraction<br>&quot;&quot;&quot;<br><br>extractor_content = &quot;&quot;&quot;<br>TITLE:<br>{title_text}<br><br>SUBMISSION:<br>{submission_text}<br><br>COMMENTS:<br>{comments_text}<br>&quot;&quot;&quot;</pre><p>Using the title, submission text, and comments (up to 100 per submission), I interacted with OpenAI’s GPT-4.1-nano via LangChain for structured sentiment extraction. Batch processing further improved efficiency, employing RangeBatchOrchestrator to manage OpenAI batch jobs—faster, cheaper, and simpler orchestration.</p><pre>+-------------------------------------------------------------------+<br>|                      SENTIMENT ANALYSIS PIPELINE                  |<br>+-------------------------------------------------------------------+<br>|                                                                   |<br>|  +---------------------+                                          |<br>|  | RedditContent       |                                          |<br>|  | (Raw Submission/Cmt)|                                          |<br>|  +---------------------+                                          |<br>|             |                                                     |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | ContentLoader       |                                          |<br>|  | (Protocol)          |                                          |<br>|  | - LocalFileContentL.|                                          |<br>|  | - S3ContentLoader   |                                          |<br>|  +---------------------+                                          |<br>|             | Loads RedditContent                                 |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | SentimentPrescorer  |                                          |<br>|  | (Abstract)          |                                          |<br>|  | - AlwaysYesPrescorer|                                          |<br>|  | - RelevancePrescorer|                                          |<br>|  +---------------------+                                          |<br>|             | Filters relevant RedditContent                      |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | SentimentExtractor  |                                          |<br>|  | (LLM Interaction)   |                                          |<br>|  | - AsyncSentimentExt.|                                          |<br>|  | - AsyncRateLimiter  |                                          |<br>|  +---------------------+                                          |<br>|             | Extracts List[SentimentResults]                     |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | SentimentDeduplicator|                                          |<br>|  | (Consolidates Sent.)|                                          |<br>|  +---------------------+                                          |<br>|             | Produces consolidated SentimentResults              |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | SentimentStorage    |                                          |<br>|  | (Protocol)          |                                          |<br>|  | - LocalFileSent.S.  |                                          |<br>|  | - IncrementalSent.S.|                                          |<br>|  +---------------------+                                          |<br>|             | Stores processed SentimentResults                   |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | TickerSentiment     |                                          |<br>|  | Aggregator          |                                          |<br>|  | (Creates DailyCons.)|                                          |<br>|  +---------------------+                                          |<br>|             | Produces DailyTickerConsensus                       |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | ConsensusStorage    |                                          |<br>|  | (Protocol)          |                                          |<br>|  | - LocalConsensusS.  |                                          |<br>|  | - S3ConsensusS.     |                                          |<br>|  | - HybridConsensusS. |                                          |<br>|  +---------------------+                                          |<br>|             | Stores DailyTickerConsensus                         |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | Orchestrators       |                                          |<br>|  | (Abstract Base)     |                                          |<br>|  | - SentimentProc.Orch.|                                          |<br>|  | - IncrementalSent.Orch.|                                          |<br>|  | - RangeBatchOrch.   |                                          |<br>|  +---------------------+                                          |<br>|             | Manages overall workflow, uses components above     |<br>|             v                                                     |<br>|  +---------------------+                                          |<br>|  | ProcessingCheckpoint|                                          |<br>|  | (Tracks Progress)   |                                          |<br>|  +---------------------+                                          |<br>|                                                                   |<br>+-------------------------------------------------------------------+<br>|                           KEY DATA MODELS                         |<br>+-------------------------------------------------------------------+<br>|                                                                   |<br>|  - SentimentResults, Sentiment, TickerRelationship                |<br>|  - ModelConfig, AsyncExtractionConfig, DeduplicationConfig        |<br>|  - ProcessingResult, BatchProcessingResult, DailyUpdate           |<br>|  - DailySentimentBatch, ProcessingCheckpoint, ConflictInfo        |<br>|  - TickerAggregateStats, DailyTickerConsensus                     |<br>|  - PathConfig, BatchPathConfig, BatchConfig, FlagsConfig          |<br>|                                                                   |<br>|  Enums: DeduplicationStrategy, ConflictResolution                 |<br>|                                                                   |<br>+-------------------------------------------------------------------+</pre><h3>Part 4: Aggregating sentiment</h3><p>Post-processing involved aggregating sentiment per ticker daily. For this, I used:</p><pre>@dataclass<br>class TickerAggregateStats:<br>    &quot;&quot;&quot;Aggregated statistics for a single ticker across all submissions&quot;&quot;&quot;<br>    ticker: str<br>    total_mentions: int<br>    sentiment_distribution: Dict[str, int]  # {&#39;bullish&#39;: 5, &#39;bearish&#39;: 2, &#39;neutral&#39;: 3}<br>    <br>    # Weighted averages<br>    avg_confidence: float<br>    avg_sentiment_intensity: float<br>    confidence_weighted_intensity: float<br>    <br>    # Categorical distributions<br>    position_distribution: Dict[str, int]<br>    conviction_distribution: Dict[str, int] <br>    time_horizon_distribution: Dict[str, int]<br>    source_distribution: Dict[str, int]<br>    <br>    # Submission-level stats<br>    unique_submissions: int<br>    submissions_as_primary: int  # How many times this was the primary ticker<br>    <br>    # Quality metrics<br>    explicit_mentions: int<br>    implicit_mentions: int<br>    avg_confidence_by_sentiment: Dict[str, float]<br>    <br>    # Consensus metrics<br>    sentiment_consensus_strength: float  # 0-1, how unified the sentiment is<br>    dominant_sentiment: str<br>    dominant_sentiment_percentage: float<br>    <br>    # NEW: Polarization and conviction metrics<br>    sentiment_polarization: float = 0.0  # 0-1, higher = more polarized<br>    sentiment_controversy_score: float = 0.0  # Custom controversy metric<br>    is_controversial: bool = False  # Flag for highly polarized stocks<br>    conviction_weighted_bullish_score: float = 0.0<br>    conviction_weighted_bearish_score: float = 0.0<br>    avg_bullish_conviction: float = 0.0<br>    avg_bearish_conviction: float = 0.0<br>    date: str = None</pre><p>TickerSentimentAggregator combined individual submission results into clean, aggregated sentiment scores for each ticker.</p><h3>Part 5: EDA of sentiment and stock prices</h3><p>Finally, after extensive pre-processing, I obtained clean, daily ticker-specific sentiments aligned with stock data. Diagnostic plots began with Apple’s 2017–2025 sentiment and price relationship:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vw9b9ekqVPweVlNERLk4dA.png" /></figure><p>Patterns observed without formal tests:</p><ul><li>Sentiment dipped from 2021 to 2023, coinciding with Apple’s 2022–2023 price decline.</li><li>Sentiment exhibited cyclical spikes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yBAVnaKv_cxATILqqPWIuA.png" /></figure><p>Examining the predictive relationship (sentiment vs. future 25-day returns) showed different trends for 2017 and 2018, indicating complexity in the sentiment-return relationship.</p><pre>+--------+--------------+------------+----------+-------------+-------------+-----------+-----------+------+<br>| ticker | horizon_days | ema window | p_value  | coefficient | signal_type | r_squared | intercept | year |<br>+--------+--------------+------------+----------+-------------+-------------+-----------+-----------+------+<br>| AAPL   |           15 |         20 | 0.000001 |    -1.55269 | Contrarian  |  0.116504 |  4.147986 | 2017 |<br>| AAPL   |           10 |          3 | 0.090545 |    0.362849 | Momentum    |  0.020471 | -0.643544 | 2018 |<br>| AAPL   |           20 |         20 | 0.000466 |   -1.834835 | Contrarian  |  0.063887 |  6.507036 | 2019 |<br>| AAPL   |            2 |          3 | 0.014975 |   -0.404165 | Contrarian  |  0.040137 |   1.22702 | 2020 |<br>| AAPL   |           20 |         20 | 0.000785 |   -1.540919 | Contrarian  |  0.058669 |  5.499994 | 2021 |<br>| AAPL   |           15 |         15 | 0.037882 |   -1.291089 | Contrarian  |  0.022607 |  0.704856 | 2022 |<br>| AAPL   |           20 |         15 | 0.000523 |    1.916608 | Momentum    |  0.063463 |   0.20881 | 2023 |<br>| AAPL   |           20 |         20 | 0.000705 |    1.856422 | Momentum    |   0.06031 | -0.433795 | 2024 |<br>| AAPL   |           20 |          7 | 0.062843 |   -1.170055 | Contrarian  |  0.038778 | -0.507907 | 2025 |<br>+--------+--------------+------------+----------+-------------+-------------+-----------+-----------+------+</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w0X3O1wDntVGWvNRxvoB-Q.png" /></figure><p>OLS regression (fwd X-day return = α + β·sent_Yd_EMA) across all years revealed statistically significant but low R² results, emphasizing sentiment&#39;s limited predictive power alone. A rolling window (220 days, EMA 45, 40-day horizon) visualized varying predictive power across periods, clearly showing the model’s effectiveness fluctuates.</p><p>Conclusion: While sentiment alone isn’t reliable for trading, adding filters (e.g., volatility) could improve usability.</p><h3>Part 6: Financial modelling</h3><p>Next, I constructed a basic trading system, strictly using sentiment-based predictions. Assumptions:</p><ul><li>Sentiment-only features.</li><li>Linear model predictions of returns using past sentiment and returns.</li><li>Long/short trading with realistic considerations (slippage, commissions).</li><li>Exits determined by take-profit, stop-loss, or holding duration defined by OLS.</li></ul><p>Let’s also outline an important thing here. A very common mistake I see in many ML projects that are based on time-series data is leaking the data or looking into the future. Here is my attempt to avoid that.</p><p>If I want to make a trade on December 15th, I can only use data available up to that date. For instance, if I’m building an OLS regression model like:</p><pre>fwd X‑day return = α + β · sent_Yd_EMA</pre><p>I <strong>cannot</strong> use sentiment data from December 15th itself to predict returns after that date. Suppose I’m predicting the 10-day forward return: my latest valid sentiment observation would be from December 5th. Thus, the December timeline would look like this:</p><pre>December timeline (each cell = 1 trading day)                     ▶ time<br>┌────────────────────────────────────────────────────────────────────────────┐<br>│   01   02   03   04   05   06   07   08   09   10   11   12   13   14   15│<br>│   S    S    S    S    S    S    S    S    S    P    •    •    •    •    ▲ │<br>└────────────────────────────────────────────────────────────────────────────┘<br><br>S  = sentiment observations we areallowed to use  <br>P  = last **price** that can appear in a *label* (because 10‑day fwd return  <br>     ending on 10 Dec is the last one you can fully observe)  <br>•  = five‑day embargo window (10 – 14 Dec) — **strictly off‑limits**  <br>▲  = trade date (15 Dec) — you’re trying to predict the 10‑day return  <br>     from 15 Dec → 25 Dec, which hasn’t happened yet</pre><p>If I train the model using a rolling window that ends before December 1st, I guarantee no leakage. Conceptually, for each trading week, I reuse the same model trained on the preceding 220 days, ending approximately two weeks prior to the trading period.</p><p>To find the best predictive parameters (“horizon days” and “EMA windows”), I do not directly optimize financial metrics such as the Sharpe ratio. Instead, I select the parameters that best explain returns for the upcoming horizon window based on historical fit. The image below visualizes how predictive power (R²) varies over time for different combinations of parameters for Apple (AAPL):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*95KGAkEGgxztjB1srr4Nbw.png" /></figure><p>From this heatmap, we clearly observe that sentiment’s predictive ability fluctuates significantly. Sentiment-based predictions were strongest during certain periods (e.g., late 2017, mid-2020), while in others (e.g., most of 2022–2025), sentiment provided little to no predictive value.</p><p>This underscores a critical point: relying solely on sentiment data for stock return predictions might not be robust. Additional signals or filtering methods (such as volatility-based filters) are likely necessary to build a dependable, production-ready trading system.</p><p>After that we use that model to make trades. Here is a diagram how we enter\exit trades</p><pre>┌──────────────────────────────────────────────────────────────────────────────┐<br>│ MAIN WALK‑FORWARD LOOP ‑‑ iterate i = 1 … len(df)                            │<br>└──────────────────────────────────────────────────────────────────────────────┘<br>        │<br>        ▼<br>┌───── 1. READ CURRENT ROW ───────────────────────────────────────────────────┐<br>│   date  = df.iloc[i][&quot;Date&quot;]                                                │<br>│   price = df.iloc[i][&quot;Close&quot;]                                               │<br>└──────────────────────────────────────────────────────────────────────────────┘<br>        │<br>        ▼<br>┌───── 2. MARK‑TO‑MARKET &amp; EXIT EXISTING POSITIONS ───────────────────────────┐<br>│  for pos in positions:                                                      │<br>│     ● MTM:                                                                  │<br>│        long →  pos_val = price * shares                                     │<br>│        short → pos_val = entry_val + (entry_px – price) * shares            │<br>│        borrow_cost += borrow_fee_per_day (short only)                       │<br>│        days_held++                                                          │<br>│     ● EXIT TEST:                                                            │<br>│        time‑based  : days_held ≥ horizon                                    │<br>│        stop‑based  : pos_val ≤ 0.85·entry_val  OR  ≥ 1.45·entry_val         │<br>│        ├─ if TRUE  → close trade                                            │<br>│        │             • slip_px, commission                                  │<br>│        │             • pnl = (exit_px – entry_px) * shares  – fees          │<br>│        │             • capital += entry_val + pnl                           │<br>│        │             • record trade dict → trades[]                         │<br>│        └─ if FALSE → keep → new_positions[]                                 │<br>└──────────────────────────────────────────────────────────────────────────────┘<br>        │<br>        ▼<br>┌───── 3. BOOK‑KEEPING ────────────────────────────────────────────────────────┐<br>│ positions   = new_positions                                                  │<br>│ equity_curve[date] = capital + Σ pos.current_val                             │<br>└──────────────────────────────────────────────────────────────────────────────┘<br>        │<br>        ▼<br>┌───── 4. ENTRY LOGIC (if signal &amp; slot free) ────────────────────────────────┐<br>│  can_enter? → row[&quot;is_significant&quot;]  AND  (allow_multi OR not positions)    │<br>│  ├─ if FALSE → skip                                                         │<br>│  └─ if TRUE  →                                                              │<br>│       pred   = row[&quot;predicted_return&quot;]   (% over horizon)                   │<br>│       sigma  = row[&quot;realised_vol_20d&quot;]                                      │<br>│       side   = long  if  pred &gt; +threshold                                  │<br>│                short if  pred &lt; –threshold                                  │<br>│       SIZE %:                                                               │<br>│          fixed pct  OR  pct∝R²  OR  Kelly = |pred| / σ²                     │<br>│          clamp between min_pct and max_pct                                  │<br>│       dollars        = pct * capital                                        │<br>│       slip_px        = price ± slippage_bps/2                               │<br>│       shares         = dollars / slip_px                                    │<br>│       commission     = commission_per_share * shares                        │<br>│       capital       -= (dollars – commission)                               │<br>│       append new position {side, entry_px, shares, …}                       │<br>└──────────────────────────────────────────────────────────────────────────────┘</pre><ul><li>Important note: we have different way to size our position: flat, kelly-based, or model-R²-based (higher R² -&gt; higher confidence in predictions -&gt; higher risk)</li><li>Important note 2: Stop loss and take profit are completely eyeballed here.</li></ul><p>What are we trying to achieve here? We are trying to beat a simple “buy &amp; hold strategy”, where just invest our capital in the beggining and wait. The stocks only go up, right? Let’s take a look at couple of interesting examples:</p><h4><strong>Example 1: Google</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uNorkj8Faw7lXaZ-a032TQ.png" /><figcaption>Fig1: R² of fitted models. Y axis: Horizon days (Windows for forward return)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*Bi0BIXZpaGRTm4aBsDTKwQ.png" /><figcaption>Fig2: Example of a suspicions window: Almost perfect correlation between return and sentiment</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lf3EBnzpd1eIGfylBaS7_w.png" /><figcaption>Fig3: Dianostic plot</figcaption></figure><p><strong>How to read Figure 3:</strong></p><ul><li><strong>Panel 1 (Top): Dashed yellow line</strong>: Google stock price; <strong>Dashed blue line</strong>: Buy-and-hold strategy performance.; <strong>Solid blue line</strong>: Sentiment-based trading strategy performance. <strong>Triangles</strong>: Red upward triangle: Enter short position; Red downward triangle: Exit short position; Green upward triangle: Enter long position; Green downward triangle: Exit long position</li><li><strong>Panel 2</strong>: Daily sentiment scores (violet dots) plotted alongside the same entry and exit triangles from Panel 1 + stock price</li><li><strong>Panel</strong>: Rolling OLS slope using historical data with a 220-day window, indicating the relationship strength between sentiment and forward returns.</li><li><strong>Panel 4</strong>: Statistical significance of the slope, shown as negative log10(p-values). Higher values indicate greater significance.</li><li><strong>Panel 5 (Bottom)</strong>: R², showing the proportion of variance in stock returns explained by sentiment.</li></ul><p><strong>Key insights from Figure 3</strong>:</p><ul><li><strong>Win rate</strong>: 65%, across 67 total trades.</li><li><strong>Information Coefficient (IC)</strong>: 0.17, indicating modest predictive strength.</li><li><strong>Strategy performance</strong>: 172% return vs. 353% for buy-and-hold.</li><li>Critically, sentiment accurately identified price trends in 2022–2023, demonstrating clear potential despite overall weaker returns compared to buy-and-hold. Given Google’s popularity, the dataset is rich and informative.</li></ul><p>Note: this strategy is far from perfect, for example, many long trades probably hurting it a lot, but the idea here was to understand if there is any signal at all.</p><h4><strong>Example 2: ACB</strong></h4><p>This is an example where Reddit sentiment fails: Once recreational‑cannabis hype faded, there was no new catalyst; social chatter persisted briefly but couldn’t move the tape. Reddit sentiment can <em>predict</em> in story‑driven names <strong>only while the story is alive </strong>(And look at ine trade that failed everything!)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IT77LNQPQpvqaAEe7sLlzw.png" /></figure><h4><strong>Example 3: Intel</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IHLzUeZ9ljBlF5uu1xrXMA.png" /></figure><p>Reddit didn&#39;t get Intel right before 2022! The crowd’s chatter helped spot when <strong>not</strong> to own Intel but provided little foresight the rest of the time.</p><h3>Final</h3><p>Reddit definitely can serve as a valuable supplementary signal in trading systems — but only when combined thoughtfully with additional market-driven features. The system demonstrated here can certainly be improved. Specifically, refining entry and exit strategies, optimizing position sizing based on sentiment confidence and market volatility, and integrating risk management could capture significant additional gains that are currently missed.</p><p>While this analysis clearly shows that sentiment tracks stock price movements, it also highlights the challenge: sentiment alone is insufficient. Future improvements might involve combining sentiment signals with traditional financial indicators such as momentum, volatility, and volume metrics, or even using sentiment as a conditional filter rather than a standalone decision driver.</p><p>Ultimately, the critical insight here is that there <strong>is</strong> a measurable predictive signal hidden within Reddit chatter. The real question moving forward is: how can we best harness this signal, systematically and efficiently, to achieve consistent trading performance?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25b5b2dc86a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating Molecular Representations for Property Prediction: A Computational Chemistry Approach…]]></title>
            <link>https://sergeykolchenko.medium.com/evaluating-molecular-representations-for-property-prediction-a-computational-chemistry-approach-93246c75be6a?source=rss-e626cc224497------2</link>
            <guid isPermaLink="false">https://medium.com/p/93246c75be6a</guid>
            <dc:creator><![CDATA[Sergey Kolchenko]]></dc:creator>
            <pubDate>Mon, 02 Oct 2023 00:57:30 GMT</pubDate>
            <atom:updated>2023-10-02T00:57:30.152Z</atom:updated>
            <content:encoded><![CDATA[<h3>Evaluating Molecular Representations for Property Prediction: A Computational Chemistry Approach Using Molfeat and AutoGluon</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nrrg6O9IM0Hh4dBlA_pwqQ.jpeg" /><figcaption>Generated witl Bing &amp; DALL-E</figcaption></figure><p>Computational chemistry is a fascinating domain where the challenges are as intriguing as the possibilities, especially for a non-specialist in computational chemistry like me. Handling small molecules brings us to an important question: how do we represent them? Traditionally, SMILES (Simplified Molecular Input Line-Entry System) strings have been the go-to option, as they offer a text-based representation of molecules. But how do you make machine learning models understand these strings? There are various methods, such as computing Morgan fingerprints, which encode groups of atoms into a unique binary vector. But the possibilities don’t end there; emerging techniques like ChemGPT can also offer latent space representations. So, which method is optimal? The answer isn’t straightforward. In this blog post, I’ll explore molfeat, a highly versatile framework for aggregating various molecular representations, in conjunction with Autogluon, an AutoML package designed to simplify the modeling process. We’ll test these various molecular representations across 18 different benchmarks from tdcommons.ai to determine which is most effective.</p><h3>Code</h3><p>Admittedly, I’m inclined towards automation (in other words, lazy), so I streamlined the data processing and model building phases.</p><ol><li>Let’s focus on SMILES standardization.</li></ol><pre>import datamol as dm<br>import anndata as ad<br>import numpy as np<br>import pandas as pd<br>from typing import List, Optional, Callable<br><br>from molfeat.trans.fp import FPVecTransformer<br>from molfeat.trans import MoleculeTransformer<br>from molfeat.trans.pretrained.hf_transformers import PretrainedHFTransformer<br>from molfeat.trans.pretrained import GraphormerTransformer<br>from molfeat.trans.pretrained import PretrainedDGLTransformer<br>from molfeat.calc.pharmacophore import Pharmacophore2D<br>import numpy as np<br>import anndata as ad<br><br>from autogluon.tabular import TabularDataset, TabularPredictor<br>from autogluon.tabular.configs.hyperparameter_configs import get_hyperparameter_config<br><br>def preprocess_smiles_str(smiles: str) -&gt; str:<br>    mol = dm.to_mol(smiles, ordered=True)<br>    mol = dm.fix_mol(mol)<br>    mol = dm.sanitize_mol(mol, sanifix=True, charge_neutral=False)<br>    mol = dm.standardize_mol(<br>        mol,<br>        disconnect_metals=False,<br>        normalize=True,<br>        reionize=True,<br>        uncharge=False,<br>        stereo=True,<br>    )<br>    return dm.standardize_smiles(dm.to_smiles(mol))<br><br>def preprocess_smiles_anndata(<br>    anndata: ad.AnnData,<br>    smiles_column: str = &#39;SMILES&#39;,<br>    clean_smiles_columns: str = &#39;standard_smiles&#39;<br>) -&gt; ad.AnnData:<br>    anndata.obs[clean_smiles_columns] = anndata.obs[smiles_column].apply(preprocess_smiles_str)<br>    return anndata</pre><p>2. Generate different representations.</p><pre><br><br>def generate_representations_for_smiles(<br>    records: np.ndarray,<br>    transformer: Callable[[str], np.ndarray],<br>    batch_size: int = 128,<br>    **kwargs<br>) -&gt; np.ndarray:<br>    features_list = []<br>    n_batches = len(records) // batch_size<br>    batch_iter = batch_indexable(records, batch_size)<br>    for batch in batch_iter:<br>        try:<br>            features = transformer.transform(batch)<br>        except Exception as e:<br>            print(&#39;Error in batch&#39;, batch, transformer, e)<br>        features_list.append(features)<br>    features_list = np.concatenate(features_list)<br>    return features_list<br><br>def generate_representations_for_anndata(<br>        input_anndata: ad.AnnData,<br>        smiles_column: str = &#39;SMILES&#39;,<br>        fpvec_models: Optional[List[str]] = None,<br>        chemgpt_models: Optional[List[str]] = None,<br>        graphormer_models: Optional[List[str]] = None,<br>        dgl_models: Optional[List[str]] = None,<br>        pharmacophore_models: Optional[List[str]] = None,<br>        hft_models: Optional[List[str]] = None,<br>        molecule_models: Optional[List[str]] = None,<br>        batch_size: int = 128,<br>        device: str = &#39;cpu&#39;,<br>        **kwargs<br>) -&gt; ad.AnnData:<br>    dm.disable_rdkit_log()<br>    fpvec_models = fpvec_models or DEFAULT_FPVEC_MODELS<br>    chemgpt_models = chemgpt_models or DEFAULT_CHEMGPT_MODELS<br>    graphormer_models = graphormer_models or DEFAULT_GRAPHORMER_MODELS<br>    dgl_models = dgl_models or DEFAULT_DGL_MODELS<br>    pharmacophore_models = pharmacophore_models or DEFAULT_PHARMACOPHORE_MODELS<br>    hft_models = hft_models or DEFAULT_HFT_MODELS<br>    molecule_models = molecule_models or DEFAULT_MOLECULE_MODELS<br><br>    records = input_anndata.obs[smiles_column].values<br>    # FPVecModels<br>    for model_type in fpvec_models:<br>        transformer = FPVecTransformer(kind=model_type, dtype=float)<br>        input_anndata.obsm[model_type] = generate_representations_for_smiles(records, transformer, batch_size=batch_size)<br><br>    # PharmacophoreModels<br>    for model_type in pharmacophore_models:<br>        transformer = MoleculeTransformer(featurizer=Pharmacophore2D(factory=model_type), dtype=float)<br>        input_anndata.obsm[model_type] = generate_representations_for_smiles(records, transformer, batch_size=batch_size)<br><br>    # MoleculeModels<br>    for model_type in molecule_models:<br>        transformer = MoleculeTransformer(featurizer=model_type, dtype=float)<br>        input_anndata.obsm[model_type] = generate_representations_for_smiles(records, transformer, batch_size=batch_size)<br>    # HFTransformers<br>    for model_type in hft_models:<br>        transformer = PretrainedHFTransformer(kind=model_type, notation=&#39;smiles&#39;, dtype=float, device=device, pooling=&#39;mean&#39;, random_seed=42, preload=True)<br>        input_anndata.obsm[model_type] = generate_representations_for_smiles(records, transformer, batch_size=batch_size)<br><br>    # ChemGPTModels<br>    for model_type in chemgpt_models:<br>        transformer = PretrainedHFTransformer(kind=model_type, notation=&#39;selfies&#39;, dtype=float, device=&#39;gpu&#39;, pooling=&#39;mean&#39;, random_seed=42, preload=True)<br>        input_anndata.obsm[model_type] = generate_representations_for_smiles(records, transformer, batch_size=batch_size)<br><br>    # GraphormerModels<br>    for model_type in graphormer_models:<br>        transformer = GraphormerTransformer(kind=model_type, dtype=float, pooling=&#39;mean&#39;, preload=True)<br>        input_anndata.obsm[model_type] = generate_representations_for_smiles(records, transformer, batch_size=batch_size)<br><br>    # DGLModels<br>    for model_type in dgl_models:<br>        transformer = PretrainedDGLTransformer(kind=model_type, dtype=float, pooling=&#39;mean&#39;, preload=True)<br>        input_anndata.obsm[model_type] = generate_representations_for_smiles(records, transformer, batch_size=batch_size)<br><br>    return input_anndata</pre><p>Generating representations for an input AnnData object involves storing them in its .obsm attribute. So what exactly is AnnData? Simply put, AnnData is a flexible structure that combines the best of numpy arrays and pandas DataFrames to manage annotated data. It allows for the storage of annotations related to your samples, features, and even multiple layers of data. In our case, we use the .obsm property to hold various matrices, each of shape (n_samples, n_representation). Although the sample order remains constant, the feature dimensions may vary among different representations. For a deeper dive into this versatile package, the official <a href="https://anndata.readthedocs.io/en/latest/">documentation</a> is a rich resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1QaOjr8AXSxYcFK2GM7Svg.png" /><figcaption>anndata structure, from <a href="https://anndata.readthedocs.io/en/latest/">https://anndata.readthedocs.io/en/latest/</a></figcaption></figure><p>Different model frameworks — be it ChemGPT, DGL, or FPVec — come with their own classes and input parameters. Therefore, we iterate through each model type and store the respective molecular representations. Specifically, we apply feature extraction on every SMILES string from the smiles_column in the input AnnData object.</p><p>3. Run AutoML</p><pre>def fit_automl(<br>        train_ad: ad.AnnData,<br>        test_ad: Optional[ad.AnnData],<br>        label_col: str,<br>        task_problem: str,<br>        task_metric: str,<br>        task_name: str,<br>        val_ad: Optional[ad.AnnData] = None,<br>        time_limit: int = 60,<br>        presets: str = &#39;good_quality&#39;<br>) -&gt; pd.DataFrame:<br>    task_results = []<br>    hyperparameters = get_hyperparameter_config(&#39;default&#39;)<br>    hyperparameters.pop(&#39;KNN&#39;)  # KNN never worked :( <br>    for representation_name in train_ad.obsm.keys():<br>        print(f&#39;Fitting {representation_name}&#39;)<br>        representation_shape = train_ad.obsm[representation_name].shape[1]<br>        feature_names = [&#39;feature_&#39; + str(x) for x in range(representation_shape)]<br>        train_dataset = pd.DataFrame(train_ad.obsm[representation_name], columns=feature_names)<br>        train_dataset[label_col] = train_ad.obs[label_col].values<br>        train_dataset = TabularDataset(train_dataset)<br>        if val_ad is not None:<br>            val_dataset = pd.DataFrame(val_ad.obsm[representation_name], columns=feature_names)<br>            val_dataset[label_col] = val_ad.obs[label_col].values<br>            val_dataset = TabularDataset(val_dataset)<br><br>        test_dataset = pd.DataFrame(test_ad.obsm[representation_name], columns=feature_names)<br>        test_dataset[label_col] = test_ad.obs[label_col].values<br>        test_dataset = TabularDataset(test_dataset)<br><br>        predictor = TabularPredictor(<br>            problem_type=task_problem,<br>            eval_metric=task_metric,<br>            path=f&#39;models/{task_name}__{representation_name}&#39;,<br>            label=label_col,<br>            verbosity=1,<br>            log_to_file=True,<br>            log_file_path=f&#39;logs/{task_name}__{representation_name}&#39;<br>        )<br>        predictor.fit(<br>            train_dataset,<br>            tuning_data=val_dataset if val_ad is not None else None,<br>            use_bag_holdout=True if val_ad is not None else False,<br>            time_limit=time_limit,<br>            hyperparameters=hyperparameters,<br>            presets=presets<br>        )<br>        results = pd.DataFrame(predictor.evaluate(test_dataset, silent=True), index=[representation_name])<br>        results[&#39;task&#39;] = task_name<br>        task_results.append(results)<br>    return pd.concat(task_results)</pre><p>The crux of the operation lies in this function: it fits an autoML model for each representation stored in the .obsm attribute of the input AnnData file. We use both training and validation data sets, especially if early stopping is a consideration. However, it&#39;s important to note that we only record the performance score of the model. The actual model type—whether it&#39;s XgBoost, logistic regression, or a decision tree—is irrelevant to our primary question: What is the optimal performance achievable using a specific molecular representation?</p><p>4. Example of running a single dataset from TDC</p><pre>task_name = &#39;DILI&#39;<br>data = Tox(name=task_name)<br>split = data.get_split()<br>label_col = &#39;Y&#39;<br>train_ad = ad.AnnData(obs=split[&#39;train&#39;])<br>val_ad = ad.AnnData(obs=split[&#39;valid&#39;])<br>test_ad = ad.AnnData(obs=split[&#39;test&#39;])<br><br>train_ad = preprocess_smiles_anndata(train_ad, smiles_column=&#39;Drug&#39;)<br>val_ad = preprocess_smiles_anndata(val_ad, smiles_column=&#39;Drug&#39;)<br>test_ad = preprocess_smiles_anndata(test_ad, smiles_column=&#39;Drug&#39;)<br><br>train_ad = generate_representations_for_anndata(train_ad, smiles_column=&#39;standard_smiles&#39;)<br>val_ad = generate_representations_for_anndata(val_ad, smiles_column=&#39;standard_smiles&#39;)<br>test_ad = generate_representations_for_anndata(test_ad, smiles_column=&#39;standard_smiles&#39;)<br><br>task_problem = &#39;binary&#39;<br>task_metrics = &#39;average_precision&#39;<br><br>task_results = fit_automl(<br>    train_ad=train_ad,<br>    test_ad=test_ad,<br>    label_col=label_col,<br>    task_problem=task_problem,<br>    task_metric=task_metrics,<br>    task_name=task_name,<br>    val_ad=val_ad,<br>    time_limit=time_limit,<br>    presets=presets    <br>)<br>task_results.to_csv(f&#39;results/{task_name}.csv&#39;)</pre><p>To illustrate, let’s consider the task of predicting DILI (Drug-Induced Liver Injury). Essentially, the goal is to ascertain if a particular molecule will cause liver damage. In this example, we obtain the data, divide it into training, validation, and test sets, preprocess the SMILES strings, generate all feasible molecular representations, and finally fit an autoML model. The target metric used here is average precision.</p><h3><strong>Results</strong></h3><p>In this section, we iterate over all processed datasets, ranking each molecular representation based on performance for a given problem. The top-performing representation for a particular dataset earns 10 points, the second-best earns 9 points, and so on, down to the tenth best, which earns 1 point. Representations beyond the tenth receive zero points. As I delved into this analysis, I implemented two key modifications.</p><p><strong>First</strong>, several tasks were trivial for any molecular representation. For instance, predicting drug absorption from the human gastrointestinal system into the bloodstream posed little challenge, with average precision scores consistently above 0.9, regardless of the representation. Consequently, I chose not to award points for such tasks. Specifically, if the best representation didn’t outperform the average by at least 10%, no points were awarded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ckrpip5BxvjkWE4jP6dVw.png" /><figcaption>Predicting human intestinal absorption is a difficult task, just not in this dataset — whatever you use, you get .9+ AP!</figcaption></figure><p><strong>Second</strong>, I introduced a scaling factor to the points awarded. If the top-performing representation is twice as good as the mean performance (across all representations), and the second-best is 1.15 times better than the average, the raw scores are then scaled by their “performance gain over the mean,” calculated as score[i] / mean(score). While statisticians might raise eyebrows at this approach, it aligns with my intuition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vYw9snrjhGkx54DVp5I0gQ.png" /><figcaption>gin_supervised_edgepred performed way better than average on this task…</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gNFP8KWanVt2KmmGDbveLA.png" /><figcaption>which resulted in extra points!</figcaption></figure><p>After processing 18 tasks, 9 were discarded for being too easy and therefore not suitable benchmarks for computational chemistry. Summing the points across the remaining 9 tasks, the top-performing molecular representations were identified:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yxKxhK970LLcsYHPgArVAA.png" /></figure><p>It might not come as a surprise for somebody that desc2D, numerical representation of chemical and physical properties of a molecule, are the best for representation when we need to predict something that is directly related to the molecular properties (that sounds like a natural choice for me). Two other interesting models, that are worth trying when building a molecular predictors, are ECFP (Extended-connectivity fingerprints) and two graph NN models coming from <a href="https://arxiv.org/abs/1905.12265">the same paper</a>. Perhaps not surprisingly, Transformer-based models didn’t perform well in this benchmark (alghough here no fine tuning was performed and I used the raw latent representation).</p><h3>Conclusion</h3><p>Molfeat is a robust library that enables various molecular representations with minimal changes to your existing codebase. Using Autogluon, you can bypass the arduous task of model selection, focusing instead on curating and understanding your dataset. By integrating Molfeat and Autogluon, I evaluated an array of molecular representations using datasets from the Therapeutics Data Commons (TDC). However, as pointed out by Pat Walters, the quest for accurate molecular property prediction demands more diverse and high-quality datasets to validate the efficacy of these models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93246c75be6a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The best approach to semantic segmentation of biomedical images]]></title>
            <link>https://medium.com/data-science/the-best-approach-to-semantic-segmentation-of-biomedical-images-bbe4fd78733f?source=rss-e626cc224497------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbe4fd78733f</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[segmentation]]></category>
            <dc:creator><![CDATA[Sergey Kolchenko]]></dc:creator>
            <pubDate>Fri, 18 Dec 2020 18:10:19 GMT</pubDate>
            <atom:updated>2020-12-21T19:09:17.738Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aWK6yDaBe7x5viez2VyeMw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@clark_fransa?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Arnold Francisca</a> on <a href="https://unsplash.com/s/photos/coding?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3><strong>Intro.</strong></h3><p>Semantic segmentation is a problem of computer vision in which our task is to assign a class to each pixel in the image using that image as an input. In the case of semantic segmentation, we don’t care if we have multiple instances (objects) of the same class, we simply label them all with their class. There are multiple courses\reviews on different computer vision problems (for example <a href="https://nanonets.com/blog/semantic-image-segmentation-2020/">https://nanonets.com/blog/semantic-image-segmentation-2020/</a>), but one picture can summarize different computer vision problems better than a description</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aeiRv2XyE4N2n1kXf47cuw.png" /><figcaption>source: <a href="http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture11.pdf">http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture11.pdf</a></figcaption></figure><p>Semantic segmentation has lots of applications for biomedical image analysis: X-ray\MRI scans, digital pathology, microscopy, endoscopy, etc. The list of challenges at <a href="https://grand-challenge.org/challenges">https://grand-challenge.org/challenges</a> is quite impressive, there are lots of different interesting and important questions to explore.</p><p>From the technical point of view, if we think about the semantic segmentation problem, for the N*M*3 (let’s assume we have an RGB image) image we want to produce the corresponding map N*M*k (where k is the number of classes). There are many architectures that can solve this problem, but here I want to talk about two particular architectures, Unet and Unet++.</p><p><strong>TL</strong>;<strong>DR: If you have a semantic segmentation problem start with Unet++ with resnest200e encoder (available at </strong><a href="https://github.com/qubvel/segmentation_models.pytorch">https://github.com/qubvel/segmentation_models.pytorch</a>)</p><p>There are many reviews on Unet, how it changed the field forever (the original publication now has 20+k citations, quite impressive). It is unitively very clear architecture, which consists of an encoder, which generates a representation of the image and a decoder, which uses that representation to build the segmentation. Two maps at each spatial resolution are concatenated (the grey arrows), so you combine two different representations of the image together. And it worked!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lvXoKMHoPJMKpKK7keZMEA.png" /><figcaption>source: <a href="https://arxiv.org/abs/1505.04597">https://arxiv.org/abs/1505.04597</a></figcaption></figure><p>The next bog thing was using a pre-trained encoder. Think about the image classification problem. Effectively, we try to build a feature representation of the image, such that different classes in that feature space can be separated. We can take (almost) any CNN and use it as an encoder, take features from that encoder and feed them to our decoder. To my best knowledge, one of the first published papers of that kind were from <a href="https://arxiv.org/abs/1801.05746">Iglovikov &amp; Shvets</a> and <a href="https://arxiv.org/pdf/1806.05182.pdf">Buslaev et al.</a>, where the group of authors used VGG11 and resnet34 respectively to generate better features for Unet decoder and boost its performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*z92nyr9VD7vkjwD-i6zMkg.png" /><figcaption>TernausNet (VGG11 Unet), source: <a href="https://github.com/ternaus/TernausNet">https://github.com/ternaus/TernausNet</a></figcaption></figure><p>Unet++ is a recent improvement of Unet architecture (a nice <a href="https://sh-tsang.medium.com/review-unet-a-nested-u-net-architecture-biomedical-image-segmentation-57be56859b20">review</a> is also posted on medium), which has multiple skip connections</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*6wzlYHGu84qCy05sFQf1-w.png" /><figcaption>source: <a href="https://arxiv.org/pdf/1807.10165.pdf">https://arxiv.org/pdf/1807.10165.pdf</a></figcaption></figure><p>According to the <a href="https://arxiv.org/pdf/1807.10165.pdf">paper</a>, Unet++ seems to outperform original Unet. Just like in Unet, multiple encoders (backbones) can be used here to generate strong features for the input image.</p><h3><strong>Which encoder should I take?</strong></h3><p>Here I want to focus on Unet and Unet++ and compare their performance with different pre-trained encoders. For that, I chose to use a chest X-ray dataset (a combination of two datasets: <a href="https://www.kaggle.com/kmader/pulmonary-chest-xray-abnormalities/">first</a>, <a href="https://www.kaggle.com/yoctoman/shcxr-lung-mask">second</a>), with quite a simple task to segment lungs. It is a binary segmentation, so we should assign each pixel a probability of a class “1”, which we can later binarize to make a mask. First, let’s look at the data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yM6Jo2XoaH_1_cTowEscBw.png" /><figcaption>Examples of labeled images from chest X-ray dataset (image by Author)</figcaption></figure><p>So these are quite large images, typically 2000*2000 pixels, with large masks and visually it seems like finding a lung is not a problem. Using a great <a href="https://github.com/qubvel/segmentation_models.pytorch">segmentation_models_pytorch </a>library we have access to <strong>100+ different pre-trained encoders for Unet and Unet++. </strong>Let’s make a quick pipeline to train a model using <a href="https://github.com/catalyst-team/catalyst">Catalyst</a> (another great library for pytorch, which helps you to train your models without having to write lots of boring code) and <a href="https://github.com/albumentations-team/albumentations">Albumentations</a> (which helps you to apply different transformations to your images)<br>After all, it is going to looks something like this<br>1. Define dataset and augmentations. We will resize images to 256*256 and apply some heavy augmentations to the train dataset</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/117ab8b73e9973725599d4153bd84657/href">https://medium.com/media/117ab8b73e9973725599d4153bd84657/href</a></iframe><p>2. Define model and loss function. Here we take Unet++ with regnety_004 encoder and train with a sum of DICE + BCE losses using RAdam + Lookahed optimizers</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77de651cd4e54ef8a673b54aec6ee314/href">https://medium.com/media/77de651cd4e54ef8a673b54aec6ee314/href</a></iframe><p>3. Define callbacks and train!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/84b174cc6831e96723dc98702d3921c0/href">https://medium.com/media/84b174cc6831e96723dc98702d3921c0/href</a></iframe><p>If we do this for Unet and Unet++ with different encoders we can look at the validation quality of each trained model and summarize it like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*wdLzQkfzOFm_OMXWtpcFQQ.png" /><figcaption>Unet and Unet++ scores on validation (image by Author)</figcaption></figure><p>The first thing we notice is that Unet++ seems to perform better than Unet with all encoders. Sure, sometimes this difference is not that big, and we can’t say if they are statistically different at all — we would need to train multiple folds and look at the score distribution, single points don’t prove anything. Second, resnest200e shows top quality while still having a reasonable number of parameters. Interestingly, if we look at <a href="https://paperswithcode.com/task/semantic-segmentation">https://paperswithcode.com/task/semantic-segmentation</a> we find that resnest200 is SOTA in several benchmarks as well.</p><p>OK, but let’s actually compare different predictions using Unet++ and Unet with resnest200e encoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3dTmiLnDS0PB0y-AGcg1Mw.png" /><figcaption>Unet and Unet++ with resnest200e encoder predictions. The left panel shows differences in predictions of two models (image by Author)</figcaption></figure><p>There are cases when Unet++ is actually worse than Unet in some individual cases. Who could expect that? (Anyone, I guess). But overall it seems to be a bit better.</p><p>In general, this dataset looks like an easy task for a segmentation network. Let’s test Unet++ on a harder task. For that, I used <a href="https://jgamper.github.io/PanNukeDataset/">PanNuke</a> dataset, which is a histology dataset with labeled nuclei (205,343 labeled nuclei, 19 different tissue types, 5 nuclei classes). Data is already split into 3 folds and stored as individual 256*256*3 images with corresponding masks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pTsNjJ_AYCEHVnsJ92EVrg.png" /><figcaption>PanNuke sample examples (image by Author)</figcaption></figure><p>We can use similar code to train Unet++ models on this dataset and this is what we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*UpNW6pj34HwoKejTwLbYtw.png" /><figcaption>Unet++ score on validation (image by Author)</figcaption></figure><p>We see the same pattern here — resnest200e encoder seems to perform better than the others. We can visualize some examples using two different models (with the best, resnest200e encoder, and the worst, regnety_002)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*cuaXYxRNtl0cmmXCztaWgQ.png" /><figcaption>resnest200e and regnety_002 predictions (image by Author)</figcaption></figure><p>We can definitely say that this dataset is a harder task — we say that not only masks are less precise, but individual nuclei are assigned to the wrong classes. However, Unet++ with resnest200e encoder still performed quite well.</p><h3>Where to next?</h3><p>This is hardly a comprehensive guide to semantic segmentation, this is more an idea of what to use to get a solid baseline. There are many more models, such that FPN, DeepLabV3, Linknet, which are quite different from Unet, there are many Unet-like architectures, for example, Unet with double encoder, MAnet, PraNet, U²-net — there are many models to choose from and some of them may perform better on your task, however, a solid starting baseline can help you to start in the right direction. The code for this benchmark is available at <a href="https://github.com/azkalot1/segmentation_benchmark">https://github.com/azkalot1/segmentation_benchmark</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbe4fd78733f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-best-approach-to-semantic-segmentation-of-biomedical-images-bbe4fd78733f">The best approach to semantic segmentation of biomedical images</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Histopathologic cancer detection as image classification using Pytorch]]></title>
            <link>https://sergeykolchenko.medium.com/histopathologic-cancer-detection-as-image-classification-using-pytorch-557aab058449?source=rss-e626cc224497------2</link>
            <guid isPermaLink="false">https://medium.com/p/557aab058449</guid>
            <category><![CDATA[image-classification]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sergey Kolchenko]]></dc:creator>
            <pubDate>Thu, 25 Apr 2019 11:50:12 GMT</pubDate>
            <atom:updated>2019-04-25T11:50:12.555Z</atom:updated>
            <content:encoded><![CDATA[<p>A few weeks ago another kaggle (playground) competition has ended. Despite lack of any reward (neither money nor kaggle medals) a decent amount of people has participated. I and Alex Donchuk got 15th place, which is quite high, given that some people found ground truth labels and submitted them, so I want to share some ideas about this competition.</p><p>The idea of the competition was classify small pathology images, in other words to label as “1” images which contain tumor tissue and “0” images without tumor. I guess this can be an easy task for an expert, but for me it was impossible to distinguish normal and cancer images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*8hKi49y9Nw6Xsi6b2M-z6g.png" /><figcaption>Can you find a difference between one class and another?</figcaption></figure><p>As for the data here, participants were provided with small (96x96 pixels) patches from whole slides, so it was quite easy to handle the data because whole patches are huge megapixel images usually with several levels of resolution and it can be quite tricky to handle them. Hopefully here we had a lot of small images, so we could focus on different tricks and models instead of spending time on creating complicated pipelines for data loading and preprocessing. As it was said by authors of this dataset,</p><blockquote>[PCam] packs the clinically-relevant task of metastasis detection into a straight-forward binary image classification task, akin to CIFAR-10 and MNIST. Models can easily be trained on a single GPU in a couple hours, and achieve competitive scores in the Camelyon16 tasks of tumor detection and whole-slide image diagnosis. Furthermore, the balance between task-difficulty and tractability makes it a prime suspect for fundamental machine learning research on topics as active learning, model uncertainty, and explainability.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/1*6GD-xThMvDXmCWvBin_T1A.png" /><figcaption>Class distribution histogram</figcaption></figure><p>Classes were almost perfectly balanced (not like 10 times difference), so we can expect that simple binary cross entropy will work (yet focal loss and class weights can also be tested), so this is almost like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*FnXYQ6OQOWxCXHImWln1RQ.png" /><figcaption>Thanos is happy with your data</figcaption></figure><p>ROC-AUC was used as a metrics so the output was probability of cancer presence, and here I can’t help but notice that this metrics would be kind of useless for a clinician and is only good for competitions. In a real case we will also need to think about false-positive cases, since missing them is extremely important. But anyway, roc-auc was used here.</p><p>Since we had a lot of data I made a dataset loader, which reads files for disks, performs augmentations and send them to the NN.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf6bfb5cb63c1d47d2b5072735b6735f/href">https://medium.com/media/cf6bfb5cb63c1d47d2b5072735b6735f/href</a></iframe><p>I was interested in how different models can perform here, so I trained several of them using same parameters (number of epochs, learning rate and learning rate scheduler). This may not be the best way (since models’ performance can change drastically with optimal hyperparameters) but at least there was a clear trend. For this I took imagenet pretrained models, splitt data into 10 folds and used Adam with learning rate 3e-4. After 5 epochs without improvement of validation loss learning rate was reduced by two, and all models were trained with Adam with default parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WmSIC-A84gNBJPgX_ILZ4Q.png" /></figure><p>Several things are clear from here. First, resnet152, resnet101, densenet169 and densenet201 quickly overfit, which could mean that either we don’t have enough data to train them or a better set of hyperparametrs and training scheme is need, like warm-up\one cycle\cycling lr or restarting. Anyway, seresnet34, densenet121 and resnet34 (not shown here) performed the best, so I focused mostly on them.</p><p>Here I have to note that this test was done before whole slides ID’s were released, so images from one slide were both in train and validation, which is a leak and led to validation, so my gap between public lb score and my validation score was huge. After that I split data using whole slides ID’s as groups, which allowed me to have correct validation and thus gap decreased.</p><p>And because of this leak, which led to overfit, correlation between local validation and public score wasn’t that high. So that just shows that even if performance on your local validation is good, you need to be very careful, because you may have a leak in your data, so even 10-folds cross validation won’t help.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*f94JnFAz146X9I0SbYLJ4Q.png" /><figcaption>validation rocauc and lb roc auc are not really correlated</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*8wOZOTuJU2LV43aK27NqGg.png" /></figure><p>Overall, since I have trained several models the idea was just to blend them together. Ideally for that you need uncorrelated models, so I looked at correlations of models predictions, where for same model I have several version with prediction, including tta (test time augmentation), retrained models (models, which I trained again with lower LR) and models, trained on original sizes of images (96x96 instead of resized to 224x224). Here we see that in general predictions are highly correlated, but that’s life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GYbZ4_CzHfHJfdTtnyOuyA.png" /></figure><p>As for tta, I used 8 additional images during inference, Dehidral group D4 + 4 random augmentations (like here), which added quite a lot to.</p><p>After all, this competition was quite useful, since it showed the importance of the correct validation scheme. There are many things which can be still done on this dataset, such as</p><ul><li>Correct comparison of the models : using whole slides IDs for folds split</li><li>Finding optimal hyperparametrs for models and test different learning rate schemes, such as one cycle vs cycling, etc</li><li>Using additional information from images for splits\training, such as stratification by staining type\brightness</li><li>Finding optimal set of augmentations, which helps models to generalize</li><li>Metric learning, which is now used for classification. Can it be successfully used here?</li><li>Explainability of predictions — why images were assigned to their class?</li><li>Pseudo labeling — can we use it here to have even more training data?</li></ul><p>All this can still be tested here, so this dataset is really usefull for topics like this.</p><p>I have also made a <a href="https://github.com/azkalot1/histopathologic_cancer_detection">repo </a>with a minimal example, correct training of resnet34 with several tricks and correct validation split.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=557aab058449" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Meets Biology (once again) or Single Cell RNA Sequencing As a Field for…]]></title>
            <link>https://sergeykolchenko.medium.com/machine-learning-meets-biology-once-again-or-single-cell-rna-sequencing-as-a-field-for-2be24f9108dd?source=rss-e626cc224497------2</link>
            <guid isPermaLink="false">https://medium.com/p/2be24f9108dd</guid>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[biology]]></category>
            <dc:creator><![CDATA[Sergey Kolchenko]]></dc:creator>
            <pubDate>Mon, 18 Feb 2019 10:11:00 GMT</pubDate>
            <atom:updated>2019-04-18T13:20:37.331Z</atom:updated>
            <content:encoded><![CDATA[<h3>Machine Learning Meets Biology (once again) or Single Cell RNA Sequencing As a Field for Unsupervised Learning</h3><p>In a past decade amount of data in biology skyrocketed, and there seems to be no limit for that, as we have more and more sequenced genomes, deeper RNA expression and epigenetics datasets. On a separate note, we have brilliant examples of the broad use of machine learning for protein structure prediction and drug discovery. Now we are also able to study the 3D organisation of the genome, which can later be used for understanding of genomic rearrangements and subsequent diseases.</p><p>But there is one relatively new field of bioinformatics where classic ML tools are particularly useful — <strong>single-cell RNA sequencing.</strong></p><p>Before we move to single cell data, let’s briefly cover what is RNA sequencing and what is RNA in general.</p><p>Some of you may have heard about central dogma of molecular biology, such as information being transformed from DNA to RNA and from RNA to proteins. Proteins are the molecular machines which are responsible for everything in the cell. Unfortunately, it is very hard to quantify the exact amount of each protein in the cell, but we can easily sequence their corresponding RNA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*bNhYj825RJgNCY85Smwl8A.jpeg" /><figcaption>The Central Dogma of Molecular Biology, <a href="http://ib.bioninja.com.au/">ib.bioninja.com.au</a></figcaption></figure><p>As stated in the name, RNA sequencing is a group of methods for quantifying the amount of RNA in a sample. The reason we want to quantify it is that different cells may express different genes on different levels. Because of that difference, cells have extremely various properties and functions and we are interested in what caused that. Typical examples of the use of RNA sequencing is a comparison of gene “activity” (amount of RNA) in two groups, for example, treatment and control, where we have N control samples, N treatment samples and we use statistical methods to estimate which genes are deferentially expressed (= have different means in populations) between our groups (this can be interpreted directly as A\B test for those who are familiar with them). That’s quite useful when we are trying to understand if, for example, our treatment has any effect. Another example is our attempt to understand the effect of different mutations of the genomic sequence on a level of gene expression</p><p>However, those methods have their limitations. Since we extract RNA from a group of cells, we lose all information about the difference between them and we can only get a mean expression of each gene. In some cases that as useful as knowing the average temperature of all patients in a hospital. That may be a number, but does it make sense? This is especially important for cancer studies, as it has been shown that a tumor is not homogeneous but in fact can consist of the cell with different morphological, phenotypic and expression profiles so drug sensitivity tests may not be 100% accurate. It doesn’t mean, though, that bulk RNA sequencing is useless, we just need to understand its limitations.</p><p>So what should we do? Well, without going deep into the details of molecular biology, we can try to keep the information about each cell, so every time we sequence an RNA molecule we also sequence a small technical sequence, which is unique for each cell. If we add additionally a small technical sequence to each unique molecule of RNA, we can also accurately quantify the real number of RNA in each cell. So for each molecule, we sequence we know the ID of the cell (barcode) it came from and its own unique ID (in bioinformatics we call it a unique molecular identifier, UMI) so we can quantify the exact amount of RNA in each cell, therefore keeping all possible information! After all, at the end of the experiment, we will have a genes-by-cells table P, where P[i,j] is an expression of gene i in cell j.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*AYUSPreiLmcDvvdjMneaSg.jpeg" /><figcaption>source hemberg-lab.github.io/scRNA.seq.course</figcaption></figure><p>But, like any technique, there are problems and limitations. And one of such problems, the most important for us, is drop out events. Due to the stochastic nature of gene expression and technical limitations from each cell we get only a subset of genes, a subset of data we theoretically can get from it. We observe a lot of zero expression of different genes in our cells and unfortunately, we can’t tell if one particular “zero” is because that gene is simply not expressed in the cell or it simply didn’t make it to our subset — we call such events “dropouts”. (Some people may notice that it looks very similar to recommendation systems, where we also have quite sparse data, as one user can’t rate all possible films\products)</p><p>Now, when we cover all details necessary for the understanding of our data, both from biological and technical points of view, we can summarize it in terms of the machine learning task.</p><p>So our data has the following properties: <br>Typical number of features (genes) is ~20,000–30,000. (Here, however, I need no notice that we usually remove genes with low expression and work with around ~5,000–7,000 genes… which is still a lot). The number of samples can vary from 5,000 cells (usual number from one run of the pipeline we use in the lab) to 400,000 cells and even 1,300,000 cells. As mentioned above, due to dropouts and nature of the samples, data is sparse. Another important point for clustering algorithms is different scales of our features (gene A can be expressed 100 times higher than gene B and yet that doesn’t mean that gene A is more important than gene B) and scales of samples. We can have a different number of UMIs (sum of expression) per cell, but that can be a biological variation (some cells during their life simply have less mRNA) or technical (two cells can have exactly the same amount of mRNA, but from one we get more data due to technical variations). Therefore proper normalization is crucial.</p><p>The typical question biologists have is usually given all K cells, how many cell states (cell clusters) do we have and what are their marker genes, which we define as statistically significant features of a particular cluster. Given that we rarely have any labels from cells we can say that this is a typical question for <strong>unsupervised learning</strong>, where we have no prior knowledge for the number of clusters and clusters can be d<em>ifferent size, shape, density and can be connected one with another</em> (For example, cell cycle adds additional variation between close cell states). Keeping in mind the dropout component, which adds additional noise to data and can increase the distance between close cells, this task becomes more and more challenging.</p><p>Another important issue here is a high dimensionality of our data because of the so-called “curse of dimensionality”. Since we work with at least 5,000 genes, it would require 2⁵⁰⁰⁰ cells to cover all points in that space even if binarize gene expression. Obviously, we can’t sequence that many cells, so we need to first reduce the number of dimensions.</p><p>Typical pipeline for that is finding highly variable features (for example, using the coefficient of variation) and then using PCA to reduce the number of components to some reasonable number and on top of that, for sake of visualization, use tSNE or <a href="https://github.com/lmcinnes/umap">UMAP</a>, or even more complex methods, as variational autoencoders and even GANs (However, this may work only with large datasets and it is hard to transfer such models from one dataset to another.)</p><p>Next step in the analysis is clustering. At this moment at least 15 different tools for single cell RNA seq clustering are widely used, using different clustering mechanism, from simple K means with several initializations to dbscan and agglomerative clustering. However, since we usually lack any labels for cells, we can try to estimate the robustness of our clusters on datasets with known labels or by using different metrics to estimate cluster performance, such as BIC, AIC or silhouette method, or we can do a smarter thing — <strong>ask biologists to help us</strong>! We can use domain knowledge, since for many important genes we know their pattern of expression and by using that we can try to analyze and identify some known cell types. This is not always possible, but this is a topic for further articles.</p><p>Let’s build a simple pipeline for single cell RNA analysis. We will analyze a public dataset of Peripheral Blood Mononuclear Cells (PBMC) from 10X Genomics, which consists of ~8000 cells, which is available <a href="http://support.10xgenomics.com/single-cell-gene-expression/datasets/3.0.0/pbmc_10k_protein_v3">here</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a15573a0313a6488cc56076cbca17c11/href">https://medium.com/media/a15573a0313a6488cc56076cbca17c11/href</a></iframe><p>Next, we load our data. The matrix is quite big with lots of zeros, so it is stored in a sparse format.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b0e7d15f1f4b47a5fd1878d011dc03c/href">https://medium.com/media/8b0e7d15f1f4b47a5fd1878d011dc03c/href</a></iframe><p>Let’s do some EDA to understand what are we working with</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e3d896377e21fc9dd6ad4b81799bf57/href">https://medium.com/media/2e3d896377e21fc9dd6ad4b81799bf57/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zh3MfLvbtL_VZY0MywzKoQ.png" /></figure><p>What we see is that we have a lot of genes with extremely low expression and some (maybe even 1 with extremely high expression) — they may be outliers. On the other hand, we also know that some transcription factors (genes which are regulating expression of other genes) can be expressed on a really low lever and filtering our genes like may lead to loosing some structure.</p><p>At the same time, working with 33555 features is <strong>quite</strong> hard, so let’s remove some of them. For that we can do a simple thing. First, we remove genes with low expression (let’s say 100), plot CV (coefficient of variation) and select a reasonable threshold select genes, which we will use for clustering.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43e92662e6037ccf37ec8649ab9aa8bb/href">https://medium.com/media/43e92662e6037ccf37ec8649ab9aa8bb/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*ccpq9jDjPtRY7wg5qui4sg.png" /></figure><p>Let’s use define highly variable and useful genes as genes with CV ≥ 10.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ed5a816b6b5117defbe4d9d194e126d/href">https://medium.com/media/1ed5a816b6b5117defbe4d9d194e126d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OYySLpmI5bCt7TE60Ins2Q.png" /></figure><p>We again need to check our cells. It may happen that after filtering genes we will end up with cells with zero expression of any genes. It is better to remove them and after log-normalize our data</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81b73b146270d39e20a52bec4fe89ee3/href">https://medium.com/media/81b73b146270d39e20a52bec4fe89ee3/href</a></iframe><p>Next our step is using UMAP. We will use it to reduce number of dimensions to 2 and we will run clustering on that. This is not a good idea, since projection only on two dimensions may not be optimal and probably won’t reflect the whole complexity of the dataset, but it has one plus — it is easy to visualize and understand results of different clustering techniques. Another thing we can try to optimize here — is n_neighbors, min_dist, learning rate and other parameters of UMAP.</p><p>Usually clustering is done after PCA and we use tSNE to project the results on a two dimensional plane. However, UMAPis better if we want to add additional data points latter. Well, UMAP is just better in general.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f58bdc3453e148c70903513725871f21/href">https://medium.com/media/f58bdc3453e148c70903513725871f21/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*_d6ZxycLE2h5iSrv0tR7HQ.png" /></figure><p>We already see some structure here, without optimization of UMAP and pca parameters and with quite naive way to select features for clustering. The only question is if this structure reflects biology. But as an exercise we can try to run several clustering methods</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/76aba88845b81850c598c91b151b4267/href">https://medium.com/media/76aba88845b81850c598c91b151b4267/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mb7nBr_j9bpXrRpy-PnKqw.png" /></figure><p>Just visually, DBSCAN and SpectralClustering look good, but as I said, there are some metrics to monitor to find “optimal” clustering. Therefore here we can tune our parameters as well. After all, there are three steps where our naive approach can be improved</p><ol><li>Accurate feature selection</li><li>Parameters tuning for UMAP+pca</li><li>Parameters tuning for clustering</li></ol><p>However, the main question here, as I said, if this structure reflects the true biological complexity. To answer this question we will need to look at marker genes of each identified clusters to try to match them with known cell types and understand underlying biology.</p><p>In the next post, we will compare our results with commonly used pipelines and will try to find “markers” for each cluster and identify them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2be24f9108dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>