<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Fanbomeng on Medium]]></title>
        <description><![CDATA[Stories by Fanbomeng on Medium]]></description>
        <link>https://medium.com/@fanbomeng23?source=rss-736e5868ba2a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9RWlG65ZOzON0SnCP54tLQ.png</url>
            <title>Stories by Fanbomeng on Medium</title>
            <link>https://medium.com/@fanbomeng23?source=rss-736e5868ba2a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:40:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fanbomeng23/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unittest from Zero to Hero in 5 mins]]></title>
            <link>https://medium.com/@fanbomeng23/unittest-from-zero-to-hero-in-5min-ff78abd61434?source=rss-736e5868ba2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff78abd61434</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[unittest]]></category>
            <category><![CDATA[pytest]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Fanbomeng]]></dc:creator>
            <pubDate>Wed, 23 Oct 2024 19:00:56 GMT</pubDate>
            <atom:updated>2024-10-23T19:02:40.635Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Getting Started with Unit Testing in Python: From unittest to pytest</strong></p><p><em>Your journey to writing reliable and maintainable Python code begins here!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RQxqLCuGk9N8p10ApuIy_Q.png" /></figure><p>[1]</p><ol><li><strong>Introduction: Why Test Your Code?</strong></li></ol><p>Have you ever written code that worked perfectly — until it didn’t? Bugs and errors are an inevitable part of programming. That’s where <strong>unit testing</strong> comes into play. Unit testing involves testing individual units or components of your program to ensure they function as intended.</p><p><strong>Why should you care about unit testing?</strong></p><p>• <strong>Catch Bugs Early</strong>: Identify and fix issues before they escalate.</p><p>• <strong>Ensure Code Quality</strong>: Maintain high-quality, reliable code.</p><p>• <strong>Facilitate Refactoring</strong>: Modify code confidently, knowing existing functionality remains intact.</p><p>• <strong>Documentation</strong>: Tests can serve as additional documentation on how your code is supposed to behave.</p><p>In this guide, we’ll explore two popular Python testing frameworks: the built-in unittest and the third-party pytest. We’ll walk you through setting up your testing environment, writing your first tests, and best practices to make your code robust and error-free.</p><p><strong>2. Getting Started with Unit Testing</strong></p><p><strong>Setting Up Your Environment</strong></p><p>Before diving into writing tests, let’s set up a clean and organized environment.</p><p><strong>Project Structure</strong></p><p>Organizing your project files makes your code more maintainable and your tests easier to run. Here’s a common project structure:</p><pre>project/<br>├── src/<br>│ └── your_code.py<br>├── tests/<br>│ └── test_your_code.py<br>└── requirements.txt</pre><ul><li>src/: Contains your source code.</li></ul><p>• tests/: Contains your test code.</p><p>• requirements.txt: Lists project dependencies.</p><p><strong>Creating a Virtual Environment</strong></p><p>Using a virtual environment ensures that your project’s dependencies are isolated.</p><pre># On Unix or macOS<br>python3 -m venv venv<br>source venv/bin/activate<br># On Windows<br>python -m venv venv<br>venv\Scripts\activate</pre><p><strong>Installing Dependencies</strong></p><p>If you have external libraries, list them in requirements.txt and install them:</p><pre>pip install -r requirements.txt</pre><p><strong>Installing Testing Tools</strong></p><p>While we’ll start with Python’s built-in unittest, installing pytest gives you access to more advanced features.</p><pre>pip install pytest</pre><p><strong>3. Understanding unittest: The Basics</strong></p><p><strong>What is unittest?</strong></p><p>unittest is Python’s built-in testing framework inspired by JUnit. It provides a class-based approach to organize your test cases. Key concepts include:</p><p>• <strong>Test Case</strong>: Smallest unit of testing; checks for a specific response to a particular set of inputs.</p><p>• <strong>Test Suite</strong>: Collection of test cases.</p><p>• <strong>Test Fixture</strong>: Preparation needed to perform one or more tests, such as creating temporary databases or directories.</p><p><strong>Writing Your First Test</strong></p><p>Let’s start by writing a simple function and its corresponding test.</p><p>Source Code (your_code.py):</p><pre># src/your_code.py<br>def add(a, b):<br>  return a + b</pre><p><strong>Test Code (</strong>test_your_code.py<strong>):</strong></p><pre># tests/test_your_code.py<br>import unittest<br>from src.your_code import add<br>class TestMathFunctions(unittest.TestCase):<br>def test_add(self):<br>self.assertEqual(add(1, 2), 3)<br>self.assertEqual(add(-1, 1), 0)<br>if __name__ == &#39;__main__&#39;:<br>unittest.main()</pre><p><strong>Explanation:</strong></p><p>• <strong>Importing </strong>unittest<strong> and the function to test</strong>.</p><p>• <strong>Creating a class</strong> that inherits from unittest.TestCase.</p><p>• <strong>Defining test methods</strong> that start with test_.</p><p>• <strong>Using assertions</strong> like self.assertEqual to check expected outcomes.</p><p><strong>Running Tests with unittest</strong></p><p>Run your tests using the following command:</p><pre>python -m unittest discover</pre><p>This command discovers all tests in your project matching the pattern test*.py and runs them.</p><p><strong>4. Moving to pytest: A Simpler Alternative</strong></p><p><strong>Why pytest?</strong></p><p>While unittest is powerful, pytest simplifies the testing process with:</p><p>• <strong>Simpler Syntax</strong>: Use plain assert statements instead of special assertion methods.</p><p>• <strong>Function-Based Tests</strong>: No need to wrap tests in classes.</p><p>• <strong>Powerful Fixtures</strong>: More flexible setup and teardown mechanisms.</p><p>• <strong>Better Reporting</strong>: Detailed info on test failures.</p><p><strong>Writing a Test with pytest</strong></p><p>Let’s rewrite the previous test using pytest.</p><p><strong>Test Code (</strong>test_your_code.py<strong>):</strong></p><pre># tests/test_your_code.py<br>from src.your_code import add<br>def test_add():<br>assert add(1, 2) == 3<br>assert add(-1, 1) == 0</pre><p><strong>Explanation:</strong></p><p>• <strong>No imports needed for testing</strong> — just import the function you’re testing.</p><p>• <strong>Define test functions</strong> that start with test_.</p><p>• <strong>Use plain </strong>assert<strong> statements</strong> for checks.</p><p><strong>Running Tests with pytest</strong></p><p>Simply run:</p><pre>pytest</pre><p>pytest will automatically discover and run all tests in files named test_*.py.</p><p><strong>5. Mocking in Unit Tests</strong></p><p><strong>Why Mock?</strong></p><p>Sometimes, your functions interact with external systems like databases, file systems, or web services. <strong>Mocking</strong> allows you to simulate these interactions, making your tests faster and more reliable.</p><p>• <strong>Isolation</strong>: Test components in isolation.</p><p>• <strong>Control</strong>: Define the behavior of external dependencies.</p><p>• <strong>Performance</strong>: Avoid slow operations like network calls.</p><p><strong>Using unittest.mock</strong></p><p>Let’s say we have a function that lists files in a directory.</p><p><strong>Source Code (</strong>your_code.py<strong>):</strong></p><pre># src/your_code.py<br>import os<br>def list_files(directory):<br>return os.listdir(directory)</pre><p><strong>Test Code (</strong>test_your_code.py<strong>):</strong></p><pre># tests/test_your_code.py<br>from unittest.mock import patch<br>from src.your_code import list_files<br>@patch(&#39;src.your_code.os.listdir&#39;)<br>def test_list_files(mock_listdir):<br>mock_listdir.return_value = [&#39;file1.txt&#39;, &#39;file2.md&#39;]<br>result = list_files(&#39;dummy/path&#39;)<br>assert result == [&#39;file1.txt&#39;, &#39;file2.md&#39;]</pre><p><strong>Explanation:</strong></p><p>• <strong>Using </strong>@patch<strong> decorator</strong> to replace os.listdir with a mock.</p><p>• <strong>Setting return value</strong> of the mock to control the output.</p><p>• <strong>Testing the function</strong> without relying on the actual file system.</p><p><strong>Understanding @patch</strong></p><p>The @patch(‘src.your_code.os.listdir’) tells the test to replace os.listdir in your_code.py with a mock object. This is crucial because you need to patch <strong>where the function is used</strong>, not where it’s defined.</p><p><strong>6. Best Practices for Writing Tests</strong></p><p>Write Clear and Simple Tests</p><p>• One Assertion per Test: Focus on a single behavior.</p><p>• Descriptive Test Names: Clearly state what you’re testing.</p><p><strong>Focus on Small, Isolated Units</strong></p><p>• <strong>Test Functions in Isolation</strong>: Avoid dependencies between tests.</p><p>• <strong>Use Mocks Wisely</strong>: Mock external interactions, but not the code under test.</p><p><strong>Use Fixtures for Setup and Teardown</strong></p><p>In pytest, fixtures can simplify setup and teardown tasks.</p><p><strong>Example:</strong></p><pre>import pytest<br>@pytest.fixture<br>def sample_data():<br>return {&#39;key&#39;: &#39;value&#39;}<br>def test_using_fixture(sample_data):<br>assert sample_data[&#39;key&#39;] == &#39;value&#39;</pre><p><strong>7. Transitioning Between unittest and pytest</strong></p><p><strong>Mixing unittest and pytest</strong></p><p>You can run unittest test cases using pytest without any modifications.</p><p><strong>Example:</strong></p><pre># tests/test_with_unittest.py<br>import unittest<br>class TestExample(unittest.TestCase):<br>def test_true(self):<br>self.assertTrue(True)</pre><p>Running pytest will discover and execute this unittest test.</p><p><strong>Gradual Migration to pytest</strong></p><p>If you have an existing suite of unittest tests, you can gradually start writing new tests using pytest while still running all tests together.</p><p><strong>8. Advanced Topics</strong></p><p><strong>Testing Asynchronous Code with pytest-asyncio</strong></p><p>For async functions, you can use the pytest-asyncio plugin.</p><p><strong>Installation:</strong></p><pre>pip install pytest-asyncio</pre><p><strong>Usage:</strong></p><pre>import asyncio<br>import pytest<br>async def async_add(a, b):<br>await asyncio.sleep(1)<br>return a + b<br>@pytest.mark.asyncio<br>async def test_async_add():<br>result = await async_add(1, 2)<br>assert result == 3</pre><p><strong>Parameterized Testing with pytest</strong></p><p>Run the same test with different inputs using @pytest.mark.parametrize.</p><p><strong>Example:</strong></p><pre>import pytest<br>@pytest.mark.parametrize(&quot;a, b, expected&quot;, [<br>(1, 2, 3),<br>(-1, 1, 0),<br>(0, 0, 0)<br>])<br>def test_add(a, b, expected):<br>assert add(a, b) == expected</pre><p><strong>9. Conclusion</strong></p><p>We’ve explored the essentials of unit testing in Python, comparing unittest and pytest. While unittest provides a solid foundation and is included out of the box, pytest offers a more streamlined and flexible approach.</p><p><strong>Key Takeaways:</strong></p><p>• <strong>Start Testing Early</strong>: Incorporate testing from the beginning of your project.</p><p>• <strong>Choose the Right Tool</strong>: Use unittest for class-based tests and pytest for simplicity and advanced features.</p><p>• <strong>Write Maintainable Tests</strong>: Keep tests clear, concise, and focused.</p><p><strong>Next Steps:</strong></p><p>• <strong>Experiment</strong>: Try writing tests using both frameworks to see which suits your style.</p><p>• <strong>Explore Plugins</strong>: Check out pytest plugins like pytest-cov for coverage reports.</p><p>• <strong>Stay Curious</strong>: Testing is a skill that improves with practice.</p><p><em>Have questions or comments? Feel free to share them below! Happy coding and testing!</em></p><p>Reference:</p><p>[1] webpage: <a href="https://www.softwaretestingstuff.com/unittest-vs-pytest">https://www.softwaretestingstuff.com/unittest-vs-pytest</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff78abd61434" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From LDA to Short text topic modeling — Negative sampling and Quantization Topic Model(NQTM)]]></title>
            <link>https://medium.com/professorbob/from-lda-to-short-text-topic-modeling-negative-sampling-and-quantization-topic-model-nqtm-3eabee5447a7?source=rss-736e5868ba2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3eabee5447a7</guid>
            <category><![CDATA[lda]]></category>
            <category><![CDATA[theory]]></category>
            <category><![CDATA[short-text]]></category>
            <category><![CDATA[topic-modeling]]></category>
            <category><![CDATA[nqtm]]></category>
            <dc:creator><![CDATA[Fanbomeng]]></dc:creator>
            <pubDate>Thu, 30 Sep 2021 01:35:22 GMT</pubDate>
            <atom:updated>2021-10-01T08:06:30.277Z</atom:updated>
            <content:encoded><![CDATA[<h3>From LDA to Short text topic modeling — Negative sampling and Quantization Topic Model(NQTM)</h3><h3>Introduction</h3><p>What is topic modeling? An example can be: there are a set of articles, but on what general topics are they belong to, what are these topics? The Machine learning(ML) models that answer these questions are topic modeling models.</p><p>Topic modeling has been used in many applications. They are vastly discussed online. Especially for the statistics-based ones, in which Latent Dirichlet Allocation (LDA) can be one of the representatives. These statistics method-based models are widely used and proofed to be very successful in many fields.</p><p>Though the success of LDA type of models in topic modeling. They are proofed to be not very effective in a short text. While short text messages are playing more and more important roles nowadays. For example, text messages, Twitter messages, movie or product reviews, etc.</p><p>NQTM is a very powerful short text topic modeling model, it can be trained through gradient descent methods like other ML models. But in order to really understand and even change the code of this model, the understanding of the mechanics of LDA will be very helpful</p><p><strong>In this blog we are going to focus on the following points:</strong></p><ul><li><strong>How LDA really works in math and in programing</strong></li><li><strong>How people can extract topics with unsupervised machine learning models? Where and how does the model get topics? Remember no tagged data are used</strong></li><li><strong>How NQTM works?</strong></li><li><strong>Extra codes that are not provided in the official git repository that give the text and topic pairs.</strong></li></ul><h3>How LDA really works in math and in programing</h3><p>LDA is an unsupervised model, in this case, the training process is also the prediction process. Essentially, a simplified way of thinking about this process, for each document,</p><ul><li>LDA is computing the probability of different topics with respect to this document. For example, if we only have three topics: (1, history), (2, art), (3, science), then Doc-A many have the following outputs after being processed with LDA (0.2,0.2,0.6), then topic three(science) is the one holds the highest probability as a topic for Doc-A</li><li>LDA also computes the matrix that holds the probability distribution P(word w| topic t). It tells for each of the topics in this matrix(say row zero), what are the probability that this topic holds this word. For example, topic science very like has a high probability of the words, like physics, experiment, theory, etc.</li></ul><p>Of course, there are more steps in the real algorithm even before passing the documents to LDA, for example, tokenization, removing stop words, do lemmatization, etc.</p><p><strong>Then the next question: in the real case, How LDA is really got set up?</strong></p><p><strong>The first question you may have is what is “Dirichlet” in LDA exactly means.</strong></p><p>The Dirichlet distribution is a statistic distribution, used in modeling the topic distribution in Document, which, we put the math definition in Appendix.</p><p><strong>Next, as mentioned above, the two main task people use LDA is:</strong></p><ul><li>Get the top topics among all of the documents P(topic | Doc)</li><li>For each of the documents, what are the main topics belong to it P(topic|Doc)* P(word| topic)</li></ul><p>This is also the final output for other topic modeling models.</p><p>A good way to understand how LDA works is to think it reversely, how to use the two distribution P(topic | Doc) and P(word | topic) to reconstruct the documents</p><p>LDA is a generative probabilistic model. The basic assumption in LDA is that Documents can be represented by random mixtures over latent topics, and topics is characterized by a distribution over words. Here the distribution of Document over topics in LDA follows the Dirichlet distribution. In steps[2]:</p><p>LDA assumes the following generative process for documents</p><ol><li>Choose N ∼ Poisson(ξ).</li><li>Choose θ ∼Dir(α).</li><li>For each of the N words wₙ : (a) Choose a topic zₙ ∼ Multinomial(θ). (b) Choose a word wₙ from $p(wₙ ∣ zₙ, β), a multinomial probability conditioned on the topic<br> zₙ</li></ol><p>If we put the above steps in the math format, it will be the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a-grOjqj7I9BYf5dUMdZmA.png" /></figure><p>For the parameters mentioned above, choose θ from the Dirichlet distribution with the given α and a selected number of topics k. The word probabilities are parameterized by a K x V matrix β which needs to be determined later. N is independent of all the other data-generating variables, it is an ancillary variable.</p><p>For the parameters mentioned above, choose θ from the Dirichlet distribution with the given α and a selected number of topics k. The word probabilities are parameterized by a K x V matrix β which needs to be determined later. N is independent of all the other data-generating variables, it is an ancillary variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*R1YsOeUuBml1Uc7Ote0_og.png" /></figure><p>The details are shown in A.3 of Reference [2]. By computing the derivatives of the KL divergence and setting them equal to zero, we can get the following relationship between ϕ and γ</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*V7BdOiYZMJYJyuNLZTr6Lg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/364/1*a0t3_Bz_TmXMHXH7yytRxg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*1ZyikR9VLEFJwiDwOdNQhQ.png" /></figure><p>With the above relationship, the optimization of parameters are document-specific by the following[2]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*hrTyHSaXYfUIqhmog7Jwtg.png" /></figure><p>So till this far, it is clear how the LDA is performed and how the optimization is done in real cases.</p><h3>Negative sampling and Quantization Topic model (NQTM)</h3><p>As mentioned at the most beginning, the motivation of NQTM is dealing with a short text. NQTM is an auto-encoding framework that addresses the un-supervised short text modeling problem. In this model two essential steps are done:</p><ul><li>mapping topic distribution into an appropriate defined embedding space.</li><li>A new negative sampling decoder to improve the topic diversity performance.</li></ul><p><strong>The performance of NQTM is very impressive compared with the unsupervised topic modeling models[3].</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7zy6C5IZ5RypAiNZaRdaw.png" /></figure><p>The evaluation matrix used here are topic coherence($C_v$) and unique score(TU)</p><p>Topic coherence metrics depend on co-occurrences of topic words learned by models in the external corpus assuming that coherent words should co-occur within a certain distance[4]. It has been proven to perform better than other coherence metrics like widely used NPMI, UMASS. Details can be seen in either reference 3 or reference 4</p><p>TU is used for topic diversity evaluation. The score ranges from 1/k to 1 and a higher value means the generated topics are more diverse due to fewer duplicated words across other topics.</p><p>Usually, in general higher TU will cause lower Cᵥ as coherent words seldom repeat, higher Cᵥ leads to lower TU as coherent words frequently repeat across topics.[3]</p><h3>NTQM model structure and processing steps</h3><p>In NQTM, MLPs(multilayer perceptron) are used in both encoder and decoder. Also, the model assumes that short text x is in the form of a bag of words which produces continuous</p><p>There are mainly three steps of the model:</p><ul><li>Short text Encoder</li><li>Topic Distribution Quantization</li><li>Negative sampling Decoder</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DuXGVXnx4hgYHTANpkImfA.png" /><figcaption>Fig: refer from [3]</figcaption></figure><p>For the short text encoder, In the adoption of a simple network structure, θₑ is calculated as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*bsJkQ1G7iGD5UuWgdIH5_A.png" /></figure><p>After the encoding step, the next step is the topic distribution quantization. A discrete embedding space <strong>e</strong>=(e₁, e₂, …, e) ∈ Rᴷˣᴮ is set. K is the number of topics and B is the size of the space. To encourage the maximum of the distance between embedding vectors, the embedding matrix is extended, the first K vectors are initialized with identity matrix and additional set, and the (eᴷ⁺¹ …eᴮ) are initialized with uniform unit scaling:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*-sDc63yJ06Z6qsgA1XHv-w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*Qt_aFRjexli7pUWjzItiEg.png" /></figure><p>This representation of the embedding matrix is optimized during the training.</p><p>The short x is encoded to the θₑ and then mapped to the embedding space as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*5_Ha9nz5lw5Z_0njVb6KLA.png" /></figure><p>θᵩ is changed to the eₖ which holds the closest distance. Then the new θᵩ can be passed to the decoder steps, which are also MLPs.</p><p>In order to generate more diverse topics, a negative sampling scheme is formulated. The words with high probabilities in other topics but not assigned to the current short text are taken as negative samples. Specifically, if top t topics are chosen, then the model removes the top t and samples one negative topic Zₙ from the left (k-t) with equal probability</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*1KToCyMjPyWdaMpm_lTn5w.png" /></figure><p>Then generate M words from the Zₙ, as the words that decoder should not include them since they are not from the top t topics.</p><p>The loss function has two components, the one from top t and the negative contributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*D5NeVhC4q_vyb6bQ9u6yCA.png" /></figure><p>here x⁽ᶦ⁾ refers to the i-th short text in the corpus. Together with the l₂ regularization, give the final form of the loss function.</p><p>These are the background of this model. In the real application, the object is to get the collection of topics and also the topics for each of the input short texts. With the embedding space, NQTM first gets the top t embedding of topics for each short text inputs and then exact the topic words associated with it. This is very similar to the two matrices that LDA used to do the generation.</p><h3>NQTM using example</h3><p>input text should be preprocessed before passing to the model, the steps are the following:</p><h3>Details in processing:</h3><ul><li>tokenize each text and remove non-Latin characters and stop words by using NLTK</li><li>filter out short texts with length less than 2</li><li>remove words with document frequency less than 5</li><li>convert all letters into lower cases</li><li>count vector from sklearn</li></ul><p>Note: more detailed procedures are in the file utils/preprocess.py</p><p><strong>Following the instruction on the official github page</strong>(<a href="https://github.com/BobXWu/NQTM">link</a>), the model could be run but the model only gives the collection of topic words for all of the documents, not in the form of (Doc, topic) pair.</p><p>An extra function is needed to be added to the package. In the function run_NQTM.py, adding in the following code, then a file will be generated with a selected number of topics and topic words associated with each doc when called in the main function.</p><pre>def print_topic_words(beta,theta, feature_names,n_topic_selection=2, n_top_words=2):<br>    top_words = list()<br>    doc_topic_word=list()<br>    for i in range(len(beta)):<br>        top_words.append(&quot; &quot;.join([feature_names[j] for j in beta[i].argsort()[:-n_top_words - 1:-1]]))<br>    for i in range(len(theta)):<br>        doc_topic_word.append(&quot; &quot;.join([top_words[j] for j in theta[i].argsort()[:-n_topic_selection - 1:-1]]))</pre><pre>    with open(os.path.join(args.output_dir, &#39;topic_words_T{}_K{}_{}th&#39;.format(n_top_words, args.topic_num, args.test_index)), &#39;w&#39;) as file:<br>        for line in doc_topic_word:<br>            file.write(line + &#39;\\n&#39;)</pre><p>Appendix session:</p><p><strong>what is Dirichlet distribution:</strong></p><p>In probability and statistics, the Dirichlet distribution, often denoted as Dir(a), is a continuous multivariate probability distribution parameterized by a vector α. The mathematics definition is as follows:</p><p>The Dirichlet distribution of order K ≥ 2 with parameters a₁, …, a&gt;0 has a probability density function with respect to Lebesgue measure on the Euclidean space Rᴷ⁻¹ given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*5-GQf4Kn-VPu-oyLYiIaJA.png" /></figure><p>B(a) is the normalization factor and is in a complex express, but the key idea of Dirichlet distribution is relatively simple as shown above, by changing α, then the shape of the distribution also changed [1]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mvtly-jqu4fVYv8DkHeHIg.png" /><figcaption>fig: reference from [1]</figcaption></figure><h3>Reference</h3><ol><li>Wikipedia, Dirichlet distribution, <a href="https://en.wikipedia.org/wiki/Dirichlet_distribution">https://en.wikipedia.org/wiki/Dirichlet_distribution</a></li><li>Latent Dirichlet Allocation, Journal of Machine Learning Research 3 (2003) 993–1022, David M. Blei, Andrew <a href="http://y.ng/">Y.Ng</a>, Michael I. Jordan</li><li>Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1772–1782, November 16–20, 2020. c 2020 Association for Computational Linguistics Xiaobao Wu, Chunping Li, Yan Zhu, Yishu Miao</li><li>Michael R¨oder, Andreas Both, and Alexander Hinneburg.2015. Exploring the space of topic coherence measures. In Proceedings of the eighth ACM international conference on Web search and data mining, pages 399–408. ACM.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3eabee5447a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/professorbob/from-lda-to-short-text-topic-modeling-negative-sampling-and-quantization-topic-model-nqtm-3eabee5447a7">From LDA to Short text topic modeling — Negative sampling and Quantization Topic Model(NQTM)</a> was originally published in <a href="https://medium.com/professorbob">Stellia.ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Topic Modeling: is NQTM a better choice than LDA for short text ?]]></title>
            <link>https://medium.com/@fanbomeng23/topic-modeling-is-nqtm-a-better-choice-than-lda-for-short-text-a99a6dcbe51b?source=rss-736e5868ba2a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a99a6dcbe51b</guid>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[short-text]]></category>
            <category><![CDATA[published-in-nov-2020]]></category>
            <category><![CDATA[nqtm]]></category>
            <category><![CDATA[topic-modeling]]></category>
            <dc:creator><![CDATA[Fanbomeng]]></dc:creator>
            <pubDate>Fri, 17 Sep 2021 13:07:16 GMT</pubDate>
            <atom:updated>2021-09-29T21:33:37.163Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://www.notion.so/NQTM-one-of-the-best-models-for-unsupervised-short-text-topic-modeling-or-Topic-Modeling-is-NQTM--4547245815644c99bc011297914d2275">Topic Modeling: is NQTM a better choice than LDA for short text ?</a></h3><p>Professorbob.ai</p><h3>Prefix</h3><p>This article is aiming to have a quick and clear introduction of the Negative sampling and Quantization Topic Model(NQTM) short text model and also providing step by step examples of quick running and testing. The theory background will not covered in great detail, for more information on theory, please refer to <a href="https://aclanthology.org/2020.emnlp-main.138.pdf">paper</a></p><h3>Introduction</h3><p>Short texts are playing a more and more important role in people’s daily life. Not only traditional text messages, but the new media online with applications on computers and smartphones, like tweets, headlines, Whatsapp. Extracting useful information from these short text messages can be helpful in many areas, for example, recommendation systems, context summarization, classifications, etc. Topic modeling is one of the useful topics among these applications. Take tweets for example, if we want to see tweets related to Covid-19 in a particular area, if a model can generate the topic for each of the tweets messages, it will be very helpful for reviewing messages on a large scale.</p><p>Conventional topic models work well on long documents but not on short text. The reason is that the co-occurrence information from short texts is extremely limited as known as the data sparsity problem. This hinders the topic models leaning and generating high-quality topics in the unsupervised learning fashion.</p><p>There are a number of models explored in the short text topic modeling, for example, yield pseudo texts, Biterm Topic Model, Urn_DMM, etc. They perform better than traditional models like LDA, but still not in great quality.</p><p>NQTM is a model that provides robust performance as un-supervise model and at the same time, no extra data beside the sentences are needed to be provided.</p><h3>NQTM</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HRRX2etx90jxUqTBEzgG3g.png" /><figcaption>Figure: general configuration of NQTM[1]</figcaption></figure><p>The motivation of NQTM is dealing with short text. NQTM is an auto-encoding framework that addresses the un-supervised short text modeling problem. There are two essential components in this model:</p><ul><li>mapping topic distribution into an appropriate defined embedding space.</li><li>A new negative sampling decoder to improve the topic diversity performance.</li></ul><p><strong>To be more specifically about the steps:</strong></p><p>Short text Encoder(step1)</p><ul><li>Take the text sequence as input and encode it into vector representation</li></ul><p>Topic Distribution Quantization(step2)</p><ul><li>Creating embedding space, which has k vectors(k is the number of topics preselected by the user), matching encoded vector produced in step1 with each of the vectors in step2, and re-assign the step1 vector to be the step2 vector, which has the closest distance between them.</li></ul><p>Negative sampling Decoder(step3):</p><ul><li>Only part of the vectors in the embedding space created in step2 are selected and remaining ones form a collection of negative vector space, the final collection of negative vectors are sampled from the negative vector space, which also contribute to the form of loss functions.</li></ul><h3>Performance of NQTM with respect to other models(before Nov, 2020)</h3><p><strong>The performance of NQTM is very impressive compared with the un-supervised topic modeling models[1].</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6QXzgB9pN8DjlBiK8-gHzw.png" /></figure><p>The evaluation matrix used here are topic coherenece Cᵥ and unique score(TU), the higher the better. More details about the definition of these two evaluation matrix, please refer to the appendix</p><h3>NQTM step by step step us and demo(official git-repo <a href="https://github.com/BobXWu/NQTM">link</a>)</h3><p><strong>A. Set ups following the official instruction:</strong></p><ol><li>create new conda env(recommended): <br>conda create -n NQTM python=3.6</li><li>Install necessary package: <br>pip install tensorflow-gpu==1.13.1 <br>pip install scipy==1.5.2 <br>pip install scikit-learn==0.23.2</li><li>Download the package: <br>git clone <a href="https://github.com/BobXWu/NQTM.git">https://github.com/BobXWu/NQTM.git</a><br>or<br>Directly clone from the following repository, <a href="https://github.com/fanbomeng/Topic_modeling_NQTM">link</a> then the code in run_NQTM.py will be modified already(if doing so, omit step B)</li></ol><p>B. <strong>Modify the code — the code providing by the official package lacks the <br>function to produce (doc,topic words) pair function. it only has the function print out all of the topic words for all of the docs, the following adding in doc → topic words production function.</strong></p><ol><li>Use the editor you are familiar with, open the file run_NQTM.py, put the following function after the function print_top_words(beta, feature_names, n_top_words=2)</li></ol><pre>def print_topic_words(beta,theta, feature_names,n_topic_selection=2, n_top_words=2):<br>    top_words = list()<br>    doc_topic_word=list()<br>    for i in range(len(beta)):<br>        top_words.append(&quot; &quot;.join([feature_names[j] for j in beta[i].argsort()[:-n_top_words - 1:-1]]))<br>    for i in range(len(theta)):<br>            doc_topic_word.append(&quot; &quot;.join([top_words[j] for j in theta[i].argsort()[:-n_topic_selection - 1:-1]]))</pre><pre>with open(os.path.join(args.output_dir, &#39;topic_words_new_T{}_K{}_{}th&#39;.format(n_top_words, args.topic_num, args.test_index)), &#39;w&#39;) as file:<br>        for line in doc_topic_word:<br>            file.write(line + &#39;\n&#39;)</pre><p><strong>Comment on parameters:</strong></p><p>print_topic_words function is used to create the (doc, topic words), parameters for this function are:</p><ul><li>beta, theta: two matrix mentioned often in LDA like models, which are (topic, topic words), (doc, topic) matrix respectively</li><li>feature_names: unique words in the whole collection of data, no need to be provided by user, it will be automatically extract and loaded by the program, as long as pass the right variable, which will be pointed out later</li><li>n_topic_selection：the number of topics selected for each of the document</li><li>n_top_words: in each of the topic, how many topic words recorded.</li></ul><p>2. Add the new function print_topic_words in train(model, train_data, vocab, config), following the line, train_theta = get_theta(model, train_data) as</p><pre>print_topic_words(beta,train_theta, vocab)</pre><p>&amp;</p><p>comment out the line: print_top_words(beta, vocab)</p><h4><strong>C. Running the package:</strong></h4><ol><li>Data processing for the training:</li></ol><pre>python utils/preprocess.py --data_path data/stackoverflow --output_dir input/stackoverflow</pre><p>2 Generating topic words:</p><pre>python run_NQTM.py --data_dir input/stackoverflow --output_dir output</pre><p>comment: This will generate generate output file, containing topic word for each role of the input doc</p><p><strong>D. User defined input file creation:</strong></p><ul><li>Preprocessing involving the following steps:<br>tokenize each text and remove non-Latin characters and stop words by using NLTK</li></ul><pre># if the file containing collection of short text doc is called example.csv<br># The following code can be reference for creating your own data:</pre><pre>import pandas as pd<br>import nltk<br>nltk.download(&#39;words&#39;)<br>from nltk.corpus import stopwords<br>from nltk.tokenize import word_tokenize<br>stop_words = set(stopwords.words(&#39;english&#39;))<br>total_words = set(nltk.corpus.words.words())</pre><pre>def processing(SC):<br>    New_collection=[]<br>    for sentence in SC:<br>        tmp=[]<br>        sentence=sentence.lower()<br>        word_tokens = word_tokenize(sentence)<br>        for W in word_tokens:<br>            if (W in stop_words) or (W not in total_words) or ( not W.isalpha()):<br>                continue<br>            tmp.append(W)<br>        New_collection.append(&#39; &#39;.join(tmp))<br>    return New_collection</pre><pre>data=pd.read_csv(&#39;example.csv&#39;)</pre><pre>data_clean=processing(data)</pre><pre>with open(&#39;texts.txt&#39;, &#39;w&#39;) as f:<br>    for term in data_clean:<br>        f.write(&quot;%s\n&quot; % term)</pre><pre>#The file texts.txt can be passed to the model for preprocessing like<br>#python utils/preprocess.py --path data/stackoverflow --output_dir input/stackoverflow</pre><p>The format in the texts.txt is each row is a cleaned doc.</p><p><strong>E. example performance on College level Calculus questions and answer(harder than normal short text topic modeling) with only 400 inputs.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GQD7zcYUvqqV207fkS45wA.png" /></figure><h3>Reference:</h3><ol><li>Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1772–1782, November 16–20, 2020. c 2020 Association for Computational Linguistics Xiaobao Wu, Chunping Li, Yan Zhu, Yishu Miao</li></ol><h3>Appendix:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ONyq46Oa6fU5PYXHZlenTQ.png" /><figcaption>reference from [1]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*222hggmirH7p6uk6stzocw.png" /><figcaption>reference from [1]</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a99a6dcbe51b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>