<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Oksana Tkach on Medium]]></title>
        <description><![CDATA[Stories by Oksana Tkach on Medium]]></description>
        <link>https://medium.com/@oksanatkach?source=rss-8d21393a57f1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*LTyQELQ-OyV-AtOO.</url>
            <title>Stories by Oksana Tkach on Medium</title>
            <link>https://medium.com/@oksanatkach?source=rss-8d21393a57f1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 22:29:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@oksanatkach/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[NLP Startup: Where to Begin]]></title>
            <link>https://medium.com/@oksanatkach/nlp-startup-where-to-begin-7e2affdec7a0?source=rss-8d21393a57f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e2affdec7a0</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[business-advice]]></category>
            <dc:creator><![CDATA[Oksana Tkach]]></dc:creator>
            <pubDate>Wed, 24 Aug 2022 01:21:48 GMT</pubDate>
            <atom:updated>2022-08-24T16:06:20.410Z</atom:updated>
            <content:encoded><![CDATA[<p>So you have an NLP idea. Here’s some advice to avoid making basic mistakes.</p><ol><li><strong>Identify that your startup is indeed in NLP</strong></li></ol><p>If you found this article, you are probably on the right track. More often than not, people don’t realize they are creating something NLP-related and that their software could benefit from NLP research. A good rule of thumb — if your idea has anything to do with processing language (text or speech, social media, chats, or articles) — you are probably in NLP. I welcome you in this field, it is incredibly frustrating.</p><p>You might be somewhere in between. For example, a popular field of development currently is processing various text documents, such as PDF. Often, there are projects that focus solely on PDF as an image in order to do all sorts of permutations to documents. In that case, you are likely not in NLP, or at least textual data is not at the forefront of your research. However, if you use the text on the page as well as the pixel position of the text in order to process the document somehow, you are both in Computer Vision <em>and </em>NLP. Good luck.</p><p><strong>2. Define your problems, tasks, and methods</strong></p><p>Very often people get overwhelmed and confused by NLP because they lack structure in their definition of the problem. This is also because the NLP field itself lacks structure, but that’s a topic for a different conversation.</p><p>This is why I wrote the article below that describes how you can break down your NLP-related problem to see more clearly what direction you should be moving in.</p><p><a href="https://medium.com/@oksanatkach/how-to-approach-your-nlp-related-problem-a-structure-guide-70fe259bfc8f">How to Approach your NLP-Related Problem: A Structure Guide</a></p><p><strong>3. Do <em>not </em>start big</strong></p><p>I mean, do start big if you have tons of labelled data. Do start big if you have tons of money to collect a dataset and label it. But, if you have either of those things, you are not a startup. What are you doing reading this article?</p><p>Here’s the problem I see time and time again.</p><p>Many machine learning specialists are math whizzes who would much rather research matrix multiplication than make an NLP demo for you. These people are amazing, but they belong in academia or in a giant company that can give them money and time for research. When you have a person like this on a team, they immediately want to solve every problem by collecting and analyzing data. This is the correct scientific approach, but for a startup, not only won’t this approach produce a viable MVP in a short time, it will likely tank your project very soon.</p><p>A startup’s initial purpose is to either attract some investment or start selling a product that produces some income. Otherwise, it’s not a startup, it’s a hobby. Your initial task should be to have some sort of a demo or a usable app as soon as possible.</p><p>Before you do any sort of custom machine learning, you must first research if there are any available pre-trained models or labelled datasets you can use. With a little bit (or a lot) of linguistic creativity, you can use what is out there to build a semi-decent pipeline for your specific NLP idea. If all else fails, you can always fall back to rule-based NLP methods. This is a great way to manually overfit your model on some custom demo dataset that showcases your app in the best light. Hey, if it was good enough for Steve Jobs, it is good enough for you.</p><p>Of course, never stop there. Rule-based NLP does <strong>not</strong> scale and does <strong>not</strong> generalize on new, previously unseen data. Like, at all. The more people use your product, the worse the experience will become. You must have a growth plan. Expand your dataset continuously and iterate your NLP methods, making them increasingly more accurate and complex. The plan should also take into account the integration and debugging efforts, which will likely take more time than the R&amp;D itself.</p><p>Just don’t try to do all of the data science at the very beginning.</p><p><strong>3. Modularize</strong></p><p>Do <em>not </em>mix NLP and other parts of your code. Always isolate everything NLP-related into a separate module or service. If possible, make the code scalable and universal by moving change-able parameters into separate config or json files. This does not need to be perfect. You’ll find the perfect solution as you improve your code base over the years. It can be primitive, as long as it’s scalable and refactor-able.</p><p>This often happens when the developers initially didn’t realize they were coding an NLP product, so every part of the code was back-end to them, and it all got mixed. This will become a huge expensive pain point once you realize you need to change your NLP methods or refactor the code.</p><p>I once did some work for a startup that had their app coded in java. They were using some rule-based methods (they of course didn’t know this is what they were doing) to count how many times certain words and phrases were occurring in chat conversations. While reviewing the code, I found the word counters in multiple different unrelated places on the backend, as well as in their database module. They literally had NLP coded in SQL. This app was impossible to improve in terms of NLP without re-writing the whole backend. They of course positioned themselves as a <em>data-driven intelligent AI something-or-other</em>. The startup ceased to exist a few months later.</p><p><strong>4. Do <em>not </em>use old technology</strong></p><p>No weird java apps from the 90s (looking at you, GATE). There are plenty of old NLP services that are often rule-based, and so might be perfect for your demo. But I would not recommend using them seriously. They might do exactly what you want, but the development will be slow and painful, connecting modern libraries to them will be impossible or very hard, and hosting them will be difficult and expensive. Then at the end, they might ask for tens of thousands of dollars for you to use the service in production. No.</p><p>Hire a good python developer who’ll recreate that same behavior in a language that’s modern and easy to integrate.</p><p>Yes, python is slow. But your demo doesn’t need to be fast. It’s just a demo. It’s not processing large amounts of data. Optimizing is important, but it is also time-consuming. Start optimizing when users start complaining, not when you don’t even have users. This is why you are modularizing everything at the beginning — to make optimizing easier later.</p><p><strong>5. Hire someone who knows linguistics</strong></p><p>Have at least a few language geeks on the team. You want those people who religiously watched all Jurafky’s lectures they pirated from torrent after coursera took them all down. You want at least two people so they can talk to each other and not be lonely. I mean so they can talk to each other and come up with new cool ideas for your NLP service. You may not understand them, but you need them. Not just machine learning people, although those are always useful to have on the team, but computational linguists who can analyze your data and make experience-driven decisions. I promise you, this works so much better than endlessly tuning parameters in some complicated model that never works that well in real life.</p><p><strong>6. Hire field consultants</strong></p><p>Your NLP business must be solving some problem in some business field. Maybe it’s an app that helps diagnose patients, in which case your business field is medicine. If your primary user base is banks, you are in banking (my condolences). Maybe it’s social media, the translation business, the beauty business. Whatever it is, you need to know what your target user base wants, expects, or how they react to whatever you’ve built. You need a trusted consultant, or a few, preferably people who aren’t in tech but deeply in that business field. They can explain every detail of their workflow and give objective feedback about your software, and even if you don’t listen to everything they say, you’ll still learn a ton.</p><p><strong>7. Do <em>not</em> promise magical AI</strong></p><p>This just doesn’t work. AI works when combined with human reasoning. Build your app in a way that <em>helps</em> a person do their work faster and more efficiently, not replaces the human.</p><p>If you promise an app that does everything for the user, you’ll be met with two problems.</p><p>The first one is rather prosaic — the technology is just not there. It’s impossible and you won’t be able to deliver on your promise.</p><p>The other reason is much more interesting — in some business fields, people <em>don’t want </em>AI to do everything for them. If it’s something very strict, like translating important documents, people don’t trust computers (and rightfully so) to not make mistakes. They want a human to check AI’s work. If “magical AI that will replace your employees” is your primary selling point, you might actually repel potential clients.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e2affdec7a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Approach your NLP-Related Problem: A Structure Guide]]></title>
            <link>https://medium.com/@oksanatkach/how-to-approach-your-nlp-related-problem-a-structure-guide-70fe259bfc8f?source=rss-8d21393a57f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/70fe259bfc8f</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[Oksana Tkach]]></dc:creator>
            <pubDate>Wed, 24 Aug 2022 00:10:48 GMT</pubDate>
            <atom:updated>2022-08-24T16:11:37.382Z</atom:updated>
            <content:encoded><![CDATA[<p>Very often people ask me for an NLP consultation for their business projects but struggle to describe where exactly they need help. This gets even harder when someone had taken one NLP course and knows some terminology, but is applying it in the wrong places. To make sense of what people want, over the years I’ve developed the following structure of how to approach NLP in business.</p><ol><li><strong>What is your business problem?</strong></li></ol><p>Try to think of the problem you are having practically, not in terms of NLP. In very simplified terms, a business problem is when you are losing value or not creating as much value as you need. From my experience, AI tends to save resources rather than generate value. So people turn to AI to automate or speed up some work they would otherwise pay for.</p><p><em>Let’s look at this example. We have an SEO company. The company employs copywriters who write articles that mention particular keywords. This is in order to fill websites with content so that google would show them higher up in their search ranking. The company decides they can’t afford to pay copywriters and they would like to somehow automate the creation of those SEO-friendly articles.</em></p><p><strong>2. What is the NLP problem?</strong></p><p>A business problem can easily be translated into an NLP problem. An NLP problem isn’t defined in terms of saving resources or generating value, it’s defined in linguistic terms. Having something described in linguistic terms makes it much easier to find the NLP task later on.</p><p>Let’s say you trade stock and you want me to build some software that analyzes the news and tells you what some publicly traded company is doing with their business on that particular day. Doing that research manually is time-consuming. That’s your business problem. The NLP problem is to get a computer to identify specific linguistic markers of whether the company is doing well or badly that day. What sort of phrases are we looking for? How would we go about extracting them? What other linguistic markers can be useful (like the tone/mood of the article)? And so on.</p><p><em>In our example, the SEO company needs to figure out how to generate text without human intervention. Not only that, they also need the text to be about a particular topic and contain specific keywords.</em></p><p><strong>3. What is the NLP task?</strong></p><p>NLP tasks are very well defined within the field. I like to separate two different groups of NLP tasks: basic and complex.</p><p>Basic tasks are rarely the solution to your NLP problem. Normally, they are at the basis of feature extraction. Many of them are also “solved”, meaning that for many languages we already have models that give decent results for the basic tasks. These include:</p><ul><li>tokenization</li><li>POS-tagging</li><li>lemmatization</li><li>word representation (e.g. embeddings)</li></ul><p>etc.</p><p>Normally, complex tasks use the basic tasks as a part of pre-processing, and will have multiple parts and pipelines. Some of these are:</p><ul><li>Named Entity Recognition</li><li>text mining</li><li>machine translation</li><li>question answering</li><li>Natural Language Understanding</li></ul><p>etc.</p><p><em>Coming back to our example, the NLP task the SEO company is trying to solve is Natural Language Generation, or text generation.</em></p><p><strong>4. What is the best NLP method to use?</strong></p><p>AI as a field of research, and NLP in particular, includes these methods:</p><ul><li>Dictionary-based (simple substring matching).</li><li>Rule-based (regular expressions and various transducers).</li></ul><p><em>A transducer in NLP is a deterministic model that takes in some string and modifies it according to the rules you coded. Unlike a statistical model, a transducer has a predictable, controllable output.</em></p><p><em>Note that the two methods above aren’t really a part of data science because they are heuristic rather than analytical. But they are still a part of AI. The two methods below is where the machine learning starts.</em></p><ul><li>Statistical (hidden Markov models, maximum entropy, SVM machines, etc.)</li></ul><p><em>I also refer to these as simpler machine learning methods. They are faster and simpler to train and require less data than neural networks to give some results. These can have workable results when your task has low variability (like very obvious linguistic patterns).</em></p><ul><li>Neural networks (LSTM, though rarely used now; CNNs, transformers).</li></ul><p><em>To get better oriented, you can think of neural networks as the same ideas and concepts as the simpler machine learning methods, but reinforced by tons of computational power and data.</em></p><p>Now, this is the part where people often get confused. <em>Any NLP task may be solved by any one of these methods. </em>The methods above are ranked in ascending order by complexity, performance, and the amount of data you’ll need. The dictionary-based method is easy to code and it doesn’t require any data, but it will have very, very low recall. However, it would still be considered AI.</p><p>Regardless of your NLP task, the method you choose depends on:</p><ul><li>how much data you have;</li><li>how much time you have;</li><li>how much money you can pay for the development;</li><li>how much control over the output you need.</li></ul><p>To dwell on the last point a little bit more. For something like a chatbot, you can use a neural network to develop it. But it will have unpredictable outputs (you don’t always know how the chatbot will reply). But if you are using a chatbot for sales, you need it to stick to a particular rhetoric, such as trying to sell the user some shoes. Because of this, chatbots are normally developed using simpler methods, more often the rule-based method. Even if you have the data, time, and money, sometimes for your business purposes you need to “dumb down” the NLP solution in order to control it.</p><p>Named Entity Recognition is a great example here because a NER application can use all of these methods all at once.</p><p>Named Entity Recognition is a task of extracting some named entities from a string of text. Usually people want the computer to identify company names, people’s names, countries, dates, amounts, etc.</p><p>Since there is a limited number of countries in the world, you can just use the dictionary-based method for this. Compile a list of all possible countries and look for them in your input text.</p><p>For dates, there are limited ways in which we write them down. Date formats are highly regular. Because of this, the rule-based method (regular expressions) would perform very well for date extraction.</p><p>Now, company names are much more unpredictable. Extraction of company names in particular is not yet fully solved, but you can often get decent results from a transformer model. You can try extracting companies using NLP rules, and you’ll get decent precision (very little garbage returned), but very low recall (you’ll only extract maybe 20% of company names). So for more complicated entities, machine learning is the better way to go.</p><p><em>Let’s go back to our SEO example. To generate text according to some keywords, the developers can use multiple different methods.</em></p><p><em>To have a quick working prototype for text generation, you can hard-code some rules where you glue together various phrases in order to construct sentences.</em></p><p><em>If you want to generate some fake movie reviews using rules, it would look something like this:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UYgFpoHWxTCMJmu5E5hm9w.png" /></figure><p><em>Is this elegant, smart, or impressive? No. But it’s quick, it doesn’t need a dataset, and with some linguistic expertise you might just fool the google algorithm.</em></p><p><em>If we have more time, we can collect a small dataset for each set of keywords we need, and train a few statistical language models.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t-kt7AOGI0l_LyM7IBtUSg.png" /></figure><p><em>The training is quick. The model generates each next word based on how frequently it appeared in the same context in your dataset (so based on the word’s probability). It may be less readable than the rule-based method but it has much more variability in the text, so might perform better in the search ranking.</em></p><p><em>As the next step, the SEO company may invest in collecting and labelling a few gigabytes of articles. They can then fine-tune a pre-trained transformer based on their custom dataset, and get a model that generates very human-like text on the topic that they want. Maybe they’d need to train multiple models. Who knows. This needs research. This also needs time and money for collecting the dataset, getting the model to work as intended, and deploying this monstrosity to make it usable by anyone in the company.</em></p><p>Hopefully, this article gives a better understanding of how to apply NLP in business. Very often, you don’t need to go full R&amp;D right off the bat. Use a simpler, more primitive method until your business is mature enough to take a more scientific approach. And yes, you can still claim that you use AI on your website.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70fe259bfc8f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In Defense of Rule-Based NLP]]></title>
            <link>https://medium.com/voice-tech-podcast/in-defense-of-rule-based-nlp-ddf2c4611dc6?source=rss-8d21393a57f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddf2c4611dc6</guid>
            <category><![CDATA[speech-recognition]]></category>
            <category><![CDATA[signal-processing]]></category>
            <category><![CDATA[audio]]></category>
            <category><![CDATA[conversational-ui]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Oksana Tkach]]></dc:creator>
            <pubDate>Tue, 23 Jul 2019 19:30:01 GMT</pubDate>
            <atom:updated>2019-07-31T19:11:49.882Z</atom:updated>
            <content:encoded><![CDATA[<h4>Not all AI is neural networks, and not all neural networks are effective</h4><p><em>I have found that we humans are, in a way, prejudiced against simplicity. We tend to believe that more complicated things are more worthy of our attention, regardless of their actual performance. A neural network trained with a terabyte of data on a dozen GPUs for a month sounds like it will have subhuman performance.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*FcAQ5VaSftebaThFfkV8DQ.jpeg" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fbingewith.com%2Farticle%2F%3A%3A_Generate_Audio&amp;dntp=1&amp;display_name=BingeWith&amp;url=https%3A%2F%2Fbingewith.com%2Farticle%2F%3A%3A_Generate_Audio&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=bingewith" width="480" height="110" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6a9e202d4ee773dbc23199fcf6d186c/href">https://medium.com/media/d6a9e202d4ee773dbc23199fcf6d186c/href</a></iframe><p>Contrary to popular opinion, artificial intelligence is not <em>only</em> neural networks. The term neural network sounds awesome because it hints that the algorithm somehow mimics the way we think (which, it doesn’t). For many problems, neural networks give results of amazing nuance. For many, but nearly not all of them.</p><p>See, a neural network is only as good as the data you train it on. In absence of large amounts of well-structured, clean, homogeneous data, your best algorithm trained for days won’t really perform.</p><p>This is why there’s never a clear distinction between computational linguistics, Natural Language Processing, Data Science and AI that understands and generates language. Because some rule-based approaches in NLP are so linguistics-based that I hesitate to count it as part of Data Science. Data Science has a process: you collect clean, relatively homogeneous data, you analyze it and get <em>objective</em> statistical information about that data, you extract features based on that analysis, you train a model on an algorithm that will extract the most informative bits from your data in the best way. It’s a descriptive, objective process that only works with and is completely reliant on good data.</p><p>In rule-based approach, you rely completely on human expertise, with our biases and blind spots. The rule-based approach is prescriptive and subjective. This is why it is also so much more usable.</p><p>Here are four reasons why rule-based approaches in Data Science, NLP in particular, work.</p><ol><li><strong>It’s a quick working solution</strong></li></ol><p>I could bet my beloved macbook pro that about 80% of startups that market themselves as “AI-powered” and “data-driven” actually just use a bunch of if-statements in their MVP.</p><p>So are they lying about their product? Not at all. Rules are still AI.</p><p>Is their product working then? Most likely. It is definitely performing better than a baseline. Their developers definitely have enough expertise in the field to come up with some heuristic solution that does work in the real world.</p><p>Why aren’t they using a neural network? Probably because they don’t have enough data yet and collecting and analyzing data is in their business plan at some future stage, when they reach some critical user mass. Most likely though, they are spending their VC money on marketing and sales.</p><p><strong>2. Because solving a Data Science problem is like trying to reach the speed of light</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j708uMCdk6bzDUQb5_znrQ.jpeg" /></figure><p>The baseline is very easily reached. Something like 80% is harder but doable, depending on a task. But as you approach the 100%, every fraction of a percent is exponentially harder to reach, in terms of amounts of data and algorithm complexity. This is because Data Science is trying to explain the real world, and the real world is <em>mostly predictable but unchangeably random. </em>Not even real life humans that label data sets can agree between each other 100%. Human results are more like 97–99% accurate.</p><blockquote>Build better voice apps. Get more articles &amp; interviews from voice technology experts at <a href="https://voicetechpodcast.com/">voicetechpodcast.com</a></blockquote><p>Take language, for example. When studying a language synchronically, i.e. in a particular moment in time, it has a limited vocabulary — only the words that we use at that moment have meaning. But if you look at a language diachronically, i.e. as a system through time, you’ll see that its vocabulary is endless. Because we come up with new words all the time, be it something that has a logical derivation, like lol, or something cultural, like covfefe.</p><p>But I digress.</p><p>Iterative development is a great way to ride this speed-of-light phenomenon. Say you’re an SEO company, and you’ve decided to ditch the copywriters. They are expensive and not always as effective as you’d like. So you want an NLP solution that will generate texts based on the keywords you provide.</p><p>You can, in a respectably short period of time, set up a rule-based system that will generate sentences based on some grammar and insert your keywords as heads of noun or verb phrases. You can add some more rules to exclude weird combinations of words. The result will be a text that’s not particularly readable, but one which could probably fool Google.</p><p>While you’ve got a solution up and working, you can create a system that is a little more sophisticated: for each set of keywords, you can find existing relevant, human-written articles by Google search, then just count how many times each word occurs in them. And voila — you’ve got a statistical language model for your particular set of keywords. You can then generate brand new sentences based on ngram probabilities. The text still won’t make much sense overall, but now you’ve excluded any weird word combinations.</p><p>As the third iteration, you can crawl websites to collect a data set with human-written articles, and use the keywords extracted from the HTML as their labels. Now you can dwell into language generation neural networks. They will likely produce texts that are coherent from start to finish. (But still read like my undergraduate thesis — with no real point to it.)</p><p><strong>3. The problem is too specific</strong></p><p>Some NLP solutions need to be deterministic.</p><p>Take Siri, for example. Siri doesn’t generate its answers to your queries based on ngram probabilities. The bot is an information retrieval system, so she has to give you logical, concrete answers. For your each question, there is a logical path of if-statements built in. That’s called a knowledge base.</p><p>A knowledge base is a structured collection of facts about the world. It can’t, by definition, be probabilistic. This is why information retrieval chatbots trained on neural networks don’t work (yet).</p><p>If a corporation wants to use a chatbot as an additional sales channel, that’s more of a design problem than a Data Science problem. Sure I could train a neural network on a bunch of conversations, but the client doesn’t care if the bot can reply “Polo” to “Marco”, they just want it to be easy to sell something through the bot. The task is more to re-create user interface, but with a conversation. It <em>has</em> to be hard-coded.</p><p>A hard-coded niche chatbot is still a linguistics problem though. You have to analyze discourse — how people tend to communicate. You have to model a conversation and all its caveats. You have to predict all users’ possible responses, using linguistic features like sentence structure, part-of-speech tags, etc.</p><p>The smartest chatbot I ever built was an enormous project written in ChatScript — a fully rule-based, offline, lightweight engine. The end result was a smart bot because the conversation I was modeling was very niche and couldn’t deviate wildly by design. I just spent A LOT of time thinking about all possible responses the users could give me, testing the conversation on other people to see what they naturally reply that I hadn’t predicted, modeling the chatbot’s words in a way that solicited exactly the response I needed to move the exchange forward, and making sure my algorithm’s dozens of if-statements didn’t conflict with each other. The end result was significantly more impressive than that of a seq2seq neural network trained on actual people’s conversations.</p><p><strong>4. Software Limitations</strong></p><p>Sofia the robot uses ChatScript to talk. This is because, I am fairly certain, Sofia is just a raspberry pi stuck inside a mannequin with a ginormous marketing team behind it. It is most likely not even connected to internet. ChatScript is fully portable, fully-rule based, lightweight engine that doesn’t rely on any external resources. It’s run as a local server out of the box and it’s very easy to install and compile. Perfect for when you don’t have enough memory to store and unpack a language model.</p><p>One of my clients has some insanely outdated software stack. At one point, we had to write a sentence boundary disambiguation algorithm that could only run on python 2.6, with no possibility to install any additional libraries or import any external files, and execution time restriction of something like a minute or two. No Data Science method, apart from rule-based, could possibly work here.</p><p>All Data Science problems start with identifying the features of your data. When you don’t have the data to chart onto a nice pandas chart, you have to turn to a tragically over-looked DS skillset component: domain knowledge. Sometimes you just have to think about how language works and <em>assume</em> what features <em>would</em> come up on a pandas chart if you had data to analyze. Then you take those assumed features and write an if-statement for each. There, you’ve got yourself a heuristic algorithm. This is why for a serious NLP project where some parts just can’t be solved with statistical classifiers or neural networks, the expertise of an analytical linguist is vital.</p><p>In a perfect world, this shouldn’t work, because any heuristic solution is subjective by definition, and science doesn’t like that. But in the real world, with humans and deadlines and business plans, it does work, and we should use it.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F6b7b35%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2F6b7b35%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a76a1650cf30355bd307195d382558a/href">https://medium.com/media/8a76a1650cf30355bd307195d382558a/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddf2c4611dc6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/voice-tech-podcast/in-defense-of-rule-based-nlp-ddf2c4611dc6">In Defense of Rule-Based NLP</a> was originally published in <a href="https://medium.com/voice-tech-podcast">Voice Tech Podcast</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Internationalize Your Flask App Like a Champ]]></title>
            <link>https://medium.com/@oksanatkach/how-to-internationalize-your-flask-app-like-a-champ-e57535185893?source=rss-8d21393a57f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/e57535185893</guid>
            <category><![CDATA[babel]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[flask-framework]]></category>
            <category><![CDATA[internationalization]]></category>
            <category><![CDATA[localization]]></category>
            <dc:creator><![CDATA[Oksana Tkach]]></dc:creator>
            <pubDate>Tue, 23 Jul 2019 14:02:18 GMT</pubDate>
            <atom:updated>2024-03-07T12:37:55.221Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Internationalize Your Flask App Like a Pro</h3><h4>How to tweak the Babel library for large i18n projects to enable pseudotranslation and iterative localization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BWrdcJMoTqTKjeBqOqGYaQ.jpeg" /></figure><p>At Metamova, we do a lot of localization engineering, and a lot of what I do at work on a regular basis is making sure that whatever work the translators did actually end up in any kind of human accessible, readable form. Unfortunately, sometimes our source files are so weird that I can’t really fix anything on the localization engineering stage. Especially when it comes for products that need continuous support, like websites and apps. Usually the reason we can’t help is because the stage before localization — <em>internationalization</em> — was done incorrectly or not at all.</p><p>Here’s a little project of mine where I internationalized my website as if it will be translated into a dozen languages.</p><p>I built my company’s website with Flask. Partly because I am loyal to Python, partly because I am so not learning PHP at this point in my life, but mostly because Flask is just plain fun to work with.</p><p>Anyway, for translation purposes, Flask uses a library called Babel. I have many problems with Babel and would like to rewrite many parts of it, but its good parts are too good. Babel has a large collection of dat files that contain specifications for dates, currency, fonts, etc. for <strong>735</strong> locales. That is very impressive and very useful.</p><p>Babel extracts your translatable content into PO files in which you can manage your translations, then tells Flask to display the strings according to what locale a user’s browser returns.</p><p>For Babel to know which content is translatable, you have to add a Babel config file to your project. The config file contains some predefined filters that tell Babel where to look for human text in your HTML template. As a rule, you need to go to your HTML templates and wrap all of your displayable text like this:</p><pre>{{ _(&#39;Some text here&#39;) }}</pre><p>The double curly braces tell Jinja web template engine that this is not HTML but some executable expression. The underscore and normal braces tell Babel that everything inside the quotes that follow is some text to be translated.</p><p>Now, here’s a crazy idea that will make the internationalization process infinitely easier for you: don’t put the actual human text in the HTML code.</p><p>Think about it. For a huge international website this makes no sense. If your product <em>is </em>the website (i.e. it’s a web-store, not just a portfolio webpage), and your product is international, some of the pages on your platform are bound to look different for different locales. You are also bound have different features and promos for different countries, AND if your product keeps growing and changing, the content will change so rapidly it makes no sense to keep updating the HTML code, or to keep rewriting it with new content. You actually risk creating such a mess you can’t even keep track of different versions of your source strings.</p><p>More than that, your source language may change. You may have different source languages, depending on the locale you are translating into, its culture and market. For instance, if you already translated English to German, maybe it would make more sense to have German be the source language for your Scandinavian expansion. Babel can’t have several source languages if the source language is already in the HTML template. It goes against the logic of Babel’s design. So let’s tweak Babel’s design a little. Here’s how.</p><p>Instead of natural language strings, it’s better to put some unique identifiers in the HTML, then get your corresponding strings by IDs, per target language. When designing and creating the HTML template, come up with a descriptive ID and put it in place of the actual string, then wrap it in Babel-friendly markup:</p><p>Before:</p><pre>&lt;li class=&quot;nav-item&quot;&gt;<br>    &lt;a class=&quot;nav-link active&quot; href=&quot;#&quot; data-scroll-nav=&quot;0&quot;&gt;<br>        {{ _(&#39;Home&#39;) }}<br>    &lt;/a&gt;<br>&lt;/li&gt;</pre><p>After:</p><pre>&lt;li class=&quot;nav-item&quot;&gt;<br>    &lt;a class=&quot;nav-link active&quot; href=&quot;#&quot; data-scroll-nav=&quot;0&quot;&gt;<br>        {{ _(&#39;nav.item_1&#39;) }}<br>    &lt;/a&gt;<br>&lt;/li&gt;</pre><p>Now, Babel functionality will treat the ID as the source, and export it into a PO file, where you can add the corresponding string as translation:</p><pre>#: app/templates/metamova.html:78<br>msgid &quot;nav.item_1&quot;<br>msgstr &quot;Home&quot;</pre><p>It’s here that I would like to mention that there are very many problems that I see with the PO format, and its design doesn’t make much sense to me. For instance, by default it treats natural language string as a unique identifier (worst idea ever), it’s kind of hard to add any extra information about the strings in a structured way, the versioning logic is unnecessarily complicated and has made me overwrite existing translations with empty strings several times before I got the grasp of how to work with these weird files. However, for all its faults, PO is the only format in the translation industry that, unlike XLIFF, offers some interoperability. Which is why I presume Babel uses it. So let’s go through what I think is the coolest way to leverage all the functionality of Babel while making your translation process as efficient as possible.</p><p>When you initialize a new locale in Babel, you get a new PO file. By our logic, our IDs are our source strings. So you’ll have PO files for each locale, all containing IDs, waiting to be connected to your translated content.</p><p>By using Babel’s built-in PO parser, we can now read all the PO files and merge them into one elegant JSON structure showing all translations for the same ID:</p><pre>&quot;nav.item_1&quot;: {<br>    &quot;en&quot;: &quot;Home&quot;,<br>    &quot;uk&quot;: &quot;Головна&quot;,<br>}</pre><p>Here’s my function for merging the PO files into JSON:</p><pre><strong>def </strong>export_strings(source=<strong>&#39;en&#39;</strong>, target=<strong>None</strong>):<br>    source_str = StringIO(open(translations + <strong>&#39;/&#39; </strong>+ source +      <br>        <strong>&#39;/LC_MESSAGES/messages.po&#39; </strong>, <strong>&#39;r&#39;</strong>, encoding=<strong>&#39;utf-8&#39;</strong>).read())<br>    source_catalog = read_po(source_str)<br>    for_tron = { message.id: {source: message.string}<br>                 <strong>for </strong>message <strong>in </strong>source_catalog <strong>if </strong>message.id }<br><br>    <strong>if not </strong>target:<br>        <strong>for </strong>locale <strong>in </strong>babel.list_translations():<br>            locale = locale.language<br>            <strong>if </strong>locale <strong>!=</strong> source:<em><br>                </em>target_str = StringIO(open(translations + <strong>&#39;/&#39; </strong>+<br>                locale + <strong>&#39;/LC_MESSAGES/messages.po&#39;</strong>, <strong>&#39;r&#39;</strong>,<br>                encoding=<strong>&#39;utf-8&#39;</strong>).read())<br>                target_catalog = read_po(target_str)<br><br>                <strong>for </strong>message <strong>in </strong>target_catalog:<br>                    <strong>if </strong>message.id <strong>and </strong>message.id <strong>in </strong>for_tron.keys():<br>                        for_tron[message.id][locale]=message.string<br>    <strong>else</strong>:<br>        target_str = StringIO(open(translations + <strong>&#39;/&#39; </strong>+ target +<br>          <strong>&#39;/LC_MESSAGES/messages.po&#39;</strong>, <strong>&#39;r&#39;</strong>, encoding=<strong>&#39;utf-8&#39;</strong>).read())<br>        target_catalog = read_po(target_str)<br><br>        <strong>for </strong>message <strong>in </strong>target_catalog:<br>            <strong>if </strong>message.id <strong>and </strong>message.id <strong>in </strong>for_tron.keys():<br>                for_tron[message.id][target] = message.string<br><br>    <strong>with </strong>open(app_path + <strong>&#39;/json_strings/strings.json&#39;</strong>, <strong>&#39;w&#39;</strong>,<br>     encoding=<strong>&#39;utf-8&#39;</strong>) <strong>as </strong>outfile:<br>        json.dump(for_tron, outfile, ensure_ascii=<strong>False</strong>)</pre><p>Now, no matter who handles your localization engineering, all they need is this one JSON file. The engineer will extract whatever language you choose to be the source, then insert the translations into the corresponding locale. You can then use the updated JSON to import the strings into your PO files:</p><pre><strong>def </strong>import_strings(filename=<strong>None</strong>, source=<strong>&#39;en&#39;</strong>, target=<strong>None</strong>):<br>    <strong>if </strong>filename:<br>        from_tron = json.loads(open(filename, <strong>&#39;r&#39;</strong>, encoding=<strong>&#39;utf-8&#39;</strong>)<br>         .read())<br>    <strong>else</strong>:<br>        from_tron = json.loads(open(app_path +<br>        <strong>&#39;/json_strings/strings.json&#39;</strong>, <strong>&#39;r&#39;</strong>, encoding=<strong>&#39;utf-8&#39;</strong>).read())<br><br>    template_str = StringIO(<br>     open(<strong>&#39;messages.pot&#39;</strong>, <strong>&#39;r&#39;</strong>, encoding=<strong>&#39;utf-8&#39;</strong>).read())<br><br>    <strong>if not </strong>target:<br>        <strong>for </strong>locale <strong>in </strong>babel.list_translations():<br>            locale = locale.language<br>            new_catalog = Catalog()<br>            <strong>for </strong>id <strong>in </strong>from_tron:<br>                <strong>if </strong>locale <strong>in </strong>from_tron[id].keys():<br>                    new_catalog.add(id, from_tron[id][locale])<br>            new_catalog.update(template)<br><em>            </em>write_po(open(translations + <strong>&#39;/&#39; </strong>+ locale +                  <br>             <strong>&#39;/LC_MESSAGES/messages.po&#39;</strong>, <strong>&#39;wb&#39;</strong>), new_catalog)<br><br>    <strong>else</strong>:<br>        new_catalog = Catalog()<br>        <strong>for </strong>id <strong>in </strong>from_tron:<br>            <strong>if </strong>target <strong>in </strong>from_tron[id].keys():<br>                new_catalog.add(id, from_tron[id][target])<br>        new_catalog.update(template)<em><br>        </em>write_po(open(translations + <strong>&#39;/&#39; </strong>+ target + <br>          <strong>&#39;/LC_MESSAGES/messages.po&#39;</strong>, <strong>&#39;wb&#39;</strong>), new_catalog)</pre><p>If you want to do your engineering in a more web 2.0-friendly way, with all of your strings exported into this JSON, you could build a server sending POST requests to any online translation service out there, in a controlled way. You can add a “date” attribute and save all the previous versions of strings that belonged to that particular ID, for any language. Et voila, your JSON file is now a translation memory as well!</p><pre>&quot;nav.item_1&quot;: {<br>    &quot;en&quot;: &quot;Home&quot;,<br>    &quot;uk&quot;: [<br>        {string: &quot;Головна&quot;, date: &quot;&quot;},<br>        {string: &quot;Домашня сторінка&quot;, date: &quot;&quot;}<br>    ]<br>}</pre><p><strong>Pseudotranslation with Babel</strong></p><p>But we are not done yet! Before you send your strings for translation, you have to check if you’ve extracted all text on your webpage. This is called pseudotranslation. We will just insert a bunch of random symbols where the translation should be, import it back to your webpage and look through it in a browser to see if there aren’t any rogue source strings anywhere that didn’t get exported for translation, like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uR7w7b80oY2_AT6AHeXH6A.png" /><figcaption>Oh no, we forgot about the form placeholders!</figcaption></figure><p>To achieve this with Babel, we’ll need to tweak its source code a little bit. We will force Babel to display the strings we want by adding a new fake locale, called “pseudo”, to its list of all possible languages.</p><p>First of all, go to folder <strong>babel </strong>in your Python lib folder, then open the folder <strong>locale-data</strong>. For Babel to load a locale, it requires a corresponding DAT file to be placed in the <strong>locale-data </strong>folder. To have a DAT file for our fake pseudo language, we can copy the <strong>en.dat </strong>file (if you want to test right-to-left languages like Arabic, you could copy one of the files for Arabic locales). We will then name the copied file <strong>pseudo.dat</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eT3lBTXxv13BRxTN6lk4aQ.png" /><figcaption>Find this location on your server and create a dummy pseudo file</figcaption></figure><pre>cd &lt;your env name&gt;/lib/python&lt;youpr python version&gt;/site-packages/babel/locale-data/<br>cp en.dat pseudo.dat<br>cp en.dat ids.dat</pre><p>Now, open the file <strong>core.py</strong> in the <strong>babel </strong>directory and find the list LOCALE_ALIASES, and add two new fake locales at the end of the list: ‘pseudo’: ‘pseudo’ and ‘ids’: ‘ids’. (I also added a fake IDs locale to be able to display our original IDs.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wOXy86JLjsArGs6KbzfBTA.png" /><figcaption>Add your fake locales to Babel’s source code</figcaption></figure><pre>nano &lt;your env name&gt;/lib/python&lt;youpr python version&gt;/site-packages/babel/core.py</pre><p>Initialize <strong>pseudo </strong>and <strong>ids </strong>as new locales for translation in your configuration file or as an environment variable on your server.</p><pre>LANGUAGES = os.environ.get(<strong>&#39;LANGUAGES&#39;</strong>) <strong>or </strong>[<br>                                        <strong>&#39;en&#39;</strong>, <strong>&#39;uk&#39;</strong>, <strong>&#39;ids&#39;</strong>, <strong>&#39;pseudo&#39;</strong>]</pre><p>Using a simple regex, insert a bunch of X-s instead of translation into the PO file and run the Babel command to compile the translation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*khQDWUPr3Q-je-lnjo4WJg.png" /><figcaption>Pseudotranslated PO file</figcaption></figure><p>Flask pulls up the locale that your browser tells it to, based on your location or your preferred language you chose in the browser. Now, to force Flask to display a specific locale for a specific page, you can use the Babel function <em>force_locale</em>. Here’s how my Flask route for pseudotranslation looks:</p><pre>@app.route(&#39;/pseudo&#39;)<br>def pseudo():<br>    with flask_babel.force_locale(&#39;pseudo&#39;):<br>        return render_template(&#39;metamova.html&#39;)</pre><p>Now I can just go to <a href="https://metamova.com/pseudo">https://metamova.com/pseudo</a> to view my website pseudotranslated at any time.</p><p>To display the original IDs, we don’t even need to initialize translation. If there’s no translation from Babel, Flask will just display the strings that are in the HTML. So you can do the same route but for IDs:</p><pre>@app.route(&#39;/ids&#39;)<br>def ids():<br>    with flask_babel.force_locale(&#39;ids&#39;):<br>        return render_template(&#39;metamova.html&#39;)</pre><p>Now I can check what ID goes where at any time just by typing <a href="https://metamova.com/ids">https://metamova.com/ids</a> into my browser.</p><p>This is very exciting, because with a little bit of javascript magic we could create an interactive in-context translation view that would insert whatever the translator types into the right place based on the ID, so the translator could see the position of the string, as well as how short or long it should be right there on the page.</p><p>As someone who used to translate websites, this would have been a dream. For Ukrainian, one simple string like «Sign Up» could be translated as an infinitive («Зареєструватись»), call-to-action imperative mood («Зареєструйтеся»), a descriptive noun («Реєстрація»), or some completely different, much shorter phrase (like «Вхід»), all depending on where the phrase appears in the UI.</p><p>These are all very simple tweaks, but can be extremely helpful if done <em>before </em>you launch your website. If you plan to go aggressively international down the road, you’ll have no time to think about an optimal localization process when you are actually in the middle of managing a complicated web app for ten to twenty different countries and languages.</p><p>And if you have no idea what you are doing with your translations or if your process is the best it can be, get in touch with us by going to <a href="https://www.metamova.com/">metamova.com</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e57535185893" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>