<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bleckwen Tech Team on Medium]]></title>
        <description><![CDATA[Stories by Bleckwen Tech Team on Medium]]></description>
        <link>https://medium.com/@bleckwen_tech_team?source=rss-20fc04793e8c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*tCx4NOBigHECkDBEmldLyA.jpeg</url>
            <title>Stories by Bleckwen Tech Team on Medium</title>
            <link>https://medium.com/@bleckwen_tech_team?source=rss-20fc04793e8c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:34:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bleckwen_tech_team/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Twitter sentiment analysis in real time, part 3]]></title>
            <link>https://medium.com/bleckwen/twitter-sentiment-analysis-in-real-time-part-3-9318233804c1?source=rss-20fc04793e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9318233804c1</guid>
            <category><![CDATA[cloud-deployment]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[docker-compose]]></category>
            <dc:creator><![CDATA[Bleckwen Tech Team]]></dc:creator>
            <pubDate>Thu, 25 Jul 2019 09:31:51 GMT</pubDate>
            <atom:updated>2019-07-25T09:31:51.822Z</atom:updated>
            <content:encoded><![CDATA[<h4>Written by <a href="https://medium.com/u/46a57fcd03f2?source=post_page---------------------------">Manar Toumi</a>, <a href="https://medium.com/u/4589f0a13a59?source=post_page---------------------------">Niels Borie</a>, <a href="https://medium.com/u/f1ca6a2fa815?source=post_page---------------------------">Quentin Coïc</a> &amp; <a href="https://medium.com/u/8136ed9be4d4?source=post_page---------------------------">Andy Touly</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uAqnFIE8upNowRfw.jpg" /></figure><p><strong>This article will cover topics as cloud deployment or automation with Jenkins, but you won’t find the corresponding parts on our public </strong><a href="https://github.com/BleckwenAI/teastream"><strong>GitHub</strong></a><strong> where we propose you to set up the application described in this series of articles locally thanks to scripts.</strong></p><p><a href="https://medium.com/bleckwen/twitter-sentiment-analysis-in-real-time-part-2-64ae1d7b0e">Previously</a>, we created a tweet sentiment analysis pipeline that gathers the tweets from the source, in real time, analyses them and stores them for visualization purposes…</p><h3>Introduction</h3><p>Wayne entreprises got wind of our little project and wants it. As crime in the city Gotham reached scary numbers, it is no longer easy to prevent it. Batman himself contacted us, he wants to use our solution to feel Gotham’s pulse through analyzing its tweets. The problem is, using our solution is not a walk in the park. It requires many manual steps: starting each service we use, specifying configuration…</p><p>Hand in hand with Bruce Wayne himself we will put in place the necessary elements to make setting up and interacting with it easier.</p><h3>The utility belt</h3><p>As the times get tough, we need to arm ourselves with some solid tools. For interacting with our app, running every component with ease and automatizing builds, so let’s do a little overview of the technologies we use, so that we can understand them better.</p><blockquote>You will always fear what you don’t understand.<br>Carmine Falcone</blockquote><h4>Flask</h4><p>Light, straightforward and easy to use, that is <a href="http://flask.pocoo.org/">Flask</a> in a nutshell. It is a microframework for python that we will use to create a simple user interface where he can specify the term he wants to track.</p><h4>Docker and Docker-compose</h4><p>In order to organise the different aspects of the application we will use <a href="https://www.docker.com/">Docker</a>. It relies on the concepts of <a href="https://docs.docker.com/engine/docker-overview/#docker-objects">Docker images and containers</a> where each piece of app is packaged with all the parts it needs. To put the pieces toghether we use <a href="https://docs.docker.com/compose/">Docker Compose</a>.</p><h4>Microsoft Azure services</h4><p>Our application will be deployed on cloud thanks to <a href="https://azure.microsoft.com/en-in/overview/what-is-azure/">Microsoft Azure</a> cloud services. More specifically, we will deploy it on a <a href="https://docs.microsoft.com/en-in/azure/virtual-machines/linux/">Linux virtual machine</a> which will be the cloud host for our containerised application and we will use a private <a href="https://docs.microsoft.com/en-us/azure/container-registry/container-registry-intro">container registry</a> to store the Docker images built for some pieces of our application. In addition, we will also use the <a href="https://docs.microsoft.com/en-us/azure/dns/dns-overview">Azure DNS</a> service to give a name for the virtual machine that can be used instead of its IP address.</p><h4>Jenkins</h4><p>For the purpose of a continuous integration, building and deployment we will use <a href="https://jenkins.io/">Jenkins</a> to build and test our project continuously making it easier for us to integrate changes to the project, to obtain fresh builds and to provide results quickly.</p><h3>The command center</h3><p>In the Batcave, Bruce has many monitors displaying our Kibana dashboards and follow the sentiment about a certain twitter topic. But specifying what word to track, stop tracking and display tracked words is quite “labor intensif”. So, in collaboration with Wayne Entreprises we designed a more user friendly control interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/0*WX83McVgptSBbCBn.jpg" /></figure><p>The interface allows tracking a new term by adding it to the tag list, stop tracking a term by simply removing it.<br>We built this web app using the python Flask framework. Adding a tag triggers sending a request track &#39;term&#39; through a kafka topic to our Scala job that communicates with Twitter and fetches the corresponding tweets.</p><h3>The Docker trick</h3><blockquote>Have you finally learned to do what is necessary?<br>Ra’s al Ghul</blockquote><p>Progressively, our application has been enriched and it is now composed of a dozen of pieces. So far, to test the different parts of our application, we had to launch every service or run everything manually. What a tedious scheme! We then have decided to use Docker Compose in order to start each piece in one command.</p><p>In order to use Docker Compose, we need to containerize every piece of the application that can be divided into two types:</p><ul><li>Services that we directly use like Zookeeper, a Kafka broker, Kafka connect, Flink, Elasticsearch or Kibana. For these ones, we just have to pull already built Docker images from the public Docker registry, Docker Hub.</li><li>For our jobs (the Twitter client and the Tweet analysis) and our web application, we setup a proper environment, to build then run our code using, a <a href="https://docs.docker.com/engine/reference/builder/"><em>Dockerfile</em></a>. Once built, the Docker images are pushed to our private Microsoft Azure container registry to be stored and easily pulled later.</li></ul><p>Then, we write a <a href="https://docs.docker.com/compose/compose-file/compose-file-v2/"><em>docker-compose.yml</em></a> file to define how our Docker containers should behave in production and that’s all. The following single command is then sufficient to start each piece of our application</p><pre>docker-compose up</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/0*_jcML3rcpHU06nXn.png" /></figure><p>This architecture is the one we use on our Microsoft Azure Linux virtual machine to deploy our application on cloud.</p><h3>Alfred’s labour</h3><blockquote>Yes master Bruce?<br>Alfred Pennyworth</blockquote><p>Well, we containerized every piece of our app, but … that’s pretty annoying to test it, especially during the development phase when you often integrate new features. Indeed, we can run the entire thing in one command, but there are several steps that need to be done before it. Sooner or later, all custom images that has been created need to be update requiring to edit the Dockerfile and build the image again.</p><p>Worse, some of the images we have created require a .jar file (for example, the “flink-job” image needs the .jar of the Flink job presented in article 2 to work). In these cases, we need to recreate the .jar, put it in the right directory, build the image again and push it.<br>These steps were time consuming, so we gave it to Alfred… hum… I mean Jenkins.<br>After <a href="https://medium.com/@teeks99/continuous-integration-with-jenkins-and-gitlab-fa770c62e88a">creating and configuring</a> our job on the Jenkins platform, it’s time to tell him what to do! And for that, we will define his roadmap through the <a href="https://jenkins.io/doc/book/pipeline/jenkinsfile/">Jenkinsfile</a>. Basically, ours looks like this :</p><pre>pipeline {<br>    agent { <br>        /* environment where the entire Pipeline will be executed */<br>    }<br>    triggers {<br>        /* when the pipeline should be re-triggered */<br>    }<br>    environment {<br>        /* set of environment variables */ <br>    }<br>    stages {<br>        /* insert all stage here */<br>     }<br>}</pre><p>It’s in the “stages” tab that we will define the tasks that the Jenkins job mush accomplish, like:</p><ul><li>creating the jars</li><li>building docker images</li><li>publishing every image on a docker registry (in our case, we publish it on a private registry on Microsoft Azure)</li></ul><p>Because the order of steps has an importance, here’s what Jenkins needs to do:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i6goaeHbBLRCglqA.png" /></figure><p>Without going into too much detail Jenkins just does two sbt assembly and several docker build / push.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hyLrfpRgUJmeptha.jpg" /></figure><p>In this way, when we modify our Flink’s job, the python Flask back-end or our Kafka producer, no need to rebuild everything: <strong>Jenkins handles it</strong>.<br>We just have to push on our git repository some modification, Jenkins will rebuild all docker images and publish it. A simple docker-compose up will update our services by retrieving the latest versions of the images.</p><h3>Application/example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RQXzU0Lg3-fgVS81.jpg" /></figure><h3>Why do we fall, Bruce?</h3><blockquote>Why do we fall, Bruce? So we can learn to pick ourselves up…<br>Thomas Wayne</blockquote><p>The main lesson we learned during this deployment phase of the project was: sometimes, it is better to take a step back on what we’re doing. This lesson applies to all parts of a project, but for this article it was crucial. <strong>Understanding first how Docker works in practice before trying to debug part of our code during deployment should have been our priority</strong>.<br>One thing that bothered us at first was understanding network interfaces in Docker/Docker Compose.<br>It’s not a difficult thing, but you have to understand that the localhost inside a Docker container is not necessarily the same localhost as outside… you know…<br>And things get more complicated when there are several containers, communicating with each other, on a newly created network.<br>It took time to find what’s wrong with our Flink’s job code: we were surprised when we realized that localhost:9092 could no longer correspond to the adress of our Kafka broker (because we are in a Docker container now) and that we had to replace it with broker:9092.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iNUhSmbA0zyR5Jot.jpg" /></figure><h3>Now what</h3><p>Throughout this project, we learned a lot. We got familiar technologies that were previously unknown to us and built a mini application of “sentiment analysis” from beginning to end. We went through code prototyping phases (mainly in scala) up to the automation and deployment phase with Docker and Jenkins. This fabulous trip allowed us to learn a little more about the software development and DevOps disciplins. We have learned to work as a team and to learn from our mistakes.<br>Our project is certainly not perfect, far from it. There is still much to be done and explored. But we are convinced that this tiny project has helped us to face the future with greater serenity.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9318233804c1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bleckwen/twitter-sentiment-analysis-in-real-time-part-3-9318233804c1">Twitter sentiment analysis in real time, part 3</a> was originally published in <a href="https://medium.com/bleckwen">Bleckwen</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Twitter sentiment analysis in real time, part 2]]></title>
            <link>https://medium.com/bleckwen/twitter-sentiment-analysis-in-real-time-part-2-64ae1d7b0e?source=rss-20fc04793e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/64ae1d7b0e</guid>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[h2o]]></category>
            <category><![CDATA[flink]]></category>
            <dc:creator><![CDATA[Bleckwen Tech Team]]></dc:creator>
            <pubDate>Thu, 27 Jun 2019 09:37:50 GMT</pubDate>
            <atom:updated>2019-06-27T09:37:50.273Z</atom:updated>
            <content:encoded><![CDATA[<h4>Written by <a href="https://medium.com/u/46a57fcd03f2">Manar Toumi</a>, <a href="https://medium.com/u/4589f0a13a59">Niels Borie</a>, <a href="https://medium.com/u/f1ca6a2fa815">Quentin Coïc</a> &amp; <a href="https://medium.com/u/8136ed9be4d4">Andy Touly</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*epFOM8jYv5Dm5vs0.jpg" /></figure><p><a href="https://medium.com/bleckwen/twitter-sentiment-analysis-in-real-time-part-1-be3fcc0bb4a2">Previously</a> on <strong>twitter sentiment analysis in real time</strong>, we managed to invite tweets that have a common topic into our base. Hundreds and thousands of tweets come before our eyes. Yet we don’t know them, we don’t know how they think or what they feel. There are just far too many of them and just… three of us!</p><blockquote>Remember that our source code is available on our <a href="https://github.com/BleckwenAI/teastream">GitHub</a>!</blockquote><h3>A dive into a tweet’s mind</h3><p>The tweets go through quite a journey before arriving to their last destination. From selection by topic using the Twitter API, shipping with the kafka broker to storage and visualisation with Elasticsearch and Kibana. So we are going to add one more stop along their way, for them to rest and for us to get some information about them. This stop will include a fantastic therapy and analysis session. Where we will get to <strong>know them better</strong>.</p><h3>Therapist toolkit</h3><p>Before jumping into the business of tweet therapy we need the right infrastructure. Our therapist is, as you might have guessed, no human. He’s a <strong>machine learning</strong> model that needs a framework to learn how to be a decent therapist. He also needs an office where tweets come and go to get <strong>processed</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Nq7pp98RTgL2EPMZ.png" /></figure><h3>Processing</h3><p>For the task of processing the tweet stream, let’s use <a href="https://flink.apache.org/"><strong>Apache Flink</strong></a> (<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.6/quickstart/setup_quickstart.html">quickstart</a>) a distributed processing framework that can be used over unbounded and bounded data streams.</p><blockquote><em>The choice of Flink can be justified by its exactly once processing (each message is processed once) and its stateful computing (we can store the state of the job and restart from it later).</em></blockquote><h3>Machine learning</h3><p>Shall we talk about <strong>AI</strong>? It’s still the main topic of the project, for God’s sake! We naturally turned to the idea of building our own machine learning model, rather than using an already trained one.</p><p>Growing up in the fabulous world of Python, we already opened a Jupyter Notebook before asking questions.<br>The harsh reality is: although the training is simple in Python, it <strong>is hard to integrate ML models into an existing non-Python, i.e., JVM product</strong>.</p><blockquote><em>For example, to be able to use a classification model of scikit-learn in real-time by a non-Python environment:</em></blockquote><blockquote><em>You need to write a wrapper that translates an input dataset into a numpy vector where you need to handle categoricals and null values, and properly sort all input values to construct a well-formed numeric array.</em></blockquote><blockquote><em>Call sklearn predict function and publish the return value as a Web service.</em></blockquote><blockquote><em>Maintain the availability of this prediction Web service in addition to the rest of your application.</em></blockquote><p>This is why we chose <a href="http://docs.h2o.ai/h2o/latest-stable/h2o-docs/index.html"><strong>H2O</strong></a> (<a href="http://docs.h2o.ai/h2o/latest-stable/h2o-docs/quick-start-videos.html">quickstart</a>), a <strong>Java ML framework</strong> built on top of Hadoop and Spark. With h2o, you can either use their stream product for serving ML models, or generate and import<strong> </strong><a href="http://docs.h2o.ai/h2o/latest-stable/h2o-genmodel/javadoc/overview-summary.html#whatisapojo"><strong>POJO</strong></a><strong>/</strong><a href="http://docs.h2o.ai/h2o/latest-stable/h2o-genmodel/javadoc/overview-summary.html#whatisamojo"><strong>MOJO</strong></a> (so that you don’t need to have an H2O cluster running) into your existing Java/Scala application and directly use the corresponding scoring function which supports null values, categoricals, and maps input values by name.</p><h3>The apprentice</h3><p>If we want better insight of our tweet sentiment, we need to find the right person. An <strong>expert of the Twitterian psychology</strong>, the guy able to pinpointing the right pathology at first sight. To do so we will have to choose the right candidate, aka the machine learning model, and train him on our Twitterverse guinea pigs who have already been diagnosed!</p><p>First of all, our little pythonistic heart will not be too broken, H2O offers the possibility to develop in several languages, and there is of course a Python API.<br>So let’s start by installing the Python client (<a href="http://docs.h2o.ai/h2o/latest-stable/h2o-docs/downloading.html">tutorial</a>).</p><p>We decided to train our model based on the <a href="http://help.sentiment140.com/for-students/">sentiment140 dataset</a>, that you can find <a href="http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip">here</a>.<br>The dataset contains 1600K annoted tweets (half positive and half negative), this is a good start to build our first model!<br>Let’s open our notebook, import H2O, load our dataset and create some magic…</p><h3>Data preprocessing</h3><p>Before playing the sorcerer’s apprentice, we need to do some cleaning!<br>In most of the cases, text datasets (especially on Twitter) are not clean. You can find word abbreviations, some HTML encoding that hasn’t been converted to text, emoticons and so on…<br>Moreover, some words aren’t useful for our study because they are far too common and do not support any valuable information, these are the <strong>stop words</strong> and we need to erase them.<br>So, without further ado, get the <strong>tokenizer</strong> function defined in our <a href="https://github.com/BleckwenAI/teastream/blob/master/python/H2O_Sentiment_Analysis_Classifier_Training.ipynb">notebook</a> and apply it to our tweets and obtain our tokens!</p><h3>Embeddings</h3><p>Because machine learning models are a little lazy and do not understand the meaning of a sentence at first sight, we have to spoon-feed them and create a representation of words in a vector space, called <strong>word embeddings</strong>.<br>There’s plenty of solution to do so, but <a href="https://en.wikipedia.org/wiki/Word2vec">word2vec</a> is probably the famous one. How perfect, <strong>H2O support word2vec</strong> so we can either choose to train our own or use an <a href="https://s3.amazonaws.com/tomk/h2o-world/megan/w2v.hex">already trained</a> one!<br>We chose the latter…</p><pre>w2v_model = h2o.load_model(&quot;./w2v.hex&quot;)</pre><p>To be able to use it later by our Flink job, let’s export this model as a MOJO object.</p><pre>w2v_model.download_mojo()</pre><h3>Classification</h3><p>So far, we have managed to get a representation of each words in a space of dimension 100, but what about our sentences? There are more advanced ways to get a representation of our sentences in this vector space but the fastest (and most intuitive) is simply to average them out!<br>What a coincidence? There’s a built in H2O function to make this average:</p><pre>w2v_model.transform(words, aggregate_method = &quot;AVERAGE&quot;)</pre><p>Put those vectors in our initial matrix (we take the opportunity to do a little train/test split).<br>Wow, we now have for each row of our matrix (our tweets) 100 columns allowing the numerical representation of our sentences and 1 column representing the sentiment of this text… Guess what? All we have to do now is <strong>train our classifier</strong>!</p><pre>predictors = tweet_texts.names[2:]<br>response = &#39;Sentiment&#39;<br>classifier = H2OGradientBoostingEstimator(stopping_metric=&quot;AUC&quot;,<br>                                          stopping_tolerance=0.001,<br>                                          stopping_rounds=5,<br>                                          score_tree_interval=10,<br>                            model_id=&quot;gbm_sentiment_classifier.hex&quot;)</pre><pre>classifier.train(x = predictors, y = response, training_frame =                                  train, validation_frame = valid)</pre><blockquote><em>We chose to use a </em><a href="http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/gbm.html"><em>GBM</em></a><em>, we made it as simple and fast as possible. You are free to use another type of classifier and tune your hyperparameters to get better results. In addition, you will find in the </em><a href="https://github.com/BleckwenAI/teastream/blob/master/python/H2O_Sentiment_Analysis_Classifier_Training.ipynb"><em>notebook</em></a><em>, a way to evaluate the performance of your model ;)</em></blockquote><p>All we have to do now is <strong>export our model</strong>, the sentiment analyzer, into <strong>POJO</strong> (because it’s better) and move on to the integration part!</p><pre>classifier.download_pojo(&#39;.&#39;)</pre><p>You can find a small map of our office and our proposal to cure tweets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XLDzUP4c-68h253t.png" /></figure><h3>My door is always open…</h3><p>Now we have a professional trained and ready to receive real life tweets. We can open for business!<br>First things first, we are using Flink, so our dependecies need to be updated with flink-streaming-scala, flink-scala and flink-core.<br>Our tweets are shipped by Kafka, to welcome them we need to <strong>connect our Flink job to the Kafka topic</strong> that will represent the <strong>input</strong> using flink-kafka connector. This library allows us to create a consumer and directy retrieve a stream of tweets.</p><blockquote><em>Before defining a Flink job we need an </em><em>ExecutionEnvironment that we will call </em><em>env, and at the end of our job description we need to execute it </em><em>env.execute()</em></blockquote><p>Our consumer listens to twitter-topic with JSONKeyValueDeserializationSchema that deserializes our tweets into a JSON ObjectNode</p><pre>val kafkaConsumer = new FlinkKafkaConsumer010(<br>  List(&quot;twitter-topic&quot;).asJava,<br>  new JSONKeyValueDeserializationSchema(true),<br>  kafkaConsumerProperties<br>  )<br>val raw_tweets = env.addSource(kafkaConsumer)</pre><p>With this part done, our tweets are in the lobby waiting for the doctor to see them.</p><h3>From theory to practice</h3><h4>Fill out the form</h4><p>Before going into the doctor’s office, the tweet should give us more information about himself. He will fill out useful imformation in the form of a Scala case class that we will call Tweet.</p><pre>case class Tweet(time: Date, <br>                text: String,<br>                hashtags: Set[String],   <br>                score: String = &quot;undefined&quot;,<br>                ...//and more information we will not be getting into<br>                )</pre><p>This information should be retrieved from the aforementioned JSON ObjectNode, to do this let’s create a companion object for the Tweet case class. This allows us to have a constructor that takes an ObjectNode and creates a Tweet. Let’s see an example of how to retrieve information from a raw Tweet.</p><pre>val value = raw_tweet.get(&quot;value&quot;)<br>text = value.get(&quot;text&quot;).asText ()</pre><blockquote><em>To get a more in depth idea of what a tweet wanna tell us about himself </em><a href="https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/intro-to-tweet-json.html"><em>refer to the detailed documentation</em></a><em>.</em></blockquote><h4>On the couch</h4><p>Our tweets are now ready to lie on the couch and our freshly trained therapist is ready to jump in the fire. It’s time for him to put all of what he learned during his training into practice and diagnose the sentiment of each tweet patient that comes through his door.<br>He will then try to put to practice what he learned at the H2O university in its own Flink professional office, that is, tokenizing, embedding tweet texts and assigning a sentiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-4yqfjZENf0e4k3I.png" /></figure><h4>The career plan</h4><p>The transition from theory to practice is not always a walk in the park and our young therapist may have some difficulties in its early stages at his office. For him to be integrated he needs a JAVA library: <a href="http://docs.h2o.ai/h2o/latest-stable/h2o-genmodel/javadoc/index.html"><em>h2o-genmodel</em></a>.</p><p>For the <strong>tokenization</strong> step, we just rewrite a Scala equivalent to our H2O Python tokenizer.</p><p>Next, we embed our pre-trained Word2Vec MOJO and classifier POJO. As its name doesn’t suggest, the <a href="http://docs.h2o.ai/h2o/latest-stable/h2o-genmodel/javadoc/index.html"><em>EasyPredictModelWrapper</em></a> class we use to embed our models doesn’t enable to make straightforward transformations or predicitons in almost one line in a Scikit-Learn style. This is why we recreate this syntaxic sugar by representing both models through a case class with (named Word2VecModel and ClassifierModel). To a certain extent, we will construct wrapper classes of wrapper classes whose goal is to be able to predict, in our Job Flink, the sentiment of each tweet of a stream in the following way</p><pre>val scoredTweets = tweets.map(ClassifierModel(&quot;classifier_model&quot;).predict(_))</pre><p>Where <em>“classifier_model”</em> is the name of the POJO H20 trained classifier (which is a Java file) placed in the same folder as our Job Flink. The complete code associated to these case classes is available on our <a href="https://github.com/BleckwenAI/teastream">GitHub repository</a> and we will now see some big steps of their construction.</p><h4>Taking stock of his knowledge</h4><p>Once the tweets tokenized, the first thing to do for our new professional therapist is to remind all what he learned and order his knowlodges. Indeed, sometimes loading an exported model is not as simple as we can think like we experienced by trying, again and again, to load, with a relative path, our pre-trained Word2Vec MOJO model by trying to make the way specified in the MOJO Quick start of the overview tab in the h2o-genmodel library documentation work (see the Well, lesson learned section for more detail). We finally found a roundabout way to achieve our purpose. Following our code then enables to import a MOJO model (which is a zip folder), placed in the resource folder of the main folder where there is our Word2VecModel case class, from its name.</p><h4>Questioning tweets</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/0*-lPfibdYRtqzuCXL.jpg" /></figure><p>Once he has clear ideas, our fresh therapist may start to collect information by questioning his patients tweets and it is really different compared to the training. Indeed, as we can see in our code we have to construct each row data (one row for one tweet) in a specific way that can be confusing. We then have to be careful with that.</p><blockquote><em>An </em>EasyPredictModelWrapper<em> object take a new row data in the form of a </em><a href="http://docs.h2o.ai/h2o/latest-stable/h2o-genmodel/javadoc/hex/genmodel/easy/RowData.html">RowData</a><em> object which is a Java </em><a href="https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html">HashMap</a><em>. In the general case, for each pair we have to put the feature name as the key and the feature value as the value. But, we have to be careful to put numerical values as strings because of the </em><a href="https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#putAll-java.util.Map-">putAll</a><em> method expected argument. Moreover, something that can be unclear is the name to give to the features (the words retained by the tokenizer) of a Word2Vec model. Actually, we have to give each word as the key and as the value in this case. The transformation will then change the value into the embedding vector of float numbers and let the word as the key.</em></blockquote><h4>Do it all yourself</h4><p>It is well known that during the training, teachers may facilitate some intermediate steps in order to help their students to learn. Indeed, at the H2O university, we were able to straightforwardly average the words embeddings of a tokenized tweet text in order to get its own embedding. In his Flink professional office, our therapist need to do it by himself. We then define in our code Scala methods for that task and must be vigilant about the possible non-predictable tweets, that is, when no word is retained by the tokenizer and/or is in the Word2Vec vocabulary.</p><blockquote><em>Our therapist does not want to misjudge, so when the probability is too close to 0.5 he returns “undecided”, and sometimes the language spoken by the tweets is not understandable so he returns “undecided” (for tweets unrecognized by our Word2Vec).</em></blockquote><h3>Diagnosed, ready to hit the road</h3><p>Now that our tweets have been diagnosed, they can pack up and continue on their way.<br>To do that we need a producer, and we choose to serealize into a string. But for now the tweets as of the type Tweet, in order to transfom this case class into a JSON string, we’ll use a library called liftweb that does just that. In the case class Tweet we’ll override the method toString</p><pre>implicit val formats = DefaultFormats<br>override def toString: String = {<br>  write(this)<br>}</pre><p>All we have to do now is transform the tweets into strings, and off they go</p><pre>scoredTweets<br>      .map(x =&gt; x.toString())<br>      .addSink(kafkaProducer)</pre><h3>The doctor said…</h3><p>A new life begins for our tweets, our therapy has been beneficial we know a little more about them!<br>This is a far cry from the dirty tweets we received in the first article! We especially have a field that did not exist before, the “score” field that came straight out of our model!<br>We’ll be able to make nice visualizations around these sentiments in <strong>Kibana</strong>.<br>In addition, the mention used by the user to track tweets is available, so we can aggregate around this and show the feelings associated with those topics.<br>Moreover, there’s a date format finally recognized by Elasticsearch, so we can make time-based aggregations, ideal if we want to visualize in real time the evolution of the number of positive or negative feelings on a particular mention!<br>So we can now pimp our previous dashboard!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Js1mvVdfMu4hI_ir.jpg" /></figure><h3>Well, lesson learned</h3><blockquote>Did you ever hear the philosophy that once a man admits that he’s wrong, he is immediately forgiven for all wrongdoings?<em><br>– Vincent Vega, Pulp Fiction</em></blockquote><p>We had the opportunity, during our MOJO loading adventure, to learn to not desperately wanting to make work a piece of code following a documentation way that is definitely not ready to work. Indeed, the documentation that we followed specifies an apparently simple way of importing a MOJO model</p><pre>new EasyPredictModelWrapper(MojoModel.load(&quot;word2vecmodel.zip&quot;))</pre><p>But, that’s a trap. In practice, the only way we found to import MOJO model in this way was to use an absolute path as an argument for the scala MojoModel.load method. All our relative path attempts failed because this method wasn’t able to find the zip folder. This can be fine for local uses of our application but when it comes to deploy it onto the cloud, being able to import the MOJO model in a relative way from the resource folder greatly facilitates the process. Thus, sometimes, it’s better to circumvent a problem and looking for different approaches than indefinitely coming up against a brick wall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*T_m9aGRXQDidkuDb.png" /></figure><h3>Now what?</h3><p>We’ve accomplished a lot so far, don’t you think? But what more can we do? You may not have missed it, but everything we’ve done so far is local… not great for a proper application. For the rest, we’re going to make things a little tougher and work on the cloud! And as we want more and more, we will do all this with adapted tools to facilitate the deployment and automation of our application, so stay tuned…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64ae1d7b0e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bleckwen/twitter-sentiment-analysis-in-real-time-part-2-64ae1d7b0e">Twitter sentiment analysis in real time, part 2</a> was originally published in <a href="https://medium.com/bleckwen">Bleckwen</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Twitter sentiment analysis in real time, part 1]]></title>
            <link>https://medium.com/bleckwen/twitter-sentiment-analysis-in-real-time-part-1-be3fcc0bb4a2?source=rss-20fc04793e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/be3fcc0bb4a2</guid>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kibana]]></category>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Bleckwen Tech Team]]></dc:creator>
            <pubDate>Wed, 12 Jun 2019 07:01:02 GMT</pubDate>
            <atom:updated>2019-06-12T07:01:02.454Z</atom:updated>
            <content:encoded><![CDATA[<h4>Written by <a href="https://medium.com/u/46a57fcd03f2">Manar Toumi</a>, <a href="https://medium.com/u/4589f0a13a59">Niels Borie</a>, <a href="https://medium.com/u/f1ca6a2fa815">Quentin Coïc</a> &amp; <a href="https://medium.com/u/8136ed9be4d4">Andy Touly</a></h4><p>We are gathered here today to witness the union of data science, software development and devOps in the sacred bonds of a mini-project.<br>If you are a part of one of these families, you are invited to this three part ceremony. Please RSVP…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fdlTGVIhns2Be5iQ.jpg" /></figure><h3>A data scientist dive into “real code”</h3><p>Before ringing the bells let us introduce ourselves, we are three young members of the data science family. There are three levels of code that we might be required to produce in our career:</p><ul><li>Scripting, mainly in R or Python this allows visualizing, model fitting, parameter tuning…</li><li>Prototyping, used to demonstrate or to check an hypothesis. It can be, however, too slow for production and hard to integrate with existing code</li><li>Production level code, the holy grail, testable, easy to deploy, version controlled, optimized, user friendly…</li></ul><p>We are comfortable in the first level, but we aspire to learn more about prototyping and production level code. To do this we must merge knowledge with the other two families throught this project.<br>We chose to address a well known data science use case: sentiment analysis in Twitter. Our app should follow a hashtag, retrieve the corresponding tweets and extract the underlying sentiment. All in real time!<br>This is a public ceremony, our source code is available on our <a href="https://github.com/BleckwenAI/teastream">GitHub</a> repository. And as a guest we want you to be able to recreate our project, to learn from our mistakes and to engage us when you notice an error of judgement.</p><p>This article is part one of a three-part series:</p><ul><li>Part 1: The opening ceremony and unwrapping the software development gifts to ship our data from source to sink efficiently</li><li>Part 2: Unwrapping the data science gifts to create a sentiment analysis model</li><li>Part 3: Unwrapping the devOps gifts to deploy on cloud and the closing ceremony with the results</li></ul><h3>The pieces of the puzzle</h3><p>The project’s architecture revolves around the data. So let’s unwrap the software development gifts to create a data flow and find a <strong>source</strong>, a <strong>shipping</strong> tool, a <strong>sink</strong> and a <strong>visualization</strong> tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/0*vk6jpemak68a-A1X.png" /></figure><h4>The environment</h4><p>Any healthy project needs :</p><ul><li>A good foundation: the Oracle JDK version 1.8 (update 31 or later) to be installed (<a href="https://docs.oracle.com/javase/8/docs/technotes/guides/install/install_overview.html">Install</a>)</li><li>Speaking the same language: Scala, which combines object-oriented and functional programming in one concise, high-level language</li><li>Having common ground: using <a href="https://www.jetbrains.com/help/idea/meet-intellij-idea.html">IntelliJ IDEA</a> (<a href="https://www.jetbrains.com/idea/download">Download</a>, <a href="https://www.jetbrains.com/help/idea/discover-intellij-idea-for-scala.html">Setup for Scala</a>) as an IDE and <a href="https://www.scala-sbt.org/release/docs/Setup.html">SBT</a> as a build tool</li></ul><h4>Shipping</h4><p>For the task of shipping messages (tweets) let’s use Apache Kafka (<a href="https://kafka.apache.org/documentation/#gettingStarted">Getting started</a>, <a href="https://www.tutorialspoint.com/apache_kafka/index.htm">Tutorial</a>) a distributed streaming platform. In a nutshell, producers publish data to topics (a particular category of messages) split into partitions that are handled by Kafka brokers (or servers) which form a cluster. Then, consumers may subscribe to one or more topics and consume messages from them. This tool is known for his ability of reliably getting a high volume of data between systems or applications in real time. Notice that Kafka uses another tool to work, Zookeeper (<a href="https://zookeeper.apache.org/">Official website</a>, <a href="https://www.tutorialspoint.com/zookeeper/index.htm">Tutorial</a>), that manages and coordinates Kafka brokers.</p><blockquote><em>You may wonder why make life so complicated and use a technology like Kafka to only transport our tweets from one point to another… and you’re probably right! The strongest reason is when you couple Kafka with an other service, if this service crashes you don’t loose any data.</em></blockquote><h4>Store</h4><p>After fetching tweets, let’s store them. For this job, we use <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html">Elasticsearch</a>, a search and analytics engine. This tool works with <a href="https://www.elastic.co/guide/en/kibana/current/index.html">Kibana</a>, an analytics and visualization platform (<a href="https://www.elastic.co/start">Download</a>). It is used to search, view, and interact with data stored in Elasticsearch. This last tool is an indexer usually used as a distributed database (an index may be considered as a database) which is known for its scalability and ability to return results quickly in combination with Kibana that enables advanced data analysis and visualization of data in a variety of charts, tables, and maps through custom dashboards.<br>In addition, many examples of sentiment analysis, using Elasticsearch and its Kibana visualization plugin, are available on the web, so we use a tool that has proven its worth for this use case.</p><h4>Connect</h4><p>Now that we have our index to store our tweets, we just have to move the tweet stream from our Kafka topic into Elasticsearch. But how can we do that?<br>The answer is: <a href="https://kafka.apache.org/documentation/#connect">Kafka Connect</a> (<a href="https://docs.confluent.io/current/connect/index.html#kafka-connect">More documentation</a>).<br>It enables to connect scalably and reliably Kafka to other systems and move large collections of data into and out of Kafka. This is done through source connectors to move data from an external system into Kafka topics and sink connectors to move data from Kafka topics into an external system. There exists connector implementations for commonly used data systems. This solution is then much more convenient when a good connector implementation is available for the data system of interest.</p><blockquote><em>Another solution would be to use the Kafka API to create a Consumer (as it will be explained with the producer to catch tweets). Once it’s created, it’s necessary to implement the shipping interface between the kafka consumer and the index using the Elasticsearch API. Unfortunately this solution is time consuming and a bit constraining because it requires mastering both the Kafka and Elasticsearch APIs.</em></blockquote><p>To install Kafka Connect but also Kafka and Zookeeper at the same time, we recommend you to install <a href="https://docs.confluent.io/current/getting-started.html">Confluent</a> Platform which contains them as well as connectors for some of the most commonly used data systems (including an Elasticsearch sink connector). (<a href="https://www.confluent.io/download/">Download</a>)</p><h3>Listen to the birds twittering</h3><p>In order to get a real time tweet stream, let’s get a <a href="https://developer.twitter.com/en.html">Twitter developer</a> account and apply for an app.<br>Once we get our request approved, we will have the consumer API keys and the access tokens.<br><a href="https://github.com/twitter/hbc">Hosebird client</a> or hbc is a Java HTTP client for consuming Twitter’s standard streaming API created by Twitter, the hbc-core module provides a messaging queue with no parsing. To use this module add the following dependency to your build.sbt</p><pre>libraryDependencies += &quot;com.twitter&quot; % &quot;hbc-core&quot; % &quot;2.2.0&quot;</pre><p>The tweets as they come will be stored in a linkedblockingqueue, this blocks the thread until the operation can succeed. For our case this is when we “pull” from the queue</p><pre>val queue = new LinkedBlockingQueue[String](100000)</pre><p>We then create an endpoint with hbc-core that communicates with the standard POST statuses/filter in the Twitter API. This endpoint allows us to fetch public statuses that match one or more filter predicates. Our filter predicate is the (Java) list of the keyword that we are interested in</p><pre>val endPoint = new StatusesFilterEndpoint()<br>endPoint.trackTerms(keyWords.asJava)</pre><p>Declaring our developer credentials in the client</p><pre>val auth = new OAuth1(&quot;consumerKey&quot;, &quot;consumerSecret&quot;, &quot;accessToken&quot;, &quot;accessTokenSecret&quot;)</pre><p>We then build our client that will get the requested tweets and put them in the queue</p><pre>val client = new ClientBuilder()<br>    .name(&quot;filter_by_keyword&quot;)<br>    .hosts(Constants.STREAM_HOST)<br>    .endpoint(endPoint)<br>    .authentication(auth)<br>    .processor(new StringDelimitedProcessor(queue))<br>    .build()<br>  client.connect()</pre><p>By doing this we will have access to tweets by running</p><pre>queue.take</pre><h3>It’s about sending a message…</h3><p>The next step, is putting the tweets into a Kafka topic. Note that for running this program need to have both a Zookeeper and a Kafka server up and running on their respective default ports.<br>In your build.sbt add the Kafka dependency (for the version you are using)</p><pre>libraryDependencies += &quot;org.apache.kafka&quot; %% &quot;kafka&quot; % &quot;2.0.0&quot;</pre><blockquote><strong><em>Tip for sbt dependencies:</em></strong><em> A problem that we’ve encountered when working with SBT dependencies, is differentiating the dependencies that come from Java and the ones that come from Scala. In fact the Java libraries are added as </em><em>groupID % artifactID % revision and Scala libraries as </em><em>groupID %% artifactID % revision. The reason behind this is that sbt adds automatically the Scala version to the Scala libraries if there’s a double percentage.<br>Before creating the producer, its properties must be specified in a manner similar to this:</em></blockquote><pre>val props = new Properties<br>props.put(&quot;bootstrap.servers&quot;, &quot;localhost:9092&quot;)<br>props.put(&quot;client.id&quot;, &quot;Producer&quot;)<br>props.put(&quot;key.serializer&quot;, &quot;org.apache.kafka.common.serialization.StringSerializer&quot;)<br>props.put(&quot;value.serializer&quot;, &quot;org.apache.kafka.common.serialization.StringSerializer&quot;)<br>val producer = new KafkaProducer[String, String](props)</pre><p>For more information on these properties, refer to this <a href="https://kafka.apache.org/documentation/#producerconfigs">documentation</a>.<br>We specify the topic that our producer will publish to</p><pre>val topic = &quot;twitter-topic&quot;</pre><p>Finally to send the tweet</p><pre>while(true) {<br>    var record: ProducerRecord[String, String] = null<br>    try {<br>      record = new ProducerRecord[String, String](topic, queue.take)<br>    }<br>    catch {<br>      case e:InterruptedException =&gt;<br>        e.printStackTrace()<br>    }<br>    producer.send(record)<br>  }</pre><p>After running this program, let’s make sure the topic “twitter-topic” is created in Kafka, and that opening a consumer in the terminal does display tweets.</p><blockquote><em>The topic is created automatically since the configuration </em><em>auto.create.topics.enable is </em><em>true by default in the kafka broker.</em></blockquote><h3>Connect the dots</h3><p>Having tweets in a Kafka topic is good, storing them is better! Let’s use Kafka Connect to create a connector that will link our Kafka topic to our Elasticsearch index.<br>Before running the connector, let’s create a configuration file that will define the behavior of the connector: call it elasticsearch-connect.properties</p><pre>name=elasticsearch-sink<br>connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector<br>tasks.max=1<br>topics=twitter-topic<br>topic.index.map=twitter-topic:twitter-index<br>connection.url=http://localhost:9200<br>type.name=kafka-connect<br>key.ignore=true<br>schema.ignore=true</pre><p>Basically, it means that we create a connector called elasticsearch-sink and that uses io.confluent.connect.elasticsearch.ElasticsearchSinkConnector sink which is responsible for sending data to Elasticsearch.<br>Furthermore, we need to define which Kafka topic, twitter-topic, we want to ship into Elasticsearch (using topics property) while topic.index.map implies that tweets are stored in an Elasticsearch index called twitter-index.<br>In this article a local Elasticsearch instance is used, so let’s put this address in the “connection.url” field.<br>To run the connector in standalone mode let’s use the connect-standalone.sh which is provided with Kafka and can be found in the bin directory. It requires two configuration files, the one that you already created and another one, connect-standalone.properties (it defines Kafka brokers list, key and value converters, whether schemas should be used… ) that can be found in /etcdirectory.<br>Once our configuration is ready, we can run the connector, with the following commands</p><pre>&lt;path-to-confluent&gt;/bin/connect-standalone &lt;path-to-confluent&gt;/etc/kafka-connect-elasticsearch/connect-standalone.properties &lt;path-to-confluent&gt;/etc/kafka-connect-elasticsearch/elasticsearch-connect.properties</pre><h3>Let’s see the results</h3><p>All we have to do now is send data into our architecture, for this part I assume you already have tweets in a Kafka topic and an Elasticsearch and Kibana instances up.<br>Let’s run our Producer with sbt and fetch tweets mentioning the keyword “Trump”.<br>Once it’s done, we can check in Kibana (on <a href="http://localhost:5601/">http://localhost:5601</a>) if we received well our tweets.</p><blockquote><em>When you launch Elasticsearch, if you see this:<br></em><em>max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]<br>Don’t burn your computer, we had the same problem.<br>Just increase your limits on mmap counts :<br></em><em>sudo sysctl -w vm.max_map_count=262144</em></blockquote><p>Go to the management tab in Kibana, create an index pattern to match your Elasticsearch index, then you can go to the “discover” tab to view the data in your index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xpxO4SXJWY_10T25.gif" /><figcaption>The discover tab in Kibana allows us to see tweets incoming, with the auto-refresh set to 5 seconds we can get close to a real time feel.</figcaption></figure><p>At the top right of the page, you can set the update of your index. To view the data arriving in real time on your Elasticsearch database, you can set a 5 second auto-refresh.<br>Now it’s time to look at the data you have to deal with.<br>I advise you to take a look at the field description page available with the twitter API. Twitter offers us a wide range of information, from describing the metadata around a tweet to describing the user who sent it.</p><p>Keep in mind that each tweet gathered is a raw JSON containing hundreds of information about its author, its type, its popularity or even its localization. Elasticsearch can manage those kind of information, by applying a mapping uppon it. The problem is that in most of the time, it creates a default mapping that doesn’t match well our fields. And this is precisely what happening here: you will may see in Kibana, timestamp of the tweet is not recognized as a date but instead as a string. This mapping problem may be tiresome for building some advanced dashboards on Kibana, but not overwhelming / insurmontable. For this first article, let’s skip this part and use only fields that are self-sufficient and doesn’t need to get a specific mapping. Let’s stick to the text, hashtags and city retrieved by users. This is enough to build a small dashboard showing us the last tweets retrieved and the location of the people who retweeted it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P7PyoQd1p9SP3FjT.png" /></figure><blockquote><em>If you don’t visualize all the tweets in Kibana: it’s probably an Elasticsearch problem. The tweets we retrieve are JSONs of variable size, the problem is that, by default, a document in an Elasticsearch index cannot exceed 1000 fields. The result is a Kafka Connect that crashes because Elasticsearch refuses to index a JSON exceeding 1000 fields. Nothing is lost! Restart your connector, and do a little curl:<br></em><em>curl -X PUT &#39;localhost:9200/your-index/_settings&#39; -H &#39;Content-Type: application/json&#39; -d &#39;{ &quot;index.mapping.total_fields.limit&quot;: 3000 }&#39;</em></blockquote><h3>Well, lesson learned</h3><p>In this first part of our journey we learned a lot.<br>We learned that estimating the time needed to complete a task is crucial. When we started, we overestimated the amount of tasks that could be accomplished in one week. We wanted to have a complete data pipeline from the twitter API to Kibana, going through all steps (Kafka, an empty streaming job, Kafka again and Elasticsearch) in a week. After realising our misjudgement, we redefined our goal and decided to only have Kafka in the middle.<br>Another important lesson was that sometimes taking the time to read some theory behind the technologies we use is crucial. We delved into many unknown technology realms, so taking a step back to understand what is a broker, what is a topic in Kafka for example made it easier to apply these concepts and to understand where the problems were coming from.</p><h3>Now what?</h3><p>At the end of this article we have tweets shipped from source to storage, but these tweets are raw. In the next article we will see how we can process them using <a href="https://flink.apache.org/">Flink</a> and score them with our sentiment analysis machine learning model, trained using <a href="https://www.h2o.ai/">h2o</a>, so stay tuned…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be3fcc0bb4a2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bleckwen/twitter-sentiment-analysis-in-real-time-part-1-be3fcc0bb4a2">Twitter sentiment analysis in real time, part 1</a> was originally published in <a href="https://medium.com/bleckwen">Bleckwen</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[About]]></title>
            <link>https://medium.com/bleckwen/about-b5b002ff70ee?source=rss-20fc04793e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5b002ff70ee</guid>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Bleckwen Tech Team]]></dc:creator>
            <pubDate>Wed, 05 Jun 2019 10:39:50 GMT</pubDate>
            <atom:updated>2019-06-05T10:39:50.533Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5zbY6JYF02Cl1D-_loLlBw.jpeg" /></figure><p>Bleckwen is a French Fintech established in 2016 and based in Paris. In partnership with financial institutions, we deliver highly sophisticated real-time behavioural analysis via AI and help deliver an effective and robust response against financial crime. Bleckwen help teams to focus on what is important when it comes to analysing data. We do reveal the hidden truth in your data.</p><p>At Bleckwen we are 35 passionate people, eager to share about Data Science innovation and Sofware Development Engineering. We hope you will enjoy reading this blog: any comments will be welcome. Do not hesitate to contact us to discuss.</p><p>We warmly invite you to check out our <a href="https://www.bleckwen.ai/">website</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5b002ff70ee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bleckwen/about-b5b002ff70ee">About</a> was originally published in <a href="https://medium.com/bleckwen">Bleckwen</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>