<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alexandre Dall Alba on Medium]]></title>
        <description><![CDATA[Stories by Alexandre Dall Alba on Medium]]></description>
        <link>https://medium.com/@alexandredallalba?source=rss-f46e6d397edf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*V51Lm6ukzh0dPiw8.</url>
            <title>Stories by Alexandre Dall Alba on Medium</title>
            <link>https://medium.com/@alexandredallalba?source=rss-f46e6d397edf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:26:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alexandredallalba/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Some Tips for Spark Code Optimization]]></title>
            <link>https://alexandredallalba.medium.com/some-tips-for-spark-code-optimization-afcb21eac4a4?source=rss-f46e6d397edf------2</link>
            <guid isPermaLink="false">https://medium.com/p/afcb21eac4a4</guid>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Alexandre Dall Alba]]></dc:creator>
            <pubDate>Sat, 15 May 2021 23:14:36 GMT</pubDate>
            <atom:updated>2021-05-15T23:14:36.840Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aYoc5uHObJc5EgUz" /><figcaption>Photo by <a href="https://unsplash.com/@jamie452?utm_source=medium&amp;utm_medium=referral">Jamie Street</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Hi everybody</strong>, today I’m going to talk a little bit about code optimization in spark environment. I don’t intend to go into depth here, but give some tips on problems I’ve faced in projects.</p><p>As the saying goes, “life is an eternal learning process” and like so many data engineers, every day we learn something new that we bring to our clients. Thinking this way, I don’t consider myself an expert on the subject, but someone who seeks to improve every day by reading and exchanging ideas.</p><h3><strong>Why optimization is important</strong></h3><p>It is very important to understand that a Spark environment runs in a cluster, i.e. several distributed machines and it is in this distributed environment that your applications will run, so, if you are programming in Python for example, you must know Pyspark to use these features.</p><p>Furthermore,understanding this will guide you how you will develop your code. The idea is always to create faster code that consumes fewer resources. This directly impacts your client’s time and financial costs.</p><p>Since every application is different and has different behaviors, some best practices will help your application make better use of Spark’s resources.</p><p>There are a number of important issues to address, in this post I will focus on some of them:</p><h3><strong>Caching</strong></h3><p>Whenever you are going to use the same dataframe multiple times, it is ideal to cache it. This way, with the component in memory, its execution will be much faster, decreasing the processing time, consequently, optimizing the cluster resources.</p><p>There are basically two ways to put your dataframe in memory:</p><p><strong>1. Cache()</strong>. Will always store the dataframe in memory</p><p><strong>2. Persist()</strong>. Keeps the dataframe in memory or on disk, depending on the need, the object size and the memory x available storage ratio.</p><p><strong>Tip</strong>: Don’t forget to unpersist in order to release memory.</p><h3><strong>Filtering</strong></h3><p>Always try to filter your dataframe as much as possible before using operations like groupBy, join, sort, etc. Start with the most selective joins. Also, move joins that increase the number of rows after aggregations if possible.</p><h3><strong>Partitioning</strong></h3><p>By default Spark creates 200 partitions during a shuffle operation. Depending on the volume of data and the relationship/operation of some dataframes (join, groupBy, etc)this process can be very time consuming.</p><p>I will not treat the shuffle concept in this post, the important thing here is to understand that this process can consume or even exhaust the cluster execution memory. Always avoid wide dependency between parent and child tables, this improves data traffic between nodes, increasing the speed of your process.</p><p>One way to deal with this issue is to properly partition your dataframe.</p><p>Always partition datasets to use narrow dependency, i.e., make parent and child processes clear and on a 1:1 format. When the dataset is not partitioned or when you use groupbykey, sort and other such functions, you have no idea how the dataset is partitioned, parents may have too many child processes and vice versa, requiring too much network traffic between nodes and this is bad for spark in terms of performance.</p><p>There are two strategies for performing dataframe partitioning:</p><p><strong>1. Repartition:</strong> The repartition does reshuffling, so I can put in a larger number of shares if I want to. i.e., if a file has 100 lines, if you do a repartition 10, it will generate 10 partitions (files) with 10 lines in each file. if you put repartition 1000, it will generate 1000 partitions but only 100 will be populated, the rest will be empty.</p><p><strong>2. Coalesce:</strong> It does not reshuffle, there is no point in putting a number greater than the number of lines of the dataframe. i.e., if a file has 100 lines, and is set 10, each partition will have 10 files (10x10 = 100), if the coalesce is 4, it will create files (partition) in each containing 25 lines within each file, so the maximum number of coalesce can be 100, where each partition would have 1 file (partition) corresponding to 1 line.</p><h3><strong>Broadcasting</strong></h3><p>When you need to perform joins between datasets, one solution to avoid data traffic is to store the smaller tables locally. When you have one dataset which is smaller than other dataset, broadcast joining is highly recommended.</p><p><strong>The last important tip is to always use the parquet format to store and process your data</strong>.</p><p>According to the official page, “Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language…” “Parquet is built to support very efficient compression and encoding schemes…”</p><p><a href="https://parquet.apache.org/">https://parquet.apache.org/</a></p><p><strong>So, why use the parquet format ?</strong></p><p>· To reduces IO operations</p><p>· To fetch specific columns that you need to access.</p><p>· To consume less space.</p><p>· To support type-specific encoding.</p><p>It is true that some of these tips are automatically implemented by the optimizer, but as any relational database, relying on the optimizer alone does not always bring the expected result. The more control you have over your code, more control you’ll have over its behavior.</p><h3>See you!!!!</h3><h3>Don’t forget to give your 👏 !</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afcb21eac4a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taking Insights with MS Power BI and Teams]]></title>
            <link>https://alexandredallalba.medium.com/taking-insights-with-ms-power-bi-and-teams-145988799a9c?source=rss-f46e6d397edf------2</link>
            <guid isPermaLink="false">https://medium.com/p/145988799a9c</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-driven]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[power-bi]]></category>
            <dc:creator><![CDATA[Alexandre Dall Alba]]></dc:creator>
            <pubDate>Fri, 14 May 2021 18:49:21 GMT</pubDate>
            <atom:updated>2021-05-14T18:49:21.515Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i07brKNgzEu97ZNAitjR_w.jpeg" /></figure><p><strong>Hi everybody</strong>, today I want to talk a little bit about a tool that will support your business in spreading the data driven culture.</p><p>Collaboration and sharing of ideas are increasingly relevant issues in teams and in the workplace. When we talk about data, this becomes even more evident.</p><p>Information must increasingly be shared and used so that decisions are guided by data and not by feelings or hunches.</p><p>The pandemic has made these issues even more latent due to the need for remote work, taking the meeting tools to another level.</p><p>Creating a data-driven culture is a multidisciplinary challenge that involves more than just making information available. It is necessary that people live the information on a daily basis and know how to use this information in their favor and in favor of their company.</p><p>Based on this approach, Microsoft launched in late 2020 several new features of integration between Power Bi and Teams.</p><p>With these features it is possible to access your dashboards without leaving Teams, making this a more integrated and collaborative experience.<br>Inside Teams, in a business meeting for example, you can open your report, share insights with colleagues, exchange ideas through chat<br>facilitating the dissemination of the data culture in your company.</p><p><strong>Among the new capabilities, we can highlight:</strong></p><ul><li>An app within Teams to open and work on the PBI</li><li>Search within Teams for PBI content</li><li>Ability to create new dashboards directly from Teams</li><li>Use chat for discussion with the facility to share report links including filters</li><li>Integration of data sources with other tools like Excel</li></ul><p>But let’s see how this works in practice. You need to be logged in with your corporate account to use the features we will see below.</p><h3><strong>Finding PBI content</strong></h3><p>From MS Teams you can easily find your apps, dashboards and reports, just use the toolbar and search for @Power BI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cM1ZyiQa0Pd3TyEdP_rWNg.png" /></figure><h3><strong>Exchanging ideas within teams using Power BI</strong></h3><p>Through chat you can send links to reports directly to a channel, person, or meeting. The links sent ensure the correct delivery of information, including filters and slicers that have been applied to the dashboard. This allows everyone in a meeting, for example, to see the same information.</p><p>This capability has been available in the Power BI service since May 2020. Now, the links open directly in the Power BI app, so users don’t have to leave Teams to view the item.</p><p>To use chat, simply open your PBI report within Teams and click “Chat in Teams”. You can also add and share comments by clicking on “Comment”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_CbpYYdSETcwxf-sXv_M8w.png" /></figure><h3><strong>Data Sharing</strong></h3><p>Another interesting feature is dataset sharing. It helps users discover available datasets from across the organization directly from the Power BI app in Teams.</p><p>Sharing datasets allows anyone to access reliable data sources to generate their own reports, thus fostering self-service BI and allowing employees to analyze and draw insights from these databases.</p><p>From there, users can quickly generate a Power BI report or choose to analyze and explore the data in Excel.</p><p>Power BI datasets are secured and governed, with permission-based access, row-level security and so on.</p><p>1.Click on datasets</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tYwV-mo73LrnDq5x1qPuCw.png" /></figure><p>2.Choose your dataset</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lumghYfMDdG3klTAUpVGxA.png" /></figure><p>3. Start to analyse it in the PBI or Excel</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zsHbPAcAbmXSG_SgYfx2Wg.png" /></figure><h3><strong>Add Power BI reports to your team’s channel</strong></h3><p>This feature is very interesting for making the most relevant day-to-day information available within your work environment.<br>And it’s extremely simple to do that, just click “+” on the top bar of your team channel, select Power BI, and then the report to be made available.<br>After finishing, the report will appear as a tab in the channel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eDbWgzqjb25tePB-KDYHxw.png" /></figure><p>As we can see, we are evolving at a rapid pace so that teams are increasingly collaborative, enabling people to have greater autonomy and assertiveness in decision making.</p><h3>See you!!!!</h3><h3>Don’t forget to give us your 👏 !</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=145988799a9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streaming Kafka Messages to Google Cloud Pub/Sub]]></title>
            <link>https://alexandredallalba.medium.com/streaming-kafka-messages-to-google-cloud-pub-sub-f3ce7ef425b0?source=rss-f46e6d397edf------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3ce7ef425b0</guid>
            <category><![CDATA[big-data-and-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Alexandre Dall Alba]]></dc:creator>
            <pubDate>Thu, 15 Aug 2019 16:00:59 GMT</pubDate>
            <atom:updated>2019-08-15T16:00:59.476Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2PvVmhWoSigW_lom" /><figcaption>Photo by <a href="https://unsplash.com/@nasa?utm_source=medium&amp;utm_medium=referral">NASA</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Hi everybody,</strong> nowadays, to be a qualified data engineer, we need to know different platforms and solution providers in order to delivery the best solution for our clients.</p><p>It’s a little bit extense but is a step-by-step so I think you’ll enjoy.</p><p>This post will show you how to use Google Pub/Sub to publish and subscribe Kafka messages.</p><p><strong>Overview</strong></p><p>In this lab, you launch a Confluent Kafka instance and use it to communicate with Cloud Pub/Sub.</p><p>In this lab, you learn how to:</p><ul><li>Initiate Cloud Launcher to create an instance of Confluent Kafka</li><li>Configure a Kafka connector to integrate with Pub/Sub</li><li>Setup topics and subscriptions for message communication</li><li>Perform basic testing of both Kafka and Cloud Pub/Sub services</li><li>Connect IoT Core to Cloud Pub/Sub</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2i2Opcx95nFZepbOn_q2Rg.png" /></figure><p><strong>Activate Google Cloud Shell</strong></p><p>Google Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Google Cloud Shell provides command-line access to your GCP resources.</p><ol><li>In GCP console, on the top right toolbar, click the Open Cloud Shell button.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*OgzimAAJRDlrBUpPERDW_Q.png" /></figure><p>2. In the dialog box that opens, click START CLOUD SHELL:</p><p>It takes a few moments to provision and connect to the environment. When you are connected, you are already authenticated, and the project is set to your PROJECT_ID. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pqz-KOpkW0H5yzwJ4SJwpA.png" /></figure><p>gcloud is the command-line tool for Google Cloud Platform. It comes pre-installed on Cloud Shell and supports tab-completion. You can list the active account name with this command:</p><pre>gcloud auth list</pre><p>Output:</p><pre>Credentialed accounts:<br> - &lt;myaccount&gt;@&lt;mydomain&gt;.com (active)</pre><p>You can list the project ID with this command:</p><pre>gcloud config list project</pre><p>Output:</p><pre>[core]<br>project = &lt;project_ID&gt;</pre><p>For this lab you will be using multiple SSH sessions. Leave these windows open during your lab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*-y4RkIkVs_cVwz1WDowcCw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MtMii5OxmHgrBZMPTs9fHA.png" /></figure><p><strong>Introduction</strong></p><p>With the announcement of the Google Cloud Confluent managed Kafka offering, it has never been easier to use Google Cloud’s great data tools with Kafka. You can use the Apache Beam Kafka.io connector to go straight into Dataflow, but this may not always be the right solution.</p><p>Whether Kafka is provisioned in the Cloud or on premise, you might want to push to a subset of Pub/Sub topics. Why? For the flexibility of having Pub/Sub as your GCP event notifier. Then you could not only choreograph Dataflow jobs, but also use topics to trigger Cloud Functions. So how do you exchange messages between Kafka and Pub/Sub?</p><p>This is where the Pub/Sub Kafka Connector comes in handy. In this post you will learn the basic steps to start working with a Kafka connector on Google Cloud. This Kafka instance connects to Cloud Pub/Sub and exchanges event messages between the two services. In the real world, Kafka would likely be run in a cluster, but for this lab you will develop a simpler usage pattern.</p><p>The following diagram illustrates the high-level architecture for this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uU9ZaCrBnnOY06cfgo_0-g.png" /></figure><p>Setting up the environment Launch a Kafka instance via Marketplace</p><ol><li>In the GCP Console, in the left-hand Navigation menu, click Marketplace.</li><li>Locate the Apache Kafka deployment by searching for Kafka.</li><li>Click on the version with the title that simply says Kafka and the description of Google Click to Deploy — Virtual Machines and read about the service provided by the software.</li><li>Click on the blue Launch On Compute Engine button.</li><li>You can accept the default setting, or make changes based on your requirements. Check off the Terms of Service box, then click Deploy.</li></ol><p>Tip: While you can customize the deployment you should ensure makes sure that the instance is deployed to a zone in the us-central1 region. The lab requires you to use the IoT Registry in a later step using the same region and that service is not yet available in all regions.</p><p>Once the deployment completes, a new VM instance named kafka-1-vm is available in the zone you selected.</p><p>Next you need to alter the Cloud API scope on the VM created to enable the cloud Pub/Sub API.</p><ol><li>Click on the instance name on the right side of the screen, then Stop the Kafka virtual machine instance. Refresh your screen and wait until you see the instance has stopped before continuing. It will take 1–2 minutes. Refresh your screen to make sure you’re looking at the most recent information. You will see a new icon next to the name indicating it is stopped.</li><li>Click Edit then scroll down and select Allow full access to all Cloud APIs in the Access scopes section, then click Save.</li><li>Restart the Kafka virtual machine instance by clicking Start. There will be a confirmation message when it has restarted. Refresh your screen to make sure you’re looking at the most recent information.</li></ol><p><strong>Configure the Kafka VM instance</strong></p><ol><li>In the Console, go to Navigation Menu &gt; Compute Engine and open a SSH cloud shell to the Kafka VM. (This is SSH Window A.)</li><li>In the SSH window, set an environment variable to the project identifier.</li></ol><pre>export PROJECT_ID=[PROJECT_ID]</pre><ol><li>Copy the generated jar file from the storage bucket to the Kafka VM.</li></ol><pre>gsutil cp gs://cloud-training/gsp285/binary/cps-kafka-connector.jar .</pre><ol><li>Create the destination sub-directory for the Kafka connector:</li></ol><pre>sudo mkdir -p /opt/kafka/connectors</pre><ol><li>Move the downloaded jar file to the directory created for the Kafka application:</li></ol><pre>sudo mv ./cps-kafka-connector.jar /opt/kafka/connectors/</pre><ol><li>Update the java connector file permissions to be executable:</li></ol><pre>sudo chmod +x /opt/kafka/connectors/cps-kafka-connector.jar</pre><ol><li>Change the current directory to /opt/kafka/config:</li></ol><pre>cd /opt/kafka/config</pre><ol><li>Using an editor create cps-sink-connector.properties:</li></ol><pre>sudo nano cps-sink-connector.properties</pre><ol><li>Add the following content, replacing PROJECT_ID with your GCP Project ID:</li></ol><pre>name=CPSSinkConnector<br>connector.class=com.google.pubsub.kafka.sink.CloudPubSubSinkConnector<br>tasks.max=10<br>topics=to-pubsub<br>cps.topic=from-kafka<br>cps.project=PROJECT_ID</pre><ol><li>Using an editor create another file named cps-source-connector.properties:</li></ol><pre>sudo nano cps-source-connector.properties</pre><ol><li>Add the following content, replacing PROJECT_ID with your GCP Project ID:</li></ol><pre>name=CPSSourceConnector<br>connector.class=com.google.pubsub.kafka.source.CloudPubSubSourceConnector<br>tasks.max=10<br>kafka.topic=from-pubsub<br>cps.subscription=to-kafka-sub<br>cps.project=PROJECT_ID</pre><p>The Kafka application is now configured to use the connector. Leave this SSH connection to the Kafka VM instance open, so we can finish the configuration and run the application later.</p><p><strong>Pub/Sub Topic and Subscription setup</strong></p><ol><li>In the Cloud Shell window, set an environment variable to the project identifier.</li></ol><pre>export PROJECT_ID=[PROJECT_ID]</pre><ol><li>Configure the Pub/Sub topics to communicate with Kafka:</li></ol><pre>gcloud pubsub topics create to-kafka from-kafka</pre><p>2. Create a subscription for the to-kafka topic:</p><pre>gcloud pubsub subscriptions create to-kafka-sub --topic=to-kafka --topic-project=$PROJECT_ID</pre><p>Pub/Sub is now configured with two topics. A subscription has also been created on the “to-kafka” topic using the PROJECT_ID variable.</p><p>This configuration allows messages to be consumed by Pub/Sub. Go look at Pub/Sub in Console</p><p>3. Now create a subscription for traffic published from Kafka:</p><pre>gcloud pubsub subscriptions create from-kafka --topic=from-kafka</pre><p><strong>Start the Kafka VM application instance</strong></p><p>Now you will set up the Kafka VM Topic.</p><ol><li>Return to the Kafka VM instance (SSH Window A.) and submit the following command:</li></ol><pre>cd /opt/kafka</pre><p>2. Create a topic that will exchange information to Pub/Sub:</p><pre>kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic to-pubsub</pre><p>3. Create a topic that will receive messages from Pub/Sub:</p><pre>kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic from-pubsub</pre><p>4. You need to make a couple of edits to the existing file located in /opt/kafka/config/connect-standalone.properties by running:</p><pre>sudo nano /opt/kafka/config/connect-standalone.properties</pre><p>5. First, uncomment the plugin path that is located at the bottom of the file and updated it to the following:</p><pre>plugin.path=/opt/kafka/connectors</pre><p>6. You need to make a couple of edits to the existing file located in /opt/kafka/config/connect-standalone.properties by running:</p><pre>sudo nano /opt/kafka/config/connect-standalone.properties</pre><p>7. First, uncomment the plugin path that is located at the bottom of the file and updated it to the following:</p><pre>plugin.path=/opt/kafka/connectors</pre><p>8. Second, toggle the key and value converter for schema enforcement to the setting false; close the editor when you’re done:</p><pre>key.converter.schemas.enable=false<br>value.converter.schemas.enable=false</pre><p>9. Move back to the user home directory:</p><pre>cd ~</pre><p>10. Use an editor to create a new file named run-connector.sh and add these contents to it:</p><pre><em>#!/bin/bash</em></pre><pre>/opt/kafka/bin/connect-standalone.sh /opt/kafka/config/connect-standalone.properties \<br>/opt/kafka/config/cps-sink-connector.properties \<br>/opt/kafka/config/cps-source-connector.properties</pre><p>11. Update the file permissions to allow it to be executed from the command line:</p><pre>sudo chmod +x ./run-connector.sh</pre><p>12. Start the connect service:</p><pre>./run-connector.sh</pre><p>The Kafka service should now be running on the VM. Leave this session open so that any errors can be seen.</p><p><strong>Data exchange between Kafka and Pub/Sub</strong></p><p>Test Kafka to Pub/Sub (producer/consumer) communication by opening a new SSH window where the Kafka commands will be run.</p><ol><li>Open a new SSH connection to the Kafka VM, this is SSH Window B. Enter the following command to initiate a Kafka console:</li></ol><pre>kafka-console-producer.sh --broker-list localhost:9092 --topic to-pubsub</pre><p>2. From the Kafka console enter the following data elements, then press Ctrl+c to terminate the command entry:</p><pre>{&quot;message&quot;:&quot;Hello&quot;}<br>{&quot;message&quot;:&quot;Big Data&quot;}</pre><p>3. Return to Cloud Shell to see the information submitted earlier:</p><pre>gcloud pubsub subscriptions pull from-kafka --auto-ack --limit=10</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zb9an3I3iaUdXUmV" /><figcaption>Photo by <a href="https://unsplash.com/@vantroostpierrick?utm_source=medium&amp;utm_medium=referral">Pierrick VAN-TROOST</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Now we can already see the magic happening!</strong></p><p>Note: You may need to run this command a couple of times to see results. Cloud Shell Example Output (do not copy)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*u4F-CIqQlN2KIOAxzFCkmg.jpeg" /></figure><p>Kafka to Pub/Sub messaging is configured and working as expected.</p><p><strong>Test Pub/Sub to Kafka</strong></p><ol><li>In SSH Window B. enter the following command:</li></ol><pre>kafka-console-consumer.sh --bootstrap-server=localhost:9092 --value-deserializer=org.apache.kafka.common.serialization.StringDeserializer --topic from-pubsub</pre><p>2. In the active Cloud Shell window, publish a topic to be consumed by Kafka</p><pre>gcloud pubsub topics publish to-kafka --attribute=data=Hello</pre><p>Check SSH Window B. for the Kafka VM Example Output:</p><pre>{&quot;message&quot;:&quot;&quot;,&quot;data&quot;:&quot;hello&quot;}</pre><p>Pub/Sub to Kafka message is configured and working as expected. Ctrl+c to stop this process.</p><p>If you go to Navigation menu &gt; Pub/Sub, you will see the two topics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*Ttn0HbdVdzb82jO4ZaZR6w.png" /></figure><p><strong>Pub/Sub to Kafka Testing</strong></p><p>Your architecture for testing Pub/Sub to Kafka is as illustrated below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5VqdO1jROro4vDAYu-r2BQ.png" /></figure><p>Note: Ensure that a Kafka instance is actually running in the background — there should still be an open window showing the output from the instance. If the application instance is not currently running, open a new SSH connection to Kafka and run the command ./run-connector.sh at the command line.</p><ol><li>In the consumer/producer SSH session (SSH Window B.) enter the following command:</li></ol><pre>kafka-console-consumer.sh --bootstrap-server=localhost:9092 --value-deserializer=org.apache.kafka.common.serialization.StringDeserializer --topic from-pubsub</pre><p>2. From Cloud Shell you’ll create some example content. Use an editor and make a text file named movies.txt, and add the following contents to it:</p><pre>Deadpool 2<br>Avengers Infinity Wars<br>Jurassic World Fallen Kingdom<br>MI6 Fallout<br>Black Panther<br>The Incredibles<br>Three Billboards Outside of Ebbing Missouri<br>A Quiet Place<br>Thoroughbreds<br>Super Trooper 2</pre><p>3. Enter the following script to publish your movie messages to the Kafka consumer:</p><pre><strong>while</strong> read i; do gcloud pubsub topics publish to-kafka --attribute=data=&quot;$i&quot;; done &lt; movies.txt</pre><p>From the above command a stream of messages should be observable on the Kafka consumer window.</p><p>In this example we have sent a steam of information between two services. As the example demonstrates, exchanging information once configured is fairly straightforward.</p><p><strong>Kafka to Pub/Sub testing</strong></p><p>Our architecture for testing Kafka to Pub/Sub is illustrated below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YKHwXZZhsd7PrDFETmtNaA.png" /></figure><ol><li>Ctrl+c in SSH Window B, then Use an editor to create a text file named tv.json and add the following contents:</li></ol><pre>{&quot;message&quot;:&quot;Archer&quot;}<br>{&quot;message&quot;:&quot;Ozark&quot;}<br>{&quot;message&quot;:&quot;Star Trek Discovery&quot;}<br>{&quot;message&quot;:&quot;Westworld&quot;}<br>{&quot;message&quot;:&quot;The Magicians&quot;}<br>{&quot;message&quot;:&quot;Legion&quot;}<br>{&quot;message&quot;:&quot;Cloak and Dagger&quot;}<br>{&quot;message&quot;:&quot;The Good Place&quot;}<br>{&quot;message&quot;:&quot;Silicon Valley&quot;}<br>{&quot;message&quot;:&quot;Mr Robot&quot;}<br>{&quot;message&quot;:&quot;Rick and Morty&quot;}<br>{&quot;message&quot;:&quot;Mindhunter&quot;}</pre><p>2. Now use the following script to publish your TV messages to the Pub/Sub consumer:</p><pre>kafka-console-producer.sh --broker-list localhost:9092 --topic to-pubsub &lt; tv.json</pre><p>3. From the Cloud Shell window, run the following command to view the messages that have been published from Kafka:</p><pre>gcloud pubsub subscriptions pull from-kafka --auto-ack --limit=10</pre><p>In this example you have sent a stream of information between two services. When passing information via Kafka, the message content is formatted as JSON.</p><p>Full documentation of gcloud is available on <a href="https://cloud.google.com/sdk/gcloud">Google Cloud gcloud Overview</a>.</p><p>In the next post we’ll learn how to simulate an IOT streaming using the Google IoT core service.</p><p>Enjoy the code, improve it if you want!</p><h3>See you!!!!</h3><h3>Don’t forget to give us your 👏 !</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3ce7ef425b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Gen, a new PPL language by MIT]]></title>
            <link>https://becominghuman.ai/introduting-gen-a-new-ppl-language-by-mit-f77397eeff3?source=rss-f46e6d397edf------2</link>
            <guid isPermaLink="false">https://medium.com/p/f77397eeff3</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[internet-of-things]]></category>
            <dc:creator><![CDATA[Alexandre Dall Alba]]></dc:creator>
            <pubDate>Sat, 20 Jul 2019 03:47:28 GMT</pubDate>
            <atom:updated>2019-08-05T14:23:17.274Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DVaK1xUPagpwSTKR" /><figcaption>Photo by <a href="https://unsplash.com/@joshriemer?utm_source=medium&amp;utm_medium=referral">Josh Riemer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Hi everybody, today I’ll introduce you a new PPL, “Gen”.</p><p>In an article presented at the conference Programming Language Design and Implementation at the end of June/19 , the researchers of MIT described a new probabilistic programming system called “Gen.”</p><p>Probabilistic modeling and inference are core tools in diverse fields including statistics, machine learning, computer vision, cognitive science, robotics, natural language processing, and <a href="https://becominghuman.ai/">artificial intelligence</a>. To meet the functional requirements of applications, practitioners use a broad range of modeling techniques and approximate inference algorithms. However, implementing inference algorithms is often difficult and error prone. Gen simplifies the use of probabilistic modeling and inference, by providing modeling languages in which users express models, and high-level programming constructs that automate aspects of inference.</p><h3>Trending AI Articles:</h3><blockquote><a href="https://becominghuman.ai/deep-learning-book-notes-chapter-1-b310837c76cf">1. Deep Learning Book Notes, Chapter 1</a></blockquote><blockquote><a href="https://becominghuman.ai/deep-learning-book-notes-chapter-2-linear-algebra-for-deep-learning-af776cf52506">2. Deep Learning Book Notes, Chapter 2</a></blockquote><blockquote><a href="https://becominghuman.ai/machines-demonstrate-self-awareness-8bd08ceb1694">3. Machines Demonstrate Self-Awareness</a></blockquote><blockquote><a href="https://becominghuman.ai/visual-music-machine-learning-workshop-for-kids-a90c957dab33">4. Visual Music &amp; Machine Learning Workshop for Kids</a></blockquote><p>Like some probabilistic programming research languages, Gen includes universal modeling languages that can represent any model, including models with stochastic structure, discrete and continuous random variables, and simulators. However, Gen is distinguished by the flexibility that it affords to users for customizing their inference algorithm. It is possible to use built-in algorithms that require only a couple lines of code, as well as develop custom algorithms that are more able to meet scalability and efficiency requirements.</p><p>Gen’s flexible modeling and inference programming capabilities unify symbolic, neural, probabilistic, and simulation-based approaches to modeling and inference, including causal modeling, symbolic programming, deep learning, hierarchical Bayesian modeling, graphics and physics engines, and planning and reinforcement learning.</p><p>Gen is a package for the Julia programming language. Gen consists of multiple modeling languages that are implemented as DSLs in Julia and a Julia library for inference programming.</p><p>“Gen is the first system to be flexible, automated and efficient enough to cover these kinds of very different examples in computer vision and data science and offers state-of-the-art performance,” says Vikash K. Mansinghka ’05, MEng ’09. , PhD ’09, a researcher in the Department of Brain and Cognitive Sciences who runs the Probabilistic Computing Project.</p><p>More information can be found in:</p><p><a href="https://probcomp.github.io/Gen/">Introduction</a></p><p>First, download Julia 1.0 or later.</p><p>Then, install the Gen package with the Julia package manager. From the Julia REPL, type ] to enter the Pkg REPL mode and then run:</p><p>In [ ]:</p><pre>pkg<strong>&gt;</strong> add <a href="https://github.com/probcomp/Gen">https://github<strong>.</strong>com<strong>/</strong>probcomp<strong>/</strong>Gen</a></pre><p>Let’s write a short Gen program that does Bayesian linear regression: given a set of points in the (x, y) plane, we want to find a line that fits them well.</p><p>There are three main components to a typical Gen program.</p><p>First, we define a generative model: a Julia function, extended with some extra syntax, that, conceptually, simulates a fake dataset. The model below samples slope and intercept parameters, and then for each of the x-coordinates that it accepts as input, samples a corresponding y-coordinate. We name the random choices we make with @trace, so we can refer to them in our inference program.</p><p>In [ ]:</p><pre><strong>using</strong> Gen</pre><pre>@gen <strong>function</strong> my_model(xs::Vector{Float64})</pre><pre>slope <strong>=</strong> @trace(normal(0, 2), :slope)</pre><pre>intercept <strong>=</strong> @trace(normal(0, 10), :intercept)</pre><pre><strong>for</strong> (i, x) <strong>in</strong> enumerate(xs)</pre><pre>@trace(normal(slope <strong>*</strong> x <strong>+</strong> intercept, 1), &quot;y-$i&quot;)</pre><pre><strong>end</strong></pre><pre><strong>end</strong></pre><p>Second, we write an inference program that implements an algorithm for manipulating the execution traces of the model. Inference programs are regular Julia code, and make use of Gen’s standard inference library.</p><p>The inference program below takes in a data set, and runs an iterative MCMC algorithm to fit slope and intercept parameters:</p><p>In [ ]:</p><pre><strong>function</strong> my_inference_program(xs::Vector{Float64}, ys::Vector{Float64}, num_iters::Int)</pre><pre><em># Create a set of constraints fixing the</em></pre><pre><em># y coordinates to the observed y values</em></pre><pre>constraints <strong>=</strong> choicemap()</pre><pre><strong>for</strong> (i, y) <strong>in</strong> enumerate(ys)</pre><pre>constraints[&quot;y-$i&quot;] <strong>=</strong> y</pre><pre><strong>end</strong></pre><pre><em># Run the model, constrained by `constraints`,</em></pre><pre><em># to get an initial execution trace</em></pre><pre>(trace, _) <strong>=</strong> generate(my_model, (xs,), constraints)</pre><pre><em># Iteratively update the slope then the intercept,</em></pre><pre><em># using Gen&#39;s metropolis_hastings operator.</em></pre><pre><strong>for</strong> iter<strong>=</strong>1<strong>:</strong>num_iters</pre><pre>(trace, _) <strong>=</strong> metropolis_hastings(trace, select(:slope))</pre><pre>(trace, _) <strong>=</strong> metropolis_hastings(trace, select(:intercept))</pre><pre><strong>end</strong></pre><pre><em># From the final trace, read out the slope and</em></pre><pre><em># the intercept.</em></pre><pre>choices <strong>=</strong> get_choices(trace)</pre><pre><strong>return</strong> (choices[:slope], choices[:intercept])</pre><pre><strong>end</strong></pre><p>Finally, we run the inference program on some data, and get the results:</p><p>Visualization Framework</p><p>Because inference programs are regular Julia code, users can use whatever visualization or plotting libraries from the Julia ecosystem that they want. However, we have paired Gen with the GenViz package, which is specialized for visualizing the output and operation of inference algorithms written in Gen.</p><p>An example demonstrating the use of GenViz for this example linear regression problem is available in the gen-quickstart repository. The code there is mostly the same as above, with a few small changes to incorporate an animated visualization of the inference process:</p><p>It starts a visualization server and initializes a visualization before performing inference:</p><p>In [ ]:</p><pre><em># Start a visualization server on port 8000</em></pre><pre>server <strong>=</strong> VizServer(8000)</pre><pre><em># Initialize a visualization with some parameters</em></pre><pre>viz <strong>=</strong> Viz(server, joinpath(@__DIR__, &quot;vue/dist&quot;), Dict(&quot;xs&quot; <strong>=&gt;</strong> xs, &quot;ys&quot; <strong>=&gt;</strong> ys, &quot;num&quot; <strong>=&gt;</strong> length(xs), &quot;xlim&quot; <strong>=&gt;</strong> [minimum(xs), maximum(xs)], &quot;ylim&quot; <strong>=&gt;</strong> [minimum(ys), maximum(ys)]))</pre><pre><em># Open the visualization in a browser</em></pre><pre>openInBrowser(viz)</pre><p>The “vue/dist” is a path to a custom trace renderer that draws the (x, y) points and the line represented by a trace; see the GenViz documentation for more details. The code for the renderer is here.</p><p>It passes the visualization object into the inference program.</p><p>In [ ]:</p><pre>(slope, intercept) <strong>=</strong> my_inference_program(xs, ys, 1000000, viz)</pre><p>In the inference program, it puts the current trace into the visualization at each iteration:</p><p>In [ ]:</p><pre><strong>for</strong> iter<strong>=</strong>1<strong>:</strong>num_iters</pre><pre>putTrace!(viz, 1, trace_to_dict(trace))</pre><pre>(trace, _) <strong>=</strong> metropolis_hastings(trace, select(:slope))</pre><pre>(trace, _) <strong>=</strong> metropolis_hastings(trace, select(:intercept))</pre><pre><strong>end</strong></pre><p>I hope you enjoyed the topic and I see you next time!</p><p>Enjoy the code, improve it if you want to!</p><h3>See you!!!!</h3><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*ujWl0s2U2-HUBFTq" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f77397eeff3" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/introduting-gen-a-new-ppl-language-by-mit-f77397eeff3">Introducing Gen, a new PPL language by MIT</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Persisting Sentiment Analysis Tweets]]></title>
            <link>https://becominghuman.ai/persisting-sentiment-analysis-tweets-ce31acfeae8f?source=rss-f46e6d397edf------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce31acfeae8f</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Alexandre Dall Alba]]></dc:creator>
            <pubDate>Thu, 04 Jul 2019 16:29:36 GMT</pubDate>
            <atom:updated>2019-07-24T13:57:01.946Z</atom:updated>
            <content:encoded><![CDATA[<h3>Persisting Sentiment Analysis of Tweets</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HUpyi0uo2ooP7vnE" /><figcaption>Photo by <a href="https://unsplash.com/@blancotejedor?utm_source=medium&amp;utm_medium=referral">Aarón Blanco Tejedor</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Hi everybody,</strong> this little snippet will show you how to use the Twitter API and do the sentiment analysis for each tweet found. We’ll use different Python libs to make this exercise.</p><p>The first thing you need to do is import the necessary libraries to use. In this case, we’ll need the TWEEPY in order to make a twitter connection, TEXTBLOB to run sentiment analysis, TIME and DATETIME to control a pause (I explain this point better bellow) and record the time that the register was inserted in a database and MYSQL.CONNECTOR in order to connect and use the MySQL.</p><h3>Trending AI Articles:</h3><blockquote><a href="https://becominghuman.ai/basics-of-neural-network-bef2ba97d2cf">1. Basics of Neural Network</a></blockquote><blockquote><a href="https://becominghuman.ai/bursting-the-jargon-bubble-deep-learning-22da43caa525">2. Bursting the Jargon bubbles — Deep Learning</a></blockquote><blockquote><a href="https://becominghuman.ai/how-can-we-improve-the-quality-of-our-data-61b55c9063d1">3. How Can We Improve the Quality of Our Data?</a></blockquote><blockquote><a href="https://becominghuman.ai/machine-learning-using-logistic-regression-in-python-with-code-ab3c7f5f3bed">4. Machine Learning using Logistic Regression in Python with Code</a></blockquote><p>In [ ]:</p><pre><strong>import</strong> tweepy<br><strong>from</strong> tweepy <strong>import</strong> OAuthHandler<br><strong>from</strong> textblob <strong>import</strong> TextBlob<br><strong>import</strong> datetime<br><strong>import</strong> mysql.connector</pre><p>After import the libs, we’ll develop a function that makes a connection, passing the parameters to mysql connect method.</p><p>In [ ]:</p><pre>connection <strong>=</strong> mysql.connector.connect(host<strong>=</strong>&#39;localhost&#39;,database<strong>=</strong>&#39;database_name&#39;,user<strong>=</strong>&#39;user&#39;,password<strong>=</strong>&#39;password&#39;)</pre><p>After that, we need to create a cursor passing the necessary insert statement.</p><p>In [ ]:</p><pre>cursor <strong>=</strong> connection.cursor(prepared<strong>=True</strong>)</pre><pre>sql_insert_query <strong>=</strong> &#39;&#39;&#39;INSERT INTO `tweets`.`tweets_table`(`Date`,`Author`, `Text`, `Translate`, `Sentiment`, `Sensation`)VALUES(%s,%s,%s,%s,%s,%s)&#39;&#39;&#39;</pre><p>Now the cursor is instantiated and the record found is committed in the database.</p><p>In [ ]:</p><pre>insert_tuple <strong>=</strong> (p_date,p_author, p_text, p_translate, p_sentiment,p_sensation)</pre><pre>result <strong>=</strong> cursor.execute(sql_insert_query, insert_tuple)</pre><pre>connection.commit()</pre><p>To each entry committed, a message is displayed.</p><p>In [ ]:</p><pre>print (&quot;Record inserted successfully into tweets_table table&quot;)</pre><p>I case of an error, a try.. except handles, makes a rollback and shows the error message.</p><p>In [ ]:</p><pre><strong>except</strong> mysql.connector.Error <strong>as</strong> error :</pre><pre>connection.rollback()</pre><pre>print(&quot;Failed to insert into MySQL table {}&quot;.format(error))</pre><p>At the end, a final block closes the cursor and database connection.</p><p>In [ ]:</p><pre><strong>finally</strong>:<br>      <em>#closing database connection.<br>      </em><strong>if</strong>(connection.is_connected()):<br>      cursor.close()<br>      connection.close()<br>      print(&quot;MySQL connection is closed&quot;)</pre><p>Below, the first part of the code completed:</p><p>In [ ]:</p><pre>import tweepy<br>from tweepy import OAuthHandler<br>import timeimport pandas as pd<br>from textblob import TextBlob<br>import datetime<br>import mysql.connector</pre><pre>def insertInTable(p_date,p_author, p_text, p_translate, p_sentiment,p_sensation):<br>    try:<br>        connection = mysql.connector.connect(host=&#39;localhost&#39;,<br>                             database=&#39;databaase_name&#39;,<br>                             user=&#39;user&#39;,<br>                             password=&#39;password&#39;)<br>        cursor = connection.cursor(prepared=True)<br>        sql_insert_query = &#39;&#39;&#39;INSERT INTO `tweets`.`tweets_table`(`Date`,`Author`, `Text`, `Translate`, `Sentiment`, `Sensation`)VALUES(%s,%s,%s,%s,%s,%s)&#39;&#39;&#39;</pre><pre>insert_tuple = (p_data,p_autor, p_texto, p_traducao, p_sentimento,p_sensacao)<br>        result = cursor.execute(sql_insert_query, insert_tuple)<br>        connection.commit()<br>        print (&quot;Record inserted successfully into tweets_table table&quot;)<br>    except mysql.connector.Error as error :<br>        connection.rollback()<br>        print(&quot;Failed to insert into MySQL table {}&quot;.format(error))<br>    finally:<br>        #closing database connection.<br>        if(connection.is_connected()):<br>            cursor.close()<br>            connection.close()<br>            print(&quot;MySQL connection is closed&quot;)</pre><p>Once the function is finished it is time to start the connections with Twitter.</p><p>I will not explain in this topic how to get the required key and token. This information can be obtained from the address below:</p><blockquote><a href="https://developer.twitter.com/">https://developer.twitter.com/</a></blockquote><p>Here you will find all the information to create an account and get the key and token, as well as find all the API documentation for several languages like Python.</p><p>Next step is to pass the necessary connection information, all obtained from the site mentioned above. The information below is fake, it’s only being used as an example.</p><p>In [ ]:</p><pre>consumer_key = &#39;Baks8re9889rwe8r9we80r9we809r8q&#39;<br>consumer_secret = &#39;219uewqiueqwiur09wq898092hewqkj&#39;<br>access_token = &#39;ifudsaoifsdafpsdaipofisdapipc098098q423&#39;<br>access_secret = &#39;rwe87r0we88re09tre9twe9rd8937423ji&#39;</pre><p>Next step is to pass the information and make a connection.</p><p>In [ ]:</p><pre>auth = OAuthHandler(consumer_key, consumer_secret)<br>auth.set_access_token(access_token, access_secret)</pre><pre>api = tweepy.API(auth)</pre><p>In [ ]:</p><pre>print (&quot;Downloading and Persisting tweets...&quot;)</pre><p>Using a cursor we will pass the necessary parameters to the search API, such as the word to be searched, the number of records per page, whether to bring retweets or not and other fields that will be stored in the table as date, author, tweeted text and so on.</p><p>In [ ]:</p><pre>for tweet in tweepy.Cursor(api.search,<br>                           q=&quot;word&quot;,<br>                           rpp=100,<br>                           tweet_mode=&#39;extended&#39;,<br>                           result_type=&quot;recent&quot;,<br>                           include_entities=True,<br>                           lang=&quot;pt-br&quot;).items(1000):<br>    if not tweet.retweeted and (&#39;RT @&#39; not in tweet.full_text):<br>        data = tweet.created_at<br>        autor = tweet.author.screen_name<br>        texto = tweet.full_text</pre><p>For each processed tweet we will use the textblob to return the sentiment analysis.</p><p>If we are working with other languages (in my case I was finding Brazilian Portuguese tweets), we can choose to translate the tweet into English and perform the feeling analysis considering that the English language is simpler and the algorithms more mature. Obviously, for very regional expressions we cannot get the correct analysis, but like the whole statistical algorithm, we will not hit 100% of the cases, but for most of the common expressions, it works very well.</p><p>You can use textblob in order to make classifications (Naive Bayes and decision trees), tokenization, n-grams, word inflection, etc.</p><p>As the website itself says: TextBlob is a Python (2 and 3) library for processing textual data. It provides a simple API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, translation, and more.</p><p>More information about TextBlob can be found here:</p><blockquote><a href="https://textblob.readthedocs.io/en/dev/">https://textblob.readthedocs.io/en/dev/</a></blockquote><p>So each tweet is passed to the textblob that does the translation.</p><p>In [ ]:</p><pre>frase <strong>=</strong> TextBlob(texto)<br>translate <strong>=</strong> TextBlob(str(frase.translate(to<strong>=</strong>&#39;en&#39;)))</pre><p>After we’ll analyze the polarity and return the sentiment.</p><p>In [ ]:</p><pre>sentiment = translate.sentiment<br>        if sentiment.polarity &lt; 0.0:<br>            sentiment2 = &#39;Negative&#39;<br>        elif sentiment.polarity == 0.0:<br>            sentiment2 = &#39;Neutral&#39;<br>        else:<br>            sentiment2 = &#39;Positive&#39;</pre><p>This time sleep is needed to prevent Twitter from rating the program as a robot and blocking access to the API. This procedure is highly recommended. I learned after being blocked several times kkkkk.</p><p>In [ ]:</p><pre>time.sleep(10)</pre><p>Finally, you just get the time of the registry entry and write everything into the MySQL database.</p><p>In [ ]:</p><pre>now = datetime.datetime.now().strftime(&#39;%Y-%m-%d %H:%M:%S&#39;)<br>        insertInTable(now, author, text, str(translate), str(sentiment), sentiment2)</pre><p>Below, the completed code.</p><p>In [ ]:</p><pre>import tweepy<br>from tweepy import OAuthHandler<br>import timeimport pandas as pd<br>from textblob import TextBlob<br>import datetime<br>import mysql.connector<br>def insertInTable(p_date,p_author, p_text, p_translate, p_sentiment,p_sensation):<br>    try:<br>        connection = mysql.connector.connect(host=&#39;localhost&#39;,<br>                             database=&#39;database_name&#39;,<br>                             user=&#39;user&#39;,<br>                             password=&#39;password&#39;)<br>        cursor = connection.cursor(prepared=True)<br>        sql_insert_query = &#39;&#39;&#39;INSERT INTO `tweets`.`tweets_table`(`Date`,`Author`, `Text`, `Translate`, `Sentiment`, `Sensation`)VALUES(%s,%s,%s,%s,%s,%s)&#39;&#39;&#39;</pre><pre>insert_tuple = (p_date,p_author, p_text, p_translate, p_sentiment,p_sensation)<br>        result = cursor.execute(sql_insert_query, insert_tuple)<br>        connection.commit()<br>        print (&quot;Record inserted successfully into tweets_table table&quot;)<br>    except mysql.connector.Error as error :<br>        connection.rollback()<br>        print(&quot;Failed to insert into MySQL table {}&quot;.format(error))<br>    finally:<br>        #closing database connection.<br>        if(connection.is_connected()):<br>            cursor.close()<br>            connection.close()<br>            print(&quot;MySQL connection is closed&quot;)</pre><pre>for tweet in tweepy.Cursor(api.search,<br>                           q=&quot;word&quot;,<br>                           rpp=100,<br>                           tweet_mode=&#39;extended&#39;,<br>                           result_type=&quot;recent&quot;,<br>                           include_entities=True,<br>                           lang=&quot;pt-br&quot;).items(1000):<br>    if not tweet.retweeted and (&#39;RT @&#39; not in tweet.full_text):<br>        data = tweet.created_at<br>        autor = tweet.author.screen_name<br>        text = tweet.full_text<br>        phrase = TextBlob(texto)<br>        translate = TextBlob(str(phrase.translate(to=&#39;en&#39;)))<br>        sentiment = traducao.sentiment<br>        if sentiment.polarity &lt; 0.0:<br>            sentiment2 = &#39;Negative&#39;<br>        elif sentiment.polarity == 0.0:<br>            sentiment2 = &#39;Neutral&#39;<br>        else:<br>            sentiment2 = &#39;Positive&#39;<br>        time.sleep(10)<br>        print(author + &quot;-&quot; + text + &quot;-&quot; + str(sentiment) + &quot;-&quot; + str(sentiment2))<br>        now = datetime.datetime.now().strftime(&#39;%Y-%m-%d %H:%M:%S&#39;)<br>        insertInTable(now, author, text, str(translate), str(sentiment), sentiment2)</pre><p>With this few lines of code, we can start storing a large amount of text for later use in other <a href="https://becominghuman.ai/">machine learning</a> analysis.</p><p>You can use this code to make a spider and store web page content as well. You can find more information on this topic:</p><blockquote><a href="https://medium.com/datadriveninvestor/hi-everybody-this-little-snippet-will-show-you-how-to-use-a-selenium-lib-in-order-to-make-an-100c041c1bf8">A Little Snippet to Automate Web Scraping using Python and Selenium</a></blockquote><p>I hope you enjoyed the topic and see you next time!</p><p>Enjoy the code, improve it if you want to!</p><h3>See you!!!!</h3><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*bNW2ttkxMALGqum4" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce31acfeae8f" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/persisting-sentiment-analysis-tweets-ce31acfeae8f">Persisting Sentiment Analysis Tweets</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Little Snippet to Automate Web Scraping using Python and Selenium]]></title>
            <link>https://medium.datadriveninvestor.com/hi-everybody-this-little-snippet-will-show-you-how-to-use-a-selenium-lib-in-order-to-make-an-100c041c1bf8?source=rss-f46e6d397edf------2</link>
            <guid isPermaLink="false">https://medium.com/p/100c041c1bf8</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Alexandre Dall Alba]]></dc:creator>
            <pubDate>Mon, 24 Sep 2018 00:21:16 GMT</pubDate>
            <atom:updated>2019-07-05T02:04:00.248Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lCxJOS9rHi1ynVIZ" /><figcaption>“grayscale photo of dew on spider web” by <a href="https://unsplash.com/@klaus_blaze?utm_source=medium&amp;utm_medium=referral">Rúben Marques</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Hi everybody </strong>, this little snippet will show you how to use a selenium lib in order to make an automated web scraping you can use to analyse data, find patterns,etc.</p><p>This snippet is the first of many other, each one will show you the next step, this one shows the automated connection in a web page, in this case, the facebook, the next will show you how to scraping a web page using beautiful soap, after we’ll to download the data and keep it in a database and so on.</p><p>According with the documentation, the selenium package is used to automate web browser interaction from Python and used to make automated tests.</p><p>You can find more information in <a href="https://pypi.org/project/selenium/">https://pypi.org/project/selenium/</a></p><p>Several browsers/drivers are supported (Firefox, Chrome, Internet Explorer), as well as the Remote protocol.</p><p><strong>Supporte Python versions: Python 2.7, 3.4+</strong></p><p>For the installation you can use on of this 3 options:</p><p>using pip:</p><p><strong>pip install -U selenium</strong></p><p>You can download the source distribution from PyPI (e.g. selenium-3.14.0.tar.gz), unarchive it, and run:</p><p><strong>python setup.py install</strong></p><p>Finally if you’re using Anaconda:</p><p><strong>conda install -c conda-forge selenium</strong></p><p>The first thing we need to do is to import the libraries we’ll use in this snippet.</p><p>In this case, for this first step, the most important one is the selenium where we’ll make the automated connection.</p><p>In [38]:</p><pre><strong>from</strong> <strong>urllib.request</strong> <strong>import</strong> urlopen <strong>as</strong> uReq<br><strong>from</strong> <strong>bs4</strong> <strong>import</strong> BeautifulSoup <strong>as</strong> soup<br><strong>from</strong> <strong>datetime</strong> <strong>import</strong> datetime, timedelta<br><strong>from</strong> <strong>selenium</strong> <strong>import</strong> webdriver<br><strong>import</strong> <strong>time</strong></pre><p>After importing the libs, in order to run the code, we need to choose the correct driver to use in.</p><p><strong>Selenium requires a driver to interface with the chosen browser.</strong></p><p>Here, we’ll use the Chromium, but many other can be used.</p><p>you can find the driver here:</p><pre><a href="https://sites.google.com/a/chromium.org/chromedriver/downloads">https://sites.google.com/a/chromium.org/chromedriver/downloads</a></pre><p>you can find more information in the Selenium project’s page.</p><p>With Chrome driver installed, we need to set some option in order to run it.</p><p>In [39]:</p><pre>options = webdriver.ChromeOptions()<br>options.add_argument(&#39;--ignore-certificate-errors&#39;)<br>options.add_argument(&quot;--test-type&quot;)<br>options.binary_location = &quot;Your drive:\Your directory\chromedriver.exe&quot;<br>driver = webdriver.Chrome(&quot;Your drive:\Your directory\chromedriver.exe&quot;)</pre><p>Next step is to set the url we’ll use and get it with the driver.</p><p>In [40]:</p><pre>my_url = &#39;https://www.facebook.com/&#39;</pre><p>In [41]:</p><pre>driver.get(my_url)</pre><p>In our case we have a form to fill in order to access the web page, so we need to get the html ids from the respective fields. It’s easy to find them using the driver methods like find-element_by_id or find_elements_by_xpath.</p><p>In [42]:</p><pre>login = driver.find_element_by_id(&#39;email&#39;);      <br>senha = driver.find_element_by_id(&#39;pass&#39;);</pre><p>In [43]:</p><pre>login.send_keys(&#39;your user&#39;)<br>senha.send_keys(&#39;your password&#39;)</pre><p>In [46]:</p><pre>submit_button = driver.find_elements_by_xpath(&#39;//*[@id=&quot;loginbutton&quot;]&#39;)[0];</pre><p>In [ ]:</p><pre>Now <strong>is</strong> just submit the information using the click method <strong>and</strong> &lt;i&gt;voilà&lt;/i&gt;, we&#39;re in.</pre><p>In [47]:</p><pre>submit_button.click()</pre><p>In the next topics we’ll learn how to get the data using beautiful soap, <a href="https://medium.com/@alexandredallalba/persisting-sentiment-analysis-tweets-ce31acfeae8f">store it in a database</a> and analyse it using some tools like pandas, matplotlib, sklearn, etc.</p><p>Enjoy the code, improve it if you want!</p><h3>See you!!!!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=100c041c1bf8" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/hi-everybody-this-little-snippet-will-show-you-how-to-use-a-selenium-lib-in-order-to-make-an-100c041c1bf8">A Little Snippet to Automate Web Scraping using Python and Selenium</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Análise Aumentada]]></title>
            <link>https://alexandredallalba.medium.com/an%C3%A1lises-aumentadas-7b42b073e6ef?source=rss-f46e6d397edf------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b42b073e6ef</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Alexandre Dall Alba]]></dc:creator>
            <pubDate>Mon, 10 Sep 2018 22:46:18 GMT</pubDate>
            <atom:updated>2018-09-11T00:13:45.032Z</atom:updated>
            <content:encoded><![CDATA[<p>O termo Smart Data Discoverer -algo como Descoberta Inteligente de Dados-introduzido em meados de 2017, propõe a criação de insights através da criação de modelos preditivos e na integração de dados.</p><p>Você pode se perguntar: Mas o que há de novo? Já estamos falando de big data, analytics, machine learning a algum tempo.</p><p>A diferença, agora, é o conceito de Análise Aumentada -Augmented Analytics-onde a fase de preparação, descoberta de padrões e criação de modelos, até então realizadas por cientistas de dados e analistas de BI, passa a ser automatizada, liberando mais tempo dos profissionais na análise das informações e menos na preparação das mesmas.</p><p>A grande sacada das próximas ferramentas de análise de dados será fornecer muito mais do que relatórios e painéis, mas entregar análise preditiva e prescritiva de maneira automática, baseadas nas informações existentes, utilizando aprendizado de máquina.</p><p>Através do conceito da análise aumentada, as ferramentas deverão preparar os dados e realizar descoberta de padrões de forma automática através da criação de seus próprios modelos.</p><p>Gestores de áreas como RH, logística, seguros, governo, dentre outros, poderão ter acesso a predições e sugestões em tempo real permitindo muito mais agilidade nas tomadas de decisões.</p><p>E como deverá acontecer a “virada” para a análise aumentada a partir de agora?</p><p>1. Através do cruzamento de bases de dados internas e externas cada vez mais disponíveis com o crescente aumento no número de fontes abertas e compartilhadas,</p><p>2. Com a utilização de bases de BI consolidadas x dados desestruturados entregando mais valor às análises permitindo a geração de novos insights,</p><p>3. Com o aumento da oferta de plataformas integradas baseadas em nuvem e opções de escolha de bancos de dados, linguagens de programação e ferramentas de análise de acordo com a necessidade e sendo consumidas via SaaS,</p><p>4. Via ferramentas de usuário cada vez mais amigáveis e com mais recursos de automatização de tarefas e visualização nas análises,</p><p>5. Utilizando cada vez mais recursos como linguagem natural e dispositivos móveis que permitem uma nova era na interação homem-máquina</p><p>Sejamos todos bem-vindos à Era da Automatização do Conhecimento.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b42b073e6ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>