<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Karyne Scheneyder Oliveira on Medium]]></title>
        <description><![CDATA[Stories by Karyne Scheneyder Oliveira on Medium]]></description>
        <link>https://medium.com/@kschneyder?source=rss-e5b34fd3616------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*q5zzHKld7A0tWrf4</url>
            <title>Stories by Karyne Scheneyder Oliveira on Medium</title>
            <link>https://medium.com/@kschneyder?source=rss-e5b34fd3616------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:06:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kschneyder/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Engineering Project — Real Time Data Processing — Kafka]]></title>
            <link>https://medium.com/@kschneyder/data-engineering-project-real-time-data-processing-kafka-2237e8de73f0?source=rss-e5b34fd3616------2</link>
            <guid isPermaLink="false">https://medium.com/p/2237e8de73f0</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[real-time-data-processing]]></category>
            <dc:creator><![CDATA[Karyne Scheneyder Oliveira]]></dc:creator>
            <pubDate>Thu, 11 Jul 2024 00:41:55 GMT</pubDate>
            <atom:updated>2024-07-11T00:41:55.062Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Engineering Project — Real Time Data Processing — Kafka</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TWvpZBnmK9ds90sorKhWlA.png" /></figure><p>This project was developed to process data in real time by using kafka, storing and analyzing in a cloud enviroment.</p><h3>Tech Stack</h3><p>▪ Python <br>▪ Kafka<br>▪ AWS Services — EC2, S3, GLUE, ATHENA</p><p>Project code: <a href="https://github.com/kscheneyder/kafka_project">Github</a></p><p>Dataset: <a href="https://www.kaggle.com/discussions/general/139062">kaggle</a></p><h3>Creating EC2 Instance</h3><p>An EC2 instance (Elastic Compute Cloud) it’s a cloud computing service that provides scalable, on-demand processing capacity. EC2 allow us to execute application on virtual server with huge control of the enviroment, including operational system. It’s highly flexible, supporting autoscaling to adapt capacity according to demand.</p><ol><li>Creating EC2 istance:</li><li>Open AWS Service site and search for EC2</li><li>Launch an instance</li><li>Set up like the image bellow</li><li>Create a pair of keys and save it on your local folder</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*LpxfJ65TNTis0BRWXVjVAg.png" /></figure><p>6. Launch the instance and wait, when it’s available click on instance ID7.</p><p>7. Click on connect &gt;&gt; SSH Client and copy the key to access the instance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*9zMPpXBqyu4d4_Zm3eLtiQ.png" /></figure><p>8. Open cmd in windows &gt;&gt; open the folder where you saved your key pairs &gt;&gt; paste SSH key in cmd</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*FmDzWNw42gcNmUz0fAEcxA.png" /></figure><p>9. With the instance connected, you have to install java 8 by typing:</p><pre>sudo yum install java-1.8.0-openjdk -y</pre><p>10. Open kafka website and copy the link of the version you want to download, better choose a binaryh file: <a href="https://kafka.apache.org/downloads">kafka web site</a></p><p>11. Back to cmd and type to download:</p><pre>wget + {paste link to download from kafka}</pre><p>12. To install:</p><pre>tar -xvf {kafka-version-src.tgz}</pre><h3>Starting zookeeper</h3><p>ZooKeeper it’s a central coordination service used by Apache Kafka to manage the seting and sincronize brokers within cluster. Keeping information about topic status, partition and replica, making sure the high disponibility and resilience.</p><ol><li>Within the instance, open the kafka folder and type:</li></ol><pre>sudo nano config/server.properties</pre><p>2. This will allow us to change the IP, to get the IP open AWS &gt;&gt; click on instance and copy the Publick IPv4 Address</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*rOrGSInYlMz994WdF69V9g.png" /></figure><p>Change this line, removing the # &gt;&gt; close and save this setup</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*YT0fqzmf72vifowBy8kZuA.png" /></figure><p>3. Type on instance connection:</p><pre>bin/zookeeper-server-start.sh config/zookeeper.properties</pre><h3>Starting Kafka server</h3><ol><li>Open a new tab in cmd to connect to the instance (repeat the step 8 from creating EC2 instance)</li><li>Type the code bellow to change the settings of how much memory heap we need available to JVM (Java Virtual Machine) to execute the kafka server.</li></ol><pre>export KAFKA_HEAP_OPTS=&quot;-Xmx256M -Xms128M&quot;</pre><p>3. Open the kafka folder and type:</p><pre>bin/kafka-server-start.sh config/server.properties</pre><p>4. Back to AWS site to change the inbound rules on AWS EC2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*BHz5dwxf2o7MamG1njuspQ.png" /></figure><h3>Creating Kafka Topic</h3><p>An Apache Kafka topic it’s a category or data feed where messages are published and read by produces and consumers. It’s usefull to organize and storage messages from event in continuos flows, allow the transmision between distributed systems.</p><ol><li>Open a new tab in cmd to connect to the instance (repeat the step 8 from creating EC2 instance)</li><li>Again, open kafka folder and type:</li></ol><pre>bin/kafka-topics.sh --create --topic kafka_project --bootstrap-server {Put the Public IP of your EC2 Instance:9092} --replication-factor 1 --partitions 1</pre><h3>Starting Producer</h3><p>An Kafka Producer it’s a component that publish message in topics of Apache Kafka. It send data for Kafka cluster, specifying the topic and the partition where the message should be storaged. Producers can configure message delivery to ensure durability and error control. They are essential to feed data flows in real time, by accepting many data sources simultaneously.</p><ol><li>Type in the same cmd of the topic:</li></ol><pre>bin/kafka-console-producer.sh --topic kafka_project --bootstrap-server {Put the Public IP of your EC2 Instance:9092} </pre><h3>Starting Consumer</h3><p>A Kafka Consumer it’s a component that read and proccess message grom Apache Kafka topics. It subscribes to one or more topics and consumes messages from their partitions, ensuring that each message is processed reliably. Consumers can be organized into groups, allowing workload balancing and scalability of message processing.</p><ol><li>Open a new tab in cmd to connect to the instance (repeat the step 8 from creating EC2 instance)</li><li>Open the kafka folder and type:</li></ol><pre>bin/kafka-console-consumer.sh --topic kafka_project --bootstrap-server {Put the Public IP of your EC2 Instance:9092}</pre><p><em>See the python code to use the consumer and producer on the top of this article.</em></p><h3>Creating a S3 Bucket</h3><p>An S3 bucket is a cloud storage container provided by Amazon S3 (Simple Storage Service). It stores objects, which can be files of any type and size, organized in a hierarchical manner.</p><ol><li>Search for S3 in AWS Services website</li><li>Click on Create Bucket</li><li>Name your S3 bucket making sure it’s a unique name (I named mine with my surname in the end ““kafka-stock-covid-data-scheneyder”</li><li>Leave the setings as standard and click on create bucket</li></ol><p>Creating an IAM user</p><ol><li>Search for IAM on AWS</li><li>Click on user &gt;&gt; add users</li><li>Give a username and allow access key</li><li>Set permissions as AdministratorAccess</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*XMzSnxSmKSABngcSAfFumw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*KBr7fYkTOfsnd4b6_qeHig.png" /></figure><p>5. Click next, then create user</p><p>6. Click on create Secret Access Key and download the csv file to your local project folder</p><h3>Running Producer/Consumer</h3><p><em>See the python code to use the consumer and producer on the top of this article.</em></p><ol><li>Run first consumer then producer.</li><li>Open AWS S3 Bucket and update, you will be able to see the data getting stored</li></ol><h3>Creating a Crawler</h3><p>O AWS Glue Crawler é um serviço que automaticamente descobre e classifica dados armazenados em várias fontes de dados na AWS, como Amazon S3, Amazon RDS, e Amazon Redshift. Ele inspeciona os dados e infere seu esquema, criando ou atualizando tabelas no AWS Glue Data Catalog. Isso facilita a catalogação de dados, tornando-os prontos para consultas e análises com ferramentas como Amazon Athena e AWS Glue ETL. O uso de crawlers automatiza a gestão de metadados, economizando tempo e esforço em comparação com a definição manual de esquemas.</p><ol><li>Search AWS Glue on AWS Service website</li><li>Click to dropdown Data Catalog &gt;&gt; Crawler &gt;&gt; Create Crawler &gt;&gt; Name it &gt;&gt; Click on add data source</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*5jFn0HKzst4re4yNlFb9DA.png" /></figure><p>4. Browse your S3 buck and click on add an S3 data source</p><p>5. Click on next and create a new IAM rule &gt;&gt; give it the ADMIN permission as well glue permission &gt;&gt; Click on NEXT</p><p>6. Add a database and name it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*KdYfF_kdD_f5fAphrmnGqg.png" /></figure><p>7. Choose the database and click on next</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*OLoWnloMROFVANFVIEU2fg.png" /></figure><p>8. Finally click on Create database</p><p>9. Select the Crawler we’ve just created and click on run</p><h3>Analyzing the data on ATHENA</h3><p>AWS Athena é um serviço de consulta interativa que permite analisar dados diretamente no Amazon S3 usando SQL padrão. Ele é totalmente gerenciado, eliminando a necessidade de infraestrutura de servidor e permite executar consultas ad-hoc com alta performance. Os usuários pagam apenas pelas consultas executadas, tornando-o uma solução econômica para análise de dados.</p><ol><li>Search for Athena on AWS</li><li>Click on launch query editor</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*eDOciF7uwf-pMjtxrLUEiQ.png" /></figure><p>3. By clicking on preview table you will see your on the screen</p><p>4. Now it’s time to have fun :D</p><p>5. Use your SQL skills to analyze the data.</p><h3>Conclusion</h3><p>Cloud services + Kafka it’s a powerful combination to deal with real time data, which can be variable in terms of size from times to times. The combination allow us to process, storage and analyze the data with security and agility.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2237e8de73f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[End to End Data Engineering Project— Containerization Docker]]></title>
            <link>https://medium.com/@kschneyder/end-to-end-data-engineering-project-containerization-docker-541a08afe78c?source=rss-e5b34fd3616------2</link>
            <guid isPermaLink="false">https://medium.com/p/541a08afe78c</guid>
            <category><![CDATA[containerization]]></category>
            <category><![CDATA[docker-image]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[data-enginnering]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Karyne Scheneyder Oliveira]]></dc:creator>
            <pubDate>Mon, 29 Apr 2024 23:17:02 GMT</pubDate>
            <atom:updated>2024-04-29T23:17:02.236Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*UiGVHpCeDdAsOV1rnOo_Jw.png" /></figure><h3>Tech Stack</h3><p>▪ Docker<br>▪ Python (SQLAlchemy, Pandas, os)<br>▪ MySQL<br>▪ Power BI</p><p>Project code: <a href="https://github.com/kscheneyder/Docker_pipeline">Github</a></p><p>Dataset: <a href="https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce/data">Kaggle</a></p><h3><strong>Building images to our project</strong></h3><ol><li>Set up a dockerfile for MySQL that will create a database with SQL command.</li></ol><pre>FROM mysql:latest<br>COPY ./database.sql /docker-entrypoint-initdb.d/</pre><p>2. Set up a Dockerfile for every script that will run our application, each dockerfile needs to install the python packages that will be used in the app, a copy of the files saved on local machine and a new folder directory to save the files within the container. (See the script on the github repository)</p><pre>FROM python:3.11<br><br>RUN pip install pandas &amp;&amp; pip install SQLAlchemy &amp;&amp; pip install mysqlclient &amp;&amp; pip install mysql-connector-python<br><br>WORKDIR /app<br><br>COPY func1.py ./<br>COPY olist_orders_dataset.csv ./</pre><p>3. Set up a docker-compose file that contains the images of your scripts and the database image. It is possible to create dependencies between the images so that one starts when the other is finished. In this project, my Python scripts will only start running when the MySQL container has a “healthy” status, which means that the database is online and ready to use.</p><pre>services:<br>  mysql:<br>    container_name: mysql-db2<br>    build: ./mysql/<br>    image: mysql:latest<br>    restart: always<br>    environment:<br>      MYSQL_DATABASE: &#39;bronzedb&#39;<br>      MYSQL_USER: &#39;scheneyder&#39;<br>      MYSQL_PASSWORD: &#39;pulse1010&#39;<br>      MYSQL_ROOT_PASSWORD: &#39;IGTExpKAKA&#39;<br>    ports:<br>      - &#39;3306:3306&#39;<br>    expose:<br>      - &#39;3306&#39;<br>    volumes:<br>      - ./my-db:/var/lib/mysql<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;mysqladmin&quot;,&quot;ping&quot;,&quot;-h&quot;,&quot;localhost&quot;]<br>      timeout: 5s<br>      retries: 160<br><br>  pythonapp1:<br>    container_name: python-script1<br>    command: sh -c &quot;python3 ./func1.py&quot;<br>    build: ./python1/<br>    volumes:<br>      - /app<br>    depends_on:<br>      mysql:<br>            condition: service_healthy<br><br>  pythonapp2:<br>    container_name: python-script2<br>    command: sh -c &quot;python3 ./func2.py&quot;<br>    build: ./python2/<br>    volumes:<br>      - /app<br>    depends_on:<br>      mysql:<br>            condition: service_healthy<br><br>  pythonapp3:<br>    container_name: python-script3<br>    command: sh -c &quot;python3 ./func3.py&quot;<br>    build: ./python3/<br>    volumes:<br>      - /app<br>    depends_on:<br>      mysql:<br>            condition: service_healthy<br><br>  pythonapp4:<br>    container_name: python-script4<br>    command: sh -c &quot;python3 ./func4.py&quot;<br>    build: ./python4/<br>    volumes:<br>      - /app<br>    depends_on:<br>      mysql:<br>            condition: service_healthy<br><br>  pythonapp5:<br>    container_name: python-script5<br>    command: sh -c &quot;python3 ./func5.py&quot;<br>    build: ./python5/<br>    volumes:<br>      - /app<br>    depends_on:<br>      mysql:<br>            condition: service_healthy<br><br>  pythonapp6:<br>    container_name: python-script6<br>    command: sh -c &quot;python3 ./func6.py&quot;<br>    build: ./python6/<br>    volumes:<br>      - /app<br>    depends_on:<br>      mysql:<br>            condition: service_healthy<br><br>  pythonapp7:<br>    container_name: python-script7<br>    command: sh -c &quot;python3 ./func7.py&quot;<br>    build: ./python7/<br>    volumes:<br>      - /app<br>    depends_on:<br>      mysql:<br>            condition: service_healthy</pre><p>With everything done, go to cmd and start the containers by typing:</p><pre>docker-compose up -d</pre><h3>Power BI — Dashboard</h3><p>After the script complete running, go to Power BI and create a conection with database to build dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*mIf9qg0dqPvee6freAAg8w.png" /></figure><p>Star Schema:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*3oYVI-XZJ7Isyl2jl6Dylg.png" /></figure><p>Dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*xAkuvbeNEY1T4099VhNZEg.png" /></figure><h3>Conclusion</h3><p>Container technology allows us to create a program divided into microservices, giving us several benefits such as: scalability, portability, speed and security. It also protects us from possible failures, since if one application stops, the others continue to work perfectly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=541a08afe78c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Engineering Project — ETL — Brazilian Stocks]]></title>
            <link>https://medium.com/@kschneyder/data-engineering-project-etl-brazilian-stocks-25894d04a19a?source=rss-e5b34fd3616------2</link>
            <guid isPermaLink="false">https://medium.com/p/25894d04a19a</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[stocks]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Karyne Scheneyder Oliveira]]></dc:creator>
            <pubDate>Fri, 19 Jan 2024 19:30:03 GMT</pubDate>
            <atom:updated>2024-01-19T19:30:03.054Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Engineering Project — ETL — Brazilian Stocks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*nX5kaIM2opA3joFiXG0iCg.png" /></figure><h3>Tech Stack</h3><p>▪ Amazon EC2<br>▪ Amazon S3<br>▪ Apache Airflow<br>▪ Python<br>▪ Yfinance</p><h3>Overview</h3><p>In this project we are going to create a pipeline that extracts data about Brazilian market stocks both daily and historical and storage this in a S3 bucket. The pipeline will run on a EC2 instance and will be orchestrated by Apache Airflow.</p><h3><strong>Extraction</strong></h3><p>To extract the data from 2020 to 2023 and also the daily data, I’ve used the python library yfinance and investpy. The <strong>yfinance </strong>library it’s an open-source tool that uses Yahoo’s publicly available APIs, and is intended for research and educational purposes.</p><p><a href="https://github.com/kscheneyder/Stocks_Project">GITHUB</a></p><h3><strong>EC2 Instance</strong></h3><p>EC2 instance is an Amazon Web Services that makes it possible to run an application on the cloud. The big advantage of this service is the costs reduction generated by physical computers and the efficiency for increase or/and decrease the computer capacity in a timely manner for changes in hardware needs.</p><p>In this step we need to create an EC2 instance that will run the application, but first we will create an security group for this instance.</p><p>Open the AWS web site and login to your root account &gt;&gt; Search for Security Groups on EC2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*nbIeL68yqZyFIzPxXkeKtw.png" /></figure><p>The security group sets up permission to allow inbound and outbound access to the instance. You can specify the source, port range, and protocol for each inbound/outbound rule.</p><p>Click on Create a security group &gt;&gt; Edit the security group according to your project (in this case I gave the permission to MyIP)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CdkLosS5nskWYTUl97pcMQ.png" /></figure><p>Click on Services &gt;&gt; Search for EC2 &gt;&gt; Click on Instances on the left side of the screen &gt;&gt; Click on Launch Instances:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nXTvlUpavCWVk_YocqPm3A.png" /></figure><p>Name the instance &gt;&gt; On Quick Start select UBUNTU:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*4IOaUbk0BzaQKnYNQWvAzA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*FB0mnSOYgjT25hk3iObAwA.png" /></figure><p><strong>Instance Type: </strong>in this step you choose the configuration of the host computer suitable for your project. You need to evaluate factors to run your application, such as: CPU, memory, storage, networking capacity and graphics hardware.</p><p>In this case I will use t3.medium which is suitable for this project.</p><p>Then we need to create a Key pair that will link the access to this EC2 instance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*RGwpDep3VOV6RoHtWYLuog.png" /></figure><p>Save your keypair in the file you have the project saved.</p><p>On network setings select the security group we’ve created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*VKwjRGEwgcIHI6iqpYtHOA.png" /></figure><p>After this steps click on Launch instance and wait.</p><p>Select the instance and click on CONNECT &gt;&gt; Click on SSH client and copy the code on “example”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*JrnR4rY-w7sNHtHeB9ZdyQ.png" /></figure><p>Open the cmd &gt;&gt; go to the folder that you have the project saved (and the keypair) &gt;&gt; paste the code &gt;&gt; type yes and you will see a screen like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*E0_lT5UfAInppZfOeVZVew.png" /></figure><p>Install and update the files to run the application by typing the code:</p><pre>sudo apt-get update<br>sudo apt install python3-pip<br>sudo pip install apache-airflow<br>sudo pip install pandas<br>sudo pip install yfinance<br>sudo pip install investpy<br>sudo pip install virtualenv</pre><h3>Storage</h3><p>This data will be storaged on a S3 bucket, so let’s create one, follow the steps bellow:</p><blockquote>Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. Customers of all sizes and industries can use Amazon S3 to store and protect any amount of data for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides management features so that you can optimize, organize, and configure access to your data to meet your specific business, organizational, and compliance requirements.</blockquote><ol><li>Open AWS website and click on Services &gt;&gt; Search for S3 &gt;&gt; Click on Create a bucket</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xkIlE0kZLxLzCaFJ_yc4Yg.png" /></figure><p>2. Name your bucket (we will not change any configuration) and click on Create Bucket</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*GgSuFkvhq3RYSi9nK9-ZnA.png" /></figure><p>3. After you created your bucket you need to go to your code (see mine on github) and change the directory to save the file like this: s3://bucket_name/file_name.csv</p><p>4. In this step we also need to give permission to the instance access the S3 bucket. So click on your instance &gt;&gt; Actions &gt;&gt; Security &gt;&gt; Modify IAM role</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d-fEZVTab2GBTZsLuBZycw.png" /></figure><p>5. Create an IAM role &gt;&gt; Create role &gt;&gt; Use case select EC2 &gt;&gt; Next &gt;&gt; type S3 and select <a href="https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/policies/details/arn%3Aaws%3Aiam%3A%3Aaws%3Apolicy%2FAmazonS3FullAccess">AmazonS3FullAccess</a> &gt;&gt; Type EC2 and select <a href="https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/policies/details/arn%3Aaws%3Aiam%3A%3Aaws%3Apolicy%2FAmazonEC2FullAccess">AmazonEC2FullAccess</a> &gt;&gt; Click on next &gt;&gt; name the role &gt;&gt; Click on create role and add this role to your instance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*hqL6fecbb9R3AlUBjUFpFw.png" /></figure><h3>Orchestration</h3><p>Now we are going to run the airflow inside the instace.</p><p><strong>Apache Airflow</strong> is an open-source tool to programmatically author, schedule, and monitor workflows. It is used in this project to automate the daily data extraction and to integrate the diferent data sources from the market stocks.</p><blockquote>With Airflow, users can author workflows as <strong>Directed Acyclic Graphs (DAGs)</strong> of tasks. Airflow’s rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed. It connects with multiple data sources and can send an alert via email or Slack when a task completes or fails. Airflow is distributed, scalable, and flexible, making it well-suited to handle the orchestration of complex business logic.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/0*-QhhbyE9QZQOMkKH.png" /></figure><p>Open the cmd and type airflow standalone,if you see the screen bellow, the airflow is already runing on the instance, copy the Login that the system automatic generated and save it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*_XNbUesKo7pzQZsKbh3MWg.png" /></figure><p>Back to AWS website click on the instance that is running copy the Public IPv4 addresses and paste on your browser adding this port “:8080”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-i_CqgN-lzq-5yfRfInjYQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LBlNz1-8uiH9q2O1uP5rwg.png" /></figure><p>Insert the Username and Password you copied from cmd.</p><p>In this step we are going to run the airflow code on the EC2 instance to make the magic happens! :)</p><ol><li>Open a new cmd window</li><li>Open the folder directory that your project is saved</li><li>Go to AWS &gt;&gt; Instance &gt;&gt; Select your Instance &gt;&gt; Click on Conect &gt;&gt; Copy the SSH client</li><li>Paste on cmd</li><li>Open the airflow fold &gt;&gt; type ls to see the folder’s inside &gt;&gt; type sudo nano airflow.cfg</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*I50FXUGVpfQ-eY4b8oqTbw.png" /></figure><p>6. Change the directory with the folder name “Dailly_dag” and save it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*lkvWrd3iFuoksE49_PkmQg.png" /></figure><p>7. Create a new folder with the code mkdir &quot;File_name&quot;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*jh4t7G2bIb0yrYwK-lfulA.png" /></figure><p>8. Create a new file “File_name.py” inside this folder and paste the dag code (see github for more information about the dag structure).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*AM-Rmp3xdbcVzCoaw27tUg.png" /></figure><p>8. Create a new file and paste the code to extract the data both dailly and historical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BGifaUlyc2W31a3ERAlbEA.png" /></figure><p>Back to the browser and update the page to see both dag inside and turn it on to start running.</p><p><a href="http://ec2-18-234-87-68.compute-1.amazonaws.com:8080/">http://ec2-18-234-87-68.compute-1.amazonaws.com:8080/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*yrDLLpveLkQBGw2wt5kxfw.png" /></figure><h3>Conclusion</h3><p>Cloud computing is a good solution for companies that needs to extract data in an automatic, self-scaling and secure way. This solution also offers the advantage of paying only for what is used at the time needed, thus meeting variations in demand and reducing operating costs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25894d04a19a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>