<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hossein Torabi on Medium]]></title>
        <description><![CDATA[Stories by Hossein Torabi on Medium]]></description>
        <link>https://medium.com/@blcksrx?source=rss-7fb27f54fe66------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yKzzGuJIdgadzCwvn1KdIg.jpeg</url>
            <title>Stories by Hossein Torabi on Medium</title>
            <link>https://medium.com/@blcksrx?source=rss-7fb27f54fe66------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 03:33:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@blcksrx/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Integrating Slack Alerts in Airflow]]></title>
            <link>https://medium.com/@blcksrx/integrating-slack-alerts-in-airflow-d091973ebfe6?source=rss-7fb27f54fe66------2</link>
            <guid isPermaLink="false">https://medium.com/p/d091973ebfe6</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Hossein Torabi]]></dc:creator>
            <pubDate>Fri, 15 May 2020 20:00:14 GMT</pubDate>
            <atom:updated>2020-05-15T20:09:02.841Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C-HfP4M0mNE3I9ZMoPeVvQ.jpeg" /></figure><p>As a Data Engineer @Alopeyk, I decided to implement slack alerts for airflow in case of falling any dag. At first, I did some google about it[As I do it always ] And I found this awesome article <a href="https://medium.com/datareply/integrating-slack-alerts-in-airflow-c9dcd155105">Integrating Slack Alerts in Airflow</a> that helped me a lot! but it was frustrating me to do <strong>copy and pates </strong>the source code of this module on each dag! so I decided to implement a module for it and this is the result of my efforts!</p><h3>Beyond airflow plugins</h3><p>The airflow plugins module tries to load any python module that exists on the plugins directory! So basically, you can write any module, for example, util.py the plugins directory and using it on your dags like this:</p><pre>from utils import Foo</pre><p>And the Foo is a function written on the utils module and this function can be a function that tries to alerts into Slack! and this function is:</p><pre>from airflow.contrib.operators.slack_webhook_operator import SlackWebhookOperator<br>from airflow.models import Variable<br><br><br>def on_failure_callback(context):<br>    &quot;&quot;&quot;&quot;<br>    This method calls a slack_operator if a task failed.<br><br>    &quot;&quot;&quot;<br>    <br>    vars = Variable.get(&#39;slack&#39;, deserialize_json=True)<br><br>    message= &quot;&quot;&quot;{mention}<br>    **Task Failed.**<br>    **Task**: ```{task}```  <br>    **Dag**: ```{dag}``` <br>    **Execution Time**: ```{exec_date}```  <br>    **Log Url**: {log_url} <br>    &quot;&quot;&quot;.format(<br>        mention=context.get(&#39;dag&#39;).owner,<br>        task=context.get(&#39;task_instance&#39;).task_id,<br>        dag=context.get(&#39;task_instance&#39;).dag_id,<br>        ti=context.get(&#39;task_instance&#39;),<br>        exec_date=context.get(&#39;execution_date&#39;),<br>        log_url=context.get(&#39;task_instance&#39;).log_url.replace(&#39;localhost&#39;, &#39;visualize&#39;)<br>    )<br><br>     slack_operator = SlackWebhookOperator(<br>        task_id=&#39;failure&#39;,<br>        http_conn_id=&#39;slack&#39;,<br>        message=message,<br>        channel=vars[&#39;channel&#39;],<br>        username=&#39;ApacheAirflow&#39;,<br>        icon_url=&#39;https://dyltqmyl993wv.cloudfront.net/assets/stacks/airflow-scheduler/img/airflow-scheduler-stack-110x117.png&#39;,<br>    )<br><br>    return slack_operator.execute(context=context)</pre><p>Now it is easy to use it on Dag! just try to import it and passed it through the da like this:</p><pre>from alopeyk.utils import on_failure_callback<br><br><br>DEFAULT_ARGS = {<br>    &#39;owner&#39;: &#39;HosseinTorabi&#39;,<br>    &#39;depends_on_past&#39;: True,<br>    &#39;schedule_interval&#39;: &#39;@daily&#39;,<br>    &#39;on_failure_callback&#39;: on_failure_callback<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d091973ebfe6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Superset in detail]]></title>
            <link>https://medium.com/@blcksrx/apache-superset-in-detail-b3744033384f?source=rss-7fb27f54fe66------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3744033384f</guid>
            <category><![CDATA[apache-superset]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Hossein Torabi]]></dc:creator>
            <pubDate>Mon, 28 Oct 2019 13:59:05 GMT</pubDate>
            <atom:updated>2019-10-28T13:59:05.367Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h3VfABPsF7TgIwAAXTcOjA.png" /><figcaption>Apache Superset logo</figcaption></figure><h3>Introduction</h3><p>Superset was started as a project at Airbnb to serve as an open &amp; fully customizable application to visualize and explore massive amounts of data in a fast &amp; intuitive way.</p><p>Since its release as an open-source project, hundreds of companies have started to use it, more than 418 developers contribute to its source code and it has been battle-tested in environments with hundreds of daily active users.</p><p>The superset is now being incubated as a project of the Apache Software Foundation and is often mentioned as an alternative to Tableau, Looker, Power BI, and other business intelligence solutions.</p><h3>Advantages</h3><ol><li><strong>create beautiful data visualizations on the fly</strong>: <br>Superset comes with tons of different, rich visualization types included. Explore, slice &amp; dice your data in a simple, visual way.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*0COl1nTeM1ZtAR3NvGJoPA.gif" /><figcaption>Apache Superset visualization explorer</figcaption></figure><p>2.<strong> Share interactive dashboards: <br></strong>Easily filter the data on the dashboard and drill deeper into the details of any chart. Organize contents in a grid layout and even on different tabs.<br>Share any chart or dashboard by simply copy &amp; paste the URL. You can even make dashboards public, use them on a TV screen with auto-refresh or embed them into your own websites or applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*ICAAuy64z3PtukyXCo21NA.gif" /><figcaption>Apache Superset dashboard</figcaption></figure><p>3. <strong>Countless Visualization Types:<br></strong>Also, please visit the <a href="https://superset.incubator.apache.org/gallery.html">superset visualization gallery</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*3hk2jqfyAnLTrewdAbvDbw.png" /><figcaption>apache superset visualization gallery</figcaption></figure><p>3. <strong>SQL editor directly in the browser:<br></strong>Superset comes with SQL Lab — a modern, feature-rich SQL IDE for deeper analysis of your data. Write queries against your database, view table metadata and visualize the results as charts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fe29lcNU_4VRtl-YG39ArA.png" /><figcaption>Apache Superset sql_lab</figcaption></figure><p>4.<strong>Connect to any database:<br></strong>Superset connects to all common SQL-speaking databases directly out of the box.<br>For more detail, please check this <a href="https://superset.incubator.apache.org/#databases">link</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*11xTYIWGsTWB2uAeUcH76A.png" /><figcaption>Apache Superset databases support</figcaption></figure><p><strong>Disadvantages</strong></p><p>Maybe this the best part of this article:D because of everywhere they only speak about superset features but they don’t about the weakness or disadvantages of it also, it’s not only about superset! it’s about almost all the products that I know and it’s a good idea to read this section before deciding to install superset!</p><ol><li><strong>Upgrading is horrible:<br></strong>Because of the lack of true software architect in superset in some cases, upgrading is horrible Almos it happens when the architect of the superset database needs to migrate.</li><li><strong>sql_lab is not perfect</strong>:<br><strong>sql_lab</strong> user interface is not really attractive and good enough for running <strong>ad-hoc queries</strong> and personally I prefer to use other tools like <strong>DataGrip </strong>or<strong> DBeaver </strong>use <strong>sql_lab. <br></strong>Especially the importance of this issue is that almost non-developers, I mean other people of the company such as BI unit, Marketing unit that uses the superset are not interested to use <strong>sql_lab </strong>and they only interested to use dashboards and charts.</li><li><strong>User Interface Desing is no perfect:<br></strong>User interface design is not really attractive and it’s not perfect enough. Other alternatives software like **<a href="https://www.metabase.com/">Metabase</a>** or **<a href="https://www.tableau.com/">Tableau</a>** have a better design.</li><li><strong>Lack of true permissions:<br></strong>permissions in the superset are not handled enough and it’s really suffering for example you can grant a permission to a user to access the charts to see it and modify it but you can’t do it with the only read access.</li></ol><p>There are more disadvantages to it but as a superset user and contributor, I found it as a good solution than alternative software. In addition, the superset is one of the <a href="https://incubator.apache.org/index.html">Apache incubating projects</a> and it’s totally understandable.</p><h3>Superset Architecture</h3><p>Apache Superset can run on the sequential and distributed mode. In sequential mode, it can only run queries that run below 60 seconds.</p><p>In distributed mode superset spread the queries(only it do it for <strong>sql_lab, </strong>not for <strong>dashboards</strong> and <strong>explore_json</strong>) between its <strong>celery</strong> workers. Superset can use <strong>RabbitMQ </strong>or <strong>Redis</strong> for distributing the tasks and it uses <strong>Redis</strong> for caching queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zOcCtR5xi-qaJLoCeeM7oQ.png" /><figcaption>Apache superset architecture in distributed mode</figcaption></figure><h3>Installation</h3><p>Superset installation is super easy just follow these steps:</p><pre>pip install superset<br><br># Initialize the database<br>superset db upgrade<br><br># Create an admin user (you will be prompted to set a username, first and last name before setting a password)<br>$ export FLASK_APP=superset<br>flask fab create-admin<br><br># Load some data to play with<br>superset load_examples<br><br># Create default roles and permissions<br>superset init<br><br># To start a development web server on port 8088, use -p to bind to another port<br>superset run -p 8080 --with-threads --reload --debugger</pre><p>Also, it can configure to work with <strong>Gunicorn</strong> and <strong>Gevent:</strong></p><pre>/usr/local/bin/gunicorn --workers 4 --worker-class gevent --timeout 60   --bind 0.0.0.0:8080 --pid /home/superset/superset.PIDFile superset:app</pre><p>For running <strong>Celery</strong> worker with <strong>Gevent</strong> just run this:</p><pre>/usr/local/bin/celery worker --app=superset.tasks.celery_app:app --pool=gevent -Ofair -c 4</pre><p>In the distributed mode, It should create <strong>superset_config.py</strong> in the <strong>SUPERSET_HOME </strong>path and run the celery workers and superset webservers.<br>The superset config file should be like the one that exists on the<a href="https://github.com/apache/incubator-superset/blob/9fc37ea9f1ff032dea923012613d4e7189ada178/superset/config.py"> superset repository</a>, In addition, it’s not necessary to overwrite all of it.<br>In general, this config will be enough:</p><pre>import os<br>from werkzeug.contrib.cache import RedisCache</pre><pre>MAPBOX_API_KEY = os.getenv(&#39;MAPBOX_API_KEY&#39;, &#39;&#39;)<br>CACHE_CONFIG = {<br>    &#39;CACHE_TYPE&#39;: &#39;redis&#39;,<br>    &#39;CACHE_DEFAULT_TIMEOUT&#39;: 60,<br>    &#39;CACHE_KEY_PREFIX&#39;: &#39;superset_&#39;,<br>    &#39;CACHE_REDIS_HOST&#39;: &#39;localhost&#39;,<br>    &#39;CACHE_REDIS_PORT&#39;: 6379,<br>    &#39;CACHE_REDIS_DB&#39;: superset,<br>    &#39;CACHE_REDIS_URL&#39;: &#39;redis://localhost:6379/2&#39;}</pre><pre>SQLALCHEMY_DATABASE_URI = &#39;mysql://superset:superset@1ocalhost:3306/superset&#39;<br>SQLALCHEMY_TRACK_MODIFICATIONS = True</pre><pre>SECRET_KEY = &#39;superset&#39;</pre><pre>SQLLAB_TIMEOUT = 60</pre><pre>SUPERSET_WEBSERVER_TIMEOUT = 60</pre><pre>class CeleryConfig(object):<br>    BROKER_URL = &#39;pyamqp://superset:superset@localhost:5672&#39;<br>    CELERY_IMPORTS = (&#39;superset.sql_lab&#39;,  &#39;superset.tasks&#39;,)<br>    CELERY_RESULT_BACKEND = &#39;redis://localhost:6379/superset&#39;<br>    CELERY_ANNOTATIONS = {&#39;tasks.add&#39;: {&#39;rate_limit&#39;: &#39;10/s&#39;}}<br>    CELERYD_LOG_LEVEL = &#39;DEBUG&#39;<br>    CELERYD_PREFETCH_MULTIPLIER = 10<br>    CELERY_ACKS_LATE = True</pre><pre>CELERY_CONFIG = CeleryConfig<br>RESULTS_BACKEND = RedisCache(<br>    host=&#39;localhost&#39;,<br>    port=6379,<br>    key_prefix=&#39;superset_results&#39;,<br>    db=superset<br>)</pre><p>In addition, I create the GitHub repository to install superset with <strong>Ansible playbooks. </strong>It’s not totally completed but it’s easy and useful.</p><p><a href="https://github.com/blcksrx/ansible-superset">blcksrx/ansible-superset</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3744033384f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streaming databases in realtime with Kafka]]></title>
            <link>https://medium.com/@blcksrx/streaming-databases-in-realtime-with-kafka-4967e44a75c9?source=rss-7fb27f54fe66------2</link>
            <guid isPermaLink="false">https://medium.com/p/4967e44a75c9</guid>
            <category><![CDATA[confluent]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[cdc]]></category>
            <dc:creator><![CDATA[Hossein Torabi]]></dc:creator>
            <pubDate>Sun, 01 Sep 2019 13:28:16 GMT</pubDate>
            <atom:updated>2019-09-01T13:51:37.260Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PZijlgbPJUEkagwTsf_nhA.jpeg" /><figcaption>Statue of Franz Kafka</figcaption></figure><h3>Introduction</h3><p>In <a href="https://en.wikipedia.org/wiki/Database">databases</a>, <strong>change data capture</strong> (CDC) is a set of software <a href="https://en.wikipedia.org/wiki/Design_pattern_(computer_science)">design patterns</a> used to determine (and track) the data that has changed so that action can be taken using the changed data. CDC is also an approach to data integration that is based on the identification, capture, and delivery of the changes made to enterprise data sources. [1]</p><p>Realtime <a href="https://en.wikipedia.org/wiki/Change_data_capture">change data capture</a> (CDC) is becoming a popular architecture for <a href="https://en.wikipedia.org/wiki/Data_integration">data integration</a> and <a href="https://en.wikipedia.org/wiki/Pipeline_(computing)">data pipelines</a>. The rise of <a href="https://docs.confluent.io/current/connect/index.html">Kafka connect</a>, in particular, has triggered a lot of interest in the subject. In realtime CDC, a database’s changes (inserts, updates, and deletes) are available as a stream of mutations for downstream consumers to tail. This architecture is showing up a lot in modern ETL and data integration implementations, and it has a number of advantages. [2]</p><h3>Kafka Connect</h3><p>Kafka Connect, an open-source component of Kafka, is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems.</p><p>Kafka Connect is focused on streaming data to and from Kafka, making it simpler for you to write high quality, reliable, and high-performance connector plugins. It also enables the framework to make guarantees that are difficult to achieve using other frameworks. Kafka Connect is an integral component of an ETL pipeline when combined with Kafka and a stream processing framework.[3]</p><p>There are many Kafka connector that you can use for common data sources and sinks to move data into and out of Kafka.</p><p>Here is a screenshot from <a href="https://www.confluent.io/hub/">Confluent hub </a>page that you can find the connectors that you want:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iyiwZDjNe6QtLngyaAk5oA.png" /><figcaption><a href="https://www.confluent.io/hub/">confluent hub page</a></figcaption></figure><p>For more information, please read the <a href="https://docs.confluent.io/3.0.0/connect/">Kafka connect </a>docs.</p><h3>How Does it work</h3><p>It’s simple! the <strong>source connector</strong>(The connector that moves data into Kafka) starts reading and tailing the<a href="https://en.wikipedia.org/wiki/Transaction_log"> binary log</a> of the data source and moving the data into Kafka. for example, if you want to use MongoDB source connector you have to enable the <a href="https://docs.mongodb.com/manual/core/replica-set-oplog/">Oplog</a> at first.</p><p>But the <strong>sink connector</strong>(The connector that moves data out of Kafka) do the reverse! it move Kafka and it consumes the topics and sends them to the destination.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*zQ5UpHKMsF4sMnBx9FS50A.png" /><figcaption>Kafka connect CDC architecture</figcaption></figure><h3><strong>Installation</strong></h3><p>It’s very easy! you should find the connector that you want and then install it on your Kafka-connect cluster(Also it can be standalone) with the confluent-hub command(I use MongoDB as an example):</p><pre>confluent-hub install debezium/debezium-connector-mongodb:0.9.5</pre><p>Now you can verify the connector installation using Kafka-connect rest API. You can see all the connectors that you installed on the Kafka-connect with this command:</p><pre>curl $KAFKA_CONNECT_ADDRESS:8083/connector-plugins</pre><p>So now that we have the MongoDB driver installed correctly, we can configure Kafka Connect to ingest data from a database. Here’s the most minimal of configs. Note that whilst it’s minimal</p><pre>curl -X POST http://localhost:8083/connectors -H &quot;Content-Type: application/json&quot; -d &#39;{   <br>&quot;name&quot;: &quot;inventory-connector&quot;, <br>&quot;config&quot;: {       &quot;connector.class&quot;:&quot;io.debezium.connector.mongodb.MongoDbConnector&quot;,                &quot;mongodb.hosts&quot;: &quot;rs0/192.168.99.100:27017&quot;,<br>&quot;mongodb.name&quot;: &quot;fullfillment&quot;,    <br>&quot;collection.whitelist&quot;: &quot;inventory[.]*&quot;}<br>}</pre><p>That&#39;s All about it!</p><p>please help me with your feedback to write better.</p><h3>Resources</h3><ol><li><a href="https://en.wikipedia.org/wiki/Change_data_capture">https://en.wikipedia.org/wiki/Change_data_capture</a></li><li><a href="https://riccomini.name/kafka-change-data-capture-breaks-database-encapsulation?source=post_page-----cd87693ec83e----------------------">https://riccomini.name/kafka-change-data-capture-breaks-database-encapsulation?source=post_page-----cd87693ec83e----------------------</a></li><li><a href="https://docs.confluent.io/current/connect/index.html">https://docs.confluent.io/current/connect/index.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4967e44a75c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>