<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eran Elbaz on Medium]]></title>
        <description><![CDATA[Stories by Eran Elbaz on Medium]]></description>
        <link>https://medium.com/@eranelbaz?source=rss-97708ee72e46------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*305kX58JGWypqZPpoQIHBA.png</url>
            <title>Stories by Eran Elbaz on Medium</title>
            <link>https://medium.com/@eranelbaz?source=rss-97708ee72e46------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 01:42:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eranelbaz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Oct 6 MNQ Trading week recap]]></title>
            <link>https://eranelbaz.medium.com/oct-6-mnq-trading-week-recap-e35d7a47725b?source=rss-97708ee72e46------2</link>
            <guid isPermaLink="false">https://medium.com/p/e35d7a47725b</guid>
            <category><![CDATA[futures-trading]]></category>
            <category><![CDATA[trading-journals]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[trading-psychology]]></category>
            <dc:creator><![CDATA[Eran Elbaz]]></dc:creator>
            <pubDate>Fri, 10 Oct 2025 20:53:13 GMT</pubDate>
            <atom:updated>2025-10-10T20:53:13.848Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UpVqfCirE9rbggL53UM2bQ.png" /></figure><p>Last week was brutal. Six trades on MNQ futures, six losses, -$255. The kind of week that makes you question everything.</p><p>This week? Three trades. Back to breakeven.</p><p>But it wasn’t luck. It was the process finally working the way it’s supposed to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lvZec5PGSDdOJVp9fpQXSw.png" /><figcaption>Forex factory</figcaption></figure><p>Looking at the Forex Factory calendar, Monday and Tuesday had no major economic events. No catalysts. Wednesday had FOMC minutes. Thursday had Powell speaking.</p><p>So I waited. Two full days on the sidelines because I assume price action will be better on the last part of the week</p><p><strong>The lesson:</strong> Sometimes the best trade is no trade.</p><h3>Trade 1: The Classic Dilemma</h3><p>Price is in extreme discount. Perfect long setup using the 12-hour lookback. Entered targeting 1-to-3, moved my stop to protect profit, got reversed out at 1-to-1.7. Then watched price hit my original target without me.</p><p>Classic trader’s dilemma: protect profit or let winners run? I chose protection. No regrets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ohxtx0FQcwKSpIxFrD4pFg.png" /></figure><h3>Trade 2: Last Week’s Lesson Applied</h3><p>24-hour lookback setup. Price went sideways after entry. Last week, I would’ve exited and missed the move. This week? Reduced risk by moving stop to break-even. Got stopped out at zero loss instead of taking a full hit.</p><p>That’s growth. That’s what learning looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GTWR3-V3umzYokr59qSL8g.png" /></figure><h3>Thursday: Context Is Everything</h3><p>Extreme premium at all-time highs. Clear short setup, but I cut my position size in half — unknown territory means reduced risk. Entered late due to technical issues, which actually let me place a tighter stop. Hit my target for 1-to-2.</p><p>Adapted. Executed. Done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L1U97Ke-iF-v0J5_j_ONbw.png" /></figure><h3>What Actually Made This Week Work</h3><p>It wasn’t one hero trade. It was three things:</p><p><strong>Planning:</strong> Filtered out low-probability days using the economic calendar. Saved me from choppy losses.</p><p><strong>Discipline:</strong> Applied last week’s lesson when price went sideways. Reduced risk instead of panic-exiting.</p><p><strong>Risk Management:</strong> Cut position size at all-time highs. Adapted to context instead of treating every trade the same.</p><h3>Want the Full Breakdown?</h3><p>I document every trade, every decision, every lesson in my weekly MNQ futures trading recaps on YouTube. Complete transparency — wins, losses, and everything in between.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FfXGc9CGH6yo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DfXGc9CGH6yo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FfXGc9CGH6yo%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/605da76eec3be3bf8ad48ea7761d3ba1/href">https://medium.com/media/605da76eec3be3bf8ad48ea7761d3ba1/href</a></iframe><p>This is the journey to consistent profitability. Week by week. Trade by trade.</p><p>If you’re tired of fake “guru” content and want honest trading documentation, subscribe and follow along.</p><p><strong>Process over profits. Always.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e35d7a47725b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Wrote Demo Trading Bot]]></title>
            <link>https://eranelbaz.medium.com/i-wrote-demo-trading-bot-ad8292347ac8?source=rss-97708ee72e46------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad8292347ac8</guid>
            <category><![CDATA[stocks]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[bots]]></category>
            <dc:creator><![CDATA[Eran Elbaz]]></dc:creator>
            <pubDate>Tue, 21 Jan 2020 15:09:16 GMT</pubDate>
            <atom:updated>2020-01-21T15:12:23.458Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*VGktNDRRJAUYmJhd-PvRvg.jpeg" /></figure><p>The idea was to write a simple Trading bot in python which buy when the stock is low and sell when high on daily basis, but soon I understood this is bigger than what I thought</p><p>The Idea was simple one, one class to represent my broker, multiple classes of strategies and one class which is the bot<br> I made another class I called StockValue that represent the value of a stock in a given time</p><h3>StockValue</h3><pre>@dataclass<br>class StockValue:<br>    price: float<br>    name: str<br>    time: datetime = datetime.now()</pre><h3>Strategy</h3><pre>class Strategy:<br>    def decide_buy(self, sold_price: StockValue, price_history: [StockValue]) -&gt; bool:<br>        raise NotImplemented()<br><br>    def decide_sell(self, bought_price: StockValue, price_history: [StockValue]) -&gt; bool:<br>        raise NotImplemented()</pre><p>I wrote only one implementation which is:</p><ul><li>calculate the <a href="https://en.wikipedia.org/wiki/Tangent">tangents</a></li><li>If the tangent changed from positive to negative return True to sell command</li><li>If the tangent changed from negative to positive return True to buy command</li></ul><p>Also taking into consideration the current hour, because if near the closing of the trade day I want to sell and not to keep the stocks<br> And a small timeout because I don’t want to buy and sell a lot in order to minimize commands and thus minimize commissions</p><h3>Broker</h3><pre>class StockBroker:<br>    def get(self, stock_name: str) -&gt; StockValue:<br>        raise NotImplemented()<br><br>    def buy(self, stock: StockValue, count: int) -&gt; float:<br>        raise NotImplemented()<br><br>    def sell(self, stock: StockValue, count: int) -&gt; float:<br>        raise NotImplemented()</pre><p>Very simple</p><ul><li>get will return you the current StockValue of the stock</li><li>buy will give command to buy and will return how much it cost</li><li>sell will give command to sell and will return how much I earn</li></ul><p>when I initiate my mock broker I give it which stocks I want to trade and it runs using a csv file of the stock value throughout the day</p><p>I took a CSV from yahoo finance and run my simple algorithm using this data and I theoretically “made” a lot of money, <br> and than I wanted to tell myself why don’t I connect it to a real broker and actually start to make money?<br> And than I read some more, there are a lot of selling and buying commands, such as do you want that your command will be valid only until the end of the day or until you send a cancel command?<br> And in real life you trade in front of real people, and maybe there is a small change in the trend before a big change, for example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*OGVCMjcf-C0nl6jE.jpg" /></figure><p>if you will sell in the first bar and than buy again, you will lose money to commissions in contrary to leave the money in and sell the stocks when the value is actually high</p><p>Also the problem is that this bot is a passive bot,</p><blockquote><em>Good investor will sell stock because the market offer an higher price, and you will never buy stock because it rises<br> The Intelligent Investor, Benjamin Graham</em></blockquote><p>So I won’t say it was a complete failure, because I learnt a lot about trends and the stock market, read a bit from Benjamin Graham, and I think I will leave this job to the real people, who know how to trade :)</p><blockquote><em>Like this post? You want to access my private github projects?<br> Support me via </em><a href="https://www.patreon.com/eranelbaz"><em>Patreon</em></a></blockquote><p><em>Originally published at </em><a href="https://dev.to/eranelbaz/i-wrote-demo-trading-bot-5ffp"><em>https://dev.to</em></a><em> on January 21, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad8292347ac8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker — Dev, Debug, Deploy]]></title>
            <link>https://medium.com/analytics-vidhya/docker-dev-debug-deploy-15b232eed53e?source=rss-97708ee72e46------2</link>
            <guid isPermaLink="false">https://medium.com/p/15b232eed53e</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[production]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Eran Elbaz]]></dc:creator>
            <pubDate>Sat, 21 Dec 2019 12:58:40 GMT</pubDate>
            <atom:updated>2020-01-19T21:10:07.360Z</atom:updated>
            <content:encoded><![CDATA[<h3>Docker — Dev, Debug, Deploy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jzQaYzFydP79r3hN" /><figcaption>Photo by <a href="https://unsplash.com/@katetandy?utm_source=medium&amp;utm_medium=referral">Kate Tandy</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Why</h3><p>As a developer who works on couple of projects, I wanted to find a way that -</p><ul><li>My setup time will be as fast as possible</li><li>It would be easy to share my code and project setup with team-mates</li><li>I want my development environment to be as close to production as possible</li></ul><p>So I started to look for a way to do that, first I thought about Git, but the problem with Git is that the setup time is not optimal -<br> How do I automatically configure Environment Variables? or syncing Git hooks for my project, what if my project is using new programming language or version? I don&#39;t want to start install and manage versions on my local PC..</p><p>And I don’t want to pay for any solution.</p><p>Than I started to think about using Docker as my development environment, I can run Docker as a &quot;Personal Development Server&quot; and configure everything I need inside the Docker Image - Compiler / Interpreter, Environment Variables, Dependencies and more, and when I finish coding, I can use the same Docker image in my production environment</p><h3>What Docker is</h3><p>“ Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. With Docker, you can manage your infrastructure in the same ways you manage your applications. By taking advantage of Docker&#39;s methodologies for shipping, testing, and deploying code quickly, you can significantly reduce the delay between writing code and running it in production.&quot;<br> From Docker official documentation, you can read more <a href="https://docs.docker.com/engine/docker-overview/">here</a></p><h3>Build</h3><p>I use this Dockerfile as a template from <a href="https://docs.docker.com/engine/examples/running_ssh_service/">here</a> in order to build my Docker Image, I install openssh-server so I can use it just like a development server and share it with my team-mates.</p><pre>FROM python:3.6<br>ARG GIT<br><br>RUN apt-get update &amp;&amp; apt-get install -y openssh-server<br>RUN mkdir /var/run/sshd<br>RUN echo &#39;root:MYSUPERSECRETROOTPASSWORD&#39; | chpasswd<br><br># According to your linux distribution this line my differ<br>RUN sed -i &#39;s/#PermitRootLogin prohibit-password/PermitRootLogin yes/&#39; /etc/ssh/sshd_config<br><br>ENV NOTVISIBLE &quot;in users profile&quot;<br>RUN echo &quot;export VISIBLE=now&quot; &gt;&gt; /etc/profile<br>EXPOSE 22<br><br>RUN echo &#39; \n\<br>git clone $GIT ~/code \n\<br>cd ~/code \n\<br>git pull \n\<br>pip install -r requirements.txt&#39; &gt; ./run.sh<br><br># Also configure hooks, environment variables and more<br><br>RUN sh ./run.sh<br><br>RUN echo &#39;/usr/sbin/sshd -D&#39; &gt;&gt; ./run.sh<br><br>CMD [&quot;sh&quot;, &quot;./run.sh&quot;]</pre><p>When executing docker build it will git clone my project and install pip dependencies and whatever I need to do, also I define the same ./run.sh file to be execute each time I use docker run in order to keep my docker up-to-date with new commits and more</p><h3>Run</h3><p>I use this command in order to run my image</p><pre>&gt; docker run -dP my_cool_docker<br>37d6e53cb27396467a10c7361d319d28d0197a7b5dc7347bb39c251dff7403dc<br><br>&gt; docker port 3<br>22/tcp -&gt; 0.0.0.0:32768<br><br>&gt; ssh root@localhost -p 3276<br>The authenticity of host &#39;[localhost]:32768 ([::1]:32768)&#39; can&#39;t be established.<br>ECDSA key fingerprint is SHA256:z4x3yWVSJZAoswgEa0utt5jSv0Mt0Ex6sMY8a4CFCnE.<br>Are you sure you want to continue connecting (yes/no)? yes<br>Warning: Permanently added &#39;[localhost]:32768&#39; (ECDSA) to the list of known hosts.<br>root@localhost&#39;s password:<br>Linux 37d6e53cb273 4.9.184-linuxkit #1 SMP Tue Jul 2 22:58:16 UTC 2019 x86_64<br><br>The programs included with the Debian GNU/Linux system are free software;<br>the exact distribution terms for each program are described in the<br>individual files in /usr/share/doc/*/copyright.<br><br>Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent<br>permitted by applicable law.<br>root@37d6e53cb273:~# ls -l<br>total 4<br>drwxr-xr-x 1 root root 4096 Dec 14 09:45 code</pre><p>So as you can see, I run the image using detach flag and expose all ports,<br> then I use ssh to localhost with the exposed port, enter your super secret root password and we have a personal development server which is actually a docker container running on your personal PC!</p><h3>Advantages</h3><ul><li>One time setup for everything I need</li><li>Same environment for dev and production</li><li>Easy to share with team-mates</li></ul><h3>Disadvantages</h3><ul><li>You can loss data if your container crash</li><li>Fit for those who use containers in production</li></ul><p>Enjoy 😋</p><blockquote>Like this post? You want to access my private github projects?<br>Support me via <a href="https://www.patreon.com/eranelbaz"><em>Patreon</em></a></blockquote><p><em>Originally published at </em><a href="https://dev.to/eranelbaz/docker-dev-debug-deploy-199l"><em>https://dev.to</em></a><em> on December 21, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15b232eed53e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/docker-dev-debug-deploy-15b232eed53e">Docker — Dev, Debug, Deploy</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Git Hooks 101]]></title>
            <link>https://eranelbaz.medium.com/git-hooks-101-dcfcb7d3ec87?source=rss-97708ee72e46------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcfcb7d3ec87</guid>
            <category><![CDATA[policy]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[production]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Eran Elbaz]]></dc:creator>
            <pubDate>Fri, 22 Nov 2019 12:17:32 GMT</pubDate>
            <atom:updated>2020-01-19T21:10:13.463Z</atom:updated>
            <content:encoded><![CDATA[<p>In this tutorial I won’t focus on what git hooks are, I&#39;ll show u simple example on how you can use them</p><p>From <a href="https://git-scm.com/docs/githooks.html#_description">git documentation</a>:</p><pre>Hooks are programs you can place in a hooks directory to trigger actions at certain points in git&#39;s execution. Hooks that don&#39;t have the executable bit set are ignored.</pre><p>and in other words:</p><ul><li>hook is program that run when event occurs</li><li>there is a directory with all the hooks</li></ul><p>All the event are listed <a href="https://git-scm.com/docs/githooks.html#_hooks">here</a></p><p>In this tutorial I want to focus on Pre-Commit event</p><p>In our .git/hooks folder we can find all the default hook samples:</p><pre>$ ls .git/hooks applypatch-msg.sample post-update.sample pre-push.sample prepare-commit-msg.sample commit-msg.sample pre-applypatch.sample pre-rebase.sample update.sample fsmonitor-watchman.sample pre-commit.sample pre-receive.sample</pre><p>In this 101 tutorial I’ll show you how to create simple git hook that prevent git commit command to master branch</p><p>Lets now create new file pre-commit without any file extension</p><p>Now because git runs under linux (for windows using MinGW) the first line needs to be <a href="https://en.wikipedia.org/wiki/Shebang_(Unix)">Shebang</a></p><p>For Windows it should be you git bash location, For example:</p><pre>#!C:/Program\ Files/Git/usr/bin/sh.exe</pre><p>next we need to execute git branch command in order to print our branch, but we want to know if we are currently on master by using grep:</p><p>Now we warp with if with message:</p><p>in order to disable the commit we need to provide exit code which is not 0:</p><p>And our hook for pre-commit looks like that:</p><pre>$ git commit -m &quot;Commit message&quot; Commits are not allowed on branch Master!</pre><p>What is the problem with most people? is that we are lazy, we want to do something only once</p><p>and we have this option using git template directory that you can find more about it <a href="https://git-scm.com/docs/git-init#_template_directory">here</a></p><p>what that this tempalte direcotry does is when you are using git init command it will copy all the files under this directory whose name do not start with a dot</p><p>now lets define our template dir:</p><pre>$ export GIT_TEMPLATE_DIR=PATH<br>$ cd PATH<br>$ mkdir hooks</pre><p>and under PATH\hooks create your hooks and enjoy :)</p><blockquote>Like this post? You want to access my private github projects?<br>Support me via <a href="https://www.patreon.com/eranelbaz"><em>Patreon</em></a></blockquote><p><em>Originally published at </em><a href="https://dev.to/eranelbaz/git-hooks-101-62l"><em>https://dev.to</em></a><em> on November 22, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcfcb7d3ec87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Becoming High Performance Team]]></title>
            <link>https://eranelbaz.medium.com/becoming-high-performance-team-bbc3e29d81ef?source=rss-97708ee72e46------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbc3e29d81ef</guid>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[scrum]]></category>
            <category><![CDATA[teamwork]]></category>
            <category><![CDATA[kanban]]></category>
            <dc:creator><![CDATA[Eran Elbaz]]></dc:creator>
            <pubDate>Mon, 20 May 2019 09:53:25 GMT</pubDate>
            <atom:updated>2020-01-19T21:10:18.065Z</atom:updated>
            <content:encoded><![CDATA[<h4>My team’s journey on becoming High Performance Team</h4><p>For a long time my team wasn’t arranged with some sort of management methodology.</p><p>In order to improve this problem, I took a course about Project Management with emphasis on— Agile, Scrum and Kanban, and tried to implement it in my team, this is OUR journey</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NWOeIVg8qzE8nKC3KNOqDg.png" /><figcaption>Kanban Board</figcaption></figure><h3>The Question</h3><p>When you want to make a change, and especially something big like changing the way that we work, in my perspective it shouldn’t come from above, it should come from the people themselves because by doing this you make a commitment from them</p><p>So we had a team meeting and I asked them <strong>“What do you think that keeps us away from progressing?”</strong></p><p>And we started to talk about what we think, about cooperation between us, about different knowledge that some have, about doing the same project for a long period of time and more, from those points we chose one action item to solve, and it was Definition of Done.</p><p>What that was good about our DOD is testing, CR and DR that we now must do, and by this to “force” ourselves for team work</p><h3>Visualize The Workflow</h3><p>One thing that Kanban really emphasizes is the workflow - Visualize and Measure the flow</p><p>We always used some sort of Task and Workflow manager, but it felt like a burden, so we build a big physical board of TODO, In Progress and Done and where? Right in front our door, so every client or colleague could see what we do and our progress</p><p>And another board of “Help” for me, each and every team member that needs my help as a “Scrum Master” can write there, those are my tasks as a servant leader</p><p>This board has evolved also after couple of days, we added another column of CR which is limited to 4 tasks at a time</p><p>Why limit the number of tasks per columns? limiting is a powerful tool, if a column doesn’t have limit it means that it is not managed,</p><p>before we had 20 “In Progress” tasks for 8 team members which actually meant — non is in progress, and another effect is members MUST help each other in order to open the jam or you as the team leader need to intervene</p><h3>Daily</h3><p>Before I took the course Daily was more like task status for me, and less for the team in order to synchronize, I wrote the Daily and sent to my costumers, and tried to understand why each and every mission is stuck and what is the new Due Date etc.. the problem with this method is that Daily isn’t Daily, its Status</p><p>Now, after I understood the meaning of Daily — time in the day for the team to synchronize about tasks</p><p>we talk about the missions in front of the physical board, moving tasks and update each other about setbacks we encountered</p><p>We managed to reduce the Daily from 30 min meeting that we hate, to 10 min meeting that we feel that we need</p><h3>Weekly</h3><p>The next thing we tried was we started to do weeklies — 30–45 minutes meeting in which we plan the next week, breaking down tasks, and choosing what tasks we are focusing on in the next week — like sprint planning but for a one week</p><p>For me as the Team Leader, it was a game changer! from big and bulky missions to small and manageable missions, now we can split the entire release to different members and by this — minimize the cycle time and delivering less tasks but faster — and in the end more tasks for the same time</p><p>The first weekly was “monthly” like, we thought that we could do 25 user stories in a week, and it was way too much, and in the next weekly we didn’t add tasks, only arranged them more, and in the next one we took on ourselves only the relevant user stories, and after that we added new user stories but took way less, we took the exact amount for a week</p><h3>Measurement</h3><p>How will you know that you are making a progress? measurement is the only way to know that,</p><p>After you and your team can split your tasks, you can choose duration between what dates you are measuring, you can now start checking your progress, and after finishing one “sprint” of measurement, choose with your team what you should do now in order to keep improving, does your Definition of Done need some changes? do you want to limit the WIP? take 1–2 action items, and now try to measure your throughput of tasks, and see how it affects your team.</p><h3>Retrospective</h3><p>We try to do Retrospective after couple of deployments and think about what was good and what was bad and how to get better,</p><p>Retrospective is a very powerful meeting, because it can be very good and productive, or useless</p><p>We write all the points we talked about and try to make them happen until the next Retro meeting.</p><p>One of my responsibility as a team leader team is to find the weak spots, to figure out how to become better</p><p>In order to sum up, it is very easy to say “Agile is nice but…” if you really want to have a better team, and more productive team, just ask them the next question each couple of weeks “<strong>What do you think that keeps us away from progressing?”</strong></p><blockquote>Like this post? You want to access my private github projects?<br>Support me via <a href="https://www.patreon.com/eranelbaz"><em>Patreon</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbc3e29d81ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating our Hadoop Cluster with Apache DistCp]]></title>
            <link>https://eranelbaz.medium.com/migrating-our-hadoop-cluster-with-apache-distcp-5c6ddb11845?source=rss-97708ee72e46------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c6ddb11845</guid>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hive]]></category>
            <dc:creator><![CDATA[Eran Elbaz]]></dc:creator>
            <pubDate>Sat, 31 Mar 2018 17:11:56 GMT</pubDate>
            <atom:updated>2020-01-19T21:10:23.194Z</atom:updated>
            <content:encoded><![CDATA[<h4>From an unsecure cluster to a remote and secure cluster</h4><h3>Whats Our Mission?</h3><p>We need to migrate our <strong>Entire Hadoop Cluster </strong>from an unsecure CDH 5.7.1 cluster to a secure CDH 5.13.1 cluster in a remote Data Center.</p><p>And we wanted to find a complete product for our use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*yPv_3AmCyKOyVZ2AZeYflA.png" /></figure><h3>The Product Demands</h3><ul><li>Migrate our data hermetically</li><li>Migrate our Apahce Hive Metastore.</li><li>Need to show BI on the transfer process</li></ul><h3>So Whats The Big Deal?</h3><p>There are couple of frameworks for handling this type of problem:</p><ul><li><a href="https://www.cloudera.com/documentation/enterprise/5-11-x/topics/cm_bdr_tutorials.html">Cloudera BDR</a></li><li><a href="https://falcon.apache.org/">Apache Falcon</a></li><li>Oracle Golden Gate with <a href="http://www.oracle.com/technetwork/middleware/goldengate/documentation/gg-java-adp-bigdata-hdfs-adp-2601138.pdf">HDFS adapter </a>and <a href="http://www.oracle.com/technetwork/middleware/goldengate/documentation/gg-java-adp-bigdata-hive-adp-2600938.pdf">Hive adapter</a></li><li><a href="https://www.linkedin.com/pulse/moving-data-across-hadoop-clusters-using-oracle-issam-hijazi/">Oracle Data Integrator</a></li><li>And there are more that we didn’t cover</li></ul><p>However with each frameworks we encounters another fallback..</p><ul><li>Cloudera BDR does not support unsecure cluster to secure cluster</li><li>Apache Falcon does not support the current Hadoop version Cloudera uses</li><li>Oracle told us that it is not the use case for Oracle Golden Gate and recommended us to use Oracle Data Integrator</li><li>Our experience with Oracle Data Integrator wasn’t that great….</li></ul><p>So we found ourselves in front this mission without any framework that will work for us and it was obvious we had to go back to the drawing board.</p><h3>What About Apache DistCp With some Python?</h3><p>We always knew that we can use Apache DistCp from unsecure to a secure cluster but we wanted a complete product tested and reviewed that works, we didn’t want to develop our own “BDR”, although it came clear that we have to develop our own product.</p><h3>Our Flow</h3><h4>Data Copy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/198/1*-ZlDA_I4wwsfV3WH7kT7bQ.png" /></figure><ol><li>Scan the Unsecure HDFS — recursively, we need to know also the sub paths to synchronize. how many files we have? path size?</li><li>Save the HDFS scan to Database Table.</li><li>Running The <a href="https://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_admin_distcp_secure_insecure.html">Hadoop DistCp command</a> from the Secure cluster.</li><li>We want the entire process to run with our team user, so we need to change the directory permissions.</li><li>Validating that each and every byte have been synchronized across the clusters</li></ol><p>We decided to run the DistCp command in batches of 10 deepest sub directories each time, this decision was made due to resource limitation…</p><h4>Apahce Hive Metastore Syncronization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/151/1*Zkszg-AJQgVuYvyPWLu8OQ.png" /></figure><ol><li>List all the Hive databases in our Unsecure cluster.</li><li>List all the tables we have</li><li>Query “show create table” for each table.</li><li>Query the Secure cluster with the same query in order to create the tables.</li><li>Run Invalidate Metadata in order to “load” the data to impala, and msck repair for Hive</li></ol><h3>Summary</h3><p>You don’t need a fancy product to migrate or copy your data, you can always use the same old Hadoop distcp command with some hive querying.</p><p>Enjoy!</p><blockquote>Like this post? You want to access my private github projects?<br>Support me via <a href="https://www.patreon.com/eranelbaz"><em>Patreon</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c6ddb11845" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>