<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zakaria DRISSI EL MALIANI on Medium]]></title>
        <description><![CDATA[Stories by Zakaria DRISSI EL MALIANI on Medium]]></description>
        <link>https://medium.com/@maliani.zakaria?source=rss-5c8c0375ac7a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3hSW3Wsb4HE-8SlCYIW0Pw.jpeg</url>
            <title>Stories by Zakaria DRISSI EL MALIANI on Medium</title>
            <link>https://medium.com/@maliani.zakaria?source=rss-5c8c0375ac7a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 06:25:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maliani.zakaria/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why You Probably Don’t Need Microservices (Yet)]]></title>
            <link>https://medium.com/@maliani.zakaria/why-you-probably-dont-need-microservices-yet-071a643900e1?source=rss-5c8c0375ac7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/071a643900e1</guid>
            <category><![CDATA[monolithic-architecture]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Zakaria DRISSI EL MALIANI]]></dc:creator>
            <pubDate>Mon, 18 Aug 2025 14:26:03 GMT</pubDate>
            <atom:updated>2025-09-20T17:15:55.268Z</atom:updated>
            <content:encoded><![CDATA[<p>Walk into any tech conference or browse through LinkedIn, and you’ll see it everywhere: M<strong><em>icroservices</em></strong>. Companies are rushing to adopt microservices architecture because it sounds sexy, modern, and cutting-edge. “We’re using microservices” has become the new “we’re using AI” — a badge of honor that makes you sound innovative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J1e4zxHSUom_pPt_us-B8A.jpeg" /><figcaption>Monolytics can be the best choice for your application</figcaption></figure><h3>Why You Probably Don’t Need Microservices (Yet)</h3><p>Creating microservices applications has become a trend for many people. The problem with trends is that when you see them everywhere, you start doing them without understanding why and without asking the right questions before diving into the adventure.</p><p>In most cases I’ve seen during my 10 years of experience, you only need a monolithic application. But the question that comes up here is: when can we afford to think about a microservices architecture?</p><h3>Learning from Netflix’s Journey</h3><p>To answer this question, we have to go back to Netflix’s story — one of the leaders in microservices architecture. They started with a monolith. They reached its limits. They realized their platform wouldn’t scale efficiently to handle their traffic growth. The solution was to divide the code into a thousand small parts.</p><p>The question that still comes up: Why not start directly with microservices since we already know the limitations of monoliths?</p><h3>The Reality Check</h3><p>The answer is simple: for several reasons.</p><p><strong>It’s very complicated.</strong> Yes, Spring Cloud makes this architecture more accessible, but going to production remains very difficult given the large number of obstacles you encounter. Think about defining each microservice with its database, managing dependencies between microservices, handling rollbacks, memory management in pods, autoscaling in Kubernetes — especially when using Java.</p><p><strong>Traffic problems are rich people’s problems.</strong> Usually we don’t have this problem at the start of our project. I recommend starting with the simplest solution and solving problems as we encounter them.</p><p><strong>It’s too expensive.</strong> Using a microservices architecture often means deploying our application on a public cloud (AWS or GCP…) on a Kubernetes cluster. The problem is if you have 100 microservices using Java, you pay for 100 JVMs (between 300 and 500MB per pod = 300*100MB) even when nobody is using the database. And I’m not even talking about autoscaling management.</p><p><strong>It takes much more time</strong> compared to developing a monolithic application. More time means more money.</p><h3><strong>The Spring Boot Scaling Trap</strong></h3><p>In the world of microservices, Spring Boot with Java is the most popular framework choice. But here’s where it gets expensive fast:</p><blockquote>a single Spring Boot application can easily consume 800MB of RAM just for the JVM at startup. Assuming that the pod is configured to have 1GB of memory, when you configure Kubernetes to autoscale at 70% memory usage, you’re looking at scaling triggers around 700MB per pod.</blockquote><p>This creates a vicious cycle: new pods spin up quickly due to the low threshold, consume massive amounts of memory immediately upon startup, then often get terminated shortly after when the actual load doesn’t justify their existence. You end up paying for a constant churn of oversized, short-lived pods.</p><p>Google’s GKE has introduced boost functionality to help mitigate this issue, but it’s still a band-aid solution. The fundamental problem remains: Java’s memory footprint makes microservices autoscaling extremely expensive and inefficient compared to running a single, well-optimized monolith.</p><p>And I’m not even talking about the complexity of properly configuring autoscaling policies to avoid this thrashing behavior.</p><h3><strong>It requires real experts</strong></h3><p>to ensure smooth development. Often we think that reading an article and using the right tools is enough, but that’s really not the case here.</p><p>When your microservices start failing in production — andi hope they will — you need people who understand distributed systems, not just Spring Boot. You need experts who can debug network timeouts, handle service discovery issues, and fix data consistency problems across multiple databases.</p><p>You need architects to design microservices properly. You need messaging services like Apache Kafka, and managing Kafka is difficult — it’s not something you just install and forget. You need experts to decide which microservice will call which other service with REST protocols. You need people who know how to handle exceptions that span across multiple microservices. When something breaks, it’s not just one place to look anymore. It’s a chain reaction across your entire system, and you need people who understand how to trace problems through this complexity.</p><p>You also need DevOps profiles to handle Kubernetes clusters, Kafka servers, and a lot of other tools, containers, and replication. This is not easy either. It’s not easy to find these kinds of profiles, and they cost a lot.</p><h3>My Take</h3><p>I could continue but this would be too much. I admit that all this goes against the current, but I write based on my experience.</p><p>Start simple. Build a monolith first. Scale when you actually need to, not when you think you might need to. Your future self will thank you.</p><p>I encourage you to come debate if you disagree or leave a comment — I’ll take pleasure in responding.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=071a643900e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is AI Really Going to Replace Developers? What I’ve Learned After 2 Years]]></title>
            <link>https://medium.com/@maliani.zakaria/is-ai-really-going-to-replace-developers-what-ive-learned-after-2-years-4c9919ba2eb1?source=rss-5c8c0375ac7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c9919ba2eb1</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[developper]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Zakaria DRISSI EL MALIANI]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 13:55:40 GMT</pubDate>
            <atom:updated>2025-07-27T13:55:40.759Z</atom:updated>
            <content:encoded><![CDATA[<p>This week, I had a very enriching discussion with our talented Sofrecom experts about the use of generative AI: <em>Anas Mastour, Mohamed Karimi, and Zakaria DRIF</em></p><p>In this article, I’m sharing a brief summary because I find the topic really interesting, especially since right now, quite a few students and even professionals are starting to worry and often ask me the question:</p><p><strong>Should we still focus on software development, or is this a good time to move into different fields?</strong></p><figure><img alt="IA VS developers" src="https://cdn-images-1.medium.com/max/1024/1*84KTyElg4s1A_3QtuprmqQ.png" /></figure><p>The key question was something many people wonder about: Are generative AIs going to replace developers? And yes, we’ve been discussing this for 2 years and it’s still a relevant question today. Our answer was unanimous: <strong><em>No, it’s not going to replace them</em></strong>, but why not?”</p><p>1 — Because generative AIs are fundamentally a statistical model that doesn’t know how to say “I don’t know,” and as a result, in many cases it makes mistakes and you always need someone who masters their subject to be able to control this tool.</p><p>2 — Because it can be very strong for from-scratch projects based on Open Source tools, but much less effective when you want to use it with internal tools. If you ask an AI to create a connection between Gmail and your Angular site, it can be very capable, but if you want to connect with your internal system, it can still be very useful but only when the person using it has a solid grasp of their subject. We’ll cover this point further later in this article.</p><p>However, this took us to another question that I don’t particularly enjoy:</p><p><strong><em>Can we now say that a company wouldn’t need senior profiles anymore if we assume that a Junior + AI can replace a senior, as several social media videos suggest with claims like “you just need a bit of knowledge to create your app with your AI”? Or is it the opposite: can a senior work alone without needing a junior because now they have an assistant that can handle simple tasks?</em></strong></p><p>For me, there’s no right answer to this question because, even though AI has progressed in recent years, its adoption remains fairly modest. So for a senior, yes, AI will help them accomplish more tasks in less time, but it will never replace the fresh perspective that a junior could bring to a project. Not to mention that junior developers learn to make decisions and become autonomous, something that an AI assistant that spits out nonsense 33% of the time will never be able to do.</p><p>And the reverse is equally not true, a Junior + AI can never replace a Senior, that’s obvious. Seniority means technical expertise, architectural choices, selecting the right technologies for the use case and matching them to the company context. For example, for a given use case, it might be smarter to write it in Java, but if everyone in the company knows PHP, then in that context it might be better to do it in PHP. That’s what making good decisions is about. And I don’t think an AI assistant could provide that for you, unless you know exactly what to ask it — and for that, you already need the expertise.</p><p><strong><em>So how can companies adapt to this new way of working?</em></strong></p><p>For me, it will definitely impact team velocity, so with the same project budget we could accomplish +25% (on average) of what we used to do before, but this doesn’t depend only on AI but on many other tools like Git, documentation, automation, good project management… and many other aspects. <strong><em>You can’t speed up what isn’t moving forward.</em></strong></p><p><strong><em>So practically speaking, what makes someone say they gain 10 or 30 or 80% when using generative AI? I’ll try to answer based only on my own experience.</em></strong></p><p>— 100% — 80%: Someone who wants to sell you the idea that developers will lose their jobs because of AI (to get more views) will write a prompt like “create a task management application for me.” There, with <strong><em>Vibe Coding</em></strong>, they’ll ask the AI to adapt its code until they get the result they want without writing a single line of code. <strong>They’ll save 100% of the time</strong>. But this is only possible because we’re creating the application from scratch and because the application isn’t too complex.</p><p>After that, it’s still totally possible and practical to create a web page with Vibe Coding — without writing a single line of code — for example for a POC or even for production, but you need to be careful to verify everything the AI has generated. And there, we’re back to square one: to do this, you need a minimum of knowledge and expertise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HDb9IrpPs_MgkcHAqfGmFA.png" /><figcaption>Generate 100% application with IA</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KjY-Pye00WrZUzlAaGEgKw.png" /><figcaption>Vibe Coding : Ask IA to cahnge the color of template from blue to red</figcaption></figure><p>— 30% — 60%: In a project, AI can be a great help for adding Java code that will, for example, sort users by number of sales. But in a business context, I prefer to create the function, set the input parameters — for example an ArrayList&lt;Users&gt; with each user’s sales list (user-&gt;sales) — and then ask the AI to generate the code. That way, I’m sure the AI will only touch the function’s code. We could ask it to do more (with edit mode or agent mode, Copilot for example can modify multiple files in the project), but honestly I prefer to behave this way with an assistant that has that lovely 33% error rate.</p><p>— 10% — 30%: A person in a business context with their own IT ecosystem can certainly benefit from AI, but by combining it with their knowledge and expertise. For example, if you want to create LDAP authentication using a component developed upstream within your company, you’ll need to already know how to use this component and understand what’s behind it, like the OAuth2 protocol, and then AI could help you write that code. That said, in this use case the integration is much more complicated than the source code, so I’d say that depending on your expertise you can save 10 to 30% of development time, but AI could waste your time if you start directly asking AI to solve your problem without understanding the context. I’ve seen plenty of people waste hours with ChatGPT and get nowhere.</p><p>— Less than 10%: There are fields like Data Engineering where AI hasn’t been widely adopted. Ask people who work extensively with Apache Nifi whether AI has really helped them or actually saved them time, or in data pipeline architecture and data infrastructure, which is my field. Here too, I think AI hasn’t helped much so far because everything depends on context. And we shouldn’t forget that AI is based on statistical calculation, so if you ask an AI whether it’s wise to use DBT or Apache Kafka in a use case, it might say yes even if there are better solutions. And when you spend a lot of time in the architecture world, we no longer ask whether a tool or technology can meet a need, but rather whether it’s the best way to meet that need, because ultimately you can do almost anything with any tool.</p><p>For example, because of its simplicity, everyone uses MongoDB, and an AI might suggest using MongoDB for an inventory management application because MongoDB will give you more fluidity. But it’s DocumentDB which really isn’t designed for joins, so in this case you need relational databases. The big question here is: how do we estimate the time lost due to a bad choice suggested by AI? (even though in this case ChatGPT tells you yes you can, but there are other choices — you need knowledge to be able to choose between the options it offers you with their pros and cons).</p><blockquote>Today, everyone only talks about the gains when using generative AI, because it’s a trending topic. <strong>WHAT ABOUT THE LOSSES</strong>?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w7joudA5e9GknDWdqO3bOA.png" /></figure><p>Another point I want to address in this article, and it’s something I’ve discussed with many people even before LLMs appeared, is that the major complexity of new platforms no longer lies in writing source code. Take Netflix as an example with 1000 microservices, so 1000 distinct mini-applications, where each microservice contains a few thousand lines of code (LoC). Here, the big problem is managing these microservices, auto-scaling, deployment automation, documentation, architectural choices, open source strategies, Cloud/OnPrem/hybrid strategy…</p><p>Do you think if I handed you just Netflix’s source code, you could recreate the same platform or achieve the same results? Obviously not. Code generation isn’t the whole story in a developer’s world — it’s just one part of the job, and Gen AIs can’t replace everything a developer does, but they can assist with certain tasks when skillfully used by the developer.</p><h3><strong>When it comes to skills, will LLMs help or not in gaining competencies?</strong></h3><p>A few years ago, we noticed there was a big technical difference between someone who had learned based on foundational knowledge from books and people who built their knowledge from articles, StackOverflow, or YouTube videos. Someone who relies on books generally has a global vision of the technology’s context, so you don’t necessarily wait to see or encounter the problem to discover it and try to understand the outcome.</p><blockquote>I’ll give a small example: when you learn to write Java based on an article, for instance, and you want to create a list with unique elements, you might initially create a LinkedList and every time you want to add an element you’ll check if it already exists in the list. But if you had previously taken the time to read and understand all the data structures before diving in, you would surely think to use a Set instead of a List, because you’d know that a Set guarantees the uniqueness of your elements.</blockquote><p>In the era of generative AI, a new generation of developers has emerged: those who only know Gen AI and find it very annoying to search on StackOverflow or even post a question and wait for an answer. They also don’t like having to explain their problem in detail to get a correct response. At the time I’m writing this article, this is very problematic because by doing this, the learning curve becomes far too slow.</p><blockquote>This excessive dependence on LLMs can paradoxically make us lose considerable time, <strong>difficult to quantify,</strong> especially since current studies focus only on time saved without measuring time lost.</blockquote><p><strong>If we assume that from now, AI will take over the role of creating and developing, we humans will gradually start losing our dev skills (skill decay). But then, who will be able to maintain these applications when there’s a major problem?</strong></p><p>Let’s imagine we created a supermarket management application with AI. The entire application is built by generative AI with vibe coding, and we’re satisfied with the result. We later add ML algorithms to determine where to place each item using AI, and we’ve added an ML algorithm to target people and launch offers directly that will interest them. After two years, we realize the algorithms are behaving in a very racist manner, and we want to manually change the rules of the game or at least understand why this behavior is occurring.</p><p><strong>The question we need to ask here is: is it responsible to give so much power and control to an AI? Even if we reach a point where it’s possible for it to be autonomous, should we give AI the right to decide, the right to classify? Are we going to let AI have the power to decide who we’ll prioritize for surgical operations or who we’ll approve for credit and who we’ll refuse without even understanding why?</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c9919ba2eb1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IA Gen d’un point de vue developement. Comment les entreprise peuvent ils gérer ce shift ?]]></title>
            <link>https://medium.com/@maliani.zakaria/ia-gen-dun-point-de-vue-developement-comment-les-entreprise-peuvent-ils-g%C3%A9rer-ce-shift-49bde255b1c8?source=rss-5c8c0375ac7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/49bde255b1c8</guid>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Zakaria DRISSI EL MALIANI]]></dc:creator>
            <pubDate>Sat, 12 Jul 2025 11:36:41 GMT</pubDate>
            <atom:updated>2025-07-20T18:16:23.675Z</atom:updated>
            <content:encoded><![CDATA[<h3>L’IA va-t-elle vraiment remplacer les développeurs ? Mon retour d’expérience après 2 ans</h3><p>Cette semaine, j’ai eu une discussion très enrichissante avec nos talentueux experts de Sofrecom autour de l’usage de l’IA générative : <a href="https://www.linkedin.com/in/anas-mastour/"><em>Anas Mastour</em></a><em>, </em><a href="https://www.linkedin.com/in/mohammed-karimi-63446578/"><em>Mohamed Karimi</em></a><em> et </em><a href="https://www.linkedin.com/in/drifzakaria/"><em>Zakaria DRIF</em></a></p><p>Dans ce post, je partage un petit résumé, parce que je trouve le sujet vraiment intéressant, surtout qu’en ce moment, pas mal d’étudiants et même des professionnels commencent à s’inquiéter et me posent souvent la question :</p><h3>Est-ce qu’il faut encore miser sur le développement logiciel, ou est-ce le bon moment pour se réorienter vers d’autres domaines ?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XxaU3dfkQazF6nHLSwyrBA.png" /><figcaption>Developer VS LLM</figcaption></figure><p>La question principale était une question que beaucoup de personnes se posent : Est-ce que les IA Gen vont remplacer les développeurs ? Et oui, ça fait 2 ans qu’on en parle et ça reste une question d’actualité. La réponse à cette question était claire pour nous tous : <strong>Non, ça ne va pas remplacer</strong>, mais pourquoi ?</p><p>1 — Parce que les IA Gen à la base sont un modèle statistique qui ne sait pas dire que je ne sais pas et du coup dans plusieurs cas il se trompe et il faut toujours quelqu’un qui maîtrisent son sujet pour pourvoir maîtriser cet outil.</p><p>2 — Parce qu’il peut être très fort pour des projets from scratch basés sur des outils Open Source mais beaucoup moins fort quand on veut l’utiliser avec des outils internes. Si tu demandes à une IA de faire une connexion avec Gmail à ton site en Angular, il peut être très fort, mais si tu veux te connecter avec ton système interne, là il peut t’être très utile aussi mais seulement quand la personne qui l’utilise maîtrise bien son sujet. Nous allons encore aborder ce point dans la suite de cet article.</p><p>Mais ceci nous a menés à une autre question que je n’aime pas trop :</p><h3><strong><em>Est-ce que maintenant on peut dire qu’une entreprise n’aurait plus besoin de profils senior si on suppose qu’un Junior + IA peut remplacer un senior comme plusieurs vidéos sur les réseaux sociaux l’indiquent “tu as juste besoin d’un peu de connaissances pour créer ton application avec ton IA” ? Ou bien c’est le contraire : un senior peut travailler tout seul sans avoir besoin d’un Junior parce que maintenant il a un assistant qui peut faire les tâches simples ?</em></strong></h3><p>Pour moi, il n’y a pas de bonne réponse à cette question puisque, justement, même si l’IA a progressé ces dernières années, son adoption reste tout de même assez modeste. Donc pour un Senior, oui, l’IA va l’aider à faire plus de tâches en moins de temps, mais ça ne va en aucun cas remplacer la fraîcheur que pourrait apporter un junior à un projet. Sans compter que ce profil junior va apprendre à prendre des décisions et à être autonome, chose qu’un assistant IA qui te balance des conneries 33% du temps ne pourra jamais faire.</p><p>Et puis inversement, un Junior + IA ne pourra jamais remplacer un Senior, c’est évident. La séniorité, c’est l’expertise technique, le choix d’architecture, le choix des technologies adéquates au use case et en adéquation avec le contexte de l’entreprise. Par exemple, pour un use case donné, il se peut qu’il soit plus judicieux de l’écrire avec du code Java, mais si dans l’entreprise tout le monde maîtrise du PHP, alors dans ce contexte il se pourrait que ce soit mieux de le faire en PHP. C’est ça, le fait de prendre les bonnes décisions. Et ça, je ne pense pas qu’un assistant IA pourrait te le fournir, sauf si tu sais exactement quoi lui demander, et pour ça il faut déjà avoir l’expertise.</p><p><strong><em>Alors comment les entreprises pourront s’adapter á ce nouveau mode de travail ?</em></strong></p><p>Pour moi, ça va sûrement jouer dans la vélocité de l’équipe et donc avec le même budget projet on pourrait faire +25% (en moyenne) de ce qu’on faisait avant, mais ceci ne dépend pas que de l’IA mais de beaucoup d’autres outils comme Git, la documentation, l’automatisation, la bonne gestion du projet… et beaucoup d’autres aspects. <strong><em>On ne peut pas accélérer ce qui n’avance pas.</em></strong></p><p><strong><em>Alors pratiquement, qu’est-ce qui fait qu’une personne dit qu’elle gagne 10 ou 30 ou 80% quand elle utilise des IA génératives ? Je vais essayer de répondre en me basant seulement sur mon vécu ?</em></strong></p><p>— 100% — 80% : Une personne qui veut vous vendre que les développeurs vont perdre leur boulot à cause de l’IA pour qu’il y ait plus de vues va écrire un prompt genre « crée-moi une application de gestion des tâches ». Là, avec du <strong><em>Vibe Coding</em></strong>, il va demander à l’IA d’adapter son code jusqu’à avoir le résultat qu’il souhaite sans écrire la moindre ligne de code. <strong>Il va gagner 100% du temps</strong>. mais déjà ceci est possible parce qu’on a créé l’application From Scratch et parce que l’application n’est pas trop complexe.</p><p>Après, ça reste totalement possible et pratique de créer une page web avec du Vibe Coding — sans écrire une seule ligne de code — pour par exemple un POC ou même pour la production, mais il faut faire attention à vérifier tout ce que l’IA a généré. Et là, on retourne encore au point de départ : pour ce faire, il faut un minimum de connaissances et d’expertise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*UlypcrpgMsTEoNsRgDIbkQ.png" /><figcaption>Generate 100% application with IA</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yhgFHcT36J3Vj4vAU_vPZw.png" /><figcaption>Vibe Coding : Ask IA to cahnge the color of template from blue to red</figcaption></figure><p>— 30% — 60% : Dans un projet, l’IA peut être d’une grande aide pour ajouter du code Java par exemple qui va trier les utilisateurs par nombre de ventes. Mais moi, dans un contexte entreprise, je préfère créer la fonction, mettre les paramètres en entrée, par exemple un ArrayList&lt;Users&gt; avec la liste des ventes de chaque user (user-&gt;ventes) et puis demander à l’IA de générer le code. Comme ça, je suis sûr que l’IA ne va toucher que le code de la fonction. On peut lui demander de faire plus (avec le mode édit ou le mode Agent, Copilot par exemple peut toucher à plusieurs fichiers dans le projet), mais franchement je préfère me comporter comme ça avec un assistant avec cette belle stat de 33% d’erreurs.</p><p>— 10% — 30% : Une personne dans un contexte d’entreprise qui a son propre écosystème informatique peut bien sûr profiter de l’IA mais en la combinant avec ses connaissances et son expertise. Par exemple, si tu veux créer de l’authentification avec LDAP en utilisant un composant développé en amont au sein de ton entreprise, tu vas devoir déjà savoir utiliser ce composant et comprendre ce qu’il y a derrière, par exemple le protocole OAuth2, et puis l’IA pourrait t’aider à écrire ce code. Mais ceci dit, dans ce use case l’intégration est beaucoup plus compliquée que le code source et donc moi je dirais qu’en fonction de ton expertise tu peux gagner de 10 à 30% de temps de développement, mais l’IA pourrait te faire perdre ton temps si tu commences directement à demander à l’IA de résoudre ton problème sans comprendre le contexte. Et j’ai pas mal de fois assisté à des personnes qui sont restées des heures devant ChatGPT sans aucun résultat.</p><p>— Moins de 10% : Il y a des domaines comme le Data Engineering où l’IA n’a pas été très adoptée. Demandez à des gens qui font beaucoup d’Apache Nifi si l’IA les a vraiment aidés ou les a vraiment aidés à gagner du temps, ou encore dans l’architecture des pipelines et de l’infrastructure des données qui est mon domaine. Là aussi, je pense que l’IA jusqu’aujourd’hui n’aide pas beaucoup parce que tout dépend du contexte. Et puis il ne faut pas oublier que l’IA est basée sur un calcul statistique, donc si tu demandes à une IA s’il est judicieux de faire du DBT ou du Apache Kafka dans un use case, elle peut te dire oui même s’il y a de meilleures solutions. Et puis quand tu passes beaucoup de temps dans le monde de l’architecture, on ne demande plus si un outil ou une technologie peut répondre à un besoin, mais plutôt est-ce que c’est ce qu’il y a de mieux pour répondre à ce besoin, parce que finalement on peut presque tout faire avec n’importe quel outil.</p><p>Par exemple, et pour sa simplicité, tout le monde utilise MongoDB, et une IA peut te suggérer de faire du MongoDB pour une application de gestion de stock, parce que MongoDB va te ramener plus de fluidité. Mais c’est du DocumentDB qui n’est vraiment pas fait pour faire des jointures et donc dans ce cas il faut des bases de données relationnelles. La grande question qui se pose ici : comment on estime le temps perdu suite à un mauvais choix suggéré par une IA ? (même si dans ce cas ChatGPT te dit oui tu peux, mais il y’ a d’autres choix, il faut de la connaissance pour pouvoir choisir entre les options qu’il t’offre avec les avantages et inconvénients).</p><blockquote>Aujourd’hui, tout le monde ne parle que des gains quand on utilise les IA Gen, parce que c’est un sujet tendance. <strong>ET LES PERTES</strong> ?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VrD_5Egv3bOkzv5vP2KBtA.png" /></figure><p>Un autre point que je veux évoquer dans cet article, et c’est un point que j’ai déjà abordé avec un grand nombre de personnes avant même l’apparition des LLM, c’est que la grande complexité des nouvelles plateformes ne réside plus dans l’écriture du code source. Prenons l’exemple de Netflix avec 1000 microservices donc 1000 mini applications distinctes, chaque microservice contient quelques milliers de lignes de code (LoC). Là, le grand problème est la gestion de ces microservices, de l’auto-scaling, de l’automatisation des déploiements, la documentation, les choix d’architecture, les stratégies open source, la stratégie Cloud/OnPrem/hybrid…</p><p>Est-ce que vous pensez que si je vous donne juste le code source de Netflix, vous pourrez avoir la même plateforme, ou le même résultat ? C’est clair que c’est non. Il n’y a pas que la génération du code dans le monde d’un développeur, c’est une partie de son travail, et les IA Gen ne peuvent pas tout faire à la place d’un développeur, mais ces IA Gen pourront aider dans les tâches qu’elles peuvent faire si elles sont bien exploitées par le développeur.</p><h3>En ce qui concerne les compétences, est-ce que les LLM vont aider ou non à gagner en compétences ?</h3><p>Il y a quelques années, on remarquait qu’il y avait une grande différence technique entre quelqu’un qui avait appris en se basant sur des connaissances de base tirées de livres et des gens qui construisent leurs connaissances sur des articles, StackOverflow ou des vidéos YouTube. Celui qui se base sur des livres a généralement une vision globale du contexte de la technologie et donc tu n’attends pas forcément de voir ou de rencontrer le problème pour le découvrir et essayer de comprendre le résultat.</p><blockquote>Je prends un petit exemple : quand tu apprends à écrire du Java en te basant par exemple sur un article, et tu veux créer une liste avec unicité des éléments, il se peut qu’au début tu vas créer une LinkedList et à chaque fois que tu voudras ajouter un élément tu vas vérifier si ce dernier existe déjà sur la liste. Mais si avant tu avais pris le temps de lire et de comprendre toutes les structures de données avant de te lancer, tu vas sûrement penser à utiliser un Set au lieu de la List, parce que tu saurais que le Set va te garantir l’unicité de tes éléments.</blockquote><p>Dans l’ère des IA génératives, une nouvelle génération de développeurs a vu le jour : ceux qui ne connaissent que les IA Gen et trouvent très embêtant d’aller chercher sur StackOverflow ou même d’y poser une question et d’attendre la réponse. Ils n’aiment pas non plus devoir expliquer en détail leur problème pour avoir une réponse correcte. Au moment où j’écris cet article, c’est très problématique car en faisant ça, la courbe d’apprentissage devient beaucoup trop lente.</p><blockquote>Cette dépendance excessive aux LLMs peut paradoxalement nous faire perdre un temps considérable, <strong>difficile à quantifier,</strong> d’autant plus que les études actuelles se focalisent uniquement sur le temps gagné sans mesurer le temps perdu.</blockquote><h3>Si on part du principe qu’à partir de maintenant les IA vont prendre le rôle de créer et de développer, nous les humains on va commencer petit à petit à perdre nos compétences de dev (skill decay). Mais du coup, qui va pouvoir maintenir ces applications quand il y aura un problème majeur ?</h3><p>Imaginons qu’on a créé une application de gestion d’un supermarché avec une IA. Toute l’application est faite par une IA Gen avec du vibe coding, on est bien satisfait du résultat. On ajoute par la suite des algorithmes ML pour savoir où mettre chaque article avec une IA, on a ajouté un algo ML pour cibler les gens et lancer des offres directement qui vont intéresser les gens. Après deux ans, on s’aperçoit que les algorithmes se comportent d’une manière très raciste, et on souhaite changer les règles du jeu manuellement ou au moins comprendre pourquoi ce comportement.</p><p>La question qu’on doit se poser ici, est-ce que c’est responsable de donner autant de puissance et de contrôle à une IA ? Même si on arrive à un moment où c’est possible qu’elle soit autonome, est-ce qu’on doit laisser à l’IA le droit de décider, le droit de classifier ? Est-ce qu’on va laisser à l’IA le pouvoir de décider qui va-t-on prioriser pour les opérations chirurgicales ou à qui on va accepter de donner un crédit et à qui on va refuser sans même comprendre le pourquoi ?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49bde255b1c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GIT From a code versionning to a projet management tool]]></title>
            <link>https://medium.com/@maliani.zakaria/git-from-a-code-versionning-to-a-projet-management-tool-2831f99c9b48?source=rss-5c8c0375ac7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2831f99c9b48</guid>
            <category><![CDATA[scrum-master]]></category>
            <category><![CDATA[agile-methodology]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Zakaria DRISSI EL MALIANI]]></dc:creator>
            <pubDate>Sun, 08 Jun 2025 22:25:00 GMT</pubDate>
            <atom:updated>2025-07-06T14:15:10.354Z</atom:updated>
            <content:encoded><![CDATA[<p>We’re all talking about AI tools like GitHub Copilot and how they can help teams work faster. But that’s not what this article is about. I want to talk about something else — how <em>HIT (Human-Intelligent Tools)</em> can improve productivity across teams in a more practical and human way.</p><p>Before talking about fancy tools and AI, I think we need to go back to something simple but powerful: Git. Most developers use it every day, but not all teams use it in a way that really helps them work better together. Used well, <em>Git</em> can make a big difference in how a team works and how fast they move.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*8g6O0QR2JugP-nng" /></figure><h3>Git Can Be a Great Project Management Tool</h3><p>Using Git doesn’t just start when a developer writes code — it starts earlier, from the project management side. For example, when a task is created in JIRA (a widely used project management tool), it can be linked to a Git branch. That’s why project managers should also know the basics of Git. It’s not just a developer tool — it can help the whole team work better. The problem with tools like JIRA is that they’re declarative, you have to update them manually, and sometimes they don’t show what’s really going on. Git, on the other hand, shows the real work: branches, commits, merges. In agile and lean environments, this real-time visibility becomes crucial for rapid iteration and eliminating waste from manual status updatesIt reflects what’s actually happening in the project.</p><p>More technically how to use effeciently Git :</p><p>In Git we talk about local and remote repository, in this section we talk only about the local repository.</p><h3>Git : The local repository</h3><h4>Don’t Forget the Staging Area</h4><p>In a local Git repository, there are three main parts: the working directory, the staging area, and the local repository (where commits are stored). In my experience, many developers forget about the staging area, even though it’s an important step in the Git workflow.</p><p>Here’s how it usually works: when you change a file, you’re working in the <em>working directory</em>. The file is modified, but not yet ready to be committed. To prepare it, you run <em>git add file.txt .</em> This moves it to the <em>staging area, </em>a kind of preparation zone. Then, when everything is ready, you run <em>git commit -m &#39;git message&#39; </em>to move the changes from the staging area into the <em>local repository</em>.</p><p>This staging step gives you control. You can choose exactly what to include in a commit, even if you’ve changed multiple files. It’s a good habit to use it well — it keeps your commits clean and your history easier to read.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*niUsju9jdql17lDi9tV2Tg.png" /><figcaption>The 3 stages of local repository</figcaption></figure><blockquote>Lesson 1 — Don’t Use git add . Without Thinking:</blockquote><blockquote><em>One common mistake is using </em><em>git add . or </em><em>git commit -am ‘commit message’right away, without checking what’s really changed. It might seem fast, but it often adds files you didn’t mean to include. Before running that command, always do a </em><em>git status. It shows you what’s been modified, deleted, or newly created.</em></blockquote><p><em>Then, choose the files you really want to add to the staging area — ideally, the ones related to the JIRA User Story you’re working on. This helps keep your commits clean and focused on one task. It also makes it easier for others to review your work later.</em></p><h4>The Commit: A Physical Action, but Also a Functional Step?</h4><p>A commit in Git is something concrete , it records a snapshot of your code. But beyond that, it also represents a step in the project. And this “step” can vary depending on how the team works.</p><p>For example, some teams ask developers to push their code to the remote repository every day, others only once a week. That’s fine — it depends on the project. The important thing is that when you merge your work into the main branch (through a pull request or merge request), you can group all your commits into a single, clean step, with a clear and useful message. This makes the Git history easier to understand and helps everyone follow the progress of the project.</p><p>That’s why the commit message is also very important. It’s not just a formality — it helps describe what was done at each step of the project. Over time, these messages tell the full story of the development. And when it’s time to prepare a release, they’re super useful to build clear and complete release notes.</p><blockquote>Lesson 2 — The commit message is not formality:</blockquote><blockquote><em>Don’t rush your commit messages. A clear message helps others (and your future self) understand what was done and why. Try to describe your changes </em>functionally<em>, not technically. You can even take inspiration from the JIRA User Story you’re working on. Here’s a simple format you can follow:</em></blockquote><blockquote><em>[JIRA-123] As a user, i must reveive a validation mail after my registration</em></blockquote><blockquote><em>This way, each commit is linked to a business need and easy to trace. Over time, it really helps when reviewing history or building release notes.</em></blockquote><h3>Git : The remote repository</h3><p>The remote repository is where all developers collaborate. It’s the medium through which they communicate. But to communicate effectively, they must deeply understand the tool that makes this possible.</p><p>Git was created by Linus Torvalds ,the same person who created Linux, because he needed a tool that would allow efficient collaboration. Ironically, Torvalds has often said he’s not great at interpersonal communication, so he built Git to <em>let the code do the talking</em>. Git became the ultimate communication channel for developers, not through meetings or messages, but through commits, branches, and pull requests.</p><p>In this phase, we go beyond the local playground and enter the shared space, the remote. Understanding how Git handles remotes is key to contributing, reviewing, and syncing changes with a team. It’s not just about git push and git pull. It’s about understanding how teams collaborate, align, and avoid chaos.</p><h4>Systeme de branches</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*ekHlzVU_p_UpS6FVF7fbDQ.png" /><figcaption>Branching system</figcaption></figure><blockquote>You can tell a lot about how mature a project is just by looking at how the team manages their Git branches. In this chapter, I’ll show you how.</blockquote><p>In the graph below, you’ll see different types of branches: <em>master (or main)</em>, <em>develop</em>, <em>feature</em>, <em>release</em>, and <em>bugfix</em>. Each one has a clear purpose and helps organise the work on the project.</p><h4><em>Feature branch</em></h4><p>Let’s start with the feature branch. We usually create one when we start working on a new task or user story from Jira. For example, imagine we have a Jira ticket with the <em>ID 14</em> and the description: “<em>As a platform admin, I want to create a new user</em>.”</p><p>When we move this ticket to “<em>In Progress</em>” we create a new Git branch to work on it. We call it: feature/14_addUser</p><h4><em>Develop branch</em></h4><p>The develop branch is where we merge all the work of the team and test it before moving to production. Generally, when we move a JIRA ticket to <strong>“</strong><em>In Progress</em><strong>”</strong>, we create a <em>feature branch</em>. Once the work is completed, we open a <em>Merge Request (MR)</em> or <em>Pull Request (PR)</em> to propose integrating our changes back into <em>develop</em>. This is where the collaborative process truly begins!</p><p>=&gt; <strong><em>So, what is a Merge Request / Pull Request?</em></strong></p><p>A <strong>Merge Request (GitLab)</strong> or <strong>Pull Request (GitHub)</strong> is a way to tell others about changes you’ve pushed to a branch in a repository. Once submitted, it lets team members <strong>review the code (peer review), Discuss potential modifications, run automated tests and checks, Approve the changes before merging into the main codebase (like </strong><strong>develop or </strong><strong>main)</strong></p><blockquote>It’s an essential step in modern Git-based workflows for ensuring code quality, collaboration, and traceability.</blockquote><p>The main branch should always reflect the current version of the application in production. This means that if a bug is reported in the production environment, we must be able to trust that the code in the main branch exactly matches what is deployed on the production server.</p><p>To ensure this consistency and stability, no one is allowed to push directly to the main branch. Instead, developers push their changes to feature branches and open merge (or pull) requests to the develop branch. Once the CI/CD pipeline succeeds and the code is reviewed and approved, the changes are merged into develop.</p><p>At the end of each sprint, a release process is initiated: a <em>merge/pull</em> <em>request</em> is created from develop to main. If the pipeline passes and the reviewers validate the changes, a <em>release note</em> is generated, and the code is merged into main.</p><p>Finally, a <strong>c<em>ontinuous deployment (CD)</em></strong><em> </em>pipeline is triggered to automatically deploy the latest version from main to the production environment.</p><h4><em>Releases</em></h4><p>At the end of each sprint, we must create a release that includes all the features developed during that sprint. A release represents a specific version of the application, a milestone in your development journey. To give meaning to this version, we create a release note that summarizes the changes, enhancements, and fixes included. This note helps everyone understand what was delivered in that particular release.</p><p>In many cases, release notes can be generated automatically using the commit messages associated with the features included in the sprint, especially if a conventional commit format is followed.</p><p>It’s important to note that we are not required to deploy every release to production. However, when we do decide to deploy a release, we create a version <strong><em>tag (e.g., v1.0.3)</em></strong>. This tag is not just a label, it often triggers the production deployment pipeline. In most <strong><em>CI/CD</em></strong> setups, the deployment to production is not initiated by the merge into the main branch, but rather by the creation of this tag.</p><p>The version tag also plays an important role in rollback strategies. For example, if we deploy a new version (e.g., v1.0.3) and encounter issues in production, we can quickly roll back to the previous stable tag (e.g., v1.0.2) and redeploy it. This ensures that the production environment remains stable while we investigate and fix the problem.</p><h4><em>Bugfix</em></h4><p>In the lifecycle of any project, we inevitably encounter bugs. Some of these bugs are not critical and can wait until the end of the sprint. When we encounter such bugs, we follow a structured process: 1- We create a bug ticket in Jira. 2- We create a new <strong><em>bugfix branch</em></strong> from the develop branch. 3- We implement the fix and then open a merge/pull request targeting develop. 4- After the usual code review and successful pipeline execution, the fix is merged.</p><p>At the end of the sprint, this bug fix will be included in the release along with the other features developed during the sprint.</p><h4><em>Hotfix</em></h4><p>Sometimes, a critical bug is discovered in the production environment that cannot wait until the end of the sprint. In such cases, we need to act quickly to restore the system. This is where the hotfix process comes in.</p><p>Unlike regular bug fixes that start from the develop branch, a <em>hotfix branch</em> is created directly from the <em>main branch</em>, since main reflects the live production code. We apply the fix in this branch, and once the issue is resolved, the changes are merged directly back into main to trigger the production deployment as quickly as possible.</p><p>In this situation, business continuity takes priority over proces, so we intentionally bypass the usual flow that involves <em>develop</em>. However, it’s critical not to forget the next step: the same <em>hotfix</em> must also be merged back into the <em>develop branch</em>. This ensures that the <em>bug fix</em> will be included in the next sprint’s release and prevents the issue from reappearing in future deployments.</p><blockquote><em>Lesson 3 — Git can be help in project management:</em></blockquote><blockquote>Understanding the meaning of the branche just by looking at its name can let Even a project manager or a Scrum master quickly see what´s really happening in the project.</blockquote><blockquote>That´s not always the case with the Jira Dashboard, which is more of a decalarative tool, it shows what people say they´re working on, but not necessarily what is actually going on in the code</blockquote><h3>Git Merge or Git Rebase</h3><p>Throughout this article, we’ve talked a lot about merging branches, for example, merging a <em>feature branch into develop</em>, or <em>develop into main</em>. But if you’ve used platforms like GitLab or GitHub, you’ve probably seen that sometimes you’re prompted to <strong>rebase instead of merge.</strong></p><p>At this point, things get interesting, because it turns out that very few people truly understand the difference between merging and rebasing, and when to use each one.</p><h4><strong><em>Merge</em></strong></h4><p>Suppose you have two branches: <em>A (your feature branch) </em>and <em>B (develop)</em>. When you merge B into A, Git brings all the changes from B into A and creates a new merge commit that reflects the combination of both histories.</p><p>This can be useful to keep the full history of the project, but it has a drawback:</p><p>Your branch A now contains not only your own changes (<strong><em>which are related to a specific User Story in Jira</em></strong>), but also changes from others in B.</p><p>This pollutes your branch’s history, and if you later open a <em>Merge Request</em> for A, it becomes harder to review because it includes unrelated commits from B</p><h4><strong><em>Rebase</em></strong></h4><p>Rebase, on the other hand, replays your changes on top of the latest version of another branch, like develop. It rewrites your commit history so that your branch appears as if it was created from the latest version of develop, without bringing all its intermediate commits into your branch.</p><blockquote><em>It’s like saying: “Take my changes and reapply them as if I had started working on the latest version of develop.”</em></blockquote><p>This keeps your branch clean, focused on your User Story, and makes the commit history easier to understand.</p><blockquote><em>Lesson 4 — Do not join a project team without understanding the difference between Merge and Rebase:</em></blockquote><blockquote><em>Before joining a project team, it’s crucial to understand the difference between </em><strong><em>merge</em></strong><em> and </em><strong><em>rebase</em></strong><em>. Without this knowledge, you might unintentionally make mistakes that could affect the source code and potentially cause issues for the entire team.</em></blockquote><h4>Conclusion</h4><p>Git is like magic when you know how to use it well. In this article, I shared 4 simple but important lessons to help you avoid mistakes and work better with your team. Just like AI helps us be more productive today, using Git the right way can really improve the way you work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2831f99c9b48" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolution of a Data Engineering Project: From Shell Scripts to Cloud Computing Architectures.]]></title>
            <link>https://medium.com/@maliani.zakaria/evolution-of-a-data-engineering-project-from-shell-scripts-to-cloud-computing-architectures-5a145c4be0fd?source=rss-5c8c0375ac7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a145c4be0fd</guid>
            <category><![CDATA[information-technology]]></category>
            <category><![CDATA[evolution]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Zakaria DRISSI EL MALIANI]]></dc:creator>
            <pubDate>Tue, 25 Mar 2025 22:18:04 GMT</pubDate>
            <atom:updated>2025-03-25T22:18:04.158Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I tell you the fascinating story of the development of a Data Engineering project, which began with a simple request regarding the transfer of web browsing files to generate dashboards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CA3X_3jyrr5nF4DMkcWtJg.jpeg" /></figure><h3>Prehistory: The Dawn of Data Transfer, Our First Steps with SCP and SSH</h3><p>It all started with a simple need: transferring data between two servers. Like early humans discovering their first tools, we turned to SCP — it just made sense. With a single command, we could copy files over the network, straight from the shell.</p><p>But doing it manually every time wasn’t going to cut it. So, we wrote a script to automate the process, using SCP to move files between machines. To avoid typing the password each time, we set up SSH keys — making the transfers smooth and secure.</p><p>With that in place, we used crontab to run the transfers automatically every hour. Simple, efficient, and a solid first step.</p><blockquote><em>It was simple but effective. Sometimes,</em> a basic setup with essential tools is more than enough to meet the needs — no need for expensive or complex solutions.</blockquote><p>Little by little, transfer and data transformation (ETL) requests started exploding, making project management increasingly complex. Maintaining the systems became a real challenge, and that’s when the need to rethink our architectures emerged. This marked the beginning of a new era — one where more scalable and adaptable solutions became essential.</p><h3>Bronze Age: The Rise of Structured Tools</h3><p>As needs evolved, we moved away from rudimentary solutions and entered a <strong>Bronze Age</strong>, where more robust and sophisticated tools emerged. This transition was marked by the adoption of <strong>Hadoop</strong>, a solid and durable platform, and <strong>Apache Pig</strong>, which introduced <strong>Pig Latin</strong>, a scripting language that made it easier to process massive amounts of data efficiently.</p><p>The process remained structured but still somewhat <strong>handcrafted</strong>:</p><ul><li><strong>Shell scripts</strong> transferred raw data to <strong>HDFS</strong>.</li><li><strong>Apache Pig</strong> read the data, processed it using <strong>MapReduce</strong> (filtering, aggregations, etc.), and saved the results back into <strong>HDFS</strong>.</li><li><strong>Shell scripts</strong> then took over again to transfer the processed data to other teams.</li></ul><p>It worked, but it was clear that we were still relying on a lot of manual scripting to keep things running.</p><p>This period marked the beginning of a more structured approach, even though many steps were still manual and struggled to keep up with growing demands. However, it laid the foundation for future industrialization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*0ZNyTdQtlMx8P4MEvPMxUw.png" /><figcaption>Figure 1: The Integration of Hadoop for Distributed Storage and MapReduce Processing</figcaption></figure><p>It’s true that with this architecture, we were able to solve the data transformation problem, but Apache Pig could be tricky — especially when creating large scripts with <strong>UDFs (User Defined Functions)</strong>. These functions, written in Java, were necessary for more complex transformations, but they added an extra layer of complexity and made maintenance harder.</p><p>Moreover, another major challenge persisted: data transfer and integration. The <strong>Shell scripts</strong> used to move data between systems, while functional, lacked reliability and traceability. Errors could easily occur, making monitoring and troubleshooting tedious. On top of that, there was a real need for real-time processing, something that was naturally impossible with Shell scripts or Apache Pig.</p><p>It was in this context that the introduction of <strong>Apache NiFi</strong> marked a turning point. Designed to simplify data transfer in a visual and automated way, NiFi offered a more reliable, scalable, and traceable solution.</p><h3><strong>Iron Age</strong>: Apache NiFi at the Heart of Our Architecture</h3><p>We began a major migration, gradually moving away from our handcrafted Shell scripts in favor of a promising new tool: <strong>Apache NiFi</strong>.</p><p>The adventure began with excitement. Drawn to its intuitive visual interface and apparent simplicity, we quickly adopted this powerful tool to orchestrate our data flows, convinced we had found a solution that was both flexible and efficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*giU43Fb0qfANnSMl3PKP2Q.png" /><figcaption>Figure 2: Apache NiFi UI showcasing its ease of use.</figcaption></figure><p>But as we explored further, we realized that this power came with a price. Behind its simplicity, it required a good understanding of best practices, performance, and resource management.</p><p>Just like in the Iron Age, where artisans had to refine their techniques to master a tough but demanding metal, we had to strengthen our expertise to fully harness NiFi. One of the first challenges we faced was isolating <strong>Dataflows</strong>: when one flow experienced heavy load, it could impact the other flows in the same cluster, making resource management more difficult. Additionally, while low-code makes pipeline design easier, a simple mistake in handling could have major consequences on business data. Lastly, automating deployments and rollbacks turned out to be more complex than expected, forcing us to adjust our practices to ensure efficient version management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*1YBcP3pFl5ckSVCTJbEjYg.png" /><figcaption>Figure 3: Dataflow Isolation in the Cloud to Prevent One Flow’s Load from Disrupting Others in the Same On-Prem Cluster.</figcaption></figure><p>However, we quickly realized that some real-time constraints required additional solutions, especially for on-premise processing. <strong>Apache Spark</strong> was then added to our architecture to meet the distributed computing needs on our local infrastructure. While Spark is based on micro-batching rather than strict real-time processing, it allowed us to effectively manage large volumes of streaming data with acceptable latency.</p><p>Moreover, adopting NiFi also opened the doors to <strong>cloud computing</strong>. With its modular and distributed approach, NiFi became a key ally in integrating our data flows into cloud environments, making scalability and system interconnection much easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*HUg4kLQqJcvpVwtTy_XBSw.png" /><figcaption>Figure 4: Hybrid Architecture with Apache NiFi as the Orchestrator.</figcaption></figure><h3>The Classical Antiquity of Data Engineering: Towards Modern and Distributed Architectures.</h3><p>This period is marked by a major migration to GCP Cloud, much like the bold journeys of ancient explorers searching for new lands and resources. Every step of this migration — from designing the architecture to defining labels — mirrors the challenges those pioneers faced. Just as they had to set rules to navigate uncharted waters, we must establish standards to manage our digital resources.</p><p>The cloud migration took place in several stages: designing the architecture, defining resource naming conventions, assigning labels, managing access rights, setting project creation rules, automating deployment with Terraform, and finally, implementing FinOps dashboards to monitor costs.</p><p>In our drive for innovation, we took our use of Apache NiFi to the next level. We adopted NiFiKop, an open-source Kubernetes operator developed by Orange, to manage our NiFi deployments. This approach allowed us to create dedicated NiFi clusters for each data flow that needed isolation, whether due to high data volume or its critical nature.</p><p>NiFiKop has transformed our dataflow management by automating deployment across both our on-premises clusters and Google Kubernetes Engine (GKE) on Google Cloud Platform (GCP). This flexibility has given us greater scalability and resilience while significantly simplifying our data engineering operations.</p><p>This cluster isolation strategy ensures that each critical data flow runs optimally without being impacted by the performance or issues of other flows. As a result, we have built a robust and flexible data processing architecture, ready to meet the challenges of the modern big data era.</p><p>The re-architecture of our platform made it easy to add new components, enabling seamless integration and experimentation with modern tools like DBT (Data Build Tool). This tool, designed for data transformation and lineage tracking, was deployed on Kubernetes via GCP (GKE), with Airflow for orchestration and BigQuery for storing transformation steps.</p><p>But not everything was perfect… Like the early Arab scholars deciphering the stars, as we moved forward, we also uncovered new mysteries.</p><p>Our architecture met our needs, but it was too complex. Finding the right talent became a real challenge — mastery of Java, GCP, Docker, Kubernetes, Hadoop, Spark, GitLab CI, and Terraform was required. These skills were rare and in high demand, making hiring difficult.</p><p>We had to rethink our approach. A new era of Data Engineering was taking shape — easier, more accessible, yet just as powerful.</p><h3>The Industrial Era of Data Engineering: Towards Data Engineering As A Service</h3><p>The goal of this era is to create a product as a service where a data enthusiast could easily create data pipelines without needing a large team of highly specialized Data Engineers. It also aims to reduce the cost of creating data flows, as it’s possible to create hundreds of data flows when factoring in maintenance and operational costs.</p><p>In this new era of Data Engineering, the goal is to make creating and managing data pipelines an easy service, reducing the need for highly skilled experts and lowering costs.</p><p>By industrializing Data Engineering, we shift from a manual model to a scalable and optimized one, where creating data pipelines becomes as simple as deploying a SaaS service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*wB5kGipwgkaWdQKnl5S5QA.png" /></figure><h3>Conclusion</h3><p>The evolution of data engineering has been rapid, moving from basic Shell scripts to sophisticated cloud computing architectures. This transformation has enabled companies to handle growing data volumes and leverage them to gain valuable insights.</p><p>Today, the data engineering landscape is continuously evolving. New tools like Apache Iceberg, Trino, and Apache Ozone, as well as platforms like Dataiku and Rakuten, offer advanced features for data processing and analysis. Additionally, concepts such as data virtualization and multi-cloud strategies are opening up new possibilities for businesses.</p><p>In conclusion, data engineering is an exciting and ever-evolving field. Companies that adapt to new technologies and approaches will be best positioned to harness the power of data and achieve their goals.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a145c4be0fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Computer science skills of 2024]]></title>
            <link>https://medium.com/@maliani.zakaria/computer-science-skills-of-2024-8f369611c498?source=rss-5c8c0375ac7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f369611c498</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Zakaria DRISSI EL MALIANI]]></dc:creator>
            <pubDate>Sun, 21 Jan 2024 12:19:18 GMT</pubDate>
            <atom:updated>2024-02-13T14:24:59.949Z</atom:updated>
            <content:encoded><![CDATA[<h3>In my 12 years of IT experience, these are the key skills that every IT specialist should possess in 2024</h3><p>At the start of the year 2024 i want to share my vision about what a computer scientist should know in 2024. It’s important to note that I am not advocating to master all of the technologies I mention, but at least know the context of their use.</p><p>This article can be useful for students who want to know the skills needed for the job market, for freshly out of school who are seeking internships or jobs, for professionals who have gaps and want to fill them, for someone who desires to completely alter their career plan, but also human resources who may use it as a reference to evaluate candidates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AiRrgZTxD0hfG16p0aCVzw.jpeg" /></figure><p><strong><em>Algorithmic</em></strong> : I heard a lot of people laughing about taking exams at universities on paper instead of using machines, I disagree. Algorithmic knowledge is requisite before starting coding in any program. I see lots of use cases that need algorithmic skills, not programming skills. For example, to perform a dichotomic search, especially in data related problems.</p><p><strong><em>Object Oriented principles</em></strong> : These principles need to be mastered in studies, and it’s rare to see someone improve these skills in its professional career. The necessity to understand these concepts may not seem super important when working on applications based on Angular, Spring, or PHP. We follow tutorials and we get results, it works so we are happy. But this can be very problematic when we integrate a team that works on complex use cases and tries to follow good practices to get optimised results. Also when using design patterns which are best practices used to solve common design problems, and are based on object-oriented concepts. Understanding these patterns and when to apply them can greatly improve the design and architecture of a system</p><p><strong><em>Programming langage</em></strong> ( Java / Python / JavaScript ) : When you master Algorithmic and programming paradigm which refers to a method or style of programming that dictates the structure and way in which programs are constructed, you can easily learn any programming langage or all of them, especially now with generative AI, we can easily switch from one language to another provided you have a very good foundations. Among these paradigms, I can mention Object-Oriented Programming (OOP), functional programming, procedural programming, reactive programming, and so on. All frontend developers use reactive programming when calling APIs with RxJS, and many of them don’t even realize that they use that paradigm, They simply used it because they found it in an article or on StackOverFlow. This is a problem because it can cause a lot of performance issues in the application.</p><p>For me, this is the most important programming language this year :</p><ul><li><strong><em>JavaScript</em></strong> : JavaScript is widely used, and will continue to be, in front end development with frameworks like Angular, React, in backend development with framworks like Express, Sails, … , also when we use cloud architecture we can write the code behind a cloud function.</li><li><strong><em>Java</em></strong> : Lately, we all fell a degradation in the level of Java in favor of Python even if Java still the most used language in enterprises, I think because of the framework SPRING which is largely used to develop APIs and microservice applications architectures and also because of the coast of the execution of Java programs in the CLOUD which is much less than Python.</li><li><strong><em>Python</em></strong> : Python is listed among the top programming languages in the 2023 ranking. Its a high popularity is attributed to its ease of use , adaptability, and a vast ecosystem of libraries and frameworks, making it a top choice in fields like artificial intelligence, machine learning, data science, and web development.</li></ul><p>I hear a lot of people complaining that they’re not proficient in coding. The binary code base is represented by 0 and 1, We then add layers. It’s very complex to understand the final layer of coding, which is classes, objects, libraries, frameworks, etc, without a good comprehension of the other layers before. I consider myself fortunate to<strong> </strong>have had the opportunity to study at Mohammed V University<strong>,</strong> where we were taught these concepts in great<strong> </strong>detail, Unlike many courses that emphasize learning by doing and neglect fundamental concepts that enable developers to rapidly progress from beginner to experienced or expert. Some of theses layers are :</p><ul><li>The process of compilation which is crucial for Translation from High-Level to Machine Language, Error Detection, Optimization, Debugging, Security, etc.</li><li>Operating system wich is vital for Resource Management, Memory Management, Process Management, Security and Access Control, Performance Optimization. Theses concepts enhances a programmer’s ability to write efficient, effective, and secure code</li></ul><p>I can’t end this part without talking about the clean code concepts. The goal is to make the software easier to write, read and maintain. Readability, simplicity, refactoring, testing, consistency, documentation and comments, error handling, code smell, and other key concepts are crucial for long-term project success and team collaboration.</p><p><strong><em>HTML /CSS</em></strong> : It’s a classic, yes. For many people, it seems very simple, but in reality, it’s complex. Try taking a simple HTML page and adapting it to smartphones and tablets formats. You will see that it is not a piece of cake. However, it is necessary for all front-end applications (Mobile, Web, Desktop, TV, Chrome extensions, etc…).</p><p><strong><em>SQL / NOSQL : </em></strong>We all agree that DATA is money, but we need to store this data, many possibilities are available to us but how do we make the right choice ! That’s why we need to know to different types of databases and what each one is used for. There are two type of databases SQL and NoSQL :</p><ul><li><strong><em>SQL or relational databases</em></strong> : ideal for complex query-intensive processes, require predefined schema, SQL databases are vertically scalable, they support ACID property and they are well-suited for applications requiring complex transactions like banking systems, to do analytics dashboard and when data integrity is critical.</li><li><strong><em>NoSQL or Non relational databases</em></strong> : they can be document-oriented, key-value pairs, wide-column stores, or graph databases, They have dynamic schemas for unstructured data, which allows for more flexibility in storing different types of data, they are horizontally scalable, they do not support ACID which can affect data consistency, and are well-suited for large data sets and real-time web applications.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*l3b6RPs0Z3bJScIWTDHIrQ.png" /><figcaption>No SQL Databases</figcaption></figure><p><strong><em>Containerization</em> </strong>( Docker / Kubernetes ) : I can’t imagine a computer scientist in 2024 who does not master containerization and Docker, and it’s not just for devops developers, but also for backend and front-end developers, full-stack, data engineers, data scientists, etc …, so why it’s so important ? Firstly, when joining a development team, you have to install a development environment. Most of these environments are made with Docker, or more specifically Docker-compose, Secondly, the entire cloud computing world is based on containerization. We cannot use cloud services effectively without having a good knowledge of these principles, and finally docker plays a significant role in modern software architectures, including those of large-scale, high-availability services like Netflix, For me, the complexity of microservices architectures lies not only in the source code part but also in the containerization part, automation, orchestration, monitoring, and cloud computing.</p><p><strong><em>DevOps</em></strong> is a concept and practice that seeks to unite software development (Dev) and IT operations (Ops). The main goal is to improve collaboration between these two areas, speed up the delivery process, and increase software quality. It is crucial to know the various key principles that guide this practice, i will name a few :</p><ul><li><strong>Automation</strong>, the principle is to automate as much as possible, including deployment, testing, and continuous integration to reduce human errors and speed up processes, we can do this with Gitlab-ci or Jenkins.</li><li><strong>Infrastructure as Code (IaC)</strong> is a way to manage infrastructure through code to ensure consistency, reproducibility, and traceability. especially when we use cloud architecture and we want to have staging, preprod and prod environment, inconsistencies between environments can be caused by redoing the manipulations of the three environments using graphical interfaces.</li><li><strong>Continuous monitoring</strong> : By implementing monitoring and feedback mechanisms to quickly detect and respond to issues in production, and by collecting and analyzing performance metrics to continuously optimize the software delivery process.</li></ul><p><strong><em>Cloud Computing</em></strong> : I don’t think there are any companies left that don’t use at least one cloud service, it can be a software as service, a backend as a service, a Platform as a Service or Container as a Service. It’s common for people to use Backend as a Service platforms such as Firebase or Signal without being aware that they are using a cloud service, the fact that they can store user personal information on servers in another country without realizing it is problematic, for that and many other reasons, I think that every developer must have a deep understanding of cloud computing and cloud services. Also, many frontend developpers use SAAS services like Google MAP API, but if you don’t understand the concept of YOU PAY AS GO, you may be surprised by the amount of the bill.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*f8roLm_bT-hVOEId9Xc4Mw.png" /></figure><p><strong><em>Security</em></strong> : A computer scientist must have knowledge of security principles, such as code-secure principles, which are practices and principles to minimize vulnerabilities and protect against potential security threats. Some of these practices include : input validation, avoiding hardcoding sensitive information, using prepared statements and parameterized queries to avoid SQL injections, etc and devsecops methodology that include SAST which help to catch security issues early in the development process and DAST which provides additional assurance by testing the application in a runtime environment. Also, any computer scientist can consume or create APIs. In both cases, we have to know the oAuth2 protocol. Furthermore, computer scientists must know how to encrypt/decrypt data, the principle behind RSA and AES algorithms, and how to use Vault. At any time, we may need to encrypt or decrypt personal data.</p><p><strong><em>Data</em></strong> : Data can now encompass a huge scope. Nowadays, everyone (doctors, engineers, electricians, etc.) must know the idea behind artificial intelligence and machine learning, Particularly with this technological singularity and the use of tools such as Bard, ChatGPT, and others, it requires understanding at least to know how to choose between them. In my view, computer scientists need to have a more in-depth understanding of how data works, backend developers are responsible for ensuring the quality of data they store in their databases, each successful project generates a significant amount of data, also, the use of Generative AI tools can be very beneficial for developers, it can save them up to 30% of development time, developers can also use machine learning APIs like Cloud Vision, which allows developers to easily integrate visual detection features into applications.</p><p>There will be readers who think this is too much and wonder how we can master all these technologies.</p><p>To me, the secret is to create programs and then implement habits, such as reading 30 minutes a day, 182,5 hours in a year equals 26 days of work per year, to be programmed in the morning or in the evening, preferably in the morning before the start of the day to be sure that it will not be deprioritized in favor of excessively long days of work or study.</p><p><em>Step 1</em> : To gain an understanding of the principles without going into too much detail, you can easily find documentation through reading articles, books, videos on YouTube, or enrolling in courses platforms like Coursera.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*thuz0n6fvvYMzmKNzEM40w.png" /></figure><p><em>Step 2</em> : the step 1 will surely help to have a minimum of understanding which will help to create a more detailed program depending on the specialty of each person, the goal here would be to dive into the details of each topic, for example for a frontEnd developper it could be :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*iJVchCis7s4-Z7FgsZ55cw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f369611c498" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>