<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Florent Moiny on Medium]]></title>
        <description><![CDATA[Stories by Florent Moiny on Medium]]></description>
        <link>https://medium.com/@florent.moiny?source=rss-fe3c5e3137d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*PhGJsYpDaX9TAmmb.</url>
            <title>Stories by Florent Moiny on Medium</title>
            <link>https://medium.com/@florent.moiny?source=rss-fe3c5e3137d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:42:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@florent.moiny/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Value and execution are absolute]]></title>
            <link>https://medium.com/@florent.moiny/value-and-execution-are-absolute-4051a5d02105?source=rss-fe3c5e3137d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4051a5d02105</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[startup-lessons]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[Florent Moiny]]></dc:creator>
            <pubDate>Sun, 12 Feb 2023 17:10:09 GMT</pubDate>
            <atom:updated>2023-02-12T17:10:09.067Z</atom:updated>
            <content:encoded><![CDATA[<h4>Everything else is noise</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P9Dy_OL8eu4bTZVr" /><figcaption>Photo by <a href="https://unsplash.com/es/@jacksondavid?utm_source=medium&amp;utm_medium=referral">Jackson David</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>The startup life</h3><p>I know next to nothing about life. It was even more true when I joined my first company after university. It was a small startup in the adtech industry, roughly 25 people, all young and cool. The company had just raised its first round, and it was time to accelerate. I was hired as a Data Scientist to extract insights from the massive amount of data the company had amassed. I thought my job was to train machine learning models, write code in Python, and play foosball. I was so dead wrong.</p><h4>Life was fun</h4><p>I was the second Data team member, right after the Head himself, so everything was to be built from the ground up. It was exciting and challenging. I spent my time reading research papers on machine learning, realizing how little I knew, and writing code like a madman. I quickly realized that school failed to teach me many of the fundamental skills needed for my day-to-day job. So I learned. Every day. I learned to write production-grade code in Python and Scala, create Spark jobs, and orchestrate them with Airflow DAGs. I learned to rebase and handle conflicts in git. I learned how to build CI/CD pipelines. I learned to use AWS. I was getting torn to pieces in code reviews by senior members. Those hundreds of comments per pull request were my crash course on data engineering.</p><p>I loved it. I had my small bubble where I could learn and create at the same time, at my own pace, because I was under the Research and Development umbrella, because “things take time in data science,” and because I was junior. So a few months went by until I started noticing something different.</p><h4>I saw the pressure</h4><p>On the management, responsible for dozens of people.<br>On back-end engineers, responsible for delivering millions of ads 24/7 in less than 50 milliseconds.<br>On infra folks, responsible for maintaining a multi-million dollar infrastructure at all times.<br>On account executives, responsible for selling the solution to customers and bringing in money.<br>On founders, responsible for the vision and the survival of the company.</p><p>I saw them waking up in the middle of the night, dealing with failures and escalations. Coffee mugs shattered to pieces, people yelling at each other, and people being roasted for not making the targets. The mood swings were epic.</p><p>That’s when I understood.</p><h3>It’s about survival</h3><p>When you’re working for a startup, it’s all about survival. And to survive, you need two things: value and execution. Everything else is noise.</p><h4>Value</h4><p>Value is the most important thing to understand, especially for tech people, as we tend to focus too much on the tech side.</p><p>Companies exist for a single reason: to solve problems that matter. That’s it.</p><p>What programming language to use? What framework to use? What text editor to use? What architecture? Should we use machine learning? That’s irrelevant. At best, it’s an implementation detail.</p><p>Value comes from problems, pain points, and users. So first, we need to understand them. Then we need to create products that will make their lives easier. Finally, we need to do it fast. But there is a catch.</p><h4>Execution</h4><p>It’s all about pace. We must find the balance between shipping something perfect too late and shipping something unusable too early.</p><p>We might be developing the greatest machine learning model ever, but as long as it’s not hitting production, it’s useless. Historically, this is how IT projects ran. Projects lasting months or years with an extensive team of developers before hitting production. Except for rare cases, it’s 1. unnecessary, and 2. inefficient. If we ship something too late, we risk running out of money without anything to show.</p><p>And what about competition? Do we want users to go to another competitor?</p><p>If we ship something unusable too early, the user experience will be awful, and people will move away from the product, even if it gets better down the line. Reputation is at stake here. So we must be careful about what we ship and how we communicate.</p><p>So, what do we do? We iterate. We focus on the essential parts and ship them as soon as possible. We collect feedback, we improve, and then we add features. Rinse and repeat until the ROI is not attractive anymore. That’s why SaaS products have private and public previews. That’s why we will have to ship half-baked products with nasty code from time to time. Our ingestion pipeline might not be the fastest or the most cost-efficient, our ML model might not be the most accurate, and our backend might not be the most scalable, but that’s how we move fast and are agile. That is true Agile.</p><p>It sounds easy, but it’s not. Good iteration requires a different mindset. We must make tradeoffs and accept to ship sub-optimal code. We must embrace temporary chaos for long-term success. And it’s hard.</p><h3>Conclusion</h3><p>Instead of focusing on the tech itself, we must look at the bigger picture and understand what the problems are for our users. What are we trying to achieve at the business level? What is the value provided?</p><p>Ideas without proper execution are a waste of time. That’s why we need to iterate quickly, get user feedback, improve, and then pivot if required.</p><p>Finally, and this is probably the most challenging part, we must come to terms with tradeoffs and shipping sub-optimal code.</p><p>Interestingly, everything said so far applies to every company, not just startups. They simply have more money.</p><p>Until they don’t.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4051a5d02105" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Openly say that you don’t know]]></title>
            <link>https://medium.com/@florent.moiny/openly-say-that-you-dont-know-ea6faa2ad289?source=rss-fe3c5e3137d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea6faa2ad289</guid>
            <category><![CDATA[professional]]></category>
            <category><![CDATA[personal-development]]></category>
            <category><![CDATA[professional-services]]></category>
            <category><![CDATA[consulting]]></category>
            <category><![CDATA[work]]></category>
            <dc:creator><![CDATA[Florent Moiny]]></dc:creator>
            <pubDate>Thu, 02 Feb 2023 19:42:26 GMT</pubDate>
            <atom:updated>2023-02-02T19:42:26.650Z</atom:updated>
            <content:encoded><![CDATA[<p>That’s how you build credibility</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FmBPSXsfxEnxqSOu" /><figcaption>Photo by <a href="https://unsplash.com/@linkedinsalesnavigator?utm_source=medium&amp;utm_medium=referral">LinkedIn Sales Solutions</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>One of the best pieces of advice I’ve ever received as a professional.</p><h3>Situation</h3><p>You’re an expert, consultant, manager, or professional. Somebody asks you a question. You don’t know the answer, but you think you should know it. You’re uncomfortable. What do you do?</p><h3>Bad</h3><p>There are different bad behaviors, all leading to an instant lack of credibility.</p><h4>You hope for the best</h4><p>You have no clue about the answer, but you still answer. There are two possible outcomes.</p><p><strong>The answer is correct</strong></p><p>You got lucky. You now have a false sense of invincibility. You will keep doing it until you get an answer wrong, which brings us to the next section.</p><p><strong>The answer is wrong</strong></p><p>The customer will never trust you again unless you openly admit your mistake. Good luck with repairing the relationship.</p><h4>You lie and pass the hot potato to a colleague</h4><p>This is the worst because you broke two relationships at once:</p><ol><li>The customer will eventually realize you lied to them</li><li>You’re putting your colleague in a very uncomfortable position by forcing your point of view and desires</li></ol><p>That move will instantly backfire because your colleague will not want to help you anymore, will say bad things about you behind your back, will escalate this to their manager, and might very well tell the customer that you lied.</p><h4>You’re not sure, and you speak vaguely for the next 5 minutes</h4><p>This situation usually arises when you think you should know something but don’t, pushing you to say random things.</p><p>It’s a red flag 101. You think you managed your way out, but you didn’t. You’ve simply lost credibility after 30 seconds and respect after 60 seconds. The last 240 seconds are pure agony.</p><p>They won’t trust you, and they won’t like you. Please don’t.</p><h3>Good</h3><p>However uncomfortable it is, you must be open about it to create long-lasting credibility.</p><h4>If you have no clue: openly say so</h4><p>When you genuinely don’t know, be straightforward and openly say so. Stay professional, don’t downgrade yourself, don’t find excuses, simply state the truth and move on: “I don’t know. I will look into it and come back to you with the answer today/tomorrow/next week.”</p><p>Whether you’re supposed to know the answer doesn’t matter at that moment. Swallow the pill and learn from it.</p><h4>If you’re not sure: say why</h4><p>A slight variation from the previous situation. If you know precisely why you’re not sure, openly say it, share your knowledge, and follow up with, “but let me check and come back to you.”</p><p>Sharing your knowledge, even if incomplete, proves that you understand what’s going on, what you’re doing, and what you need to do.</p><h3>Bonus situation</h3><p>You answered whether you thought you were right or not. Then, a few minutes/hours/days later, you realize this was false. What do you do?</p><p>You should know the answer by now: be proactive, reach out, and openly say that you were wrong. If a piece of information threw you off and made you choose the wrong answer, say so and explain yourself so that people don’t fall into the same trap. If not, don’t find excuses. Instead, be straight to the point.</p><h3>Conclusion</h3><p>Sometimes it’s okay not to know, and sometimes it’s not. In both cases, take the hit, admit it, say that you will look into it, look into it, and come back with the answer. That’s how you build knowledge and trust at the same time.</p><p>Don’t protect your ego; above all, don’t bullshit people.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea6faa2ad289" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Cost Reduction Cheat Sheet]]></title>
            <link>https://medium.com/@florent.moiny/databricks-cost-reduction-cheat-sheet-126be465e09?source=rss-fe3c5e3137d------2</link>
            <guid isPermaLink="false">https://medium.com/p/126be465e09</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[cost]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cheatsheet]]></category>
            <dc:creator><![CDATA[Florent Moiny]]></dc:creator>
            <pubDate>Sun, 22 Jan 2023 11:49:33 GMT</pubDate>
            <atom:updated>2023-01-22T11:49:33.851Z</atom:updated>
            <content:encoded><![CDATA[<h4>No, Databricks is not super expensive</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p0AQEgtghQBI_Geq" /><figcaption>Photo by <a href="https://unsplash.com/@jpvalery?utm_source=medium&amp;utm_medium=referral">Jp Valery</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Here is a simple and straight-to-the-point 101 cheat sheet to dramatically increase your ROI on Databricks.</p><h3>Streaming</h3><h4>Do you need 24/7 streams?</h4><p>Does the business need 24/7 streams? Real-time data ingestion? Near real-time predictions?</p><p>Yes? Have your clusters run 24/7.</p><p>No? Don’t have your clusters run 24/7. How? Use <a href="https://docs.databricks.com/structured-streaming/triggers.html#configuring-incremental-batch-processing">incremental batch processing</a> via <em>trigger(once=True)</em> or <em>trigger(availableNow=True)</em>.</p><p>Databricks blog post: <a href="https://www.databricks.com/blog/2017/05/22/running-streaming-jobs-day-10x-cost-savings.html">Running Streaming Jobs Once a Day For 10x Cost Savings</a>.</p><h4>Tune the node count and size properly</h4><p>Especially for streaming clusters running 24/7. I’ve seen one customer running four streams with slightly more and bigger nodes than necessary. After two hours of basic tuning, I reduced the bill by $200k annually.</p><p>Databricks documentation: <a href="https://docs.databricks.com/clusters/cluster-config-best-practices.html">Best practices: Cluster configuration</a>.</p><h4>Don’t use autoscaling with standard streaming workloads</h4><p>They will automatically scale to the maximum number of nodes and stay there for the duration of the job. Instead, read what’s next.</p><h4>Use Delta Live Tables with enhanced autoscaling</h4><p>To have proper autoscaling with your streams on Databricks, use DLT with the cluster mode set to <a href="https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-concepts.html#databricks-enhanced-autoscaling">enhanced autoscaling</a>.</p><h4>What’s your trigger interval?</h4><p>Before <a href="https://docs.databricks.com/release-notes/runtime/8.0.html#new-structured-streaming-default-trigger-interval-reduces-costs">DBR 8.0</a>, the default <a href="https://docs.databricks.com/structured-streaming/triggers.html">trigger interval</a> for streaming queries was 0 ms, which could quickly get very expensive when pinging the underlying cloud storage. DBR 8.0 set the default to 500 ms, which can still be costly in some cases.</p><p>Pick a trigger interval that makes sense from the business perspective.</p><p>NB: that’s another reason to use new DBR runtimes as fast as possible.</p><h4>Autoloader: Be careful with directory listing</h4><p>With many files in cloud storage, <a href="https://docs.databricks.com/ingestion/auto-loader/directory-listing-mode.html">directory listing mode</a> can be costly.</p><p>Recommendation: use directory listing when developing/testing the application or in production when files are generated with lexical ordering. Use <a href="https://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html">file notification</a> in other cases.</p><h3>Clusters</h3><h4>All-purpose vs Job clusters</h4><p>All-purpose compute costs way more compared to job compute. See pricing pages for <a href="https://www.databricks.com/product/aws-pricing">AWS</a>, <a href="https://www.databricks.com/product/azure-pricing">Azure</a>, and <a href="https://www.databricks.com/product/gcp-pricing">GCP</a>.</p><p>Use all-purpose clusters when performing ad-hoc analysis, data exploration, or development. Use job clusters for all production jobs.</p><h4>Use aggressive auto-termination with all-purpose clusters</h4><p>When using an all-purpose cluster, the default <a href="https://docs.databricks.com/clusters/clusters-manage.html#automatic-termination-1">auto-termination</a> is 120 minutes. Be more aggressive: 15 or 30 minutes should be fine. Note that the current minimum is 10 minutes.</p><h4>Use aggressive auto-termination with Personal Compute</h4><p><a href="https://www.databricks.com/blog/2022/10/11/reduce-overhead-and-get-straight-work-personal-compute-databricks.html">Personal</a> <a href="https://docs.databricks.com/clusters/personal-compute.html">Compute</a> is great, but auto-termination is set at 72 hours. That doesn’t make any sense. Be aggressive and use 15–30 minutes.</p><h4>Use Databricks SQL Serverless</h4><p>The <a href="https://www.databricks.com/blog/2021/08/30/announcing-databricks-serverless-sql.html">serverless</a> version provides instant compute and is more cost-efficient.</p><h4>Use cluster policies</h4><p>Less of a low-hanging fruit, but provides guardrails. See the <a href="https://docs.databricks.com/administration-guide/clusters/policies.html?_ga=2.48650342.1572395865.1672842631-320820963.1664177213">documentation</a>.</p><h3>Runtimes</h3><h4>Upgrade runtimes</h4><p>Take advantage of newer features, bug fixes, and security patches. Some features will drastically improve your workloads, e.g., when switching to DBR 7.x and Spark 3 (and Adaptive Query Execution, see below). Most of the time, you don’t have any code changes to make.</p><p>At the very least, use the latest LTS version.</p><h4>Use Photon</h4><p>Using <a href="https://docs.databricks.com/runtime/photon.html">Photon</a> raises the number of DBUs per hour but also accelerates workloads dramatically (most of the time), reducing overall your total cost.</p><h3>Data</h3><h4>Use Delta</h4><p>Just use Delta for your tables. It’s superior in pretty much all aspects.</p><h4>Compact your tables</h4><p>Having thousands and millions of small files can kill your performance (that famous small files problem). Use <a href="https://docs.databricks.com/delta/optimize.html">OPTIMIZE</a>.</p><h4>Use Z-ordering to improve data skipping</h4><p>Less of a low-hanging fruit, but essential, especially for <a href="https://www.databricks.com/blog/2018/07/31/processing-petabytes-of-data-in-seconds-with-databricks-delta.html">needle-in-the-haystack queries</a>. Associated concept: <a href="https://docs.databricks.com/tables/partitions.html">partitioning</a>.</p><h4>Vacuum your tables</h4><p>I’ve seen terabytes and terabytes of data that should have been <a href="https://docs.databricks.com/sql/language-manual/delta-vacuum.html">vacuumed</a> months ago. Don’t pay extra for something that you don’t need.</p><h4>Use Low Shuffle Merge</h4><p>Migrate to DBR 10.4+ (remember, use the latest DBR LTS version available) and enjoy <a href="https://docs.databricks.com/optimizations/low-shuffle-merge.html">Low Shuffle Merge</a> being enabled by default.</p><h4>Set the number of shuffle partitions to auto</h4><p>Use <a href="https://docs.databricks.com/optimizations/aqe.html">Adaptive query execution</a> (enabled by default since DBR 7.3 LTS) and use <em>spark.conf.set(“spark.sql.shuffle.partitions”, “auto”)</em>. AQE will then handle the number of shuffle partitions at every pipeline step.</p><h3>Cloud</h3><h4>Use spot instances</h4><p>When it makes sense. If you’re stressed about VM availability and workload duration, you might not want to use them.</p><p>If you want more security at a potential extra cost, use the <a href="https://docs.databricks.com/clusters/cluster-config-best-practices.html#on-demand-and-spot-instances">Spot fall back to On-demand</a> feature.</p><h4>Use reserved instances and instance pools</h4><p>If you know what your VM consumption will look like in the next year(s), using reserved instances (<a href="https://aws.amazon.com/ec2/pricing/reserved-instances/">AWS</a>, <a href="https://azure.microsoft.com/en-us/pricing/reserved-vm-instances/">Azure</a>) in conjunction with <a href="https://docs.databricks.com/clusters/instance-pools/index.html">instance pools</a> can be interesting.</p><h4>Pre-purchase Azure Databricks commit units</h4><p>The same idea as reserved instances but applied to DBUs. See the <a href="https://learn.microsoft.com/en-us/azure/cost-management-billing/reservations/prepay-databricks-reserved-capacity">documentation</a>.</p><h4>Be paranoid about networking</h4><p>Be very careful about cross-region <a href="https://aws.amazon.com/blogs/architecture/overview-of-data-transfer-costs-for-common-architectures/">data transfer</a>.</p><p>Use <a href="https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html#configure-regional-endpoints-recommended">regional VPC endpoints</a> on AWS or <a href="https://www.databricks.com/blog/2020/02/28/securely-accessing-azure-data-sources-from-azure-databricks.html">Private Link/Service endpoints</a> on Azure.</p><p>Recommendation: if you don’t know what you’re doing when it comes to networking, or if you’re not sure, then 1. no shame in that, this is super hard, and 2. ask your infra team about all those magic keywords.</p><h3>Further reading</h3><p>Definitely read Databricks’ blog post <a href="https://www.databricks.com/blog/2022/10/18/best-practices-cost-management-databricks.html">Best practices for cost management on Databricks</a> by Tomasz Bacewicz and Greg Wood. I shamelessly took some of their points.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=126be465e09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What a crazy 2022 year working at Databricks]]></title>
            <link>https://medium.com/@florent.moiny/what-a-crazy-2022-year-working-at-databricks-1a66a461fa75?source=rss-fe3c5e3137d------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a66a461fa75</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[certification]]></category>
            <category><![CDATA[professional-services]]></category>
            <category><![CDATA[consulting]]></category>
            <dc:creator><![CDATA[Florent Moiny]]></dc:creator>
            <pubDate>Mon, 16 Jan 2023 07:42:27 GMT</pubDate>
            <atom:updated>2023-01-16T07:42:27.724Z</atom:updated>
            <content:encoded><![CDATA[<h4>And it’s just the beginning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xl5ZLViAk8YSi-Hx" /><figcaption>Photo by <a href="https://unsplash.com/@spacex?utm_source=medium&amp;utm_medium=referral">SpaceX</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Nobody is interested in a long and tedious list of facts, so let’s pick the most interesting ones.</p><h3>Starting the year exhausted</h3><p>March was the end of a 6-month engagement that pushed my limits, both psychologically and professionally.</p><p>The customer was creating a massive Delta lake to enable the creation of data products. But the project was dragging, complexity kept increasing, and scalability issues prevented any release to production.</p><p>The mission was clear: de-risk the project, accelerate it and get the team back on track.</p><h4>The fun part: the tech</h4><p>The core part of the project was a significant ingestion pipeline with 10TB of historical data (billions of JSON files) that would perform daily Change Data Capture with 100 million records. Constraints were part of the game: “Reuse existing code as much as possible,” aka “You will have to use legacy code,” and “Things must move forward,” aka “Prioritize and good is enough.”</p><p>Starting point: the initial load takes more than 20 hours and is unstable. The more we wait, the worse it gets. Also, the current code doesn’t update records efficiently. Good luck.</p><p>That was the beginning of a long, challenging, and thrilling journey to optimize everything we could. First, I used Delta’s Optimize and ZOrder capabilities, rewrote the merge conditions, and drastically reduced the target file size for the merges. Then, I added Photon and Low Shuffle Merge on top of it. Next, I discussed the intricacies of Delta and Low Shuffle Merge with the engineering team. We realized that, at the time, Low Shuffle Merge was not efficiently bulk deleting files in ADLS Gen2 (DBR 10.3 fixed it, and now it’s bloody fast). Finally, I created a custom batch processing to handle updates with billions of rows.</p><p>Those exchanges are what I love at Databricks: collaborating, getting help from people you’ve never talked to before, just for the sake of it, and improving the product from both sides while helping customers.</p><h4>The not-so-fun part: the political issues</h4><p>Working as a consultant in Professional Services is about more than just the tech. It’s also about managing the environment you’re working in and navigating political issues.</p><p>This engagement was the most intense I’ve ever had in that regard. I will not go into details, but let’s say that I ramped up when it comes to being a proper buffer between customers and partners, especially when they are not on good terms.</p><h4>Reflections</h4><p>A few learnings from my successes and mistakes, in gradual order of importance:</p><p><strong>Spend (a lot of) time with business people</strong></p><p>They know the data best, and I wrongly assessed the complexity of the pipeline’s data transformations. Long story short, it was a nightmare.</p><p><strong>Delegate and spend time where you will have the most impact</strong></p><p>Instead of writing transformations, I could have had a more significant impact by focusing on optimization, CI/CD, and Databricks-related issues.</p><p><strong>Work as a true team</strong></p><p>As soon as you become a team member, irrespective of your title or company, your responsibility is to make it healthy and prosperous.</p><p>On the outside, the team should be a single entity with clear goals, tasks, timelines, and ownership. Nobody should be able to point fingers directly at one member. Instead, shield the team, side with it, and push back the customer when necessary.</p><p>On the inside, each member should have clear goals, tasks, timelines, and ownership. No finger-pointing (see a pattern here?). Improve processes and accountability constructively.</p><p><strong>Be correctly accountable</strong></p><p>Accountability is a good trait until it turns into the hero syndrome. Don’t wake up at 2 AM and do 14-hour days. You’re simply setting yourself up for burnout. You might be fine in the short term, but this is a recipe for disaster in the long run. Be smarter. A lousy environment, artificial deadlines, or insufficient delegation should be fixed and not part of the game.</p><p><strong>Be open</strong></p><p>Being open is one of the most important traits you can have as a professional. Don’t keep people in the dark. Instead, raise issues. Give status reports, estimations, and timelines. Do you want to create trust, get people off your back, prevent micro-management, and reduce the number of meetings? Then, be open.</p><p>I gathered all those reflections in a single 70-slide deck that I presented to our internal EMEA and Global Performance SME groups. I even got an EMEA Professional Services MVP award for my job, which is nice.</p><h3>The biggest project in my life</h3><p>I started in March a new engagement with one of our biggest customers, Shell, that promises to be massive: create a Databricks Center of Excellence within Shell and make it thrive.</p><p>Shell’s scale is phenomenal, and the number of teams working with Databricks is super high. We need to centralize knowledge, share best practices, enable people, and accelerate high-value projects. We need a Center of Excellence.</p><p>I won’t go into great details as this is just the beginning of our journey, but I’m part of a fantastic team of Databricks and Shell people, where we tackle everything related to Databricks and more. Terraform, Unity Catalog, Delta Live Tables, performance optimizations, best practices, large-scale enablement, enabling the creation of data products based on a data mesh architecture, you name it. The impact is going to be huge.</p><p>I’m so excited to be part of this adventure and see the impact down the line. Stay tuned for more information and learnings I will gather along the way.</p><h3>Internal impact: hiring and enablement</h3><p>Hiring has been an important topic for me this year. As a result, I spent considerable time running interviews and enabling people to run them.</p><h4>Running interviews</h4><p>I ran over 50 interviews, ranging from grading coding assignments, checking Spark and Data Engineering technical skills, assessing architectural knowledge from a Data Engineering and MLOps perspective, looking for consulting skills, and evaluating company values understanding.</p><p>The more I run them, the more I realize that seniority level doesn’t correlate with interviewing capabilities. I’ve seen all potential combinations. Interviewing doesn’t have to be rocket science; instead, we can “take a step back and use common sense. I will write about it in another post.</p><h4>Enablement</h4><p>Running interviews is one thing. Scaling myself is another. My time is finite, Databricks is growing like crazy, and we need more people enabled to run interviews.</p><p>That’s why I spent considerable time on the coding assignment used worldwide to hire post-sales Solutions Architects, not just grading it but also enabling coworkers to grade it. Shadowing, reverse-shadowing, and debriefing provide trainees with structure, guidelines, and, most importantly, confidence in running the interviews. The discussions that arise are invaluable for the trainees and the trainer as they provide new perspectives and better ways of running the interviews. Simply put, everybody improves in the process.</p><h3>Certifications</h3><p>I finally took the time to pass three of the most exciting certifications out there: AWS Solutions Architect Associate, HashiCorp Terraform Associate, and Databricks Data Engineer Professional.</p><h4>AWS Solutions Architect Associate</h4><p>I studied for two weeks for this certification in March 2020 during the first lockdown, right before joining Databricks. I scheduled it, but the examiner never showed up. After that, my Databricks journey started, and I never tried to pass it again. Not sitting the exam was bugging me for two years until I couldn’t take it anymore and decided to go for it. I got a decent grade of 844 out of 1000:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/0*pDVuHBx9tEgbVkpQ" /><figcaption>AWS Solutions Architect Associate certification results</figcaption></figure><p>This certification is one of the best for learning about cloud concepts and getting practical experience. If you’re new to the cloud, go to ACloudGuru, study the associated courses, use the labs, and pass the certification. You will not regret it. Bonus point: since cloud concepts are fundamentally the same across all clouds, this will even help you when working with Azure and GCP.</p><h4>HashiCorp Terraform Associate</h4><p>There is beauty in efficiency and automation, which is why I’ve always loved creating CI/CD pipelines. Learning and using Terraform was just a matter of time.</p><p>Terraform is the best tool for infrastructure as code and integrates perfectly with Databricks via the <a href="https://registry.terraform.io/providers/databricks/databricks/latest/docs">Databricks Terraform provider</a>. So if you work with Databricks and infrastructure, use it. The quality is top, the documentation is on point, and the people working on it are crazy good.</p><p>The learning process is always the same, no matter the subject: read the doc, follow tutorials, try on dummy examples, have fun, create your pet project, and then use it at your job. For an extra challenge, book the associated certification in advance before starting, which I did.</p><p>I passed with a 96% grade:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/0*fTa6MannYAEVqY7-" /><figcaption>HashiCorp Terraform certification results</figcaption></figure><p>That’s cool, but the coolest part was that I got to use what I learned to deploy Databricks resources in production with my customers and help people internally at Databricks.</p><h4>Databricks Data Engineer Professional</h4><p>I saved the best for last. Of course, I’m biased as I work at Databricks, but the material covered by the certification is extensive and valuable. I use it daily.</p><p>It’s simple: if you want to use Databricks properly and efficiently, go to the Databricks Academy, land on the <a href="https://www.databricks.com/learn/certification/data-engineer-professional">Databricks Certified Data Engineer Professional</a> page, look at the Preparation section, do the specified self-paced courses, pass the certification to check your knowledge, apply the skills on your day-to-day job, and then thank me.</p><p>I completed the certification with a 96.66% grade:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/0*ipG5cnJi2DdQoftY" /><figcaption>Databricks Data Engineer Professional certification results</figcaption></figure><p>Interestingly, you will learn quite a lot of new stuff even if you use Databricks and read the doc daily.</p><p>Another interesting point: grounds covered by the <a href="https://www.databricks.com/learn/certification/data-engineer-associate">Associate</a> and Professional certifications are different. Same for the <a href="https://www.databricks.com/learn/certification/machine-learning-associate">ML certifications</a>. Do yourself a favor and study for them. I certainly will.</p><h3>What’s next</h3><p>My objective for 2023 is straightforward: to scale.</p><p>I first need to scale the impact of the Databricks Center of Excellence at Shell. The team made significant progress in 2022, but we will take it to another level this year. To have a lasting impact and to reach even more people and projects, we will make the Center of Excellence less of a bottleneck and more of a reusability asset.</p><p>Databricks is growing absurdly fast regarding business, employees, knowledge, and processes, bringing its own set of issues that must be dealt with as quickly as possible. Hiring and working processes are of utmost importance, so I want to scale my impact on those topics.</p><p>With the people we have at Databricks, the products, the roadmap, the leadership, the vision, and the execution, I sincerely think that Databricks is here to stay and will keep thriving for the next ten years, at minimum. I want to scale my impact externally by blogging regularly and enabling people on Databricks. But Databricks is part of a broader ecosystem, so I will also tackle topics on Python, Data Engineering, Architecture, Computer Science, Consulting, etc. I’ve already got a few ideas and can’t wait to share them with you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a66a461fa75" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>