<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marcelo Pedroni da Silva on Medium]]></title>
        <description><![CDATA[Stories by Marcelo Pedroni da Silva on Medium]]></description>
        <link>https://medium.com/@marcelopedronidasilva?source=rss-c54afacde5e5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bj7NpbdY82YY5A9yub9dCQ.jpeg</url>
            <title>Stories by Marcelo Pedroni da Silva on Medium</title>
            <link>https://medium.com/@marcelopedronidasilva?source=rss-c54afacde5e5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:17:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@marcelopedronidasilva/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Connecting to the Anthropic API: Direct Endpoint, Corporate Proxies, and Security Done Right]]></title>
            <link>https://medium.com/@marcelopedronidasilva/connecting-to-the-anthropic-api-direct-endpoint-corporate-proxies-and-security-done-right-50b2bc7a189d?source=rss-c54afacde5e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/50b2bc7a189d</guid>
            <dc:creator><![CDATA[Marcelo Pedroni da Silva]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 20:52:37 GMT</pubDate>
            <atom:updated>2026-07-10T20:57:17.982Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A practical guide for developers integrating Claude into their projects — safely.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/1*qX-jBOMo7kTq2Kg4ATtuDA.gif" /></figure><h3>Why This Matters</h3><p>Most tutorials on using the Anthropic API stop at “paste your API key here.” That’s fine for a 5-minute demo, but in any real project — especially inside a company — there are two questions you need to answer before writing a single line of code:</p><ol><li><strong>Where are requests going?</strong> Directly to Anthropic, or through your company’s proxy/gateway?</li><li><strong>Where are your credentials?</strong> In your code? In your repo? Or somewhere safe?</li></ol><p>This article walks you through both, using code patterns you can drop into a real project today.</p><h3>The Anthropic Python SDK</h3><p>When you instantiate the Anthropic client, three parameters determine everything:</p><pre>from anthropic import Anthropic<br><br>client = Anthropic(<br>    api_key=&quot;sk-ant-...&quot;,<br>    base_url=&quot;https://api.anthropic.com&quot;,<br>    default_headers={&quot;Authorization&quot;: &quot;Bearer sk-ant-...&quot;}<br>)</pre><ul><li><strong>api_key</strong> — authenticates you with the endpoint.</li><li><strong>base_url</strong> — the host that receives your requests. This is what you swap when routing through a proxy (from your company for example).</li><li><strong>default_headers</strong> — merged into every request. Corporate proxies often require an Authorization header even when the downstream authentication mechanism differs. It can be done using Bearer (like the example) or auth-key.</li></ul><p>These three parameters are the levers. The rest of the setup is about feeding them values without ever hardcoding them.</p><h3>Option 1: Direct Connection to Anthropic</h3><p>The simplest case — you’re an individual developer or a small team hitting api.anthropic.com directly.</p><pre>import os<br>from anthropic import Anthropic<br>from dotenv import load_dotenv, find_dotenv<br><br>load_dotenv(find_dotenv())<br>client = Anthropic(<br>    api_key=os.getenv(&quot;ANTHROPIC_API_KEY&quot;),<br>)<br><br>client = Anthropic(<br>    api_key=os.getenv(&quot;ANTHROPIC_API_KEY&quot;),<br>)</pre><p>Your .env file:</p><pre>ANTHROPIC_API_KEY=sk-ant-api03-...<br>BASE_MODEL=claude-sonnet-4-5</pre><p>No base_url needed — the SDK defaults to https://api.anthropic.com. One key, one endpoint, done.</p><h3>Option 2: Routing Through a Corporate Proxy</h3><p>This is the real-world scenario. Your company operates an API gateway that sits between your code and Anthropic. This is done for several legitimate reasons:</p><ul><li><strong>Cost governance</strong> — central billing, quota enforcement per team or project.</li><li><strong>Audit logging</strong> — every request is logged with the identity of the caller.</li><li><strong>Data loss prevention</strong> — the proxy can inspect and block requests containing PII or classified data before they leave your network.</li><li><strong>Key abstraction</strong> — developers never hold the actual Anthropic key; they hold a proxy-issued token.</li></ul><p>The change in code is minimal:</p><pre>import os<br>from anthropic import Anthropic<br>from dotenv import load_dotenv, find_dotenv<br><br>load_dotenv(find_dotenv())<br><br>client = Anthropic(<br>    api_key=os.getenv(&quot;ANTHROPIC_API_KEY&quot;),<br>    base_url=os.getenv(&quot;BASE_URL&quot;),<br>    default_headers={&quot;Authorization&quot;: f&quot;Bearer {os.getenv(&#39;ANTHROPIC_API_KEY&#39;)}&quot;}<br>)</pre><p>Your .env file now carries the proxy address:</p><pre>ANTHROPIC_API_KEY=proxy-issued-token-abc123<br>BASE_URL=https://llm-gateway.your-company.com/anthropic<br>BASE_MODEL=claude-sonnet-4-5</pre><p>The SDK sends the exact same shaped request — the only difference is where it lands. The proxy handles the downstream forwarding and authentication with Anthropic on your behalf.</p><blockquote><strong><em>Note on </em></strong><strong><em>default_headers:</em></strong><em> Some proxies require the </em><em>Authorization: Bearer header in addition to (or instead of) </em><em>api_key. If your proxy is compatible with the Anthropic SDK&#39;s native auth header (</em><em>x-api-key), you can drop </em><em>default_headers entirely. Check your proxy&#39;s documentation.</em></blockquote><h3>Making the Environment Switchable</h3><p>The real power of this pattern is that a single codebase works in both scenarios — dev hitting Anthropic directly, production hitting the corporate gateway — just by changing .env values.</p><p>A clean wrapper class that encapsulates this:</p><pre>import os<br>from anthropic import Anthropic<br>from dotenv import load_dotenv, find_dotenv<br><br>class AnthropicClient:<br>    def __init__(self):<br>        load_dotenv(find_dotenv())<br>        self.model = os.getenv(&quot;BASE_MODEL&quot;)<br>        self.client = Anthropic(<br>            api_key=os.getenv(&quot;ANTHROPIC_API_KEY&quot;),<br>            base_url=os.getenv(&quot;BASE_URL&quot;), # None → defaults to api.anthropic.com<br>            default_headers={<br>                &quot;Authorization&quot;: f&quot;Bearer {os.getenv(&#39;ANTHROPIC_API_KEY&#39;)}&quot;<br>            }<br>        )<br><br>    def chat(self, messages, system=None, temperature=0.02, max_tokens=1000):<br>        params = {<br>            &quot;model&quot;: self.model,<br>            &quot;max_tokens&quot;: max_tokens,<br>            &quot;messages&quot;: messages,<br>            &quot;temperature&quot;: temperature,<br>        }<br>        if system:<br>            params[&quot;system&quot;] = system<br>        response = self.client.messages.create(**params)<br>        return response.content[0].text</pre><p>No if proxy else direct branching. The base_url=None path just uses the SDK default. Swap the .env, nothing else changes.</p><h3>The Security Layer: Credentials Done Right</h3><p>This is where most tutorials cut corners. Here’s the full picture.</p><h3>Rule 1: Never Hardcode Credentials</h3><pre># Wrong — this is in your source code<br>client = Anthropic(api_key=&quot;sk-ant-api03-abc123...&quot;)<br><br># Right - resolved at runtime from the environment<br>client = Anthropic(api_key=os.getenv(&quot;ANTHROPIC_API_KEY&quot;))</pre><p>Hardcoded credentials end up in git history, log files, error traces, and eventually in a breach report. There are no exceptions to this rule.</p><h3>Rule 2: .env Files Are Never Versioned</h3><p>Your .gitignore must include:</p><pre>.env<br>.env.*</pre><p>The .env file is a local artifact — it lives on your machine (or is injected by your CI/CD system at deploy time). It never enters version control.</p><p>If it has already been committed — even once — the credential is compromised. <strong>Rotate it immediately</strong> and <strong>purge</strong> it from git history with git filter-repo or BFG Repo Cleaner.</p><h3>Rule 3: Provide a .env.example Template</h3><p>Since .env is gitignored, new contributors need to know what variables to define. Check in a template with no real values:</p><pre># .env.example — commit this, NOT .env<br>ANTHROPIC_API_KEY=your_api_key_here<br>BASE_URL=https://api.anthropic.com<br>BASE_MODEL=claude-sonnet-4-5</pre><p>This documents the required variables without exposing any secrets.</p><h3>Rule 4: find_dotenv() Instead of Hardcoded Paths</h3><pre># Fragile — breaks if you run the script from a different directory. <br># It&#39;s only acceptable in local study projects, but, please, avoid!<br>load_dotenv(&quot;/home/user/project/.env&quot;)<br><br># Robust - walks up the directory tree to find .env automatically<br>load_dotenv(find_dotenv())</pre><p>find_dotenv() searches from the current working directory upward until it finds a .env file. This works regardless of how or where the notebook or script is invoked — essential for Jupyter environments where the kernel&#39;s working directory isn&#39;t always predictable.</p><h3>Rule 5: Validate Presence, Never Print Values</h3><p>When debugging configuration issues, log that a variable exists — not its value:</p><pre>import sys<br><br>required = [&quot;ANTHROPIC_API_KEY&quot;, &quot;BASE_URL&quot;, &quot;BASE_MODEL&quot;]<br>missing = [v for v in required if not os.getenv(v)]<br>if missing:<br>    print(f&quot;Missing required env vars: {missing}&quot;)<br>    sys.exit(1)<br><br># Never do this:<br># print(f&quot;Using key: {os.getenv(&#39;ANTHROPIC_API_KEY&#39;)}&quot;)</pre><p>Any partial key value in a log file is a leak vector.</p><h3>Production Considerations</h3><p>When moving beyond local development:</p><p><strong>CI/CD systems</strong> (GitHub Actions, GitLab CI, Azure DevOps) all have native secret stores. Inject credentials as environment variables at pipeline level — never via .env files checked into the repo.</p><p><strong>Cloud environments</strong> (Azure Key Vault, AWS Secrets Manager, GCP Secret Manager) are the production-grade solution. Your application fetches the secret at startup, holds it in memory, and never writes it to disk.</p><p><strong>Rotation</strong> — API keys should be rotated on a schedule. The pattern above makes rotation a .env update (or a secret store update) with zero code changes.</p><p><strong>Principle of least privilege</strong> — if your proxy supports it, issue scoped tokens per service or team. A token that can only call Claude with a 100k-token monthly budget does far less damage when compromised than a root-level Anthropic key.</p><h3>The Full Picture in One Diagram</h3><pre>Local Dev                     Production (Corporate)<br>─────────────────────         ──────────────────────────────────<br>.env  ──► os.getenv()         CI Secret ──► env var injection<br>         │                               │<br>         ▼                               ▼<br>   Anthropic()                     Anthropic()<br>   api_key=...                     api_key=proxy-token<br>   base_url=None  ──► api.anthropic.com<br>                       base_url=gateway ──► LLM Gateway ──► api.anthropic.com</pre><p>Same code. Different .env. Different traffic path.</p><h3>Conclusion</h3><p>Connecting to the Anthropic API is three lines of code. Connecting to it safely — in a way that works on your laptop, in CI, in production, and behind your company’s gateway — takes a bit more thought, but the pattern is straightforward once you see it.</p><p>The core insight: <strong>your code should have zero knowledge of the actual credential values.</strong> It knows the variable names. The runtime environment provides the values. That separation is what makes the proxy swap a config change instead of a code change, and what keeps your keys out of git history.</p><p>The pattern shown here — python-dotenv + find_dotenv() + .gitignore + a thin SDK wrapper — is lightweight enough to use in a Jupyter notebook and solid enough to carry into production.</p><p>For more, please check the github repo (<a href="https://github.com/mpedronidasilva/Claude_Code_Certification_Study/tree/main">https://github.com/mpedronidasilva/Claude_Code_Certification_Study/</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*fQhbdpx2NwNhWS51_fvEHw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50b2bc7a189d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[(Basic) Complex data types to optimize your Master Data table.]]></title>
            <link>https://medium.com/@marcelopedronidasilva/basic-complex-data-types-to-optimize-your-master-data-table-b140f78e86c5?source=rss-c54afacde5e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b140f78e86c5</guid>
            <category><![CDATA[dimensional-modeling]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[master-data]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Marcelo Pedroni da Silva]]></dc:creator>
            <pubDate>Fri, 17 Jan 2025 13:17:41 GMT</pubDate>
            <atom:updated>2025-01-17T13:17:41.607Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Disclaimer</strong>: This content was based on the Data Engineering bootcamp available on <a href="https://bootcamp.techcreator.io/program/free-youtube-boot-camp/details">DataExpert.io</a> developed by <a href="https://www.linkedin.com/in/eczachly/">Zach Wilson</a>. It’s free and available on Dataexpert.io platform or even on Youtube, please check it out!</p><p>When studying Data Engineering I’ve came across the fundamentals of OLAP and OLTP systems, and I was wondering if when I’d do some modeling I’d had to choose between one of them as a guideline.</p><p>But, the problem is that depending on the problem, the modeling needs to be in a <em>middle-ground</em> between OLAP and OLTP. Since OLTP is optimized for processing low volume of data and usually the data is on the 3rd NF (to have low latency as well) and on the other hand OLAP’s designed to optimize larger volumes to avoid multiple JOINS that may add some unnecessary complexity.</p><p>But, what if I need to develop a dataset that will serve as a source of truth for many other dowstreams? How can I choose between OLAP or OLTP if I don’t know all the needs from the consumers of this data? Well, that’s when the <strong>Master Data</strong> comes in!</p><p>The major point here is to balance performance and volume of data. Among several techniques available, the most basic and helpful is the use of Complex data types. That is very helpful for cumulative design tables, let’s see some example:</p><p>Problem:</p><p>I have to re-think the model for some NBA data. And to start this, I have a table called “<em>player_seasons” </em>that goes like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FW4H--KcS_oW8neMfI8elQ.png" /></figure><p>So, the first thing to do here is to separate what are the player’s attributes and what are the season’s ones. So I came to this list for player’s: <em>player_name, height, college, country, draft_year, draft_round, draft_number, seasons_stats, current_season.</em></p><p>So I can create a separate table with those attributes and call it just <em>“players” </em>(very original).</p><p>Now comes the fun part! How can I implement some logic to make this table cumulative, but keeping it as short as I can. So in other terms, instead of creating a row for each player for each season, how can I append new seasons into the same row that already exists?</p><p>This is done (one of the ways) by using the <strong>complex data types.</strong> In this case I’ll use an array of struct, but there are some options like those two and Map. Their uses will depend on the business rules and the way the model is beeing think.</p><p>So, once this Master data will be cumulative, the logic to do this is something that I’ve learned recently, which is the use of a historical and a current CTE’s. The historical I can call it “<em>yesterday</em>” and the current is “<em>today</em>” (again very original).</p><p>Those two CTE’s will be very simple, like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*d7HgFQg_oCp1HxwpY3VUlg.png" /></figure><p>Since the minimum year is 1996 on my <em>player_seasons </em>table, in the first interaction the <em>yesterday</em> table will be empty and <em>today</em> will have all the values that has season = 1996, of course.</p><p>Moving forward, after the CTE’s what I’ll select to be the content of my new “<em>player</em>” table, goes like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*csbhWhNtkI3J-OB-oIlxxA.png" /></figure><p>First, I do some <em>coalesce </em>on the player’s attributes, so if the attributes from <em>taday</em>’s table is null it will get them from <em>yesterday</em>. It means that we can have a player that is retiring on 1997 and only played in 1996. So when the next interaction happens it won’t update this player row. In other hand, if there’s newer values coming in <em>today</em> for the player, it will get this data.</p><p>But the most interesting part is after that. When I created the “<em>player</em>” table I created a column named “<em>seasons_stats”</em>(<em>the plural wasn’t intended, it’s an error, sorry :-P</em>) and this is an <strong>ARRAY, </strong>which means that we can fill this with a <strong>STRUCT</strong> containing key and values that will have the datatype as <strong>seasons_stats. (</strong><em>If this is the first time you see this kind of thing this should blow your mind, at least happened to me years ago when I face something similar</em>).</p><p>So, in the CASE statements that follows in the query I’m covering three possible situations:</p><ul><li>a) It’s the first interaction so there’s no historical data or it’s the first time that the player comes in, like a rookie for example. This means that <em>seasons_stats</em> for this player will be null on <em>yesterday</em> table, then all the struct data will come from <em>today</em>’s table.</li><li>b) There’s data for the player in both, <em>yesterday</em> and <em>today</em>’s tables. So I concatenate this struct data inside of the array, keeping both values in the same row (mind blowing, isn’t it? -yes-)</li><li>c) The player has retired or didn’t played for some reason. So it will have data on <em>yesterday</em> but not on <em>today</em>’s table. So it will keep only <em>yesterday</em>’s data.</li></ul><p>That explained, let’s see after some interaction how our data looks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kYMfkbDPeDb_2emW9kAX-Q.png" /></figure><p>So here I can see the different scenarios, for players who got later (2000 for example) and players that were playing since 1996. And a powerfull thing about it is that this can track players that made some pause in their NBA carrer and then came bak, like Michael Jordan as it shows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uGnJbsOJzMBYzt1O5CrCqw.png" /></figure><p>So, imagine what this can do for systems that you need to track informations that relies on time dimensions, such as calendars tables for example. This is a helpful and not to complex optimization to create the master data that every Data Engineer can access and easily UNNEST this data and work properly according to his/her demand.</p><p>Hope this was helpful to you as it was for me!<br>See ya next time! =-)</p><p>current_season</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b140f78e86c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to install and run Pyspark locally integrated with VSCode via Jupyter Notebook (on Windows).]]></title>
            <link>https://medium.com/@marcelopedronidasilva/how-to-install-and-run-pyspark-locally-integrated-with-vscode-via-jupyter-notebook-on-windows-ff209ac8621f?source=rss-c54afacde5e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff209ac8621f</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Marcelo Pedroni da Silva]]></dc:creator>
            <pubDate>Thu, 18 Jul 2024 19:22:23 GMT</pubDate>
            <atom:updated>2024-07-18T19:22:23.626Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*YvzdjbwFW3MzYS1E.jpg" /></figure><p>If you ever wonder, how can I pratice or just do some coding using PySpark on windows, besides using Databricks (because it is not cheap ̶,n̶o̶t̶ ̶e̶v̶e̶n̶ ̶c̶l̶o̶s̶e̶, or just because you’re not everybody and you just don’t want to use it), you must have faced several issues during the process of installation and/or configuration and (like me) just (rage)quit after a few frustrated tryes.</p><p>Well, this is going to be a sum up of a several contents that I’ve gather during this search that I followed and that finally worked for me!</p><h3><strong>Step 1.</strong></h3><p><em>Many installations required, the easiest part.</em></p><p>First of all, you need to be sure that you have <strong>Java Developer Kit</strong> installed. I always recommend to keep the newest version if you’re not a hard user or a programmer that needs some exacly version.</p><p><strong>JAVA INSTALLATION</strong></p><ul><li>For me, the most recent Java version was <strong>jdk 22</strong> that is available on: &lt;<a href="https://www.oracle.com/java/technologies/downloads/#jdk22-windows">https://www.oracle.com/java/technologies/downloads/#jdk22-windows</a>&gt;</li><li>The <strong>JAVA</strong> installation is pretty much straight-forward by clicking <em>next, next, agree, continue </em>buttons and choosing a right path to install. By default the installer recommends the path like “<em>C:\Program Files\Java\jdk-22&quot;. </em>It is important to know exactly where it was installed to make sure that we configure the JAVA_HOME environment variable correctly later on.</li></ul><p><strong>PYTHON INSTALLATION</strong></p><ul><li>This is pretty much the same as for java, and probably everyone had it done, but is no harm to describe here. <br>So, download Python version at &lt;<a href="https://www.python.org/downloads/windows/">https://www.python.org/downloads/windows/</a>&gt; (I usually use a version before the most recent, but it will depend on what are you going to do, if it is for work or study, maybe your work needs such specific version ’cause of some lib dependency, lib deprecated, or other particularity.</li><li>The installation process is pretty much the same, but note that on installation screen has an option to add Python to a PATH, make sure you’re keeping this option marked, just to enshort our effort.</li><li>If everything happens accordingly as expected, you can run this command on a cmd <em>python — version </em>and you should be able to see the version you’ve just installed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*SN9a7_PRRvfPBm07aHF6eA.png" /></figure><p><strong>APACHE SPARK INSTALLATION</strong></p><ul><li>Here we want to download Spark from Apache (who maintains it) from here &lt;<a href="https://spark.apache.org/downloads.html">https://spark.apache.org/downloads.html</a>&gt;</li><li>By the time of this writing, this are the most recent version available.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*ce17Cb_joZuUxzttuINXrw.png" /></figure><p>So you’ll just choose the version, i recommend the most recent, for Spark release and for package type. Then click on the link at bullet list #3.</p><ul><li>You should see a screen like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zp_DCQ9pY769LJ4A9CyDgw.png" /></figure><p>Just click on the first link and it will download a zip file that contains Apache Spark.</p><ul><li>Extract its content to a folder under <em>C: </em>folder. On my installation I extract on the <em>C:</em> folder itself. Like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UC-Nl--89QmZVApDv6wlnA.png" /></figure><h3>Step 2.</h3><p>Now that all installations are done. we’re going to make some configurations.</p><ul><li>Create a new folder named <em>hadoop, </em>i recommend it to be besides spark folder, just to be more easier on this first time, later you can refactor and follow some folder pattern that you have. Inside this hadoop folder, create a <em>bin</em> folder, like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*LgFW5JT2SJQ2p5Q5IEcglQ.png" /></figure><ul><li>Inside the bin folder, we will put the <em>winutils.exe</em> that is necessary to perform some spark changes and allows using resources such as HDFS. To download this, you need to check the version of Hadoop that your Spark makes use (see image of Apache Spark Instalation, on option of <em>package type </em>that you’ve just downloaded and go to &lt;<a href="https://github.com/cdarlint/winutils/tree/master">https://github.com/cdarlint/winutils/tree/master</a>&gt; and navigate through the version that matches with yours. On my case, it is Hadoop 3.3, so I can go on <a href="https://github.com/cdarlint/winutils/tree/master/hadoop-3.3.5">hadoop-3.3.5</a> and search for the <a href="https://github.com/cdarlint/winutils/blob/master/hadoop-3.3.5/bin/winutils.exe">winutils.exe</a>, download this file and put it on <strong><em>C:\hadoop\bin</em></strong> folder.</li></ul><h3>Step 3.</h3><p><em>Messy part, Environment Variables configuration.</em></p><p>The last and definitely not the least part, we need to configure all the Environment Variables so the system can locate all the installations that we did, for python, java, spark and hadoop.</p><ul><li>On windows search bar, just type Environment Variables and you should see the <em>System Properties</em> and on this screen, it has the <em>Enviroment Variables.</em></li><li>On the <em>User variables for &lt;user&gt;</em> click on <strong><em>New</em> </strong>buttom and create the following: <br>- <strong>JAVA_HOME -&gt;<em> C:\Program Files\Java\jdk-22</em></strong> (or the path you choose during the JAVA installation process).<strong><br>- HADOOP_HOME -&gt; <em>C:\hadoop</em><br>- SPARK_HOME -&gt; <em>C:\spark</em><br>- SPARK_LOCAL_HOST_NAME -&gt; <em>localhost</em></strong></li><li>Then, the final step is to set up those variables into the Path one, so on the same <em>User variables for &lt;user&gt; </em>session, you should see th <em>Path</em> variable, select it and edit it. Click on <em>New </em>to create one by one of these:<br>- <strong>%JAVA_HOME%\bin<br>- %SPARK_HOME%\bin<br>- %HADOOP_HOME%\bin</strong></li><li>It should be like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*5ZOyfTF7Wbi_3tizpT7srQ.png" /></figure><p><strong>STEP 4</strong></p><p>Lets test the whole thing!</p><p>On a new cmd, run the commands:</p><ul><li><em>java --version </em>to see if java is properly set.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*DM-3EKKm4rPcB5WN-fxgvQ.png" /></figure><ul><li><em>pip install pyspark</em> to install PySpark lib that allows us to use Spark on our codes. (In my case it was already installed, but for you it must take a while untill it gets installed).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*7cEuYxiYNLSCVaHq4_XmHA.png" /></figure><ul><li><em>pyspark --version </em>to see if everything is installed and configurated okay. If it did, you should see something like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*X0XKW6Vm2kddAcCi8q8dxQ.png" /></figure><p><strong>STEP 5</strong></p><p>Now that all steps are done, you can work locally using PySpark, also can see the <strong>UI </strong>just<strong> </strong>typing <strong><em>pyspark </em></strong>on a cmd, it should give you a link to the UI, like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IgUPHg2hPQlKVAm-F49eXg.png" /></figure><p>You can past it on a web browser to acces de UI for Spark. It is going to be like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BfUMq54jFcLiDiYzwPgVHQ.png" /></figure><p>Also, you can work on PySpark using VSCode integrated with Jupyter Notebook. To do this you just need to download Python extension for VSCode and Jupyter (i don’t remember if it comes along with Python extension, just in case, search on the extensions tab and install if it not already).</p><p>Then, just create a file with the extension .ipynb that is the one for Jupyter.</p><p>And the final step, lets test this using some code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jNY2IFOkX1zsdOxRz-3C9w.png" /></figure><p>What this code actually does, it confirms the Paths to the SPARK_HOME and HADOOP_HOME, takes the path to JAVA_HOME that you have configured and throws a message if any of this configs are not properly set. Then it try to start a new Spark Session, if everything proceed the way it is supposed to, you shoul get the confirmation message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*PdchRtXvKO46-iGK93L5Uw.png" /></figure><p>Hope you find it usefull and enjoy your journey on the PySpark world!</p><p>See ya!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff209ac8621f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Notas rápidas e dicas de BigQuery - #1 A função UNPIVOT: Transformando várias colunas em duas…]]></title>
            <link>https://medium.com/@marcelopedronidasilva/notas-r%C3%A1pidas-e-dicas-de-bigquery-1-a-fun%C3%A7%C3%A3o-unpivot-transformando-v%C3%A1rias-colunas-em-duas-3c8d705c1bd8?source=rss-c54afacde5e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c8d705c1bd8</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[unpivot]]></category>
            <dc:creator><![CDATA[Marcelo Pedroni da Silva]]></dc:creator>
            <pubDate>Mon, 10 Jul 2023 10:51:41 GMT</pubDate>
            <atom:updated>2023-07-10T10:51:41.746Z</atom:updated>
            <content:encoded><![CDATA[<h3>Notas rápidas e dicas de BigQuery - #1 A função UNPIVOT: Transformando várias colunas em duas (“chave e valor”)</h3><p>Imagine que em seu projeto exista uma tabela que possui várias colunas que representem chaves e seus conteúdos são os valores (independente do data type) como o exemplo abaixo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L8K8MQU7WektZQ2bBCF16A.png" /></figure><p>Este tipo de formatação de tabela é amplamente utilizado pelas empresas, dada a facilidade de visualizar e comparar os valores entre as diferentes transportadoras (em nosso exemplo).</p><p>Porém, quando falamos de projetos de BI, onde o resultado final que será visualizado pelo <em>stakeholder </em>será um dashboard (em tableau, power BI, data studio, ou qualquer outra ferramenta de visualização de dados), na maioria dos casos, possivelmente, é mais interessante normalizar as colunas que representam as mesmas categorias de dados (coluna “chave”) e outra coluna que representem seus respectivos valores (coluna “valor”).</p><p>Esta “normalização” traz alguns prós e contras, sendo um “contra” o fato de ter-se maior quantidade de dados, pois é necessário criar uma coluna para cada registro de Carrier, conforme imagem abaixo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*d8wmIETr_s7fmQZYFPre-A.png" /></figure><p>Imagine que esta tabela tenha 1M de registros, sendo normalizada aumentaríamos para 8M de linhas (contando que sempre haverão dados para as 8 Carriers).</p><p>Mas, felizmente com as ferramentas de BigData talvez esta troca seja ainda vantajosa na maioria dos casos, assim, vejamos como podemos fazer isso utilizando o BigQuery.</p><p>Ideia 1: A primeira ideia que qualquer pessoa que inicia na área de dados é criar o que chamamos de componente, uma coluna que represente a coluna que estamos normalizando e uma coluna “Value” com seus respectivos valores.</p><p>Esta ideia não está errada, mas é bastante comum que se busque primeiro alcançar o resultado esperado, sem se preocupar com a forma utilizada para tal. Assim é comum nos depararmos com esta ideia:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*f4veKZDe7KBF1lwZR9rG7A.png" /></figure><p>Esta ideia, embora alcance o resultado final esperado, reproduz o que chamamos de <em>código sujo, </em>se baseando na repetição de código, com uma mínima alteração na coluna Value e nos valores atribuídos na coluna <em>Carrier </em>em cada subquerie (que neste caso seriam 8 subqueries, agora imagine uma tabela com 100 colunas…!)</p><p>Para resolver este problema e tornar o código em um código “limpo” podemos utilizar a função <strong>UNPIVOT </strong>e trazer toda solução em apenas 1 subquerie.</p><p>Ideia 2: Utilizando a função <strong>UNPIVOT </strong>o que buscamos é “copiar” as colunas que se repetirão para cada chave-valor e atribuir “dinamicamente” de maneira inequívoca as chaves na coluna Carrier e seus valores na coluna Value sem erro.</p><p>Portanto, basta que coloquemos tudo em uma subquerie e a partir dela, faremos a função não pivotante utilizando uma coluna de valor (coluna Value) que receberá os valores provenientes do laço de repetição que itera sobre as colunas de transportadoras, da seguinte maneira:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M_BsMJJwauW56YqIYueydA.png" /></figure><p>De certa forma, fazemos um CROSS JOIN entre os valores da subquerie Carriers e atamos à coluna Value somente os valores provenientes da coluna que está sendo iterada naquele instante.</p><p>O resultado final será o mesmo, mas como é visível o código é extremamente mais reduzido, fácil de ser lido, compreendido e mais importante de tudo, mais fácil de realizar sua manutenção.</p><h4><strong>Notas finais:</strong></h4><p>Simplificar o código é uma tarefa que é incessante e que traz muito aprendizado. Com isso em mente trago neste espaço algumas dicas rápidas de melhorias de códigos que não encontro tão facilmente em blogs e sites de programação e que me ajudam cotidianamente em meus projetos profissionais e pessoais.</p><p>Espero que gostem.</p><p>Att,</p><p>Marcelo</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c8d705c1bd8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Projeto de Visão Computacional utilizando a Lib OpenCV do Python 3.]]></title>
            <link>https://medium.com/@marcelopedronidasilva/projeto-de-vis%C3%A3o-computacional-utilizando-a-lib-opencv-do-python-3-be0ec0ef2030?source=rss-c54afacde5e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/be0ec0ef2030</guid>
            <category><![CDATA[opencv]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Marcelo Pedroni da Silva]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 16:59:55 GMT</pubDate>
            <atom:updated>2020-07-31T17:24:22.505Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Como projeto de finalização da disciplina de Algoritmos, durante o curso de graduação em Ciência da Computação, o grupo que era composto por mim e outros 4 colegas propôs uma revisão e aplicação do trabalho elaborado pelo Adrian Rosebrock.</blockquote><p>Nossa proposta de trabalho introduzimos os conceitos de visão computacional e suas funcionalidades no desenvolvimento de tecnologias como as de reconhecimento facial, reconhecimento de distâncias entre os veículos (utilizado nas tecnologias de <em>park assists</em>) e até mesmo no reconhecimento de expressões faciais para identificar sentimentos que as pessoas possam estar vivenciando.</p><p>Nosso objetivo, no entanto era mais simples, desejávamos que nosso programa fosse capaz de identificar uma bolinha e que fizesse o rastreio da trajetória enquanto a bola se movesse.</p><p>Para isso, o primeiro passo é fazer com que o código fosse capaz de binarizar a bolinha, a fim de diferenciar a bolinha do resto do cenário do campo de visão da webcam do computador. Para isso, tirei uma foto da bolinha utilizando a webcam do próprio notebook e no arquivo intitulado <em>range_detector.py</em>. (disponível em: <a href="https://github.com/Marcelopedroni/marcelo.pedroni/edit/master/.github/workflows/range_detector.py).">https://github.com/Marcelopedroni/marcelo.pedroni/edit/master/.github/workflows/range_detector.py)</a> indiquei o path da mesma para que ele aplicasse o filtro HSV, binarizando a imagem a minha escolha.</p><p>Aqui a binarização acontece:</p><figure><img alt="Binarização realizada com o código range detector" src="https://cdn-images-1.medium.com/max/1024/0*_zhyZP9b62yJ6FXl" /></figure><p>A ideia é que essa binarização nos gere valores que definam o diâmetro da bola e que separe-a dos outros elementos que compõem a imagem. Este processo é manual, tive algumas tentativas frustradas onde o fundo não ficava inteiramente preto, causando interferência na hora de realizar a identificação da bola no programa principal.</p><p>Em poucas palavras, é realizado uma série de tratamentos na imagem por meio de dilatações e erosões até obtermos a possibilidade de identificar somente a bolinha.</p><blockquote>Dica: Use uma bola de cor chamativa e que seja bem diferente das cores do plano em que ela está e se possível use uma bola que não tenha detalhes para evitar falhas de reconhecimento, lembrando que a ideia é identificar totalmente e somente a bola do resto da imagem. Quando for rodar o programa principal, use sempre a mesma bola que foi reconhecida neste processo.</blockquote><p>Com os valores de <em>HSV </em>obtidos, aplicamos os valores no código principal, intitulado de <em>ball_tracking.py </em>(disponível em: <a href="https://github.com/Marcelopedroni/marcelo.pedroni/blob/master/.github/workflows/ball_tracking.py">https://github.com/Marcelopedroni/marcelo.pedroni/blob/master/.github/workflows/ball_tracking.py</a>) e aqui fazemos a importação de todas as libs que utilizaremos: cv2, imutils (lib que o Adrian criou, como uma espécie de compilado de funções do OpenCV que seriam úteis neste projeto e a lib Numpy).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*9T4SI0jLzS7ziMN67ojzPQ.png" /></figure><p>Os dados de redLower e redUpper são os dados obtidos em HSV, após isto há a referência ao uso da webcam como entrada de vídeo, que irá captar os frames.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*f5MPVa4MRmW5fHkfsJ1LBA.png" /></figure><p>Aqui, fazemos a aplicação da máscara que irá definir a bola quando a mesma aparecer no campo de visão da webcam. A bolinha tem seu centróide (como se fosse um centro de massa, de um objeto 2D, que retorna valores x,y) identificado para que se desenhe o círculo à volta da bolinha.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*4S9pPE6B-qEOGONKQioK-A.png" /></figure><p>Neste for i in range(1, len(pts)): percorre a lista de pontos rastreados, então primeiramente ele lê se há algo na lista (caso a bolinha não esteja no campo de visão, nada será mapeado e a lista de pontos rastreados serão considerados nulos, que em Python é None).</p><p>Na linha 82, fazemos o cálculo da espessura da linha que mostra os pontos pelos quais a bolinha se movimentou e nos devolve tal rastreamento em real time na própria visão da webcam até que o usuário pare o programa ou aperte a tecla “<em>q</em>”.</p><p>Assim, o programa encerra o uso da webcam e encerra todas as janelas abertas pelo programa.</p><p>Este projeto foi possível após delinearmos uma trilha de aprendizagem de Python 3, por meio de cursos disponíveis online como o Curso em Vídeo do Gustavo Guanabara, o qual obtive 3 certificações e um curso rápido da Udemy de OpenCV para tratamento de imagens e vídeos.</p><p>That’s all Folks!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*eidi6EyIsQSvxCgP" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be0ec0ef2030" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>