<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[humanmanaged - Medium]]></title>
        <description><![CDATA[Human Managed is where machines and humans come together to think. - Medium]]></description>
        <link>https://medium.com/humanmanaged?source=rss----8d657b91fd65---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>humanmanaged - Medium</title>
            <link>https://medium.com/humanmanaged?source=rss----8d657b91fd65---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:48:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/humanmanaged" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Trust by Design: How Federated Learning Makes Collective Intelligence Possible]]></title>
            <link>https://medium.com/humanmanaged/trust-by-design-how-federated-learning-makes-collective-intelligence-possible-c921cbeaf3b0?source=rss----8d657b91fd65---4</link>
            <guid isPermaLink="false">https://medium.com/p/c921cbeaf3b0</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[federated-learning]]></category>
            <dc:creator><![CDATA[Karen Kim]]></dc:creator>
            <pubDate>Wed, 23 Apr 2025 05:34:06 GMT</pubDate>
            <atom:updated>2025-04-23T05:33:54.730Z</atom:updated>
            <content:encoded><![CDATA[<h4>(Part 3 of 3 in the “Building a Secure, AI-Ready Digital Future” series)</h4><h3>Collective Intelligence Needs Collective Learning</h3><p>In cybersecurity, no single organization — or even country — has a complete view of evolving threats.</p><p>Every detection is a partial glimpse. Every alert is a fragment. Every incident, a datapoint in a much larger, faster-moving story.</p><p>The challenge isn’t just collecting more data. It’s learning together without compromising trust, sovereignty, or privacy.</p><p>That’s where <strong>Federated Learning</strong> enters — and why it’s foundational to building a collective, AI-ready digital future.</p><h3>What Is Federated Learning?</h3><p><strong>Federated Learning (FL)</strong> is a decentralized machine learning approach where models are trained collaboratively across many organizations — without exchanging raw data.</p><p>Here’s how it works:</p><ul><li>Each participant (whether a bank, telecom provider, or government agency) trains a local version of a model on their own data.</li><li>Only the model updates — not the underlying data — are shared and aggregated to improve a global model.</li><li>Each organization retains full control over its sensitive information, while still contributing to shared intelligence.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wTzKix3G2mS_bXOCVOevRw.png" /><figcaption>Unlike traditional machine learning where data is aggregated into a central location for processing, federated learning trains algorithms across decentralized devices or servers that hold local data samples, without exchanging them.</figcaption></figure><p>In essence:<br> <strong>The data stays private.<br> The learning gets shared.</strong></p><blockquote>This is how we can scale collective defense across sectors and borders — while respecting the realities of privacy, data sovereignty, and regulatory diversity.</blockquote><h3>Why Federated Learning Matters for ASEAN and Beyond</h3><p>In regions like ASEAN, where cybersecurity readiness levels vary and national sovereignty is paramount, traditional intelligence sharing models break down.</p><p>Federated Learning offers a fundamentally different approach:</p><ul><li><strong>Privacy-preserving collaboration:</strong> Sensitive customer, citizen, or enterprise data never leaves its source.</li><li><strong>Improved threat detection:</strong> By aggregating insights from diverse environments, detection models become more accurate, more adaptive, and more resilient against sophisticated threats.</li><li><strong>Resource optimization:</strong> Smaller or less-resourced organizations can benefit from shared learning without needing massive internal teams or infrastructure.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WnnD2SpAhxXuo_y_lYObDg.png" /></figure><blockquote>In sectors like banking, digital trade, and critical infrastructure — where cyber risks are systemic — Federated Learning could be the missing layer that makes <strong>true regional coordination</strong> possible.</blockquote><h3>Inside Human Managed: Federated Learning in Practice</h3><p>At Human Managed, we see Federated Learning as a natural evolution of our work with Decision Intelligence Platforms (DIPs).</p><p>While DIPs help organizations act faster and smarter at the local level, Federated Learning helps the ecosystem learn faster and smarter collectively.</p><p>We’re implementing architectures where:</p><ul><li>Local DIPs feed anonymized, model-updated insights into sector-specific federated hubs.</li><li>These hubs aggregate intelligence across banks, telcos, or ports without ever accessing private datasets.</li><li>Human-in-the-loop governance remains embedded — ensuring that models evolve based on not just statistical patterns, but real-world business context, regulatory constraints, and operational priorities.</li></ul><h3>Human-in-the-Loop: The Invisible Backbone of Collective Learning</h3><p>As with Decision Intelligence Platforms, human oversight is critical in Federated Learning.</p><p>Machines may surface anomalies. Algorithms may find patterns. But only humans can contextualize meaning, validate assumptions, and ensure that models serve real strategic and societal goals.</p><p>In our federated designs, human governance is built into every layer:</p><ul><li><strong>Context injection:</strong> Analysts define what risks matter and why.</li><li><strong>Model review:</strong> Updates are reviewed not just for accuracy, but for strategic alignment and explainability.</li><li><strong>Adaptive feedback loops:</strong> Human decisions continuously recalibrate how models prioritize, detect, and respond.</li></ul><blockquote>Without this human scaffolding, collective intelligence would drift, fragment, or worse — reinforce blind spots.</blockquote><h3>Looking Ahead: Toward a Federated Future</h3><p>The future of cybersecurity isn’t a single dashboard or a monolithic AI.</p><p>It’s a network of decision-making systems — distributed, privacy-preserving, learning together — and grounded in human judgment.</p><p>ASEAN is taking early, bold steps toward this future. Through frameworks like the Digital Economic Framework Agreement (DEFA) and the ASEAN Norms Implementation Checklist, the region is creating space for collective cybersecurity innovation rooted in trust, sovereignty, and shared resilience.</p><p>But technology alone isn’t enough.</p><p>Building a federated future will require:</p><ul><li><strong>Technical infrastructure:</strong> Decision platforms, federated hubs, privacy-enhancing technologies.</li><li><strong>Policy frameworks:</strong> Governance models that enable responsible collaboration without forcing centralization.</li><li><strong>Cultural mindset shifts:</strong> Moving from compliance-driven security to capability-driven resilience.</li></ul><p>At Human Managed, we’re committed to helping build this future — one decision, one dataset, one collective learning loop at a time.</p><h3>Final Thought: Collective Intelligence Is a Choice</h3><p>In the end, collective intelligence isn’t a technical inevitability. It’s a <strong>strategic and ethical choice</strong>.</p><p>A choice to collaborate rather than compete.<br>A choice to embed human judgment into machine learning.<br>A choice to design cybersecurity systems that are not just faster — but wiser.</p><p>The building blocks are already here.<br>The next step is ours to take.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c921cbeaf3b0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/humanmanaged/trust-by-design-how-federated-learning-makes-collective-intelligence-possible-c921cbeaf3b0">Trust by Design: How Federated Learning Makes Collective Intelligence Possible</a> was originally published in <a href="https://medium.com/humanmanaged">humanmanaged</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision Intelligence Platforms: Embedding Human Judgment into AI-Enhanced Cyber Defense]]></title>
            <link>https://medium.com/humanmanaged/decision-intelligence-platforms-embedding-human-judgment-into-ai-enhanced-cyber-defense-a4fb1cf061e1?source=rss----8d657b91fd65---4</link>
            <guid isPermaLink="false">https://medium.com/p/a4fb1cf061e1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Karen Kim]]></dc:creator>
            <pubDate>Wed, 09 Apr 2025 04:59:48 GMT</pubDate>
            <atom:updated>2025-04-09T05:02:42.986Z</atom:updated>
            <content:encoded><![CDATA[<h4>(Part 2 of 3 in the “Building a Secure, AI-Ready Digital Future” series)</h4><h3>From Insights to Action: The Real Cyber Bottleneck</h3><p>In cybersecurity, it’s no longer about having more data — it’s about making faster, smarter and better decisions with the data we already have.</p><p>At Human Managed, we’ve seen how the traditional security stack struggles to keep up. Alerts escalate. Incidents repeat. Dashboards blink but don’t guide. Teams know something’s wrong, but not what to do next. And the result is the same, wherever you are in the world: <em>decision paralysis</em>.</p><p>This is where <strong>Decision Intelligence Platforms (DIPs)</strong> change the game. They don’t just centralize information — they operationalize it. And most importantly, they do so with <strong>humans embedded at the heart of the system</strong>, not just as reviewers, but as active contributors of business logic, context, and meaning.</p><h3>What Is a Decision Intelligence Platform?</h3><p>A <strong>DIP</strong> is a real-time decision-making engine that connects data, context, and action — while ensuring that <strong>human oversight is embedded across the lifecycle</strong>, not just at the point of exception.</p><p>Here’s what that means in practice:</p><ul><li><strong>Human context-setting:</strong> Business logic, operational constraints, and strategic priorities aren’t hard-coded — they’re continuously shaped by human input.</li><li><strong>Real-time threat detection:</strong> Signals are continuously evaluated, escalated, and acted on — automatically or by teams, depending on context.</li><li><strong>Predictive analytics:</strong> Past outcomes, human decisions, and environmental trends are looped back into the system to improve accuracy and timing.</li><li><strong>Decision transparency:</strong> Every automated or human-guided action is auditable — not just what was done, but <em>why</em> and <em>based on whose logic</em>.</li><li><strong>Continuous feedback loops:</strong> Analysts and operators don’t just execute playbooks — they help evolve them.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E7wFckSB6L9NWzF_QaAdaQ.png" /></figure><blockquote>Human-in-the-loop governance in this model isn’t just about catching what machines miss. It’s about <strong>building the very foundation the machines learn from</strong>.</blockquote><h3>Balancing Innovation and Privacy in ASEAN</h3><p>In federated regions like ASEAN, building a shared cybersecurity fabric doesn’t start with data centralization — it starts with <strong>trust</strong>, <strong>governance</strong>, and <strong>contextual intelligence</strong>.</p><p>That’s why DIPs are so valuable. They:</p><ul><li>Operate locally while aligning to regional norms</li><li>Respect sovereignty and data privacy while enabling shared situational awareness</li><li>Offer real-time, risk-aligned decisions that reflect the reality on the ground</li></ul><blockquote>Most importantly, DIPs allow <strong>human context to coexist with machine speed</strong> — so each country, agency, or enterprise can act fast without losing sight of what matters most to them.</blockquote><h3>Inside Human Managed: Decision Intelligence in Practice</h3><p>We deployed a DIP for a leading ASEAN bank that needed to secure a complex, distributed cloud environment, improve fraud detection, and reduce incident response time — all while maintaining internal and regulatory alignment.</p><p>We didn’t just plug in analytics. We co-designed a <strong>decision framework</strong> with their teams — mapping internal processes, compliance logic, and risk appetite into the platform itself.</p><p>Then we:</p><ul><li>Cataloged digital assets and security controls</li><li>Built dynamic graphs to connect users, systems, and threats</li><li>Developed decision paths that routed actions to the right team, at the right time</li><li>Embedded human approvals, learning checkpoints, and retrospective reviews to evolve models</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIK-1xXAKPlQL4kqd_wZmA.png" /></figure><p>The results:</p><ul><li><strong>90% reduction</strong> in network security violations</li><li><strong>97.8% reduction</strong> in mean time to respond to phishing</li><li><strong>97.9% reduction</strong> in mean time to respond to DDoS attacks</li></ul><blockquote>More importantly, the system didn’t just <em>react faster</em> — it <em>learned smarter</em>, because humans were feeding it meaning every step of the way.</blockquote><h3>The Real Opportunity: Meaningful Automation at Scale</h3><p>The cybersecurity future we need isn’t just fast — it’s <strong>aware</strong>.</p><p>Decision Intelligence Platforms provide the backbone for a new kind of security model:</p><ul><li>Where <strong>automation is contextualized</strong></li><li>Where <strong>decisions are governed</strong>, not guessed</li><li>Where <strong>humans and machines collaborate</strong> in a constant feedback loop of improvement</li></ul><p>This is the foundation of collective intelligence — not just signal-sharing or model optimization, but shared decision frameworks that adapt as we do.</p><h3>What’s Next</h3><p>In <strong>Part 3</strong>, we’ll explore how <strong>federated learning</strong> allows organizations and countries to train shared intelligence models without exposing raw data — and how human-in-the-loop governance plays a critical role there, too.</p><p>Because as powerful as machine learning is, it can’t know what matters — unless we teach it. And unless we stay in the loop.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4fb1cf061e1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/humanmanaged/decision-intelligence-platforms-embedding-human-judgment-into-ai-enhanced-cyber-defense-a4fb1cf061e1">Decision Intelligence Platforms: Embedding Human Judgment into AI-Enhanced Cyber Defense</a> was originally published in <a href="https://medium.com/humanmanaged">humanmanaged</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Collective Intelligence: The Next Frontier in AI-Enabled Cybersecurity]]></title>
            <link>https://medium.com/humanmanaged/collective-intelligence-the-next-frontier-in-ai-enabled-cybersecurity-part-1-of-3-in-the-4439181aa0a5?source=rss----8d657b91fd65---4</link>
            <guid isPermaLink="false">https://medium.com/p/4439181aa0a5</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[intelligence]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Karen Kim]]></dc:creator>
            <pubDate>Mon, 07 Apr 2025 06:41:53 GMT</pubDate>
            <atom:updated>2025-04-09T04:20:23.383Z</atom:updated>
            <content:encoded><![CDATA[<h4>(<em>Part 1 of 3 in the “Building a Secure, AI-Ready Digital Future” series)</em></h4><h3><em>Series Introduction: Building a Secure, AI-Ready Digital Future</em></h3><p><em>As digital economies accelerate across the globe, the tension between growth and security is becoming impossible to ignore.</em></p><p><em>Nowhere is this more visible than in ASEAN — a region sprinting toward a digitally integrated future under the </em><a href="https://asean.org/asean-defa-study-projects-digital-economy-leap-to-us2tn-by-2030/"><strong><em>Digital Economic Framework Agreement (DEFA)</em></strong><em>.</em></a><em> Yet the region’s cybersecurity posture still lags, constrained by uneven readiness, siloed infrastructure, and unresolved questions around data privacy and sovereignty.</em></p><p><em>We at Human Managed believe that </em><strong><em>cybersecurity can’t scale without collaboration</em></strong><em>. While regional initiatives like the </em><a href="https://www.csa.gov.sg/news-events/press-releases/singapore-moves-ahead-to-establish-the-asean-regional-cert-to-strengthen-regional-cybersecurity"><strong><em>ASEAN Regional CERT</em></strong><em> </em></a><em>and the </em><a href="https://www.csa.gov.sg/news-events/press-releases/singapore-and-asean-member-states-deepen-commitment-to-enhance-collective-cybersecurity-in-the-region"><strong><em>Norms Implementation Checklist</em></strong></a><em> lay the groundwork, implementation is the sticking point — especially when it comes to intelligence sharing between governments, enterprises, and critical infrastructure operators.</em></p><p><em>This three-part series explores how </em><strong><em>composite AI</em></strong><em> — especially </em><strong><em>decision intelligence platforms</em></strong><em> and </em><strong><em>federated learning</em></strong><em> — can help regions like ASEAN, and markets globally, move beyond technical patchwork to </em><strong><em>true collective intelligence</em></strong><em>. By enabling privacy-preserving collaboration and real-time decision support, these technologies can help build a cybersecurity model that is </em><strong><em>distributed, resilient, and scalable</em></strong><em>.</em></p><p><em>In highly digitalized sectors like banking, digital trade, and telco, the need is urgent — and the opportunity is real.</em></p><h3>The Decision Dilemma: Noise vs. Signal</h3><p>When you build a tech startup — or anything operational for that matter — you quickly learn that <em>more doesn’t always mean better</em>.</p><p>At Human Managed, we work across borders every day — with clients, suppliers, technology providers — and one challenge shows up everywhere: <strong>there’s too much data, and not enough shared knowledge for consistent actions</strong>.</p><p>Whether you’re sitting in India, Manila, or Singapore, the pattern is glaringly familiar. Dashboards are overloaded or stale. Cyber threats are evolving. Teams are drowning in telemetry and alerts. Meanwhile, decision-makers are still asking the most basic questions:</p><blockquote>“What should we do next?”<em><br> </em>“Can we trust this intel?”<em><br> </em>“Why didn’t we catch this earlier?”</blockquote><h3>The Digital Dilemma: Growth vs. Risk</h3><p>The Association of Southeast Asian Nations (ASEAN) region is digitizing fast. The <strong>Digital Economic Framework Agreement</strong> (DEFA) is a leap forward — creating pathways for trade, data flows, and digital services across 10+ nations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lpIdbzyEupH-tuqAFGZPDw.jpeg" /><figcaption>The DEFA seeks to offer a comprehensive roadmap to empower businesses and stakeholders across ASEAN, through accelerating trade growth, enhancing interoperability, creating a safe online environment, and increasing participation of MSMEs. Key topics such as digital trade, cross-border e-commerce, cybersecurity, digital ID and digital payments are considered along with emerging topics such as AI to ensure a future-proof DEFA.</figcaption></figure><p>But the flip side is clear: more connections = more threats. Cyber threats aren’t just growing in number — they’re becoming more coordinated, more distributed, and harder to detect in time.</p><p>And while the region has made progress with initiatives like the <strong>ASEAN Regional CERT</strong> and the <strong>Norms Implementation Checklist</strong>, we still face implementation friction:</p><ul><li>National readiness is uneven</li><li>Trust between agencies and enterprises remains tentative</li><li>Privacy and sovereignty concerns stall intelligence sharing</li></ul><p>Sound familiar? That’s because this isn’t just ASEAN’s challenge — it’s a <strong>global</strong> one.</p><h3>What’s Missing: Collective Intelligence</h3><p>Here’s what we’ve learned in the last six years of building and operating a data service platform: <strong>you can’t protect what you can’t understand, and you can’t act fast without shared context</strong>.</p><p>Cybersecurity can no longer be a solo sport. It’s time to shift from fragmented defenses to <em>collective intelligence</em> — an approach where insights are:</p><ul><li>Contextualized to each organization&#39;s unique business logics and decision models</li><li>Correlated across sectors and borders</li><li>Translated into prioritized decisions in real time</li><li>Governed by human oversight and system learning</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uTThjZxyBDuprRmvBNjaUA.png" /></figure><blockquote>We need more than tools. We need an <strong>architecture of trust.</strong></blockquote><h3>Composite AI: A System for Collective Thinking</h3><p>That’s where <strong>composite AI</strong> comes in. It’s not a single technology — it’s a layered approach to building intelligence that’s collaborative, explainable, and secure.</p><p>At Human Managed, we think about it in three layers:</p><ol><li><strong>Decision Intelligence Platforms (DIPs):</strong> They unify data flows, business context, and real-time response. Less dashboard, more direction.</li><li><strong>Federated Learning:</strong> A way to share intelligence <em>without</em> sharing raw data — ideal for privacy-first collaboration.</li><li><strong>Human-in-the-loop governance:</strong> The practice of embedding human judgment into every stage of decision automation — not just to review, but to shape, contextualize, and evolve it.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sxRAA4bx0HvWD09JrK1ERA.png" /></figure><blockquote>This is what enables <strong>collective intelligence</strong> — and it’s what lets businesses and nations collaborate without compromising sovereignty.</blockquote><h3>Why This Matters Beyond ASEAN</h3><p>If you’re building systems in the EU, Africa, Latin America, or within federal markets like the U.S. or India, you’re facing the same trade-offs:</p><ul><li>How do we collaborate without exposing raw data?</li><li>How do we stay secure while moving fast?</li><li>How do we align tech and policy to respond in real time?</li></ul><p>ASEAN might be where we are getting real-life use cases, but <strong>this is the future of cyber coordination everywhere</strong>.</p><h3>What’s Next</h3><p>In Part 2, I’ll go deeper into <strong>Decision Intelligence Platforms</strong> — the real-time operating system behind proactive cybersecurity, automated playbooks, and context-rich decision-making.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4439181aa0a5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/humanmanaged/collective-intelligence-the-next-frontier-in-ai-enabled-cybersecurity-part-1-of-3-in-the-4439181aa0a5">Collective Intelligence: The Next Frontier in AI-Enabled Cybersecurity</a> was originally published in <a href="https://medium.com/humanmanaged">humanmanaged</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quick-Start to Integrating OpenTelemetry with FastAPI (Part 2)]]></title>
            <link>https://medium.com/humanmanaged/quick-start-to-integrating-opentelemetry-with-fastapi-part-2-8901f685b472?source=rss----8d657b91fd65---4</link>
            <guid isPermaLink="false">https://medium.com/p/8901f685b472</guid>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[fastapi]]></category>
            <dc:creator><![CDATA[Jesum]]></dc:creator>
            <pubDate>Fri, 11 Aug 2023 13:43:31 GMT</pubDate>
            <atom:updated>2023-08-11T13:43:31.128Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7BMYCdFdd3FbrDuPM18UuQ.jpeg" /></figure><p>The Otel Python SDK is still a work-in-progress piece of technology. These are gaps which I discovered during my journey and how I managed to work around them.</p><h3>HTTPException Logging — Missing details!?</h3><p>The first one was where I couldn’t get Otel to log the error details when I raised a <strong>HTTPException</strong> (see <a href="https://fastapi.tiangolo.com/tutorial/handling-errors/#use-httpexception">https://fastapi.tiangolo.com/tutorial/handling-errors/#use-httpexception</a>). We use NewRelic (and DataDog) for our Otel needs, and in NewRelic, our FastAPI traces would merely show up as a generic HTTP error when <strong>HTTPException</strong> was raised.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*jVvrTi-cP68-TBAVrnmIsA.png" /><figcaption>No details!</figcaption></figure><p>See the <strong>otel.status_description</strong> and <strong>otel.status_code</strong> fields? Nothing useful for me!! Ok, to be fair, you *could* click on the text below that says <strong>There was 1 span event exception</strong> to see the details, but it’s a bit of work and I’m lazy. Here’s what that click will show you:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z-aIVUG159ifNZ5D_5kR_Q.png" /><figcaption>Still not very useful</figcaption></figure><p>Still not very useful. I want to see what’s inside <strong>HTTPException </strong>because that is what gets returned to the browser / API caller.</p><p>The workaround turned out to be simple. I ended up overriding the default HTTPException handler and you can read about it here <a href="https://fastapi.tiangolo.com/tutorial/handling-errors/#override-the-httpexception-error-handler">https://fastapi.tiangolo.com/tutorial/handling-errors/#override-the-httpexception-error-handler</a>. Here’s the code for it:</p><pre>from starlette.exceptions import HTTPException as StarletteHTTPException<br><br># For otel compatibility. It seems HTTPException is only trapped by Otel with otel.status_description<br># set to the string &quot;HTTPException:&quot;, which is not very informative for us. Here, we implement a custom<br># error handler for HTTPException class.<br>@app.exception_handler(StarletteHTTPException)<br>async def http_exception_handler(request, exc):<br>    current_span = trace.get_current_span()<br>    if (current_span is not None) and (current_span.is_recording()):<br>        current_span.set_attributes(<br>            {<br>                &quot;http.status_text&quot;: str(exc.detail),<br>                &quot;otel.status_description&quot;: f&quot;{exc.status_code} / {str(exc.detail)}&quot;,<br>                &quot;otel.status_code&quot;: &quot;ERROR&quot;<br>            }<br>        )<br>    return PlainTextResponse(json.dumps({ &quot;detail&quot; : str(exc.detail) }), status_code=exc.status_code)</pre><p><em>Note the extra code to check if there is a current Otel span recording. This is because the FastAPI production code I wrote can be configured to turn Otel tracing on / off (for debugging purposes).</em></p><p>And now I can see this!! Yay to lazy people!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*fmw_xVv02bGAz-mpdxU6RQ.png" /><figcaption>Much better!</figcaption></figure><p>Just in case you are wondering, <strong>otel.status_description </strong>takes the value directly from <strong>HTTPException.detail</strong> in FastAPI. So code like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-KAQgHa95RRFEe3Lw5TXAg.png" /><figcaption>From <a href="https://fastapi.tiangolo.com/tutorial/handling-errors/#use-httpexception">https://fastapi.tiangolo.com/tutorial/handling-errors/#use-httpexception</a></figcaption></figure><p>Will end up having “Item not found” inside <strong>otel.status_description.</strong></p><h3>Log formatting ignored!</h3><p>The venerable Pypi module called <em>logging</em> has nice integration with the OpenTelemetry SDK.</p><p>With <em>logging</em>, you can specify <strong>Formatter</strong> objects (see <a href="https://docs.python.org/3/library/logging.html#formatter-objects">https://docs.python.org/3/library/logging.html#formatter-objects</a>) which allows you to control what fields end up in your output log text. If you look at <a href="https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/logging/logging.html#envvar-OTEL_PYTHON_LOG_FORMAT">https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/logging/logging.html#envvar-OTEL_PYTHON_LOG_FORMAT</a>, it says you can change the format to whatever you desire. Well, that didn’t work for me!</p><pre>OTEL_PYTHON_LOG_CORRELATION=true<br>OTEL_PYTHON_LOG_FORMAT=%(pathname)s:%(funcName)s:%(lineno)d:%(levelname)s:%(message)s<br>OTEL_PYTHON_LOG_LEVEL=info</pre><p>If you have specify OTEL_PYTHON_LOG_FORMAT like above, I would expect my otel logs to have the full pathname to my Python file, the function name where the log was generated, the line no that generated the log, etc. In my testing, <strong>OTEL_PYTHON_LOG_FORMAT </strong>was ignored.</p><p>So if I do this in my code:</p><pre> logging.info(&quot;Hi there, Mr. Pineapple!&quot;)</pre><p>I get this otel JSON being spit out:</p><pre>{<br>    &quot;body&quot;: &quot;Hi there, Mr. Pineapple!&quot;,<br>    &quot;severity_number&quot;: &quot;&lt;SeverityNumber.INFO: 9&gt;&quot;,<br>    &quot;severity_text&quot;: &quot;INFO&quot;,<br>    &quot;attributes&quot;: {<br>        &quot;otelSpanID&quot;: &quot;2e94f2400eb2f2c9&quot;,<br>        &quot;otelTraceID&quot;: &quot;1b63d84a5a4faec3b1dd77477d1d52df&quot;,<br>        &quot;otelTraceSampled&quot;: true,<br>        &quot;otelServiceName&quot;: &quot;my-fruit-service&quot;<br>    },<br>    &quot;timestamp&quot;: &quot;2023-06-19T06:48:28.605903Z&quot;,<br>    &quot;trace_id&quot;: &quot;0x1b63d84a5a4faec3b1dd77477d1d52df&quot;,<br>    &quot;span_id&quot;: &quot;0x2e94f2400eb2f2c9&quot;,<br>    &quot;trace_flags&quot;: 1,<br>    &quot;resource&quot;: &quot;BoundedAttributes({&#39;telemetry.sdk.language&#39;: &#39;python&#39;, &#39;telemetry.sdk.name&#39;: &#39;opentelemetry&#39;, &#39;telemetry.sdk.version&#39;: &#39;1.18.0&#39;, &#39;service.instance.id&#39;: &#39;dev&#39;, &#39;service.name&#39;: &#39;my-fruit-service&#39;}, maxlen=None)&quot;<br>}</pre><p>Do you see what’s in <strong>body</strong>????? It’s just plain text! Arrrrgggggh.</p><p>The solution was to create a new class that inherits from <strong>LoggingHandler </strong>that respects <strong>Formatter </strong>objects. You can read about the solution from the post I made at Github: <a href="https://github.com/open-telemetry/opentelemetry-python/issues/3353">https://github.com/open-telemetry/opentelemetry-python/issues/3353</a></p><h3>Turn console logging on and off for debugging</h3><p>You could set up your own otel backend for debugging purposes, but hey, I’m lazy and that’s extra work and resources on my laptop. I could also export the data to a SaaS provider like NewRelic or DataDog but the ingest takes a few minutes to show up on the dashboard, and I’d have to open another account (oh gawd, those extra mouse clicks and key clicks!!).</p><p>So you might want to consider turning on console output for your otel JSON payloads (these are the ones that get shipped to your otel backend). In Python, that’s easy to do:</p><pre>from opentelemetry.sdk.trace.export import SimpleSpanProcessor, BatchSpanProcessor, ConsoleSpanExporter<br>from opentelemetry.sdk._logs.export import BatchLogRecordProcessor, SimpleLogRecordProcessor, ConsoleLogExporter<br><br>def otel_trace_init():<br>    trace.set_tracer_provider(<br>       TracerProvider(<br>           resource=Resource.create({}),<br>       ),<br>    )<br>    if DEBUG_LOG_OTEL_TO_PROVIDER:<br>        otel_endpoint_url, otel_http_headers = otel_get_env_vars()<br>        otlp_span_exporter = OTLPSpanExporter(endpoint=otel_endpoint_url,headers=otel_http_headers)<br>        trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_span_exporter))<br>    if DEBUG_LOG_OTEL_TO_CONSOLE:<br>        trace.get_tracer_provider().add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))</pre><p>The code snippet above shows how 2 variables — <strong>DEBUG_LOG_OTEL_TO_PROVIDER </strong>and <strong>DEBUG_LOG_OTEL_TO_CONSOLE </strong>control whether to send those JSONs to the SaaS provider or to the console. You can define these are environment variables and initialize them in Python like so:</p><pre>DEBUG_LOG_OTEL_TO_CONSOLE = os.getenv(&quot;DEBUG_LOG_OTEL_TO_CONSOLE&quot;, &#39;False&#39;).lower() == &#39;true&#39;<br>DEBUG_LOG_OTEL_TO_PROVIDER = os.getenv(&quot;DEBUG_LOG_OTEL_TO_PROVIDER&quot;, &#39;False&#39;).lower() == &#39;true&#39;</pre><p>I also highly recommend you set these environment variables and use them in your Python code:</p><pre>OTEL_ENDPOINT_HTTP_HEADERS=api-key=(really? You&#39;d think I&#39;d leave this here????!)<br>OTEL_ENDPOINT_URL=https://otlp.eu01.nr-data.net:4317<br>OTEL_PYTHON_LOG_CORRELATION=true<br>OTEL_PYTHON_LOG_FORMAT=%(pathname)s:%(funcName)s:%(lineno)d:%(levelname)s:%(message)s<br>OTEL_PYTHON_LOG_LEVEL=debug<br>OTEL_RESOURCE_ATTRIBUTES=service.instance.id=production,service.name=api.invicta.io,telemetry.sdk.name=opentelemetry,telemetry.sdk.language=python,telemetry.sdk.version=1.18.0<br>DEBUG_LOG_OTEL_TO_CONSOLE=true<br>DEBUG_LOG_OTEL_TO_PROVIDER=true</pre><p>You can read more about the <strong>OTEL_PYTHON_* </strong>variables here (<a href="https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/logging/logging.html">https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/logging/logging.html</a>).</p><p>Oh, please note you <em>can </em>have the = symbol in environment variables like in <strong>OTEL_ENDPOINT_HTTP_HEADERS</strong> above. It works just fine and as you’d expect it. :)</p><p>Don’t forget to read <a href="https://medium.com/p/f45cbbb27b58">part 3</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8901f685b472" width="1" height="1" alt=""><hr><p><a href="https://medium.com/humanmanaged/quick-start-to-integrating-opentelemetry-with-fastapi-part-2-8901f685b472">Quick-Start to Integrating OpenTelemetry with FastAPI (Part 2)</a> was originally published in <a href="https://medium.com/humanmanaged">humanmanaged</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quick-Start to Integrating OpenTelemetry with FastAPI (Part 1)]]></title>
            <link>https://medium.com/humanmanaged/quick-start-to-integrating-opentelemetry-with-fastapi-part-1-2be4fec874bc?source=rss----8d657b91fd65---4</link>
            <guid isPermaLink="false">https://medium.com/p/2be4fec874bc</guid>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Jesum]]></dc:creator>
            <pubDate>Mon, 10 Jul 2023 06:05:03 GMT</pubDate>
            <atom:updated>2023-07-10T06:05:03.176Z</atom:updated>
            <content:encoded><![CDATA[<p>No long useless rants here. Just code. Let’s go!! (That snack saw me through some of the code below, so it deserves a place here!)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L545r0KbZQDG3lFYYASr0A.jpeg" /></figure><pre>from opentelemetry import trace<br>from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor<br>from opentelemetry.sdk.trace import TracerProvider<br>from opentelemetry.sdk.resources import Resource<br>from opentelemetry.sdk.trace.export import SimpleSpanProcessor, BatchSpanProcessor, ConsoleSpanExporter<br>from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter<br><br>from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter<br>from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler <br>from opentelemetry.sdk._logs.export import BatchLogRecordProcessor, SimpleLogRecordProcessor, ConsoleLogExporter<br>from opentelemetry.instrumentation.logging import LoggingInstrumentor<br>from opentelemetry._logs import (<br>    SeverityNumber,<br>    get_logger,<br>    get_logger_provider,<br>    std_to_otel,<br>    set_logger_provider<br>)<br><br># The default Otel SDK completely ignores formatters when outputting the message being logged.<br># We overcome this by creating our own LoggingHandler class which respects formatters.<br>class FormattedLoggingHandler(LoggingHandler):<br>    def emit(self, record: logging.LogRecord) -&gt; None:<br>        msg = self.format(record)<br>        record.msg = msg<br>        record.args = None<br>        self._logger.emit(self._translate(record))<br><br>def otel_get_env_vars():<br>    otel_http_headers = {}<br>    try:<br>        decoded_http_headers = os.getenv(&quot;OTEL_ENDPOINT_HTTP_HEADERS&quot;, &quot;&quot;)<br>        key_values = decoded_http_headers.split(&quot;,&quot;)<br>        for key_value in key_values:<br>            key, value = key_value.split(&quot;=&quot;)<br>            otel_http_headers[key] = value<br>    except Exception as e:<br>        print(f&quot;Error parsing OTEL_ENDPOINT_HTTP_HEADERS: {str(e)}&quot;)<br>    otel_endpoint_url = os.getenv(&quot;OTEL_ENDPOINT_URL&quot;, None)<br>    <br>    return otel_endpoint_url, otel_http_headers<br><br>def otel_trace_init():<br>    trace.set_tracer_provider(<br>       TracerProvider(<br>           resource=Resource.create({}),<br>       ),<br>    )<br>    if DEBUG_LOG_OTEL_TO_PROVIDER:<br>        otel_endpoint_url, otel_http_headers = otel_get_env_vars()<br>        otlp_span_exporter = OTLPSpanExporter(endpoint=otel_endpoint_url,headers=otel_http_headers)<br>        trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_span_exporter))<br>    if DEBUG_LOG_OTEL_TO_CONSOLE:<br>        trace.get_tracer_provider().add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))<br><br>def otel_logging_init():<br>    # ------------Logging<br>    # Set logging level<br>    # CRITICAL = 50<br>    # ERROR = 40<br>    # WARNING = 30<br>    # INFO = 20<br>    # DEBUG = 10<br>    # NOTSET = 0<br>    # default = WARNING<br>    log_level = str(os.getenv(&quot;OTEL_PYTHON_LOG_LEVEL&quot;, &quot;INFO&quot;)).upper()<br>    if (log_level == &quot;CRITICAL&quot;):<br>        log_level = logging.CRITICAL<br>        print(f&quot;Using log level: CRITICAL / {log_level}&quot;)<br>    elif (log_level == &quot;ERROR&quot;):<br>        log_level = logging.ERROR<br>        print(f&quot;Using log level: ERROR / {log_level}&quot;)<br>    elif (log_level == &quot;WARNING&quot;):<br>        log_level = logging.WARNING<br>        print(f&quot;Using log level: WARNING / {log_level}&quot;)<br>    elif (log_level == &quot;INFO&quot;):<br>        log_level = logging.INFO<br>        print(f&quot;Using log level: INFO / {log_level}&quot;)<br>    elif (log_level == &quot;DEBUG&quot;):<br>        log_level = logging.DEBUG<br>        print(f&quot;Using log level: DEBUG / {log_level}&quot;)<br>    elif (log_level == &quot;NOTSET&quot;):<br>        log_level = logging.INFO<br>        print(f&quot;Using log level: NOTSET / {log_level}&quot;)<br><br>    # ------------ Opentelemetry loging initialization<br><br>    logger_provider = LoggerProvider(<br>        resource=Resource.create({})<br>    )<br>    set_logger_provider(logger_provider)<br>    if DEBUG_LOG_OTEL_TO_CONSOLE:<br>        console_log_exporter = ConsoleLogExporter()<br>        logger_provider.add_log_record_processor(SimpleLogRecordProcessor(console_log_exporter))<br>    if DEBUG_LOG_OTEL_TO_PROVIDER:<br>        otel_endpoint_url, otel_http_headers = otel_get_env_vars()<br>        otlp_log_exporter = OTLPLogExporter(endpoint=otel_endpoint_url,headers=otel_http_headers)<br>        logger_provider.add_log_record_processor(BatchLogRecordProcessor(otlp_log_exporter))<br><br>    otel_log_handler = FormattedLoggingHandler(logger_provider=logger_provider)<br><br>    # This has to be called first before logger.getLogger().addHandler() so that it can call logging.basicConfig first to set the logging format<br>    # based on the environment variable OTEL_PYTHON_LOG_FORMAT<br>    LoggingInstrumentor().instrument()<br>    logFormatter = logging.Formatter(os.getenv(&quot;OTEL_PYTHON_LOG_FORMAT&quot;, None))<br>    otel_log_handler.setFormatter(logFormatter)<br>    logging.getLogger().addHandler(otel_log_handler)<br><br><br>def fastapi_init():<br>    app = FastAPI(<br>        docs_url = None,<br>        redoc_url = None,<br>        title = &quot;Placeholder&quot;,<br>        version = &quot;Placeholder&quot;,<br>        description = &quot;Placeholder&quot;<br>    )<br>    <br>    @app.get(&quot;/redoc&quot;, include_in_schema=False)<br>    def overridden_redoc():<br>        return get_redoc_html(openapi_url=&quot;/openapi.json&quot;, title=&quot;Human Managed&quot;, redoc_favicon_url=&quot;https://invicta.io/favicon.ico&quot;)<br><br>    @app.get(&quot;/docs&quot;, include_in_schema=False)<br>    def overridden_swagger():<br>        return get_swagger_ui_html(openapi_url=&quot;/openapi.json&quot;, title=&quot;Human Managed&quot;, swagger_favicon_url=&quot;https://invicta.io/favicon.ico&quot;)<br><br>    # For otel compatibility. It seems HTTPException is only trapped by Otel with otel.status_description<br>    # set to the string &quot;HTTPException:&quot;, which is not very informative for us. Here, we implement a custom<br>    # error handler for HTTPException class.<br>    @app.exception_handler(StarletteHTTPException)<br>    async def http_exception_handler(request, exc):<br>        current_span = trace.get_current_span()<br>        if (current_span is not None) and (current_span.is_recording()):<br>            current_span.set_attributes(<br>                {<br>                    &quot;http.status_text&quot;: str(exc.detail),<br>                    &quot;otel.status_description&quot;: f&quot;{exc.status_code} / {str(exc.detail)}&quot;,<br>                    &quot;otel.status_code&quot;: &quot;ERROR&quot;<br>                }<br>            )<br>        return PlainTextResponse(json.dumps({ &quot;detail&quot; : str(exc.detail) }), status_code=exc.status_code)<br>    FastAPIInstrumentor.instrument_app(app)        <br>    return app<br><br>DEBUG_LOG_OTEL_TO_CONSOLE = os.getenv(&quot;DEBUG_LOG_OTEL_TO_CONSOLE&quot;, &#39;False&#39;).lower() == &#39;true&#39;<br>DEBUG_LOG_OTEL_TO_PROVIDER = os.getenv(&quot;DEBUG_LOG_OTEL_TO_PROVIDER&quot;, &#39;False&#39;).lower() == &#39;true&#39;<br>otel_trace_init()<br>otel_logging_init()<br>app = fastapi_init()<br>app.include_router(apirouter.api_router_v1)<br>app.include_router(apirouter.invict_api_router_v1)<br>app.include_router(apirouter.api_router)<br>app.add_middleware(GZipMiddleware, minimum_size=1000000)</pre><p>And there you have it. This is what’s in <strong>main.py </strong>in my implementation (ok, I lied, there are a few more bits of missing code removed because they aren’t relevant to the otel + FastAPI discussion :P ). Short &amp; sweet!</p><p>Before you go jumping into this code, I recommend you read the <a href="https://medium.com/p/8901f685b472">2nd part </a>of this series as well. It expands on the code above.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2be4fec874bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/humanmanaged/quick-start-to-integrating-opentelemetry-with-fastapi-part-1-2be4fec874bc">Quick-Start to Integrating OpenTelemetry with FastAPI (Part 1)</a> was originally published in <a href="https://medium.com/humanmanaged">humanmanaged</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we mint JWTs]]></title>
            <link>https://medium.com/humanmanaged/how-we-mint-jwts-b2c22f8925e?source=rss----8d657b91fd65---4</link>
            <guid isPermaLink="false">https://medium.com/p/b2c22f8925e</guid>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[identity-management]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[authorization]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Jesum]]></dc:creator>
            <pubDate>Thu, 06 Apr 2023 05:42:54 GMT</pubDate>
            <atom:updated>2023-04-06T05:45:00.159Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SjO62DlhMsTqS2Utzo7_5A.png" /></figure><p>A while ago, I wrote about how we integrate <a href="https://medium.com/u/7b847836aa03">Okta</a> and <a href="https://medium.com/u/9ef9638b3092">Auth0</a> using Auth0’s Social Connector (it runs OAuth2 behind the scenes). Today I will get into a bit more detail about how it’s done behind the scenes.</p><p>Our applications are registered directly with Auth0 as a “Regular Web Application”. This first layer of registration allows us to control <strong>application level authorization</strong>. In other words, only applications authorized by our platform can communicate with our backend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CDaKiYeaYYvET--y" /></figure><p>The second level authentication and authorization that we do is at the <strong>user-level</strong>. This is where the Okta + Auth0 integration comes into play (there’s also <strong>Cerbos</strong> but we’ll get into that as a separate topic). The user is redirected to Okta to complete sign-in (enter your username, MFA, etc.) and then returned to Auth0 to continue with the login flow.</p><p>Auth0 then executes the <strong>Fetch User Profile</strong> script for that Social connector we configured to talk to Okta. In this script, the Auth0 runtime populates a number of parameters with useful return information from Okta. You get 3 things:</p><ul><li>the user’s access token</li><li>a ctx or context object which gives you access to the id token</li><li>and a callback, cb, for you to return a profile object to Auth0 that we use to construct the JWT</li></ul><p>The access token is specific to the user who just completed the logon in Okta. We use this access token as part of the Authorization Bearer scheme to call the Okta OIDC UserInfo API (<a href="https://developer.okta.com/docs/reference/api/oidc/#userinfo">https://developer.okta.com/docs/reference/api/oidc/#userinfo</a>). This allows us to fetch additional attributes about the user.</p><p>It is highly recommended to use an Okta custom authorization server for this purpose (<a href="https://developer.okta.com/docs/guides/customize-authz-server/main/">https://developer.okta.com/docs/guides/customize-authz-server/main/</a>). Why? Because the custom authorization server allows more control over the JWT that Okta mints (<a href="https://developer.okta.com/docs/guides/customize-authz-server/main/">https://developer.okta.com/docs/guides/customize-authz-server/main/</a>). It’s also possible to move some of the custom claims being generated in Auth0’s <strong>Fetch User Profile</strong> script to an Okta custom authorization server (it’s really up to you, but we prefer Auth0’s approach because Okta’s expression language pales in comparison to raw Javascript :) )</p><p>The return JSON from the Okta API call makes it very easy to parse properties and do additional processing. For example, here is part of our <strong>Fetch User Profile </strong>script:</p><pre>function(accessToken, ctx, cb) {<br> function get_user_profile()<br> {<br>  request({<br>    url: &quot;https://&lt;our okta domain&gt;.okta.com/oauth2/&lt;id of our custom auth server&gt;/v1/userinfo&quot;,<br>    method: &quot;GET&quot;,<br>    headers: {<br>     &quot;Authorization&quot;: &quot;Bearer &quot; + accessToken,<br>     &quot;Content-Type&quot;: &quot;application/json&quot;<br>    }<br>   },</pre><pre>function(e, r, b) {<br>    if (e) return cb(e);<br>    if (r.statusCode !== 200) return cb(new Error(&#39;StatusCode: &#39; + r.statusCode));<br>    var profile = JSON.parse(b);</pre><p>The code above is part of our <strong>Fetch User Profile</strong> script in Auth0. The <strong>profile</strong> object is what we return to Auth0 as part of our <strong>Fetch User Profile </strong>script. This object is used by Auth0 to create the user profile and issue a JWT. This can be done like so at the end of the script:</p><pre>cb(null, profile);</pre><p>To add a new claim to the JWT that Auth0 mints, all you have to do is to modify the <strong>profile</strong> object to your heart’s content. For example:</p><pre>profile.geo_ip_country = &quot;Unknown&quot;;<br>if ((&#39;geoipcountry&#39;) in idToken) {<br> profile.geo_ip_country = idToken.geoipcountry;<br>}</pre><p>You will now have a claim called <strong>geo_ip_country</strong> in your JWT. What is <strong>idToken</strong> then? That is simply the user’s OIDC token and we can get that from one of the function parameter that Auth0 passes to us i.e. the <strong>ctx </strong>object:</p><pre>var idToken = jwt.decode(ctx.id_token);</pre><p>Now that you have both the access token and id token, you can manipulate the final JWT that gets issued! Oh, one more thing — how do you remove a claim? Easy as pie:</p><pre>delete profile.user_type;</pre><p>When you view your the JIT created user in Auth0, you will see something similar to the image posted below where each field is generated from the <strong>profile </strong>object that you manipulate. These fields can be used for 2 purposes — for auditing (you have a snapshot of the user the last time they logged on, so it’ really up to you what fields you want to populate for auditing purposes), and finally for JWT generation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/0*l6uz-v1mJF6M_jyZ" /></figure><p>To get these fields added to the JWT token that Auth0 mints, you use another wonderful Auth0 feature called the login action flow (<a href="https://auth0.com/docs/customize/actions/flows-and-triggers/login-flow">https://auth0.com/docs/customize/actions/flows-and-triggers/login-flow</a>).</p><p>Here’s a sample of how you can do that:</p><pre>exports.onExecutePostLogin = async (event, api) =&gt; <br>{<br>  var ManagementClient = require(&#39;auth0&#39;).ManagementClient;<br>  var management = new ManagementClient<br>  (<br>    {<br>      domain: event.secrets.domain,<br>      clientId: event.secrets.clientId,<br>      clientSecret: event.secrets.clientSecret<br>    }<br>  );<br><br>  var params = { id: event.user.user_id };<br>  let user_attributes = await management.getUser(params);<br>  <br>  if (&#39;geo_ip_country&#39; in user_attributes)<br>  {<br>    api.accessToken.setCustomClaim(&quot;geo_ip_country&quot;, user_attributes[&#39;geo_ip_country&#39;]);<br>  }</pre><p>If you need more information, feel free to reach out to us at <a href="https://www.linkedin.com/company/humanmanaged/">Human Managed</a>. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2c22f8925e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/humanmanaged/how-we-mint-jwts-b2c22f8925e">How we mint JWTs</a> was originally published in <a href="https://medium.com/humanmanaged">humanmanaged</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Human Managed Platform handle Cloud agnostic Object storage needs]]></title>
            <link>https://medium.com/humanmanaged/how-human-managed-platform-handle-cloud-agnostic-object-storage-needs-e0186194a3e7?source=rss----8d657b91fd65---4</link>
            <guid isPermaLink="false">https://medium.com/p/e0186194a3e7</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[platform]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Vishnukumar N]]></dc:creator>
            <pubDate>Thu, 06 Apr 2023 03:11:36 GMT</pubDate>
            <atom:updated>2023-04-06T03:11:36.553Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zBBelPCg8qWGf8oWpYn6qg.jpeg" /></figure><p><strong>Where we store the collected processed data?</strong><br>We picked MinIO as Cloud agnostic Highly Available Object Storage for our Platform that can be Scaled Up or Scaled Out on demand to store the Processed logs. MinIO Object storage acts a Data lake layer of the Human Managed Platform. Life Cycle management policies are applied in MinIO such that only the fast access to hot tier data lives in MinIO and the rest are further extended to the Cloud’s native Object storage service like S3, Cloud Storage etc.</p><p><strong>How do we manage MinIO?</strong><br>We created custom Cloud agnostic Helm chart and templates to deploy the appropriates that are needed first for MinIO. MinIO Tenants are then deployed on respective Kubernetes Clusters in various Clouds and are replicated amongst each other. Hence even if a whole Cloud solution fails, HM Platform’s Object storage solution would still function as they are replicated across other Tenants in other Clouds. This supports seamless continuation of service consumption. Each MinIO Tenant is separately monitored by Monitoring team using Prometheus metrics and Grafana dashboards. Human Managed uses Istio Ingress Gateway to effectively utilize the Cloud Load Balancers to support ingress to the Object storage.</p><p><strong>Duration to bring a Tenant to live?</strong><br>Provisioning Object storage solution in Human Managed is automated and would take approximately 30 minutes to come to live.</p><p><strong>Note: </strong>All product names, logos and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names, logos and brands does not imply endorsement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0186194a3e7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/humanmanaged/how-human-managed-platform-handle-cloud-agnostic-object-storage-needs-e0186194a3e7">How Human Managed Platform handle Cloud agnostic Object storage needs</a> was originally published in <a href="https://medium.com/humanmanaged">humanmanaged</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A startup’s first ever conference that transformed them.]]></title>
            <link>https://medium.com/humanmanaged/a-startups-first-ever-conference-that-transformed-them-f956a60c7c15?source=rss----8d657b91fd65---4</link>
            <guid isPermaLink="false">https://medium.com/p/f956a60c7c15</guid>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[events]]></category>
            <category><![CDATA[teamwork]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Karen Kim]]></dc:creator>
            <pubDate>Mon, 14 Nov 2022 09:06:10 GMT</pubDate>
            <atom:updated>2022-11-14T09:06:10.702Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>We launched our platform and the Human Managed I.DE.A (Intelligence, DEcisions, Actions) at GovWare Conference. This is our transformative experience in running an exhibition booth and many other things for the first time.</strong></h4><p>Back in June 2022, we took the decision to be a <a href="https://www.govware.sg/govware/expo-sponsors">Platinum Plus Sponsor</a> of <a href="https://www.govware.sg/home">GovWare</a>, Asia’s largest cybersecurity event held in Singapore. This year was its 7th edition — the first in-person conference since 2019, and was anticipating over 10,000 participants (and in the end, did have that many participants!).</p><p>Initially there was push back from the team: “We are not ready for a stage as big as GovWare” / “We’ve never run an event before” / “Why don’t we start with a smaller scale event first?”</p><p>These were all valid concerns, but we had a stronger case to go for it:</p><ul><li>COVID-19 hit soon after our inception. Borders shut down and we were forced to build and grow 100% remotely. This equipped us with a lot of valuable learnings on distributed work, but it also slowed us down, especially when it came to creative, synchronous work. So we missed a few opportunities to share our work in the market in a way that was impactful enough. Now was not the time to wait a little longer – it was time to go big.</li><li>Immediate market feedback from customers, suppliers and partners is invaluable — we wanted to see how people who don’t know us interact with our products with zero prior knowledge.</li><li>Immense learning opportunity to be surrounded by such a rich variety of cybersecurity players in the ecosystem, from industry behemoths to startups.</li><li>External pressure, when applied appropriately, can boost motivation and effectiveness. This was the moment for our entire team to jump straight into the pit and show what we’ve got.</li></ul><p>And oh, are we glad we did it.</p><h3>The I.DE.A that solves the problem of the Information Age</h3><p>There are countless problems in the Information Age. The particular one we have chosen to solve is <strong>indecision and inaction from information overload</strong>.</p><p>Our problem statement became clearer and simpler over the years, after continuously researching the market and observing directly from our customers.</p><p>Organizations today have a lot of data.</p><p>What they do not have, however, are what comes after the data. “Now you have it, what do you do with it?” :</p><ol><li><strong>Intelligence. </strong>They are not getting relevant intelligence from their data when they need it.</li><li><strong>Decision. </strong>They are unable to take effective and responsible decisions from data.</li><li><strong>Action. </strong>They are not making progress on executing their decisions because things get dropped or get blocked by inefficient processes.</li></ol><p>The best intelligence and decisions are of no use if they are not followed up with actions. So, we built a platform that doesn’t do just one thing but many things. At our core, we: generate intelligence, recommend decisions, and track actions.</p><p>We call this the Human Managed I.DE.A. — and this naturally became our leading message for GovWare. Check out our 30 seconds <a href="https://www.youtube.com/watch?v=n7R7OA9uNEI">teaser video</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1CvdHuTWy357V9UuAiOE2g.png" /><figcaption>Our service flows turns data to intel (Discover, Profile, Detect) and intel into decision ( Triage, Course of Action) and decision into action (React or Resolve).</figcaption></figure><h3>The booth for design thinking and open source</h3><p>For our booth, we wanted to openly share our thinking and execution behind our products and services.</p><p>We use flow charts and sticky notes whenever we do design thinking, so it made perfect sense to use our booth’s double-sided walls to display the <a href="https://pages.qwilr.com/resilience-lQxe93hTxatW">functions, logics, and flows</a> that literally run our platform services.</p><p>We don’t have teams for marketing, branding or events, but by sticking to our core values, we had our DIY booth design that was distinctively ‘HM’ in a matter of a few weeks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zgDezqyjg8RrJDBF" /><figcaption>This was a fun and spontaneous session: finalizing our artworks and planning their placements based on the traffic flow in our booth.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xx8GQ3QjP9OuZoYX" /><figcaption>The team got hands-on to test mounting our wall displays in our office.</figcaption></figure><p>Inside the walls, our booth area was divided into two to represent our ‘front stage’ and ‘back stage’.</p><p>The front stage was set up like a studio for our platform’s <a href="https://www.linkedin.com/feed/update/urn:li:activity:6962574387975778304/?utm_source=share&amp;utm_medium=member_desktop">Present stack</a> — our web application. We had multiple touch screens for visitors to interact with and ask questions on our product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qJufPoFEOS_DJAET" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LU9xFYGqad_usPWW" /></figure><p>In contrast to the clean and simple front stage, the back stage was designed for interactive and creative conversations. We had physical whiteboards and sticky notes covering an entire wall for design thinking discussions, workshops, and brainstorming sessions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a5ex1T_7rpOA-jKh" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lgPQsfqJXNdeDHqG" /></figure><p>Put the components all together and, voila! There it was: the Human Managed platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oEz5m6tHHnWAkrf9" /></figure><p>It was a surreal and emotional moment to see our ideas come to life in the middle of the Marina Bay Sands Expo and Convention Centre — the largest exhibition venue in Singapore.</p><h3>The interactions between the ecosystem players</h3><p>Our goals for GovWare were simple: to share our I.DE.A with our ecosystem, and get up to five quality conversations with potential users, customers, and partners. We are proud to have achieved them and much more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yid2LTktDahSYth7" /><figcaption>The energy and interest from the crowd were beyond our expectations.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1jhNdGeURA1wwaZm" /><figcaption>We showed what’s under the hood, like our Kubernetes cluster deployment and management.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wUng-83b5Z4lHraN" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6YYz8vbIKuM-aztQ" /><figcaption>Private and public sectors from various countries strongly resonated with the problem we are solving.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AYgMj1AzQqGz0c8H" /><figcaption>And of course, we didn’t go without some engaging whiteboard sessions.</figcaption></figure><h3>The highs of facing the unknowns</h3><p>GovWare was an event of many firsts for us, stretching ourselves in ways we couldn’t imagine until we found ourselves breaking through the only things that were imagined in the first place: limits.</p><p>A few highlights to note:</p><ul><li>I gave a talk on <a href="https://www.govware.sg/govware-2022-full-agenda/the-invisible-face-of-digital">how businesses can achieve their digital transformation expectations through cybersecurity</a>, in a world that is totally distributed and ‘invisible’. The entire process from preparation to delivery was not easy, but extremely rewarding and fun. More than anything, I was excited to share HM’s perspective on a topic that is a challenge for anyone running a digital business today. More on this in a separate blog soon.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JSJ41KBLzpe1FmTP" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aZSd4fFAkc6Kzodh" /><figcaption>A slide from my talk that illustrates the huge amount of data exchange that happens throughout a distributed supply chain of assets of a digital business. Here I highlight the data flow throughout the ‘invisible’ side of the business that lies behind the front-facing web application (i.e. between the machine-to-machine interactions such as APIs and code.)</figcaption></figure><ul><li>All of the team got in front of the camera to speak about their domain — unscripted and on the fly. Their initial hesitation didn’t last long, and in no time they were sharing confidently about their work, in their own words and styles. <a href="https://www.youtube.com/channel/UCiMUhacEjl4z_lu8TFFEztQ">Follow our YouTube channel</a> where these content will be uploaded soon!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aXZsa2n-TdGHqkOk" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uxlHuCvvFME3Srl2" /><figcaption>All smiles for lights, camera, action!</figcaption></figure><ul><li>We demonstrated a deployment of our partner <a href="https://www.crowdstrike.com/products/observability/falcon-logscale/">Crowdstrike’s Falcon LogScale log management solution</a> in less than three minutes on Kubernetes, using: a) <a href="https://crossplane.io/docs/v1.9.html">Crossplane</a> (an infrastructure control plane) to provision compute resources independent of cloud provider. b) a CD pipeline with <a href="https://argo-cd.readthedocs.io/en/stable/">ArgoCD</a> to install the LogScale operator.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B4eZaTS8Yh00lKOH" /><figcaption>Our Chief Engineer (aka. MicroBuilder) Jesum destroys and builds a log management service in less than 3 minutes using Crossplane and ArgoCD.</figcaption></figure><h3>Learnings</h3><p>The months leading up to GovWare were the most stimulating experience I’ve had to date at Human Managed.</p><p>Everything we had shown at GovWare, we have built ourselves, from the messaging, to the web app, to the booth artworks. And to get market reaction and feedback real time on something that you’ve poured your everything into is an indescribable experience.</p><p>We’ve learned so many valuable lessons. Here are a few.</p><ul><li>Build for internal resiliency, not external glory. Videos and photos only tell a small, curated portion of the reality. There were so many curveballs and multiple things did not go to plan. We had technical issues at the eleventh hour before the event. In the end, some team members (including those that were going to lead some key demos and workshops) were not able to attend the conference at all. But we backed each other up. Some things were completely missed, but some other things turned out better than plan A. There were no unexpected curveballs that our team was not capable of handling and that was the much more important part, not whether everything was presented as we intended.</li><li>Doubts (both self-inflicted and external) will always be there. But keep going and give it your best. You never know who might be inspired by your example. Startups less than two years old came up to our booth to learn from our journey — they said they were motivated by our presence, especially as a fellow Singaporean startup. A student came to ask whether we had any work opportunity because she got inspired by talking to a few of our team members. Getting this kind of feedback was awesome. Going forward, we hope to inspire many more in our ecosystem as well as continue to learn from them.</li><li>You will never be fully ready, but you are (almost always) more ready than you think.</li><li>Create a challenging but supportive environment where your team can try new things on their own. An event or conference is a great one, but so are other initiatives, like an impromptu video shoot ;)</li></ul><h3>Looking ahead</h3><p>We felt like we’ve aged a few years by the time the conference was over. Internally, we joke about “Life Before GovWare” and “Life After GovWare”. There is definitely some truth to this, in that the conference was a marker of a state change for us.</p><p>We’ve been sprinting for as long as we can remember. But we all know that life after GovWare is just another starting line, with more spring in our sprints and a growing conviction that we are on the right path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DLw5VJocqa0AOKA6" /></figure><p>Thank you to everyone who has interacted with us. Your feedback, encouragements and critiques gave us so much clarity on the next steps to improve on our products and services. We would love to have more of you along on our journey.</p><ul><li>Want to conduct an I.DE.A workshop or build a prototype in less than two weeks? Have an idea for collaboration or partnership? Feedback on our products and <a href="https://pages.qwilr.com/resilience-lQxe93hTxatW">services</a>? <a href="https://calendly.com/humanmanaged/discovery">Book a meeting</a> with us or <a href="/service/mailto:hello@humanmanaged.com">email us</a>.</li><li>Follow us on <a href="https://www.linkedin.com/company/humanmanaged/mycompany/?viewAsMember=true">LinkedIn</a>, <a href="https://twitter.com/humanmanaged?lang=en">Twitter</a>, and <a href="https://www.instagram.com/humanmanaged/?hl=en">Instagram</a>.</li><li>Subscribe to our <a href="https://www.youtube.com/channel/UCiMUhacEjl4z_lu8TFFEztQ">YouTube</a> channel.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f956a60c7c15" width="1" height="1" alt=""><hr><p><a href="https://medium.com/humanmanaged/a-startups-first-ever-conference-that-transformed-them-f956a60c7c15">A startup’s first ever conference that transformed them.</a> was originally published in <a href="https://medium.com/humanmanaged">humanmanaged</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Orchestrating Data Collection via Airbyte]]></title>
            <link>https://medium.com/humanmanaged/orchestrating-data-collection-via-airbyte-670ddae7c41?source=rss----8d657b91fd65---4</link>
            <guid isPermaLink="false">https://medium.com/p/670ddae7c41</guid>
            <category><![CDATA[airbyte]]></category>
            <category><![CDATA[data-ingestion]]></category>
            <category><![CDATA[human-managed]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Michael C. Reyes]]></dc:creator>
            <pubDate>Wed, 10 Aug 2022 02:35:54 GMT</pubDate>
            <atom:updated>2022-08-10T02:35:54.421Z</atom:updated>
            <content:encoded><![CDATA[<h3>Orchestrating Data Collection with Airbyte</h3><h4>To get anything out of big data, you first need to ingest data from multiple sources. Big data collection is notoriously difficult. But does it have to be?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZdOffhFycGVns2sGSKqIEQ.png" /></figure><p>Imagine having one smartphone that can only handle one mobile application at a time. It would be manageable if you only use 1 or 2 applications, but what if you have 50 mobile applications: One smartphone for Google News, another one for Facebook, another one for Instagram, and so on. Sounds awful, right?!</p><figure><img alt="Hands of people using mobile apps in smartphones" src="https://cdn-images-1.medium.com/max/1024/1*QVAYHs9T6ovagG_z3oirjg.jpeg" /><figcaption>Phone illustration vector created by <a href="https://www.freepik.com/vectors/phone-illustration">pch.vector</a><em> on </em><a href="https://www.freepik.com/"><em>FreePik</em></a></figcaption></figure><p>That is one of the main problems we’ve encountered in trying to develop the data ingestion for our Collect Microservice at Human Managed. Our platform generates intelligence and action from customers’ data, which means we work with many data sources as our input for analysis(currently standing at 30+ data sources and counting!)</p><p>Initially, we developed numerous bash and python scripts running via cron job or AWS lambda just to automatically fetch the data from multiple APIs and data sources. However, this became difficult for us to manage once we tried to scale up and replicate for each data source and customer that we have.</p><p>We had to find a way to address this with something that we can <strong>manage, maintain, scale, and modify </strong>for our use cases.</p><h3>Airbyte Open-Source ELT Platform</h3><p>Introducing <a href="https://airbyte.com/"><strong>Airbyte</strong></a>, an open-source data integration platform that syncs data from applications, APIs, and databases and transfer them to the destination that you want such as database and object storage. Remember the analogy I gave earlier? Airbyte is <em>the type of smartphone that can handle multiple mobile applications, which makes data integration simple, secure, and extensible.</em></p><p>We utilized existing Airbyte connectors that fits into our use cases and also developed our own connectors when necessary. Since Airbyte is an open-source platform, the connectors available are developed by the community where you can also <a href="https://airbyte.com/connector-requests">request</a> and <a href="https://docs.airbyte.com/contributing-to-airbyte/">contribute your own</a><strong> (We plan to contribute our custom connectors and improvements in existing ones so that we could also give back to the community, so stay tuned!)</strong>.</p><p>Once we converted our automation scripts into Airbyte connectors, we were able to easily implement and organize the batch collection that happens within our Collect microservice. In replicating the same connection but for another customer, we just set the configuration in the existing connector, set the destination, and that’s it! We were able to integrate the another data source into our platform in just minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*5TjW4cfVjJTdPLwYSbE1FA.png" /><figcaption><a href="https://airbyte.com/">Airbyte ELT Overview</a></figcaption></figure><p>Referring back to the analogy of having multiple smartphone, with Airbyte, we have now a <strong>one single powerful smartphone that can handle and receive data from multiple mobile applications </strong>(You may see the available connectors <a href="https://docs.airbyte.com/integrations/">here</a>). This solved the problem of having to maintain, and organize the automation of ingesting data which can also be customized, scaled, and managed however suits our needs.</p><h3>Drawbacks and Limitations</h3><p>There will always be a setback in using any other tool, even if it’s Airbyte. The first thing that comes into mind is the learning curve required to develop and maintain custom airbyte connectors. If you are not familiar with the concepts, you’ll have to figure out a lot of these essential concepts to grasp in order to properly understand and implement an Airbyte connector. But once figured out, most of the development will be easier than before because of the <a href="https://airbyte.com/connector-development-kit">Airbyte CDK</a> thanks to Airbyte Team.</p><p>As for its limitation, the current lack of support for data backup of raw data in-between the Extract (E) and Load (L) is not yet possible. This conflicts with our architecture of storing the raw logs without the metadata added by Airbyte.</p><h3>Conclusion</h3><p>Even though I’ve also mentioned the cons of implementing Airbyte. The benefits far outweigh the disadvantages and gives you the following capabilities:</p><ul><li>Open source and free for all to contribute</li><li>Cloud Agnostic</li><li>Customizable, Modular, Scalable, Secured, and Compliant</li><li>Set Connection Syncs and Incremental Stream State</li><li>Unlimited Sync Frequency [No Tier]</li></ul><p>The capacity and features of Airbyte to unify your data integration pipelines under one fully managed platform is fit to develop our Collect microservice since it enables us to easily integrate data sources, develop custom connectors when necessary, and manage these connections without worrying too much on security and compliance.</p><p>With a fast growing community and contributions, more and more connectors will be made and improved in the future which will continue to innovate and push to evolve the data ingestion capabilities of Airbyte.</p><p>You could learn more about Airbyte and get involved in the open-source platform by visiting the <a href="https://airbyte.com/"><em>official Airbyte website</em></a>.</p><blockquote>About Human Managed</blockquote><blockquote>Human Managed is a data company with a purpose to empower responsible decisions. Founded in 2018, we are headquartered in Singapore and operate across Hong Kong, Philippines, and India. As a self-funded ASEAN startup, we currently have more than 40 employees and a growing gig community, serving global customers in the essential services sectors to improve their cyber, risk and digital maturity.</blockquote><blockquote>Today, organizational success is driven by quick and effective decisions from an abundance of data. To enable this consistently, our platform finds answers and provides recommendations based on intuitive models and collective intelligence. Our products are built for people who process, analyze, triage, communicate, and make decisions from high volume of information each day.</blockquote><blockquote>We are always keen to explore new ways to build, co-create and solve problems. Come say <a href="/service/mailto:hello@humanmanaged.com">hello@humanmanaged.com</a> and follow us on <a href="https://www.linkedin.com/company/31544499/">LinkedIn</a>, <a href="https://www.instagram.com/humanmanaged">Instagram</a> and <a href="https://twitter.com/humanmanaged">Twitter</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=670ddae7c41" width="1" height="1" alt=""><hr><p><a href="https://medium.com/humanmanaged/orchestrating-data-collection-via-airbyte-670ddae7c41">Orchestrating Data Collection via Airbyte</a> was originally published in <a href="https://medium.com/humanmanaged">humanmanaged</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating code management with Git]]></title>
            <link>https://medium.com/humanmanaged/automating-code-management-with-git-7b89147a73ef?source=rss----8d657b91fd65---4</link>
            <guid isPermaLink="false">https://medium.com/p/7b89147a73ef</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Jared Villanueva]]></dc:creator>
            <pubDate>Tue, 02 Aug 2022 00:52:53 GMT</pubDate>
            <atom:updated>2022-08-02T00:52:53.678Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fZ0TPSlZABLDUPz-qWby8Q.png" /></figure><p>I’ve joined Human Managed — a data company that empowers decision — in late December 2021 as a front end developer. In a short amount of time, I went through a steep learning curve, working with technologies and processes that I have never worked with before.</p><p>One of those is Git, a distributed, open-source version control (VCS) that enables you to:</p><ul><li>store code</li><li>track revision history</li><li>merge code changes</li><li>revert to earlier code version when needed</li></ul><p>Back in my college days, I used to code without any version control tool like Git. What I did was copy and paste source code folders and rename it with a version number. My reasoning back then is, since I do code alone, then there’s no point in using a version control system. Lo and behold, this idea backfired to me in my thesis project. There was a time when I needed to update a block of code for a fix, and it took me hours to do so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1gxiSNtl07iKhi2ItrKgZw.png" /><figcaption>I used to agree with the guy at the bottom😂</figcaption></figure><p>I knew that I needed to learn git when I joined HM. I won’t be coding alone anymore since I’ll be collaborating with my colleagues. When I was trying to learn it during my first few weeks, I initially thought it was really complicated since there are a lot of commands that you need to understand. But when I started to contribute in our internal codebase, I (also with the help of my colleagues) managed to learn it along the way. I have lots of realizations on how this tool can benefit myself before and could’ve made me more productive back then, how I wish I learned about it sooner!</p><h3>How Human Managed uses Git</h3><p>On a day-to-day basis, we use git and <a href="https://about.gitlab.com/">GitLab </a>to store our source code, keep track of our changes for every apps per microservices and run a CI/CD pipeline to ensure that stages are automated and done correctly.</p><p>Through git, we have a commit ID which keeps track of our changes. We can always go back to whichever version we have, in any cases that we wish to.</p><p>For example, imagine you are working on a feature within a webapp and made around six commits already. You tested the application, then found that there is a bug in your app. Upon checking, you realized that the bug was caused by the initial commit you had in the source code. With git, you can revert a change and undo it, so you can go back to the original state of the source code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/0*YaPO15Ra1El8X4m5" /></figure><p>In our applications, we use GitLab as our git-based web platform environment for version control and source code management. Every day, there are changes to be made in the codebase. A developer can work concurrently with the other one through git since we utilize its branching functionality and it being a decentralized version control system. We create a branch for a specific update in our webapp, and merge it to the main ‘production’ source code once it’s finished. This way, we are not dependent on each other if we want to make changes to our applications. We can do things seamlessly without the worry of having any conflicts with what we are doing.</p><h3><strong>CI/CD Automation with GitLab</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*23wV60Mvjt8UQk0-G4tKqA.jpeg" /></figure><p><a href="https://about.gitlab.com/company/">GitLab </a>is used as our DevOps platform. Since it is git-based, we use GitLab as our remote repository to store our source code. An event is triggered via git push/commit to the remote which will execute a CI/CD pipeline through GitLab. We use it to run automated build, test, and deployment. Through gitlab, we can deploy apps immediately without the need to wait for someone to finish something, since steps are already automated.</p><p>Every action we do within our internal repositories are tracked via webhook which is configured in a CI channel wherein relevant individuals are added. We use this as our collaboration method to notify and make everyone aware on what’s happening to our apps in microservices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*hnHyo9s-o4qVdpc6f6Y9Ig.png" /><figcaption>webhook notification in slack</figcaption></figure><p>Using GitLab enables us to automate majority of version control and source code management. It’s beneficial for software engineering since manual steps are already taken out of the picture through its automated devops functionalities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*XE-_7nqWIiA7D4mEWw938A.jpeg" /></figure><p>Imagine doing manual build, test, and deployment for your applications. You’ll go back to the days where it’s not as efficient as compared to the automated stages that a devops tool like GitLab can offer. For sure it must have brought a lot of headaches to every developer before! Lucky enough, we live in an advanced tech-era now where everything can be automated already.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/0*5lZncdzXZEm7QLcZ" /><figcaption>don’t be like this guy</figcaption></figure><p>The screenshot below is an example of <em>.gitlab-ci.yml file</em>. In layman’s term, this is the set of instructions that you need to configure to automate all the actions/stages that you’ll be automating in your application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fCdYNLzq7L0_XdbAYYJJcA.png" /></figure><p>Someone just needs to understand how to write this CI file, and all the end-to-end stages for CI/CD will be automated already. That’s how handy and powerful this file is!</p><p>This is just the beginning of our code automation story. Just like git, this narrative can be further <em>branched out</em> to areas like application security, automated deployments, and gitops. But that would be for another blog! 😊</p><blockquote>About Human Managed</blockquote><blockquote>Human Managed is a data company with a purpose to empower responsible decisions. Founded in 2018, we are headquartered in Singapore and operate across Hong Kong, Philippines, and India. As a self-funded ASEAN startup, we currently have more than 40 employees and a growing gig community, serving global customers in the essential services sectors to improve their cyber, risk and digital maturity.</blockquote><blockquote>Today, organizational success is driven by quick and effective decisions from an abundance of data. To enable this consistently, our platform finds answers and provides recommendations based on intuitive models and collective intelligence. Our products are built for people who process, analyze, triage, communicate, and make decisions from high volume of information each day.</blockquote><blockquote>We are always keen to explore new ways to build, co-create and solve problems. Come say <a href="/service/mailto:hello@humanmanaged.com">hello@humanmanaged.com</a> and follow us on <a href="https://www.linkedin.com/company/31544499/">LinkedIn</a>, <a href="https://www.instagram.com/humanmanaged">Instagram</a> and <a href="https://twitter.com/humanmanaged">Twitter</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b89147a73ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/humanmanaged/automating-code-management-with-git-7b89147a73ef">Automating code management with Git</a> was originally published in <a href="https://medium.com/humanmanaged">humanmanaged</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>