<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by poga on Medium]]></title>
        <description><![CDATA[Stories by poga on Medium]]></description>
        <link>https://medium.com/@poga?source=rss-afe8ba83ff7a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MBAlXToya9FNMM-YfiCagw.jpeg</url>
            <title>Stories by poga on Medium</title>
            <link>https://medium.com/@poga?source=rss-afe8ba83ff7a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:57:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@poga/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Crescendo Lab Engineering Evolution Part 4: Our Engineering Principles in Practice]]></title>
            <link>https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog/crescendo-lab-engineering-evolution-part-4-our-engineering-principles-in-practice-56431629433a?source=rss-afe8ba83ff7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/56431629433a</guid>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[poga]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 10:56:04 GMT</pubDate>
            <atom:updated>2025-03-13T10:56:04.417Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post is part of a 4-post series:</em></p><ol><li><a href="https://medium.com/p/41b9a700df6e"><em>Crescendo Lab Engineering Evolution Part 1: Building Confidence Through Velocity</em></a></li><li><a href="https://medium.com/p/95e8bf48ea8a"><em>Crescendo Lab Engineering Evolution Part 2: Reliability — Ensuring Uninterrupted Customer Experiences</em></a></li><li><a href="https://medium.com/p/2cf9fa28e7ed"><em>Crescendo Lab Engineering Evolution Part 3: Efficiency — Maximizing Engineering Impact</em></a></li><li><a href="https://medium.com/p/56431629433a"><em>Crescendo Lab Engineering Evolution Part 4: Our Engineering Principles in Practice</em></a><em>(this post)</em></li></ol><p>In the previous three parts of this series, we explored how our engineering organization transformed through focusing on confidence, reliability, and efficiency. These journeys weren’t just about implementing new tools or processes — they were about fundamentally changing how we approach our work. In this final installment, we’ll recap the engineering principles that emerged from these experiences and now guide our daily work.</p><h3>The Formation of Our Engineering Principles</h3><p>Our engineering principles weren’t created in a vacuum or copied from another organization’s handbook. They emerged organically from our experiences, challenges, and successes. As we implemented the changes described in the previous posts, patterns began to emerge in what worked for our team and what didn’t.</p><p>These principles developed through an iterative process, with each new challenge providing an opportunity to refine our understanding of who we are as an engineering organization. Early versions were tested, revised, and sometimes completely reimagined as we put them into practice.</p><p>Most importantly, we view these principles as living documents rather than static rules. They evolve as we grow, as technology changes, and as we learn from both our successes and failures. This adaptive approach ensures our principles remain relevant and continue to guide us effectively.</p><h3>Our Core Pillars</h3><p>We optimize for four key metrics that define our success:</p><ul><li><strong>Reliability</strong>: Our systems should work consistently and predictably</li><li><strong>Scalability</strong>: Our solutions should handle growth gracefully</li><li><strong>Observability</strong>: We should understand what’s happening in our systems at all times</li><li><strong>Cost</strong>: We deliver value while being mindful of resources</li></ul><h3>Production is the Only Truth</h3><ul><li>We validate our work in production using controlled rollouts. We use feature flags and canary deployments to minimize risk.</li><li>Every feature ships with monitoring and the ability to roll back with feature flags</li><li>A feature is only considered complete when validated in production and actively delivering customer value.</li></ul><h3>Embrace Simplicity</h3><ul><li>We start simple and iterate rather than building complex solutions upfront</li><li>Before creating abstractions, we implement similar solutions multiple times to truly understand the problem space</li><li>Code should be modular enough to be easily removed as requirements change</li><li>Incorrect abstraction is way more expensive than no abstraction</li></ul><h3>Knowledge Sharing is Key</h3><ul><li>Pair programming is our primary method of knowledge transfer</li><li>We believe the best code comes from collaborative work</li><li>Regular pairing reduces documentation burden and spreads expertise across the team</li><li>Complex problems are solved faster when tackled together</li></ul><h3>Conclusion</h3><p>The transformation of our engineering organization has been about more than just improving our technical capabilities — it’s been about creating an environment where engineers can do their best work. By establishing clear principles rooted in real experiences, we’ve built a foundation that guides our decisions and shapes our culture.</p><p>These principles — focusing on our core pillars, embracing production as the source of truth, pursuing simplicity, and sharing knowledge collaboratively — have become the compass that guides our engineering journey.</p><h3>Join Our Engineering Revolution</h3><p>Are you passionate about building reliable systems that make a real impact? Do you believe in the power of simplicity, collaboration, and continuous improvement? We’re looking for engineers who don’t just write code, but who want to be part of shaping a progressive engineering culture.</p><p>Our team is expanding, and we’re seeking like-minded engineers who resonate with our principles and want to build meaningful technology in an environment that values both technical excellence and human connection. If you’re excited about working somewhere where your ideas matter and where you’ll be empowered to do your best work, we want to hear from you.</p><p>Check out our <a href="https://www.yourator.co/companies/CrescendoLab">careers page</a> to learn more about open positions and how you can become part of our engineering evolution.</p><p><em>This concludes our four-part series on Engineering Evolution. We welcome your thoughts, questions, and experiences as we all continue to evolve our engineering practices.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56431629433a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog/crescendo-lab-engineering-evolution-part-4-our-engineering-principles-in-practice-56431629433a">Crescendo Lab Engineering Evolution Part 4: Our Engineering Principles in Practice</a> was originally published in <a href="https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog">漸強實驗室 Crescendo lab — 職涯部落格 Career blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crescendo Lab Engineering Evolution Part 3: Efficiency — Maximizing Engineering Impact]]></title>
            <link>https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog/crescendo-lab-engineering-evolution-part-3-efficiency-maximizing-engineering-impact-2cf9fa28e7ed?source=rss-afe8ba83ff7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cf9fa28e7ed</guid>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[poga]]></dc:creator>
            <pubDate>Wed, 12 Mar 2025 05:56:35 GMT</pubDate>
            <atom:updated>2025-03-12T05:56:35.246Z</atom:updated>
            <content:encoded><![CDATA[<h3>Crescendo Lab Engineering Evolution Part 3: Efficiency — Maximizing Engineering Impact</h3><p><em>This post is part of a 4-post series:</em></p><ol><li><a href="https://medium.com/p/41b9a700df6e"><em>Crescendo Lab Engineering Evolution Part 1: Building Confidence Through Velocity</em></a></li><li><a href="https://medium.com/p/95e8bf48ea8a"><em>Crescendo Lab Engineering Evolution Part 2: Reliability — Ensuring Uninterrupted Customer Experiences</em></a></li><li><a href="https://medium.com/p/2cf9fa28e7ed"><em>Crescendo Lab Engineering Evolution Part 3: Efficiency — Maximizing Engineering Impact</em></a><em> (this post)</em></li><li><a href="https://medium.com/p/56431629433a"><em>Crescendo Lab Engineering Evolution Part 4: Our Engineering Principles in Practice</em></a></li></ol><p><em>This is the third post in our four-part series on engineering evolution. </em><a href="https://www.notion.so/cresclab/link-to-part-1"><em>Part 1</em></a><em> covered how we built confidence through velocity, and </em><a href="https://www.notion.so/cresclab/link-to-part-2"><em>Part 2</em></a><em> focused on enhancing reliability for uninterrupted customer experiences.</em></p><h3>The Challenge: Stalled Product Development</h3><p>Several months ago, our engineering organization faced a sobering reality: despite our best intentions, product development was slowing down. Features weren’t being delivered fast enough, critical system improvements were being deprioritized, postmortem action items were languishing unaddressed, and knowledge remained siloed within individual teams. The results were predictable but painful — repeated incidents, declining system performance, and growing technical debt.</p><p>We needed a fundamental shift in how we approached engineering work — not just in our processes, but in our culture. The answer wasn’t working harder; it was working smarter by maximizing the impact of every engineering hour.</p><h3>Engineering Autonomy as a Multiplier</h3><p>We recognized that the key to unlocking efficiency lay in empowering our engineers with greater autonomy. Traditional approaches where engineers waited for permission to make improvements or required layers of approval before deploying changes were creating bottlenecks throughout our development process.</p><p>Our solution was a redefinition of ownership: <strong>Your feature is only considered finished when it’s rolled out to all customers.</strong> This principle completely transformed how our engineers thought about their work. It was no longer sufficient to simply write code and hand it off; engineers became responsible for understanding the full impact of their changes, implementing appropriate guardrails with feature flags, and monitoring the resulting effects on customers.</p><p>This ownership mindset eliminated numerous handoffs and approval gates that had previously slowed us down. Engineers gained the confidence to make decisions independently while remaining accountable for outcomes. The result was faster iteration cycles and more thoughtful implementation, as engineers considered the entire lifecycle of their features from the beginning.</p><h3>The Small PR Revolution: Speed Through Smaller Bites</h3><p>While we covered our deployment improvements extensively in <a href="https://www.notion.so/cresclab/link-to-part-2">Part 2</a> of this series, it’s worth briefly highlighting how our shift to smaller, more frequent deployments fundamentally changed our efficiency. By breaking work into smaller chunks and deploying more often, we reduced merge conflicts, simplified code reviews, and decreased the cognitive load on our team.</p><p>This approach paired perfectly with our emphasis on engineering autonomy, giving teams the ability to ship improvements continuously rather than waiting for major releases.</p><h3>Postmortem Discipline: Learning from Every Incident</h3><p>Perhaps no area better exemplified our efficiency challenges than our handling of postmortems. Prior to our transformation, we had a telling pattern:</p><ol><li>An incident would occur</li><li>We’d conduct a thorough postmortem</li><li>We’d identify action items to prevent recurrence</li><li>Most of those actions would be ignored</li><li>The incident would eventually repeat</li></ol><p>This cycle was not only inefficient — it was demoralizing and dangerous. Our solution came through two parallel approaches:</p><p>First, we established true accountability by having staff engineers review postmortem actions and ensuring they were prioritized. Second, we eliminated a common blocker by making it clear that “I need to communicate with PM” was no longer an acceptable excuse for delaying critical system improvements.</p><p>We formalized the expectation that engineers would use their 10% time (more on this below) to address postmortem actions without requiring additional permission. The results were remarkable:</p><ul><li>Our postmortem action execution rate improved from approximately 30% to 100%</li><li>We were able to move several critical self-hosted infrastructure components to managed solutions</li><li>Repeat incidents for the same root causes dropped to nearly zero</li></ul><p>A particularly impactful example was our migration from self-hosted RabbitMQ to a managed solution. After experiencing repeated incidents with our message broker infrastructure, rather than just applying band-aid fixes, the team used their 10% time to plan and execute a complete migration to a managed service. This strategic shift not only eliminated an entire class of infrastructure-related issues but also freed up valuable engineering time previously spent on maintenance, allowing the team to focus on higher-value work.</p><h3>The Scout Rule and 10% Improvement PRs</h3><p>To systematically address technical debt and system improvements, we borrowed the Scout Rule from the programming community: “Leave the code better than you found it.” But we went a step further by formalizing this practice through dedicated improvement time.</p><p>We asked engineers to proactively protect 10% of their time (effectively one day per two-week sprint) specifically for system improvements. This wasn’t time that needed to be “earned” or negotiated for — it was an expected part of everyone’s work.</p><p>During sprint planning, engineers explicitly reserved this time and identified improvement targets, which typically fell into three categories:</p><ol><li><strong>Maintenance work</strong>: Upgrading dependencies and environments, addressing security vulnerabilities</li><li><strong>Performance optimization</strong>: Identifying and resolving bottlenecks, improving response times</li><li><strong>Refactoring and testing</strong>: Paying down technical debt, improving code quality, and enhancing test coverage</li></ol><p>By formalizing this practice, we made it clear that system improvement wasn’t an afterthought — it was an integral part of our engineering culture. The compounding effect of these incremental improvements has been remarkable, with system performance, stability, and maintainability showing steady improvements quarter over quarter.</p><h3>Self-Formed Optimization Teams: Catching Low-Hanging Fruit</h3><p>While structured processes drove many of our improvements, some of the most impactful work came from an unexpected source: informal, self-organized optimization teams.</p><p>A group of senior engineers who deeply understood both our systems and products began identifying and addressing “low-hanging fruit” — minor improvements that would significantly enhance the customer experience but weren’t prioritized in the product roadmap. These autonomous, self-driven engineers took advantage of their deep knowledge to deliver rapid improvements.</p><p>These teams operated with high autonomy, responding quickly to customer requests for minor UX improvements, additional options, or detailed settings. The approach dramatically improved customer trust and eliminated lengthy communication cycles between customers, product managers, and engineering.</p><p>The impact is best captured by this quote from one of our customers:</p><blockquote><em>“Omg! Your team is incredible for being able to release the new functions so quickly. I truly feel that choosing MAAC was the best decision I made!”</em></blockquote><p>These self-formed teams demonstrated how autonomy, when combined with deep system knowledge, could significantly multiply engineering impact.</p><h3>Knowledge Amplification Through Pairing</h3><p>The final piece of our efficiency puzzle addressed a persistent challenge: knowledge silos. When critical information about systems or practices resided with just one or two team members, we created invisible bottlenecks and single points of failure.</p><p>We implemented a structured pair programming approach that has transformed how knowledge flows through our organization:</p><ol><li>Engineers pair once or twice a week, typically for a full afternoon session</li><li>During sprint planning, we explicitly identify tickets that would benefit from pair programming with a simple flag: “This would be a good ticket for pairing”</li><li>Pairing sessions are scheduled as regular team activities, not as exceptional events</li></ol><p>The results went far beyond our initial goal of reducing onboarding time (which did improve by 60%). Regular pairing led to:</p><ul><li>Improved grasp of the overall system architecture across the engineering team</li><li>More consistent use of best practices, patterns, and tools</li><li>Reduced time spent on code reviews, as many issues were caught during pairing</li><li>More resilient teams that could handle absences or departures without disruption</li><li>Higher-quality code through real-time collaboration and knowledge sharing</li></ul><p>Contrary to initial concerns that pairing might reduce productivity by tying up two engineers, we found that the quality improvements and knowledge transfer benefits far outweighed any short-term productivity considerations. In fact, pairing often accelerated work on complex tasks by combining complementary perspectives and skills.</p><h3>Key Metrics That Transformed Our Organization</h3><p>Our efficiency initiatives yielded measurable improvements across several critical metrics:</p><ul><li><strong>10 Forward Deploys per engineer per sprint</strong>: Enabling rapid iteration and feedback</li><li><strong>≤10-minute release cycle time</strong>: Reducing the friction of shipping code</li><li><strong>100% Postmortem execution rate</strong>: Ensuring we learn from every incident</li><li><strong>60% reduction in onboarding time</strong>: Getting new team members productive faster</li><li><strong>20% reduction in development cycle time</strong>: Delivering features more efficiently</li></ul><p>But perhaps the most meaningful metrics came from our customers, who consistently commented on our improved responsiveness and the noticeable enhancements to our product’s performance and reliability.</p><h3>The Journey Continues</h3><p>Our efficiency transformation wasn’t achieved overnight, and it required constant reinforcement of the cultural shifts we were trying to establish. There were moments of skepticism and setback, but the cumulative impact of these changes has created a virtuous cycle — engineers experiencing the benefits of these practices became their strongest advocates.</p><p>As we look ahead, we continue to refine our approach. We’re exploring ways to further reduce coordination overhead, enhance our knowledge sharing practices, and measure the long-term impact of our 10% improvement time.</p><p>In our final post in this series, we’ll explore how these experiences shaped our core engineering principles and how we’re putting those principles into practice every day. We’ll dive into our philosophy that “Production is the Only Truth” and how we’ve embraced simplicity as a core value in our engineering approach.</p><p><em>What efficiency practices have transformed your engineering organization? We’d love to hear your experiences in the comments below.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cf9fa28e7ed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog/crescendo-lab-engineering-evolution-part-3-efficiency-maximizing-engineering-impact-2cf9fa28e7ed">Crescendo Lab Engineering Evolution Part 3: Efficiency — Maximizing Engineering Impact</a> was originally published in <a href="https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog">漸強實驗室 Crescendo lab — 職涯部落格 Career blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crescendo Lab Engineering Evolution Part 2: Reliability — Ensuring Uninterrupted Customer…]]></title>
            <link>https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog/crescendo-lab-engineering-evolution-part-2-reliability-ensuring-uninterrupted-customer-95e8bf48ea8a?source=rss-afe8ba83ff7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/95e8bf48ea8a</guid>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[poga]]></dc:creator>
            <pubDate>Wed, 12 Mar 2025 05:56:30 GMT</pubDate>
            <atom:updated>2025-03-12T05:56:30.147Z</atom:updated>
            <content:encoded><![CDATA[<h3>Crescendo Lab Engineering Evolution Part 2: Reliability — Ensuring Uninterrupted Customer Experiences</h3><p><em>This post is part of a 4-post series:</em></p><ol><li><a href="https://medium.com/p/41b9a700df6e"><em>Crescendo Lab Engineering Evolution Part 1: Building Confidence Through Velocity</em></a></li><li><a href="https://medium.com/p/95e8bf48ea8a"><em>Crescendo Lab Engineering Evolution Part 2: Reliability — Ensuring Uninterrupted Customer Experiences</em></a><em> (this post)</em></li><li><a href="https://medium.com/p/2cf9fa28e7ed"><em>Crescendo Lab Engineering Evolution Part 3: Efficiency — Maximizing Engineering Impact</em></a></li><li><a href="https://medium.com/p/56431629433a"><em>Crescendo Lab Engineering Evolution Part 4: Our Engineering Principles in Practice</em></a></li></ol><p>In our <a href="https://www.notion.so/cresclab/link-to-part-1">previous post</a>, we discussed how building confidence through velocity transformed our engineering culture. Today, we’ll dive into the next chapter of our journey: elevating reliability to ensure our customers can count on our platform when they need it most.</p><h3>The Wake-Up Call</h3><p>Our reliability journey began with pain. Within a short timeframe, we experienced multiple incidents that shook our customers’ confidence in our product. Two types of issues were particularly concerning:</p><ol><li><strong>Silent feature failures</strong>: Critical features were breaking without anyone noticing for extended periods. The frustrating reality was that many customers simply stopped using these features rather than reporting issues, creating an invisible erosion of trust.</li><li><strong>Regional configuration blind spots</strong>: Changes that worked perfectly in our primary regions would cause outages in specific geographic areas due to different feature configurations and availability settings. These region-specific issues were particularly challenging to catch with traditional testing.</li></ol><p>As a B2B company, our product isn’t something users interact with constantly throughout the day, but it absolutely needs to work flawlessly when they do need it. These incidents made it clear: we needed a systematic approach to reliability.</p><h3>The Reliability Task Force</h3><p>We formed a cross-functional Reliability Task Force with a clear charter: transform our approach to system stability and create a framework for continuous reliability improvement.</p><p>The task force’s first action was a brutally honest assessment of our current state. We mapped out our critical systems, identified monitoring gaps, and documented our reactive (rather than proactive) incident response patterns. The gap between where we were and where we needed to be was substantial, but having clarity on the challenge was the first step toward addressing it.</p><h3>Setting Customer-Aligned SLOs</h3><p>One of our most important realizations was that reliability targets are meaningless if they don’t reflect what matters to customers. As a B2B solution, our reliability needs are somewhat unique — our platform isn’t used constantly like a social network, but it absolutely must function correctly when customers need it.</p><p>We aligned our Service Level Objectives (SLOs) directly to our customers’ experience by tying them to the success of smoke tests on critical user journeys. This approach ensured we were measuring what truly mattered: could customers successfully accomplish what they came to our platform to do?</p><p>This customer-centric approach to SLOs fundamentally changed how we thought about reliability. It wasn’t about abstract uptime percentages; it was about consistently delivering on our promises to customers.</p><h3>What Are Smoke Tests?</h3><p>Before diving into our implementation, let’s clarify what we mean by “smoke tests.” In software engineering, smoke tests are a subset of test cases that cover the most critical functionality of a system. The name comes from hardware testing: if you turn on a circuit board and see smoke, you know something is seriously wrong without needing further testing.</p><p>In our context, smoke tests are lightweight, automated tests that verify our core user journeys are functioning correctly. Unlike unit or integration tests that focus on code correctness, smoke tests focus on the end-to-end experience from the customer’s perspective. They don’t verify that the code is perfect — they verify that the product works.</p><h3>Smoke Tests: The Customer Perspective</h3><p>We knew we needed a way to continuously verify that our most critical user journeys were working correctly. Our solution was a comprehensive suite of smoke tests that simulated real customer interactions with our platform.</p><p>We initially focused on eight critical paths:</p><ol><li>User login</li><li>SMS/LINE broadcast functionality</li><li>Auto-reply mechanisms</li><li>Rich menu functionality</li><li>Deeplink operations</li><li>OpenAPI (critical for customer integrations)</li><li>Marketing automation user journeys</li><li>Message inbox functionality</li></ol><p>These smoke tests began simply but evolved to incorporate the nuances of real-world usage. They ran continuously, providing an early warning system for issues that might otherwise go undetected until customers reported them.</p><h3>Real-World Impact: Catching the Invisible</h3><p>The impact of our smoke tests was immediate and significant. In one particularly memorable case, we deployed a configuration change that seemed harmless in our testing environments. Within one minute of deployment, our smoke tests detected that users in a specific region couldn’t access a critical feature due to a configuration error that only manifested in that geographic context.</p><p>Without the smoke tests, this issue might have gone undetected for hours or even days, as it affected only a subset of our users and wouldn’t have triggered our traditional monitoring alerts. Because we caught it immediately, we were able to roll back the change before most customers in the affected region even attempted to use the feature.</p><h3>From Alert Fatigue to Alert Precision</h3><p>Our monitoring systems were originally designed with good intentions, but over time, they had evolved into a cacophony of notifications that were increasingly ignored by our engineering team. Alert fatigue had set in, with critical issues sometimes being lost in the noise.</p><p>We undertook a comprehensive review of our alerting system with three simple principles:</p><ol><li>If an alert requires no action, disable it.</li><li>If an alert requires action, it must ping the relevant engineers and link to a clear playbook.</li><li>Every alert must be acknowledged and resolved by an engineer.</li></ol><p>This disciplined approach reduced our total alert volume by eliminating noise while ensuring that genuine issues received immediate attention. More importantly, it established a culture where alerts were taken seriously because they always indicated something truly requiring attention.</p><h3>The Results: Measurable Improvement</h3><p>The impact of our reliability initiatives was substantial and measurable:</p><ul><li>Our Mean Time To Resolution (MTTR) dropped dramatically from multiple hours to tens of minutes.</li><li>Our Service Level Objective (SLO) achievement increased to 99.5%.</li><li>Most importantly, customer-reported incidents decreased significantly, with many potential issues being caught and resolved before customers ever noticed them.</li></ul><p>The cultural impact was equally important. Our engineering team developed a shared sense of ownership for reliability, with everyone understanding how their work contributed to the customer experience. Reliability was no longer seen as a separate concern or someone else’s responsibility — it became integral to our definition of quality engineering.</p><h3>Lessons Learned</h3><p>Our reliability journey taught us several valuable lessons:</p><ol><li><strong>Reliability must be measured through the customer lens.</strong> Abstract system metrics are meaningless if they don’t reflect the actual customer experience.</li><li><strong>Customer-focused testing reveals what internal testing misses.</strong> Smoke tests from the customer perspective caught issues our internal testing missed.</li><li><strong>Alert quality matters more than quantity.</strong> Every alert should be actionable and important. Anything less trains your team to ignore notifications.</li><li><strong>Reliability is a continuous journey.</strong> Our 99.5% SLO achievement isn’t the end — it’s a milestone on an ongoing path of improvement.</li></ol><h3>Looking Forward</h3><p>As we continue our engineering evolution, reliability remains a cornerstone of our approach. In our next post, we’ll explore how we’ve optimized for engineering efficiency to maximize the impact of our team while maintaining the reliability standards we’ve established.</p><p>We believe that reliability isn’t just a technical concern — it’s the foundation of customer trust. In a B2B environment where our platform must work when customers need it, reliability is perhaps the most visible manifestation of our commitment to customer success.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95e8bf48ea8a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog/crescendo-lab-engineering-evolution-part-2-reliability-ensuring-uninterrupted-customer-95e8bf48ea8a">Crescendo Lab Engineering Evolution Part 2: Reliability — Ensuring Uninterrupted Customer…</a> was originally published in <a href="https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog">漸強實驗室 Crescendo lab — 職涯部落格 Career blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crescendo Lab Engineering Evolution Part 1: Building Confidence Through Velocity]]></title>
            <link>https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog/crescendo-lab-engineering-evolution-part-1-building-confidence-through-velocity-41b9a700df6e?source=rss-afe8ba83ff7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/41b9a700df6e</guid>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[poga]]></dc:creator>
            <pubDate>Wed, 12 Mar 2025 05:56:22 GMT</pubDate>
            <atom:updated>2025-03-12T07:39:17.565Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post is part of a 4-post series:</em></p><ol><li><a href="https://medium.com/p/41b9a700df6e"><em>Crescendo Lab Engineering Evolution Part 1: Building Confidence Through Velocity</em></a><em> (this post)</em></li><li><a href="https://medium.com/p/95e8bf48ea8a"><em>Crescendo Lab Engineering Evolution Part 2: Reliability — Ensuring Uninterrupted Customer Experiences</em></a></li><li><a href="https://medium.com/p/2cf9fa28e7ed"><em>Crescendo Lab Engineering Evolution Part 3: Efficiency — Maximizing Engineering Impact</em></a></li><li><a href="https://medium.com/p/56431629433a"><em>Crescendo Lab Engineering Evolution Part 4: Our Engineering Principles in Practice</em></a></li></ol><p>At Crescendo Lab, our journey toward engineering excellence began with a simple but powerful realization: <strong>confidence in change is the bedrock of innovation</strong>. Like many growing organizations, we found ourselves trapped in a cycle that limited our ability to evolve rapidly. Deployments were lengthy, infrequent, and often anxiety-inducing events.</p><p>Two-hour deployment windows, occurring just once per sprint, weren’t just inefficient — they were actively hindering our ability to iterate, learn, and deliver value to our customers. This post shares how we transformed our approach to software delivery, establishing the foundation for a more reliable and efficient engineering organization.</p><h3>The Starting Point: Slow and Uncertain</h3><p>Our deployment process used to look familiar to many engineering teams:</p><ul><li>Each deployment took approximately 2 hours</li><li>Teams deployed only once per sprint</li><li>We bundled multiple changes into a single deployment</li><li>Every deployment required extensive reviews and coordination</li><li>Verification of implementations in production was delayed by weeks</li></ul><p>The consequences were predictable: engineers hesitated to make changes, product managers struggled to plan feature releases with certainty, and customers waited longer for both new features and bug fixes. Our innovation velocity suffered, and so did our confidence.</p><h3>Accelerating Delivery Cycles</h3><p>To break free from this pattern, we focused on several key initiatives:</p><h4>1. Streamlining CI/CD Pipelines</h4><p>We systematically analyzed and optimized our CI/CD configuration, starting with test execution. By refactoring our test suite to run in parallel and removing redundant tests, we significantly reduced build times. The technical improvements included:</p><ul><li>Optimizing test execution to run faster and more reliably</li><li>Fine-tuning our CI/CD configurations for maximum throughput</li><li>Caching dependencies to avoid repetitive downloads</li><li>Parallelizing jobs that could run independently</li></ul><p>These technical changes laid the groundwork, but the real transformation came from our process changes.</p><h4>2. Small Pull Requests</h4><p>Perhaps the most significant shift in our approach was enforcing a “small PR” philosophy. Instead of combining numerous changes into massive deployment packages, we established a new standard:</p><ul><li>Each PR should be focused on a single logical change</li><li>Changes must be protected by feature flags whenever possible</li><li>Deployments happen as soon as a PR is approved, not at scheduled intervals</li></ul><p>This approach dramatically reduced the scope of each code review and minimized the cognitive load on reviewers. What once required hours of careful examination could now be reviewed in minutes, accelerating the entire delivery pipeline.</p><h4>3. Feature Flags as a Deployment Strategy</h4><p>Feature flags became our secret weapon in decoupling deployment from release. By wrapping new functionality behind toggles that could be enabled or disabled without code changes, we transformed deployment from a high-stakes event into a routine operation.</p><p>Our engineers could now deploy code without immediately exposing new features to users. This separation meant we could:</p><ul><li>Deploy code multiple times per day without disrupting users</li><li>Test new features in production with controlled access</li><li>Quickly disable problematic features without reverting deployments</li><li>Run A/B tests without complex deployment orchestration</li></ul><h4>The Safety Net: Tiered Release Strategy</h4><p>With feature flags in place, we implemented a graduated release process that drastically reduced risk while accelerating feedback cycles:</p><ol><li><strong>Internal verification</strong>: New features are first enabled only for test accounts, allowing us to verify functionality in the production environment without impacting real users.</li><li><strong>Tier 1 release</strong>: We then roll out to our “tier 1” customer accounts — typically smaller organizations with whom we have strong relationships. Their feedback is invaluable and comes early enough to make meaningful adjustments.</li><li><strong>Graduated expansion</strong>: Based on performance and feedback, we progressively expand to tier 2, tier 3, and eventually all customer accounts.</li></ol><h4>Case Study: The Message Builder Rewrite</h4><p>This tiered approach proved its value during a critical rewrite of our message builder component — a core piece of our application’s UI. Instead of launching the completely rewritten component to all users simultaneously, we used feature flags to control its rollout:</p><ol><li>First, we enabled it only for internal test accounts, immediately uncovering several edge cases our tests had missed.</li><li>After fixes, we expanded to tier 1 accounts and gathered feedback on usability and performance.</li><li>During the tier 2 rollout, we identified additional edge cases specific to larger accounts’ usage patterns.</li></ol><p>By the time we completed the full rollout, we had addressed numerous issues that would have caused significant disruption had we deployed traditionally. What could have been a critical outage affecting all users became a controlled, iterative improvement process.</p><h4>Measuring Success: The Metrics That Matter</h4><p>To guide our journey, we adopted DORA (DevOps Research and Assessment) metrics as our north star, with a particular focus on deployment frequency. The results were transformative:</p><p><strong>Before</strong>: 1 deployment per team per sprint</p><p><strong>After</strong>: 10 deployments per engineer per sprint</p><p>This 10x (or greater) improvement in deployment frequency had cascading benefits:</p><ul><li>Reduced lead time for changes, from weeks to hours</li><li>Faster feedback cycles for product and UX decisions</li><li>Improved system understanding across the engineering team</li><li>Dramatically reduced coordination overhead between teams</li></ul><p>One valuable outcome was how quickly we could validate or adjust features based on real-world usage. For example, after deploying an early version of our new reporting feature to test accounts, UX designers identified a critical use case we had missed during the design phase.</p><p>In our previous deployment model, this discovery would have come weeks later, potentially requiring a complex refactor after wide release. Instead, we were able to adapt the feature before most customers ever saw it.</p><h3>The Cultural Transformation</h3><p>Perhaps the most meaningful change wasn’t technical at all, but cultural. As deployments became routine rather than events, we observed:</p><ol><li><strong>Reduced anxiety around change</strong>: Engineers gained confidence in making system changes, knowing they could be quickly verified and, if necessary, disabled.</li><li><strong>Improved collaboration</strong>: Product managers and engineers developed a shared language around feature rollouts, with clearer expectations about the feedback and iteration cycle.</li><li><strong>System-wide understanding</strong>: Frequent small changes forced engineers to develop a more holistic understanding of how their work impacted the entire system.</li><li><strong>From gatekeeping to enablement</strong>: Our governance shifted from restrictive approval processes to enabling frameworks that balanced autonomy with safety.</li></ol><h3>Setting the Stage for Reliability</h3><p>This newfound confidence in our ability to change and deploy code quickly became the foundation for our next focus area: reliability. By establishing deployment velocity as a core capability, we created the conditions necessary to implement more sophisticated reliability practices.</p><p>With the ability to deploy fixes quickly and safely, we could now address issues as they arose rather than accumulating technical debt. This velocity became our first line of defense in maintaining a reliable system — a topic we’ll explore in depth in the next post in this series.</p><h3>Key Takeaways</h3><p>If your organization is at the beginning of a similar journey, consider these lessons from our experience:</p><ol><li><strong>Start with deployment velocity</strong>: The ability to deploy quickly and safely unlocks improvements in all other areas.</li><li><strong>Embrace feature flags</strong>: Feature flags are not just for A/B testing; they’re a fundamental deployment strategy.</li><li><strong>Think small</strong>: Small, focused changes are easier to review, deploy, and if necessary, revert.</li><li><strong>Measure what matters</strong>: DORA metrics, particularly deployment frequency, provide actionable insights into your delivery capabilities.</li><li><strong>Build confidence gradually</strong>: Use a tiered release approach to build confidence in changes before wide deployment.</li></ol><p>In our <a href="https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog/crescendo-lab-engineering-evolution-part-2-reliability-ensuring-uninterrupted-customer-95e8bf48ea8a">next post</a>, we’ll explore how we built on this foundation to enhance system reliability through smoke tests, precision alerting, and customer-centric monitoring. Stay tuned!</p><p><em>This post is part of our Engineering Evolution series, documenting Crescendo Lab’s journey toward building a reliable, efficient organization. We’re sharing these insights not as a destination but as waypoints on a continuous path of improvement.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41b9a700df6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog/crescendo-lab-engineering-evolution-part-1-building-confidence-through-velocity-41b9a700df6e">Crescendo Lab Engineering Evolution Part 1: Building Confidence Through Velocity</a> was originally published in <a href="https://medium.com/%E6%BC%B8%E5%BC%B7%E5%AF%A6%E9%A9%97%E5%AE%A4-crescendo-lab-engineering-blog">漸強實驗室 Crescendo lab — 職涯部落格 Career blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[搬家]]></title>
            <link>https://medium.com/@poga/%E6%90%AC%E5%AE%B6-2eda0680e13?source=rss-afe8ba83ff7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2eda0680e13</guid>
            <dc:creator><![CDATA[poga]]></dc:creator>
            <pubDate>Sat, 06 Jul 2019 10:48:05 GMT</pubDate>
            <atom:updated>2019-07-06T10:48:05.673Z</atom:updated>
            <content:encoded><![CDATA[<p>medium 看起來前途茫茫，頁面上干擾的東西也越來越多。</p><p>以後的文章會發到 <a href="https://devpoga.org/">https://devpoga.org</a> 這裡。平台是自架的靜態 blog <a href="https://gohugo.io/">Hugo</a>。</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2eda0680e13" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[為什麼現代遊戲都要有一堆不同的貨幣在裡頭？]]></title>
            <link>https://medium.com/@poga/%E7%82%BA%E4%BB%80%E9%BA%BC%E7%8F%BE%E4%BB%A3%E9%81%8A%E6%88%B2%E9%83%BD%E8%A6%81%E6%9C%89%E4%B8%80%E5%A0%86%E4%B8%8D%E5%90%8C%E7%9A%84%E8%B2%A8%E5%B9%A3%E5%9C%A8%E8%A3%A1%E9%A0%AD-abb784317211?source=rss-afe8ba83ff7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/abb784317211</guid>
            <dc:creator><![CDATA[poga]]></dc:creator>
            <pubDate>Sat, 05 Jan 2019 03:52:57 GMT</pubDate>
            <atom:updated>2019-01-05T03:52:57.053Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xR_TfPgqdy1w-maQ2OGzkA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/4mdjeo4J2Z8?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Keegan Houser</a> on <a href="https://unsplash.com/search/photos/currency?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>金幣、鑽石、寶石、水晶？</p><h3>Daniel Solis on Twitter</h3><p>Earnest questions: What&#39;s up with modern video games having so many different currencies? (Gold, dust, gems, caps, etc.) Where did this start? What is the intended effect on player psychology?</p><p>大家都已經知道的理由：<strong>為了混淆遊戲貨幣跟現實貨幣的對價關係</strong>。但是其他的沒有辦法用現金換貨幣的遊戲呢？</p><p>Triple Town 作者 Daniel Cook 的看法：</p><p><strong>這樣比較容易平衡遊戲經濟</strong>。每種貨幣可以對應到某個遊戲面向，如果那個面向經濟效益太強大，只要調整那種貨幣的產出跟消耗就能解決，而不會影響整個遊戲。</p><h3>Daniel Cook on Twitter</h3><p>1. Easier economy balancing. Each currency maps to a functional area of the game. If one area of the economy is imbalanced, you can tweak sources and sinks for that unique currency. And not imbalance other areas.</p><p><strong>更容易設定玩家目標</strong>：每種貨幣都可以清楚的提示玩家目標跟他目前的進度。</p><h3>Daniel Cook on Twitter</h3><p>2. Better player goal setting. You want to clearly signal player goals and progress towards those goals. A specialized currency enables: A. Players see thing they want B. Grok currency needed C. Do the activity to gets the currency. D. See progress as currency accumulates.</p><p><strong>讓玩家有數個清楚的發展路徑</strong>：貨幣即是進度、如何有效產出貨幣則是玩家的策略。如果全部只有一種貨幣，發展路徑也被全部混在一起，徒增玩家選擇的障礙。</p><h3>Daniel Cook on Twitter</h3><p>3. Player agency through a few key paths. Currency = progress towards goal. Multiple sources for that currency = player choice (and strategy). Too many choices = confusion, so player&#39;s economic choices are much muddier with 1 currency.</p><p>當然，多種貨幣會讓遊戲變的比較複雜，不容易上手。<strong>但是 Game-as-Service 的目標是要提供數百小時以上的遊戲時間</strong>，這樣的經濟系統是讓熱情玩家去鑽研用的。同時，這樣的系統也容易凝聚遊戲社群，讓玩家一起解讀。</p><h3>Daniel Cook on Twitter</h3><p>4. What you lose here is some elegance and learnability. But game-as-a-service a meant to be played for hundreds of hours and the economic loops are for avid, long term players. And learning about the economy can become a community activity. So not necessarily a net negative.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abb784317211" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[我選擇生產力工具的基準]]></title>
            <link>https://medium.com/@poga/%E6%88%91%E9%81%B8%E6%93%87%E7%94%9F%E7%94%A2%E5%8A%9B%E5%B7%A5%E5%85%B7%E7%9A%84%E5%9F%BA%E6%BA%96-8bdb0f56dfd2?source=rss-afe8ba83ff7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8bdb0f56dfd2</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[gtd]]></category>
            <category><![CDATA[omnifocus]]></category>
            <category><![CDATA[getting-things-done]]></category>
            <dc:creator><![CDATA[poga]]></dc:creator>
            <pubDate>Thu, 27 Sep 2018 05:32:08 GMT</pubDate>
            <atom:updated>2018-09-27T05:44:42.850Z</atom:updated>
            <content:encoded><![CDATA[<p>先說結論，我用 <a href="https://www.omnigroup.com/omnifocus/">OmniFocus</a> 用了五年以上了，目前依然滿意。</p><p>現在的生產力工具實在太多了。為了在焦慮的身心與環境中活下來，我試過許多工具，看到<a href="https://www.facebook.com/brucehsu13/posts/10217244664726255"> Bruce 討論到</a>，分享一些心得。</p><h4>快速的收納臨時出現的想法跟待辦事項。</h4><p>現實中大部分的事情都是突然想到的。吃飯時突然想到一則文章的標題、工作時想到明天是老婆生日要買禮物、買咖啡時想到苦思許久的 bug 要怎麼解。</p><p>這種時候，能快速的收納這些小事就很重要。GTD 很重要的 Inbox 觀念便是如此：清空腦中那些大量漂浮的雜事。收納的門檻越低，越能保持頭腦清楚。</p><p>OmniFocus Mac 版的全域熱鍵跟 iOS app 都能達到這個目標。</p><h4>快速的看到需要做的事。</h4><p>重點是「看到」而不是「搜尋到」。待辦事項應該在收納時，或是每週 review 時整理好。平時應該要能直接「看到」當下可以做的事情。而不是花額外的力氣去「搜尋」出要做的事。</p><p>以 OmniFocus 而言，就是用 perspective 去彙整 Tag（以前叫做 Context）。</p><p>準備一個叫 Low Energy 的 Perspective，在腦力不足但是又睡不著時，可以找點雜事來做。</p><p>或是一個叫 Writing 的 Perspective，在滿肚子牢騷不想寫 code 時可以直接找個主題來寫。</p><h4>隱藏不該看到的事項。</h4><p>這點我覺得是很多工具漏掉的部分。</p><p>在現在高度 context-switch 的環境，人腦早就被訓練的極易分心。如果沒有一個獨立的輸入介面，導致輸入資料時會看到其他不相關的事項，這些都會造成額外的腦力負擔。原本只是要記下購物清單，結果剛好看到明天有個會議要開。好不容易從腦中清掉的資訊又再度回到腦中，徒增焦慮。</p><p>而 OmniFocus 的輸入介面夠乾淨，不會看到額外的資訊。整理的功能也夠完整。輸入之後不需要另外搜尋，該看到這些事情時自然就會看到。</p><p>—</p><p>不過，生產力問題本質上是個性問題，不習慣的方法也不用強求自己，那也只是徒增焦慮而已。</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bdb0f56dfd2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[如何整合 Rust 與 Lua]]></title>
            <link>https://medium.com/@poga/%E5%A6%82%E4%BD%95%E6%95%B4%E5%90%88-rust-%E8%88%87-lua-3cb38238dc72?source=rss-afe8ba83ff7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cb38238dc72</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[actix]]></category>
            <category><![CDATA[lua]]></category>
            <dc:creator><![CDATA[poga]]></dc:creator>
            <pubDate>Mon, 03 Sep 2018 23:30:04 GMT</pubDate>
            <atom:updated>2018-09-03T23:40:07.230Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xOKVZGes-9VXtqrJkQca4Q.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/4izt8TxQmEs?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Anders Jildén</a> on <a href="https://unsplash.com/search/photos/lua?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>在系統中遷入一個動態輕巧的 scripting language 一直是個常見的設計。像 <a href="https://www.rust-lang.org/">Rust</a> 這樣的系統語言，雖然效能好，但是上手門檻較高。這時若是能遷入一個像 <a href="https://www.lua.org/">Lua</a> 一樣動態型別，簡單易懂的語言，便能大幅提高系統彈性。</p><p>最近為了實做 <a href="https://github.com/poga/actix-lua">actix-lua</a>，研究了一下 Rust 跟 Lua 之間的介接，順便學了不少 Rust 跟 Lua 的設計，筆記在此。</p><h4>Lua Binding 的選擇</h4><p>Rust 現在有數套 Lua binding，比較常被人提到的是 <a href="https://crates.io/crates/lua">lua</a>, <a href="https://github.com/tomaka/hlua">hlua</a> 與 <a href="https://github.com/kyren/rlua">rlua</a> 。</p><p><a href="https://crates.io/crates/lua">lua</a> 基本上是直接把 Lua 的 C API 直接移植，沒有做多餘的包裝。所以需要絕對的效能的話，這可能是你的最佳選擇。不過安全性跟 UB 就要自己處理了。</p><p><a href="https://github.com/tomaka/hlua">hlua</a> 提供是比較高階的介面，不讓你直接存取 Lua Stack，可以視為 lua API 上的再一層包裝。彈性較低，可能不適合某些需求。</p><p><a href="https://github.com/kyren/rlua">rlua</a> 是由知名遊戲工作室 <a href="https://blog.chucklefish.org/about/">chucklefish</a> 開發。延續 Rust 對安全性的要求，設計介面時也是以安全性為最高原則。在使用 rlua 的 API 時，不會產生任何 UB（由於 lua API 跟內部的運作方式，這種 API 真的很難做到…）也許犧牲了一點效能，不過對於 Rust 的使用者來說，這樣的 tradeoff 應該是蠻值得的。</p><p>最後我用的是 rlua。</p><h4>老問題：Lifetime &amp; ownership</h4><p>剛開始用 rlua 的時候，很容易就撞到 lifetime 的問題。不過，就跟一般使用 Rust 一樣：如果遇到很難解的 lifetime 跟 ownership 問題，很可能架構設計上就錯了。</p><p>rlua 作者很詳盡的解釋了 rlua 許多的設計概念，也深入剖析了 Lua 本身設計帶來的影響。不過，這些解釋大多分散在各 github issue 之中。這邊稍微整理一下幾個重點：</p><ol><li>rlua 提供的所有的 reference type，像是 Function、 Table 這些指向 Lua 內部元素的參照，都不應該被另外存起來。</li><li>rlua 提供的 reference type 也不應該被存到任何 Lua 內的 userdata 或是 Rust callback 之中。</li><li>傳進 rlua 中的任何值都必須是 Send （因為 lua 本身是 Send）跟 &#39;static（因為你沒辦法知道 Lua 何時會 GC 掉這些值，難以找出精確的 lifetime。）</li><li>3 的限制讓傳值給 lua 變的複雜許多。rlua 提供了 scope API，可以用一個暫時的 scope 將不符合條件的值傳給 lua。傳過去的資料在 scope 結束後就會被銷毀，避免複雜的 lifetime 問題。</li></ol><h4>最後的成果：actix-lua</h4><p><a href="https://actix.rs/">actix</a> 是個 Rust 上的 actor framework，前陣子因為極高的效能而小有名氣。由於目前 Rust async/await 的支援還不算友善，actor model 算是能幫助簡化 async 程式的架構。</p><p>而 <a href="https://github.com/poga/actix-lua">actix-lua</a> 便是將 actix 與 Lua 整合，讓人能用 Lua 實做系統中某些需要彈性、熱抽換、方便修改的部分。想法來自於 <a href="https://openresty.org/en/">openresty</a> 這套整合 nginx 與 Lua 的軟體。目前主要被我用來處理即時資料的分析。</p><p>雖然寫的過程碰了蠻多壁，不過最後的成果我還算滿意。目前還在初步的階段，歡迎各種建議與回饋。</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cb38238dc72" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[軟體設計模式 — Data-Oriented Design]]></title>
            <link>https://medium.com/@poga/data-oriented-design-9288178d3ea5?source=rss-afe8ba83ff7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9288178d3ea5</guid>
            <category><![CDATA[系統架構]]></category>
            <category><![CDATA[遊戲開發]]></category>
            <category><![CDATA[科普]]></category>
            <dc:creator><![CDATA[poga]]></dc:creator>
            <pubDate>Mon, 16 Jul 2018 06:19:38 GMT</pubDate>
            <atom:updated>2018-09-14T12:58:36.863Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nRuzaWkMbosnTZscueO5AA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/BpqDaDxG48w?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Rebecca Oliver</a> on <a href="https://unsplash.com/search/photos/game?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>遊戲開發對很多開發者來說是個陌生的領域。遊戲對於效能的極高要求跟規格的不確定性，產生出了許多特有的系統架構。Data-Oriented Design 便是個有趣的設計模式。</p><p>相較於其他設計模式，Data-Oriented Design 深受硬體快取（cache）架構影響。對於現代的高度 pipeline、高速的 CPU 架構而言，資料的存取方式對效能有非常大的影響。比起 L1、L2 cache，對主記憶體的一次存取帶來的是數百倍的效能損耗。</p><p>為了避免太過抽象，就用個實際的例子來解釋吧。</p><p>假設我們的遊戲中有許多的球，每個球有顏色、位置、半徑等等資訊。對於學過物件導向的人而言，很可能直覺的設計成這樣：</p><pre>class Ball {<br> Point position;<br> Color color;<br> double radius;<br>}</pre><p>這樣的作法很符合人對世界的理解：每個球是獨立的個體，有自己的屬性。</p><p>不過，如果遊戲中有數百萬個球在移動，這樣的作法的效能就不太好了。每個球的座標都經過物件導向的層層封裝，分散在記憶體四處，spatial locality 非常差，在更新的過程中產生了大量的 cache miss。</p><h4>There’s no ball</h4><p>如果採用 Data-Oriented Design 的話，作法會變成：</p><pre>class Balls {<br> vector&lt;Point&gt; positions;<br> vector&lt;Color&gt; color;<br> vector&lt;double&gt; radius;<br>}</pre><p>於是我們的程式中不再有「獨立的球」這個設計存在，所謂的球，只是透過一個 index，含蓄的存在遊戲世界中。</p><p>這樣的作法，對人而言並不直覺，但是對硬體而言，效能好上許多。大多數的 vector 實做，都會將其中的內容放在一段連續的記憶體空間中。因此 spatial locality 很好，對 CPU 而言，他能很輕易的猜到接下來要存取的記憶體位址，省下許多猜錯而損失的 CPU cycle。</p><h4>SoA 與 AoS</h4><p>這兩種作法又分別被稱做 Array of Structs 跟 Structs of Arrays。前者是用一個陣列存放許多獨立的 struct（class），後者是用一個 struct（class）存放許多陣列。</p><p>實際開發時，很難臨時在這兩種模式中轉換。對程式而言，model 遊戲世界的方式完全不同。所以常常一開始用了直覺的 AoS 開發，發現效能不好，需要換成 SoA 時卻無從下手。</p><p>遊戲開發在介面設計跟系統架構上，都有很多很經典的範例啊…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9288178d3ea5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Studio Moniker：受限的群眾與無限的創作]]></title>
            <link>https://medium.com/@poga/studio-moniker-%E5%8F%97%E9%99%90%E7%9A%84%E7%BE%A4%E7%9C%BE%E8%88%87%E7%84%A1%E9%99%90%E7%9A%84%E5%89%B5%E4%BD%9C-56cf834a0c23?source=rss-afe8ba83ff7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/56cf834a0c23</guid>
            <category><![CDATA[網路]]></category>
            <category><![CDATA[藝術]]></category>
            <category><![CDATA[群眾]]></category>
            <category><![CDATA[資訊設計]]></category>
            <dc:creator><![CDATA[poga]]></dc:creator>
            <pubDate>Tue, 03 Jul 2018 01:45:57 GMT</pubDate>
            <atom:updated>2018-07-03T02:44:41.114Z</atom:updated>
            <content:encoded><![CDATA[<p>第一次看到 Studio Moniker 的作品，是個叫做 Do not touch 的群眾外包式的 MV（？），透過記錄瀏覽者的滑鼠軌跡，共同創作出 MV 的內容。</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FW3ZjY8YD_NY%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DW3ZjY8YD_NY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FW3ZjY8YD_NY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/53b975f9ac05bf4c7bebafabedd9f566/href">https://medium.com/media/53b975f9ac05bf4c7bebafabedd9f566/href</a></iframe><p>MV 中隨著音樂，不斷出現指示，例如要求游標指向你的所在地、你想去的地方、要你跟著綠色指標移動…等等</p><p>比起單純的，工廠式的群眾外包。這樣的協作方式更留下了個人風格。大多數的滑鼠游標都是乖乖的依照畫面指示動作，也有些游標刻意反其道而行。</p><p>當畫面出現裸女，即使指示寫著「不要遮住她」，許多游標也硬是要幫畫面中的人物遮點。</p><p>當無數的游標跟隨著畫面上的指示流動，群眾的本質也昭然若揭。</p><p>—</p><p><a href="http://studiomoniker.com/">Studio Moniker</a> 是個來自阿姆斯特丹的互動設計工作室。作品大多以科技與群眾為主。與許多知名組織都有合作過。像是與 Mozilla Foundation 合作，為網路中立性而戰的 <a href="http://paperstorm.it/">Paperstorm.it</a>；以及與 Google Data Art Team 合作的，在 VR 空間中協作舞蹈的 <a href="https://tonite.dance/">Dance Tonite</a>。</p><p><a href="https://studiomoniker.com/about">Conditional Design</a> 是他們所提出的設計宣言：與其被媒體所限制，他們選擇透過制訂規則，讓參與者在共通的規則下自由發揮，來產生出各種奇妙的成果。</p><p>怎樣的規則呢？譬如與 Yorkshire Sculpture Park 合作的 <a href="http://sculpture.cam/">sculpture.cam</a> ，他們替公園中的雕塑各準備 120 張不同角度的剪影，讓遊客自行對照剪影，找到正確的角度後拍照。便得到了由群眾協作而成的雕塑的 3D 模型。前述的 paperstorm.it 也是透過單純的「散發傳單」這個動作，讓人自行在地圖上塗鴉。</p><p>獲獎無數的 <a href="http://radio.garden/">Radio Garden</a> 也是如此，透過簡單的將各地的電台呈現在 3D 的地球上，產生了一種新的探索各種實況資源的感受。原本各地從來不曾聽過的電台突然變的垂手可得。極簡的介面也讓探索的門檻大幅降低。</p><p>像是透過一扇窗，偷看當地人們的生活。也許那邊某個餐館正放著這些音樂，或是某個無法入睡的少年也正在聽著這個電台？</p><p>—</p><p>比起光影炫目的多媒體資訊藝術，這種低調、群眾的藝術更符合現代網路的狀況：被誘導的群眾、由細節透露出來的個人風格、以及網路特有的，不斷疊加而成的表現方式等，都是有趣的議題。</p><p>數年前被 Do not touch 衝擊後，意外在去年的 Dutch Design Award 又看到了他們的作品，從此之後再也回不去原本的世界了。</p><p>另外，他們網址的品味相當不錯，radio.garden、tonite.dance 跟 sculpture.cam 都是超棒的網址啊 XD。</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56cf834a0c23" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>