<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Emplifi - Medium]]></title>
        <description><![CDATA[We want to give you a sense of all the systems and technologies that power Emplifi and introduce our thinking, principles and our tech stack. - Medium]]></description>
        <link>https://medium.com/socialbakers-engineering?source=rss----881fbb2b4996---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Emplifi - Medium</title>
            <link>https://medium.com/socialbakers-engineering?source=rss----881fbb2b4996---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 21:32:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/socialbakers-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why Scrum Works Better in Kyoto Than in Prague]]></title>
            <link>https://medium.com/socialbakers-engineering/why-scrum-works-better-in-kyoto-than-in-prague-dbc38a13f915?source=rss----881fbb2b4996---4</link>
            <guid isPermaLink="false">https://medium.com/p/dbc38a13f915</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[scrum]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[engineering-leadership]]></category>
            <dc:creator><![CDATA[Mehmet Ekmekci]]></dc:creator>
            <pubDate>Wed, 23 Jul 2025 07:01:52 GMT</pubDate>
            <atom:updated>2025-07-23T07:01:51.614Z</atom:updated>
            <content:encoded><![CDATA[<p><em>(…And What Erin Meyer Can Teach Us About Your Painfully Long Stand-ups)</em></p><p>Let me ask you a spicy question:<br>Have you ever tried running Scrum with a multi-cultural team and ended up with daily stand-ups that lasted longer than waiting for your number to come up at the post office?</p><p>No? Lucky you.<br>Yes? Welcome to the club.</p><p>Scrum, bless its sticky-note-covered heart, has been sold to us as a universal truth. A framework to end all frameworks. The way, the truth, and the light for software development. But here’s the thing nobody told you at your Certified Scrum Master course:</p><p>Scrum wasn’t born in San Francisco.<br>It was born in Japan. And not just anywhere in Japan, inside the factories of Toyota.</p><p>And you know what? That matters. <strong><em>A lot.</em></strong></p><h3>Chapter 1 — Scrum Was Born in a Factory, Not a Zoom Call</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*3zjcZlkepBBuSF7R" /></figure><p>Let’s go back to the 1980s. While many of us were trying to figure out how to rewind VHS tapes without ruining the tape, a couple of Japanese companies were quietly reinventing the idea of teamwork and process optimization. At the heart of it all: <strong><em>Toyota</em></strong>. A company that, at the time, didn’t just build cars. It built systems. One of them was called <strong>“The Toyota Production System”</strong>, and it gave us concepts like <em>kaizen </em>(continuous improvement), <em>jidoka </em>(quality control built into the process), and respect for people.</p><p>These weren’t just buzzwords. They were reflections of deeply rooted Japanese values:</p><ul><li>Harmony over confrontation</li><li>Long-term thinking</li><li>Role-based respect</li><li>Implicit communication (saying more by saying less)</li><li>Consensus over individualism</li></ul><p>Fast forward a few years, and Western academics like Hirotaka Takeuchi and Ikujiro Nonaka started to analyze how these Japanese teams worked. In their 1986 Harvard Business Review paper titled <em>“The New New Product Development Game,”</em> they introduced the metaphor of a rugby team passing the ball back and forth as they sprint down the field together.</p><p>That metaphor? It became the <strong>foundation of Scrum.</strong></p><p>And it was Jeff Sutherland and Ken Schwaber who packaged it for software teams. But while the framework made its way across the world, the culture didn’t.</p><ul><li>We got the sprint, but not the subtlety.</li><li>We got the ceremonies, but not the consensus-building rituals underneath.</li><li>We got the sticky notes, but not the social glue.</li></ul><h3>Chapter 2 — When Scrum Meets Culture: A Real-Life Tale of 50-Minute Dailies</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*zA6M__uGgi9ePHNI" /></figure><p>One of the first teams I ever managed was a rainbow-colored mix of cultures: Brazilians, Czechs, Slovaks, and a formerly-engineer-product leader. And in theory, we were running Scrum. In practice? Our daily stand-ups lasted 50 minutes. Yes, actual 5–0. Every. Single. Day.</p><p>Why? Well, everyone had different expectations of what a “daily” should be:</p><ul><li>The Czechs wanted deep dives; otherwise, what’s the point of meeting?</li><li>The Brazilians were focused on operations — logs, tests, and keeping things smooth.</li><li>The Product Owner wanted deadline updates.</li><li>The QA wanted to check if their ideas would block anyone.</li></ul><p>But here’s the twist: nobody wanted to rock the boat.</p><p>No one said <strong><em>“hey, this is a bit much”</em></strong> because they didn’t know how to bring it up across such a culturally colorful room. We were trying to keep <em>everyone</em> happy, so we ended up with a meeting that made <em>no one</em> happy.</p><p>Scrum isn’t the problem. Treating it like a fixed religion is.</p><p>We eventually brought a timer. We talked <em>(gasp!)</em> about the meeting itself. And slowly, painfully, we got that 50-minute monster down to a very human 15-minute check-in. But it wasn’t easy, and it wouldn’t have worked if we hadn’t started addressing <em>how</em> we worked, not just <em>what</em> we were doing.</p><h3>Chapter 3 — What Erin Meyer Can Teach You About Your Awkward Sprint Planning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0YTjp97PAdLGZ0-j" /></figure><p>Enter Erin Meyer, the quietly brilliant mind behind <em>The Culture Map</em>. She didn’t invent culture clashes, but she helped a lot of us name them, and once you can name them, you can laugh at them. Or cry, depending on how your last retro went.</p><p>Her book maps how different cultures approach feedback, trust, leadership, decisions, and disagreement. It’s like the IKEA manual for assembling multicultural teams, except you don’t get the tiny Allen wrench, and someone’s probably offended by the first page.</p><p>Let’s break it down with some real-life collisions you’ve probably experienced (or will, soon):</p><p><strong>1. Leading and Deciding: Commanders vs. Campfires</strong></p><p>Some cultures like decisions to come from the top. Others expect a cozy round-table where everyone is heard. Trouble is, when you mix them, no one knows what’s expected.</p><ul><li>In <strong>Russia</strong>, <strong>China</strong>, or <strong>India</strong>, top-down leadership is the norm. The boss decides, and the team executes.</li><li>In <strong>the Netherlands</strong>, <strong>Sweden</strong>, or <strong>Denmark</strong>, decisions are made by consensus. Everyone has a voice. Sometimes, too many voices.</li><li>In <strong>Japan</strong>, the real decision happens in the hallway before the meeting. The meeting just makes it official.</li></ul><p>Put a Dutch engineer and a Russian team lead together, and you’ll see sparks. The Dutch colleague wants to debate architecture over lunch. The Russian manager just wants to move forward and stop wasting time.</p><p>Now drop Scrum into this stew, with its cheerful “the team decides together!” principle. You might get harmony… or a standoff that lasts until the sprint ends. Or the quarter.</p><p><strong>2. Trust: Built on Tasks or Built on Tea?</strong></p><p>Another dimension Meyer explores is trust, and whether you build it through work or over time through personal bonds.</p><ul><li>In <strong>America</strong>, <strong>Germany</strong>, or <strong>Australia</strong>, trust is task-based. “If you deliver, I trust you. Let’s go.”</li><li>In <strong>Brazil</strong>, <strong>India</strong>, or <strong>Saudi Arabia</strong>, trust is relationship-based. “Let’s get coffee first. Maybe dinner. Then I’ll tell you what I really think.”</li></ul><p>Now imagine doing a remote retro with a blend of both. The American devs want to jump straight into what went wrong in the last sprint. The Brazilians are still warming up, feeling out the room. No one’s wrong. But the rhythm is wildly different.</p><p><strong>3. Feedback: Direct, Blunt, and Ouch vs. Subtle, Gentle, and Mysteriously Vague</strong></p><ul><li>A <strong>German</strong> engineer might say, “This is a poor solution.” It’s not an attack, just honesty.</li><li>A <strong>Japanese</strong> engineer might say, “This part might need a bit more refinement,” which, translated, actually means “please rewrite this entire module.”</li><li>A <strong>Turkish</strong> teammate might say nothing, but three days later, the code is rewritten and renamed.</li></ul><p>In teams where feedback styles vary, retrospectives can either become fireworks or fog. One group’s idea of “constructive criticism” is another group’s “emotional breakdown.” And unless you create shared norms and rituals, you’ll never know who’s holding back. Or who just insulted someone without realizing it.</p><p><strong>4. Disagreement: Conflict-Averse vs. Conflict-Is-a-Sport</strong></p><ul><li>In <strong>France</strong>, open disagreement is expected. It’s how ideas are tested.</li><li>In <strong>Indonesia</strong> or <strong>Thailand</strong>, disagreement is subtle, layered, or entirely silent. The most polite “yes” might mean “absolutely not.”</li><li>In <strong>Czechia</strong>, people often disagree plainly, but only if they trust you. Otherwise, they go silent and wait.</li></ul><p>Now picture your retro. The Scrum guide says it should be an “open, honest” conversation. But what if honesty, in your culture, involves coded language, soft suggestions, and saving face?</p><p>So what happens when you throw all these styles into a room, slap on a Scrum label, and expect everyone to magically<strong><em> “self-organize”?</em></strong></p><p>Well… you get 50-minute stand-ups. Or dead silence in retros. Or meetings where no one’s quite sure who’s in charge. Or worse, everyone’s pretending it’s fine, while quietly googling job openings.</p><p>The lesson here isn’t “Scrum doesn’t work.”<br>It’s “Scrum isn’t a culture-neutral operating system.”<br>It assumes certain defaults, and if your team doesn’t share them, the framework may fall flat.</p><h3>Chapter 4 — When Frameworks Backfire (Or: Why Your Retro Feels Like a Silent Retreat No One Asked For)</h3><p>Let’s talk about something that happens far too often.</p><p>You roll out Scrum.<br>You run the ceremonies.<br>You read <em>The Five Dysfunctions of a Team</em> because someone recommended it in a LinkedIn post with far too many emojis.<br>You host the retro, expecting a brave, open dialogue.</p><p>But instead, you get… polite nods, side-eyes, long silences, and zero action items.<br>The dysfunctions are there, oh, they are very much there, but calling them out feels like lighting a match in a room full of dry paper.</p><p>Why?</p><p>Because frameworks like Scrum and Lencioni’s model <strong>assume</strong> a certain cultural baseline.</p><p>Let’s take Lencioni’s famous pyramid:</p><ul><li><strong>Trust</strong> (the kind where people feel safe being vulnerable)</li><li><strong>Conflict</strong> (healthy, productive conflict)</li><li><strong>Commitment</strong></li><li><strong>Accountability</strong></li><li><strong>Results</strong></li></ul><p>Looks clean. Feels logical. But again, this structure is deeply Western in flavor. It assumes that trust comes from openness. That conflict is something to be embraced. That accountability is individual, not collective.</p><p>Try telling a group of Turkish or Japanese engineers to “lean into conflict.” You’ll either get blank stares or an awkward chuckle. In many cultures, <strong>avoiding conflict is</strong> how you show trust. And “being vulnerable” with your manager? That’s not psychological safety. That’s a career-limiting move.</p><p>Or imagine telling an Italian engineer and a Dutch engineer to have an “open debate” about a technical decision. The Italian may speak with passion, gesture wildly, and never quite get to the point. The Dutch engineer may offer blunt, surgically precise criticism with zero sugarcoating. Both think they’re being helpful. Both leave the meeting slightly offended.</p><p>Now sprinkle in Scrum’s core principle: <strong>self-organizing teams.</strong></p><p>Sounds great, right? But in practice, it can lead to chaos if the team is used to a clear hierarchy and structure.</p><ul><li>In some cultures, people actively expect the engineering manager to take the lead.</li><li>In others, stepping up without being asked feels like breaking protocol.</li></ul><p>So when you ask the team to “just decide together,” what you get is a group of smart people quietly waiting for someone, <strong>anyone</strong>, to go first.</p><p>This isn’t dysfunction. It’s a cultural mismatch. You didn’t build the wrong team. You just forgot to check the wiring before flipping the switch.</p><h3>Chapter 5 — So What’s a Leader Supposed to Do With All This?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uMlukuhQ6SKkY-rE" /></figure><p>The good news? You don’t need to abandon Scrum. You just need to treat it more delicately.</p><p>Here’s what’s worked for me and others who’ve stumbled through these culture curves:</p><h4>1. Map Your Team</h4><p>Literally.</p><p>Use Erin Meyer’s Culture Map and plot where your team members sit on key scales: Decision making. Feedback. Trust. Disagreement.</p><p>This isn’t about stereotyping. It’s about awareness. Once you see the spread, you can adjust how you communicate, facilitate, and lead.</p><h4>2. Localize the Framework</h4><p>Scrum isn’t a religion. It’s a recipe. Change the spice levels. Swap ingredients.</p><ul><li>If your team hates dailies, maybe aim to make them shorter or less operational.</li><li>If retros feel like a silent group retreat, give people ways to contribute differently, or bait the quiet(er) team members to speak up.</li><li>If no one wants to volunteer for tasks, experiment with rotating ownership instead of waiting for self-selection.</li></ul><h4>3. Teach the Why, Not Just the What</h4><p>When you explain <em>why</em> we have retros, or <em>why</em> the team decides together, people are more likely to engage. Otherwise, they’ll just mimic the motions and call it a day.</p><h4>4. Talk About Culture Out Loud</h4><p>This one’s underrated. Make it normal to talk about how people give feedback, how they prefer decisions to be made, or what they find stressful in meetings. Culture is there whether you name it or not. Naming it gives you options.</p><h3>Closing Thoughts: Scrum Is a Mindset, Not a Script</h3><p>In the end, agility isn’t about whether your sprint board uses Jira or sticky notes.</p><p>It’s not about whether your retro follows the Start–Stop–Continue format.</p><p>It’s about people. And people come with habits, values, expectations, and a whole lot of cultural context.</p><p>Scrum, in its purest form, came from the quiet consensus of Japanese teams working in harmony. It was translated into a Western framework built on openness and self-expression. Now it’s used by global teams that might value harmony over honesty, or loyalty over speed.</p><p>So next time your retro feels off, or your stand-up feels like a hostage situation, maybe take a step back and ask:</p><p><strong>Are we stuck because of Scrum… or because we’ve forgotten to account for who’s in the room?</strong></p><p>Because once you understand the culture, you can start designing a way of working that <strong><em>actually </em></strong>works. Not just in theory, but in practice, with the very real humans on your team.</p><p>And that’s when the magic happens.</p><p>Fewer 50-minute stand-ups. More real progress.</p><p>And maybe, <em>just maybe</em>, a little less pain.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbc38a13f915" width="1" height="1" alt=""><hr><p><a href="https://medium.com/socialbakers-engineering/why-scrum-works-better-in-kyoto-than-in-prague-dbc38a13f915">Why Scrum Works Better in Kyoto Than in Prague</a> was originally published in <a href="https://medium.com/socialbakers-engineering">Emplifi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We (Slowly) Built a League of Software Architects — and How It Helped Everything]]></title>
            <link>https://medium.com/socialbakers-engineering/how-we-slowly-built-a-league-of-software-architects-and-how-it-helped-everything-b48f227a3c3c?source=rss----881fbb2b4996---4</link>
            <guid isPermaLink="false">https://medium.com/p/b48f227a3c3c</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[career-development]]></category>
            <dc:creator><![CDATA[Mehmet Ekmekci]]></dc:creator>
            <pubDate>Tue, 13 May 2025 08:01:46 GMT</pubDate>
            <atom:updated>2025-05-13T08:01:44.123Z</atom:updated>
            <content:encoded><![CDATA[<h3>How We (Slowly) Built a League of Software Architects — and How It Helped Everything</h3><p>If you’d told us three years ago that one day we’d have a council of twelve architects meeting biweekly, joining business reviews, and being key voices in product direction, we probably would’ve laughed and gone back to trying to convince someone to finally retire that one undocumented monolith in the corner.</p><p>But here we are. And the story of how we got here starts, like all good tech origin stories, with a problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y1ZF8RtoqvtPO2ra" /><figcaption>Hard-working engineering leaders in action</figcaption></figure><h3>Once Upon a Backlog: The Birth of an Idea</h3><p>A couple of years ago, as directors of engineering, we found ourselves in a loop. Teams were shipping features, velocity graphs looked decent, sprint rituals were happening… and yet, something felt off.</p><p>The internal technical work — the kind that moves mountains slowly, like reducing tech debt, upgrading frameworks, or improving observability — was often buried under a pile of JIRA tickets labeled “low priority” or “next sprint.”</p><p>And our engineering managers? They were holding it all together, like the CEOs of their own little startups. Product managers were flying the product planes, so to speak. But who was sitting in the CTO seat of each team, helping steer the technical direction?</p><p>That was our “aha” moment.</p><p>We didn’t need <em>more</em> process. We needed <em>architecture</em>, but not in the old-school, PowerPoint-heavy, top-down kind of way. We needed architects who were embedded, proactive, and product-aware. Not ivory tower engineers… more like experienced spelunkers, helping the team navigate through the dark caves of complexity.</p><p>So we asked: what if we just… formalized the folks who were already doing this?</p><h3>Writing the Job Description (Backwards)</h3><p>Truth be told, the first few months were a little wobbly. People interpreted the role in different ways. One person thought they needed to write 80% of the code; another feared they’d be spending their life in meetings explaining why maintaining an infrastructure is hard.</p><p>So we did what any good team does: we watched, we listened, and <em>then</em> we wrote the job description.</p><p>We focused on behaviors, not buzzwords. We asked: What <em>should</em> an architect be doing to make a team better, not just more productive? Things like:</p><ul><li>Mentoring and growing future tech leaders.</li><li>Helping product managers think through feasibility and cost.</li><li>Representing the technical voice in strategy meetings.</li><li>Translating “we should rewrite this” into something that can survive a prioritization meeting.</li></ul><p>And slowly, the architects found their footing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yl4OAnKL2xTKx9bLJEQTXQ.png" /><figcaption>Architects devouring food after a workshop</figcaption></figure><h3>Expansion Pack: Architects for Everyone</h3><p>After about a year of letting the platform architects forge ahead — sometimes gallantly, sometimes a bit confused — we expanded the initiative to product and data teams.</p><p>With some debate, some iteration, and yes, the occasional <em>“Wait, so do I own the backlog now or not?”</em> we now have <strong>12 architects</strong> embedded across the company.</p><p>They meet regularly to raise cross-team concerns, share tools and practices, and even occasionally agree on things. They join quarterly business reviews, speak up in roadmap meetings, and help ensure that our technical reality matches our product ambitions. Most importantly, they form a crucial third leg in the team leadership stool, right alongside the engineering and product managers.</p><h3>Lessons Learned (And a Few Honest Moments)</h3><p>There were some fun hiccups along the way.</p><p>One engineer paused after hearing the expectations and said, <em>“So I have to present things to people outside my team? Like… regularly?”</em></p><p>Another mistook the role for “lead implementer” and heroically took on <em>all</em> the hard tasks, before we gently helped them discover the beauty of delegation.</p><p>And one of the more interesting realizations came when some architects started to see that the resistance they occasionally faced from managers wasn’t about blocking good ideas — it was about asking the hard questions: <em>Why should we invest in this now? What problem are we solving? What will this cost us later?<br></em>At first, it felt like a “management thing” — a clash of priorities. But then they began to disagree with <em>each other</em>, too. Even among experienced architects, perspectives differed. That’s when it clicked: this wasn’t about managers versus engineers. It was about navigating complexity from different angles — and having to justify your thinking, no matter who you’re talking to.</p><p>Another quiet turning point came when some architects realized that even non-technical improvements — things like improving the team’s ways of working or nudging the culture forward — weren’t quick wins. They took energy. They took patience. They took the kind of persistence that often goes unnoticed. And suddenly, the hurdles they once attributed to someone else’s “lack of drive” started to look a lot more like… just how tricky real change can be.</p><h3>Where We’re Headed</h3><p>Today, our architects are not just system designers. They’re trust-builders, mentors, strategic partners, and product whisperers.</p><p>We’re still learning. The role continues to evolve. But we’re proud of what’s emerged: a group of engineers who don’t just write code — they shape how we <em>work</em>, how we <em>build</em>, and how we <em>think</em>.</p><p>So if your team is feeling the pressure of growing complexity or the weight of tech debt, no one’s quite sure how to tackle… consider planting the seed.</p><p>You might just grow your own league of architects. No capes required.</p><p><strong>P.S.</strong> To the amazing architects reading this: you’ve come a long way. Thanks for jumping in, leveling up, and showing us what’s possible when technical leadership is grounded in care, curiosity, and community. Keep being awesome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b48f227a3c3c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/socialbakers-engineering/how-we-slowly-built-a-league-of-software-architects-and-how-it-helped-everything-b48f227a3c3c">How We (Slowly) Built a League of Software Architects — and How It Helped Everything</a> was originally published in <a href="https://medium.com/socialbakers-engineering">Emplifi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering Highlights of 2024]]></title>
            <link>https://medium.com/socialbakers-engineering/engineering-highlights-of-2024-fec66ad00aca?source=rss----881fbb2b4996---4</link>
            <guid isPermaLink="false">https://medium.com/p/fec66ad00aca</guid>
            <category><![CDATA[emplifi]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[team]]></category>
            <dc:creator><![CDATA[Peter Krejzl]]></dc:creator>
            <pubDate>Wed, 05 Mar 2025 11:18:25 GMT</pubDate>
            <atom:updated>2025-03-05T11:18:25.218Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QvlPdfMnuCG6RZhrT78K6A.png" /></figure><p>Now that 2024 is behind us, we want to take a moment to reflect on the amazing achievements and milestones our engineering teams accomplished in 2024. That year brought us moments of innovation, collaboration, and progress that have shaped our journey — both within the company and in how we’ve supported our clients and the industry.</p><p>From groundbreaking projects to fostering a stronger team culture, there’s so much to celebrate.</p><p>Let’s take a look back at some of the highlights.</p><h3>Building the Next Generation of Engineers</h3><p>Our collaboration with the University of West Bohemia (UWB) reached exciting new heights in 2024. Several talented students joined us as interns, bringing fresh ideas and enthusiasm to important projects like Care and various public API initiatives, making a real impact on our ongoing efforts. What makes this partnership even more special is that some of these students transitioned into full-time positions, becoming integral members of our team.</p><p>Not only did these students excel in their assigned roles, but a few went above and beyond, volunteering to support other teams, tackling challenges, and expanding their own skills. Their proactive approach and eagerness to learn have been truly inspiring, reminding us of the value that fresh perspectives bring.</p><p>This collaboration also proved to be a fantastic opportunity for our senior engineers, who embraced the role of mentors, sharing their expertise and guiding these students as they grew. It’s deeply rewarding to work with new talent and teach them how we approach challenges, laying the foundation for their success while reinforcing our own best practices and culture.</p><p>Encouraged by this success, we’ve decided to extend the program into 2025, welcoming even more bright minds to join us. Initiatives like this highlight our commitment to nurturing the next generation of engineers, empowering them to thrive, and driving meaningful contributions to our projects and company goals.</p><h3>Experimentation That Drives Innovation</h3><p>Experimentation is a crucial part of growth. It’s not always easy, and there are times when experiments don’t go as planned. But in today’s fast-paced environment, standing still and waiting for solutions to come to us is not an option. Experimentation allows us to learn, adapt, and uncover new opportunities — and in 2024, the results were truly transformative:</p><h3>Five-Sprint Experiment</h3><p>Focus and clear priorities are essential for achieving excellence, and this experiment has underscored just how impactful they can be. The five-sprint experiment helped us answer critical questions about the flexibility of our delivery structure and how much we can — and should — focus on specific priorities at once. By balancing structure and adaptability, we saw measurable improvements in both team satisfaction and speed of delivery, paving the way for more intentional and effective workflows.</p><h3>AITF Becomes a Full Team</h3><p>What began as an experimental task force dedicated to integrating AI into our products has now evolved into a fully operational team. This transition was marked by significant achievements, including the successful deployment of multiple AI-driven features. Notable projects such as <strong>Librarian</strong>, <strong>AI Image Scoring</strong>, and <strong>Asset Recognition</strong> demonstrated the team’s ability to deliver solutions, reinforcing the value of AI in enhancing our product offerings. The results speak for themselves, highlighting the importance of investing in specialized teams to drive innovation.</p><h3>Verticalization Week</h3><p>This dedicated cross-functional collaboration enabled us to tackle challenges specific to our business domains and deliver tailored solutions faster. With the involvement of a small yet highly focused team of engineers, supported by designers and product owners, the initiative achieved notable deliveries in just a few days.</p><p>Beyond its immediate successes, the verticalization approach will significantly contribute to shaping our future strategy and redefining how we approach projects. This initiative set a new standard for efficiency and impact in our workflows.</p><h3>Delivering Impactful Products</h3><p>Our teams delivered several features that not only enhanced our product suite but also showcased our ability to scale and innovate in response to growing demand.</p><p>2024 was a transformative year for our Care product, which has become an increasingly crucial part of our platform. Usage grew by several orders of magnitude, driven in part by the onboarding of major clients. With advancements in automated ticket handling and smarter customer workflows, Care is now indispensable for many of our partners.</p><p>The data teams also hit remarkable new heights in 2024, processing vast volumes of data at an unprecedented scale. We now parse over 20M social media profiles, while our data lake expanded to well over a petabyte of data. These capabilities power critical features across our platform and lay the foundation for even greater innovations.</p><p>In 2024, our AI infrastructure pushed boundaries, achieving incredible scale and performance. Our models processed 100–200M predictions per day in streaming mode and several millions more predictions per day for non-streaming solutions.</p><p>Finally, strengthening relationships with key clients was another significant focus of ours. We dedicated substantial engineering time to understanding their needs and incorporating their feedback into our solutions. The overwhelmingly positive response reflects the value of close collaboration and our commitment to client success.</p><p>Together, these projects contributed to a major milestone: Forrester named us a Leader in the <strong>Forrester Social Wave 2024</strong>, recognizing our excellence in the social space. We also ranked the highest in customer feedback, a testament to the trust and satisfaction of our clients.</p><p>These achievements highlight the collective effort of our teams to build solutions that truly make an impact, setting the stage for even greater accomplishments in the future.</p><h3>Connecting with the Industry</h3><p>Throughout 2024, our teams actively participated in industry events and collaborated closely with our partners and clients, including:</p><h3>Quarterly Business Reviews (QBRs)</h3><p>We hosted insightful workshops with key clients, analysts, and partners. While we can’t name them all, we worked with some high-profile brands across industries, including automotive, hardware, and consumer packaged goods (CPG). These sessions were invaluable for strengthening relationships and aligning our solutions with partners’ needs.</p><h3>Presentations and Events</h3><p>Our engineers shared their expertise at renowned events like WebExpo and ML Prague. At ML Prague, we organized two sold-out workshops and hosted a booth that drew significant interest from attendees. Similarly, at WebExpo, we had a booth to showcase our innovations, creating opportunities to connect with the broader tech community. Additionally, We led an exceptional session at Bootcamp (Thanks, Tim!).</p><h3>Client Support in Action</h3><p>As part of our commitment to our clients’ success, we sent one of our engineers to Spain to assist with the onboarding of a large client into our Care solution. This hands-on support ensured a smooth transition and highlighted our dedication to providing personalized service.</p><p>Through these engagements, we reinforced our presence in the industry, shared knowledge, and gained valuable insights that will continue to drive our innovations forward.</p><h3>Fostering Team Culture: Moments That Defined Us</h3><p>2024 wasn’t just about work — it was about building a thriving, collaborative culture that brought us closer together and set the foundation for future success.</p><p><strong>Team-Building Activities</strong><br>From exciting game nights to unforgettable team-building trips, we created opportunities to connect outside of work, strengthening the bonds that make our teams so resilient. These moments weren’t just fun — they played a big role in building trust and collaboration across teams.</p><p><strong>Engineering Logo and Merch</strong><br>We launched our new engineering logo and merch, which quickly became a huge success. The stylish designs sparked enthusiasm and a sense of unity within the team. It’s been great to see our engineers sporting the merch at events, in the office, and even during casual meetups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6hw4YIMAfK7txlmWOjoi2A.png" /></figure><p><strong>Welcoming New Leadership</strong><br>In 2024, we welcomed a new manager from Canada, bringing fresh ideas and valuable expertise to our team. As a global company with development teams across America, Europe, and Asia, their leadership has helped us strengthen our processes and inspire the team as we continue to grow together.</p><p><strong>Github Copilot and AI-Supported Development</strong><br>But that’s not all! In 2024, we also began exploring the potential of AI tools such as Github Copilot, Cursor, and ChatGPT to assist with coding, debugging, brainstorming, and other aspects of software development. While these tools are not yet widely adopted across all teams, the initial results are promising. They’ve shown the potential to enhance efficiency and creativity, helping engineers focus more on solving complex challenges and less on repetitive tasks.</p><p>As we continue to experiment and integrate these tools into our workflows, supporting and expanding AI-powered development will remain a key focus for 2025. By leveraging tools like Github Copilot, Cursor, ChatGPT, and others, we aim to unlock their full potential and make them an integral part of our development process.</p><p>This combination of cultural initiatives and technical advancements defined 2024 as a year of connection, growth, and progress.</p><h3>Thank You, Team!</h3><p>None of this would be possible without the incredible talent and dedication of our engineering teams. Thank you for making 2024 a year to remember. Here’s to continuing to innovate, build, and grow together in 2025.</p><p><em>Stay tuned for more updates as we continue to innovate, build, and grow together in 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fec66ad00aca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/socialbakers-engineering/engineering-highlights-of-2024-fec66ad00aca">Engineering Highlights of 2024</a> was originally published in <a href="https://medium.com/socialbakers-engineering">Emplifi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Storing Content Evolution of Social Media: Optimizing Space While Preserving Precision]]></title>
            <link>https://medium.com/socialbakers-engineering/storing-content-evolution-of-social-media-optimizing-space-while-preserving-precision-172486c9911d?source=rss----881fbb2b4996---4</link>
            <guid isPermaLink="false">https://medium.com/p/172486c9911d</guid>
            <dc:creator><![CDATA[Robert Fiser]]></dc:creator>
            <pubDate>Tue, 18 Feb 2025 09:24:21 GMT</pubDate>
            <atom:updated>2025-02-18T09:41:24.741Z</atom:updated>
            <content:encoded><![CDATA[<p>At Emplifi, we analyze content from social media platforms such as Facebook, Instagram, LinkedIn, Twitter, YouTube, and more. Once a post is published, it lives its own life, collecting interactions (likes, comments, shares, views, and even negative reactions). Over time, activity on the post naturally declines as the platform stops showing it to users.</p><p>This process can be represented as a graph showing the growth of interactions over time, also known as the <strong>evolution of a post</strong>. Typically, the curve starts with a gradual increase and eventually stabilizes. However, if the post’s author invests in promoting it (boosting), the graph reveals a sudden spike in interactions. Analyzing this evolution helps us visualize engagement in our product and serves as a data source for further analysis, such as estimating whether the interactions were organic or paid.</p><p>This article will explore storing these evolution data efficiently, minimizing storage requirements while preserving accuracy.</p><h3>Snapshots and the Storage Challenge</h3><p>We create a snapshot every time we capture the current number of interactions on a post. The more snapshots we store, the more accurate the resulting evolution curve. However, this approach comes with a trade-off: more snapshots require more storage space.</p><p>In the picture, you can see the shape of a common evolution of a social content post. The X axes represent time and the Y axes represent a count of interactions on the post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AUfUncePcYgw4Mtj3yBqAQ.png" /></figure><p>Originally, we stored the evolutions in HBase. Every snapshot was saved as a new row in a separate table. Because those tables were huge, we were looking for a more efficient way to store them when we decided to migrate from HBase to DynamoDB.</p><p>The idea was to save the evolution data in the same record as the body of a post. Because more data in DynamoDB means more money to pay for the service, and there is a 400 KB document size restriction, we started looking for some compression.</p><h3>Change of the format</h3><p>The output format of the evolution REST API looks like this:</p><pre>{<br>  &quot;contentId&quot;: &quot;123456&quot;,<br>  &quot;evolution&quot;: [<br>    {<br>      &quot;time&quot;: 1571748849,<br>      &quot;likes&quot;: 12,<br>      &quot;comments&quot;: 21<br>    },<br>    {<br>      &quot;time&quot;: 1571752446,<br>      &quot;likes&quot;: 16,<br>      &quot;comments&quot;: 27<br>    }<br>  ]<br>}</pre><p>When we have hundreds of evolution snapshots for the content, you can imagine that the data is very redundant. To keep the data as small as possible we decided to change the database format of evolution data:</p><pre>&quot;evolution&quot;: {<br>  &quot;header&quot;: [&quot;likes&quot;, &quot;comments&quot;],<br>  &quot;data&quot;: [<br>    &quot;0,0,0&quot;,<br>    &quot;30,12,21&quot;,<br>    &quot;60,16,27&quot;<br>}</pre><p>The header field contains a list of stored metrics and in the data field, each row represents a snapshot evolution record in comma-separated-value(CSV) format where the first value represents a relative time from a created time of the record. The type of data field is DynamoDB StringSet, which is why the row type is String CSV. We were considering having a two-dimensional array of numbers, but DynamoDB is a Document database and it stores the structure inside each document, and in that case, it would take much more space.</p><pre>&quot;string-set-field&quot;: {<br>  &quot;SS&quot;: [<br>    &quot;0,0,0&quot;<br>  ]<br>},<br>&quot;two-demensional-array-field&quot;: {<br>  &quot;L&quot;: [<br>    {<br>      &quot;L&quot;: [<br>        {<br>          &quot;N&quot;: &quot;0&quot;<br>        },<br>        {<br>          &quot;N&quot;: &quot;0&quot;<br>        },<br>        {<br>          &quot;N&quot;: &quot;0&quot;<br>        }<br>      ]<br>    }<br>  ]<br>}</pre><h3>Ramer–Douglas–Peucker Algorithm (RDP)</h3><p>We don’t need to store all points if the evolution curve is linear or nearly linear. Instead, we can keep only the <strong>endpoints</strong> (the first and last points) and a few critical points where the curve significantly changes. This way, we can drastically reduce storage requirements without losing essential information about the curve’s shape.</p><p>This problem has been tackled before by Ramer, Douglas, and Peucker, who developed the <strong>Ramer–Douglas–Peucker (RDP) algorithm</strong>. This algorithm simplifies a curve while preserving its overall shape.</p><p>RDP works iteratively as follows:</p><ol><li>Identify the first and last points of the curve.</li><li>Find the point farthest from the straight line connecting these two points.</li><li>If the distance of this point exceeds a given threshold <strong>epsilon</strong> (the tolerance parameter), keep the point and split the curve into two segments.</li><li>Repeat the process for each segment until no points outside the epsilon tolerance remain.</li></ol><h3>Storing Metric Evolution</h3><p>To store the evolution of a single metric efficiently, we use a simple data structure with two numeric arrays for the X and Y axes:</p><ul><li><strong>X</strong> represents the relative time since the post was published.</li><li><strong>Y</strong> represents the number of interactions.</li></ul><p>Each time a new snapshot is taken, a new point is added to these arrays. We then run the RDP algorithm on the updated curve, removing unnecessary points while preserving the curve’s shape within the epsilon tolerance.</p><h3>Optimizing Parameters</h3><p>The epsilon parameter plays a critical role in balancing accuracy and storage size. A higher epsilon value results in a less precise curve but smaller data size. Conversely, a lower epsilon value ensures greater precision at the cost of increased storage.</p><p>To have an epsilon in the same scale for every post, we have to normalize the data to have them in a range from 0 to 1. The normalization happens only for the elimination purpose which means the data is stored in the original scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*6cETWjg8t0Sl8ZGVso8dKA.gif" /><figcaption>Simplified curve by RDP, source: <a href="https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm">wiki page</a>.</figcaption></figure><h3>Conclusion</h3><p>Efficiently storing the evolution of social media posts is a challenging problem in optimizing storage. The Ramer–Douglas–Peucker algorithm enables us to reduce the number of stored points without significantly affecting the curve’s accuracy. This approach allows us to manage and analyze the interaction trends of a vast number of posts while minimizing storage costs. In extreme cases, we’re dealing with documents hitting 1MB, and after implementing this solution we keep evolution data for every content under 5KB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r4JwxPPtvTUhixGVvetgFg.png" /></figure><p>We not only save infrastructure costs but also enable faster and more efficient analyses, benefiting both our analytical teams and the users of our products.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=172486c9911d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/socialbakers-engineering/storing-content-evolution-of-social-media-optimizing-space-while-preserving-precision-172486c9911d">Storing Content Evolution of Social Media: Optimizing Space While Preserving Precision</a> was originally published in <a href="https://medium.com/socialbakers-engineering">Emplifi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How in-house facade helped migrating from Mesos to Kubernetes]]></title>
            <link>https://medium.com/socialbakers-engineering/how-in-house-facade-helped-migrating-from-mesos-to-kubernetes-21c993949fb2?source=rss----881fbb2b4996---4</link>
            <guid isPermaLink="false">https://medium.com/p/21c993949fb2</guid>
            <dc:creator><![CDATA[Martin Bydzovsky]]></dc:creator>
            <pubDate>Mon, 23 Sep 2024 12:13:59 GMT</pubDate>
            <atom:updated>2024-09-23T12:13:59.455Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This article is about how we smoothly migrated almost 3000 containers from Mesos to Kubernetes with zero downtime and almost no impact on our development teams.</blockquote><h4>History window</h4><p>Years ago, when we started implementing the microservice architecture, we immediately felt an urge to automate the whole build and deploy process to make it as easy as possible for our developers. It was 2014, and we were deciding between <a href="http://mesos.apache.org/">Mesos</a> and <a href="https://kubernetes.io/">Kubernetes</a>. Both tools were very immature, Kubernetes (even though growing fast) was still before its huge 1.0 milestone, while Mesos had already established itself as a powerful tool with some large companies using it (i.e. Twitter, <a href="https://highscalability.com/how-uber-manages-a-million-writes-per-second-using-mesos-and/">Uber</a> or <a href="https://www.theverge.com/2015/4/26/8499669/software-behind-siri-named-jarvis">Apple</a>). After some evaluation and with this in mind, we chose to go the Mesos way. Nowadays, it may seem like an unwise choice, but Mesos did provide us with great service over time, and we don’t regret the choice one bit.</p><h4>Wrapper</h4><p>At that time, we had approximately 80 developers, hundreds of services, each running from its git repository, some exposing an HTTP web server, some queue/batch processors, some cron jobs but usually, each microservice was a combination of all process types.</p><p>In Emplifi (and previously Socialbakers), we always encouraged our developers to be completely responsible not only for their code but also for the deployments. Each developer has the ability to release a new version of each service (those he works on) and ship it to production. It brings many benefits:</p><ul><li><strong>Accountability</strong> — Developers are more thorough as they feel the power and responsibility</li><li><strong>Substitutability</strong> — When one senior or DevOps engineer gets sick or leaves on vacation, you can seamlessly continue deploying</li><li><strong>Knowledge</strong> — since everyone can participate in the whole lifecycle and suggest improvements, the general “skillset” of all developers can grow much more easily</li></ul><blockquote>Unlike Kubernetes where you use one API to schedule all workloads, Mesos uses a concept of Frameworks (schedulers). To run something in the cluster, you don’t directly call the Mesos API but rather the Framework’s. There are different Frameworks for long-running jobs, crons, etc. They run as a standalone software and submit the jobs to the Mesos cluster). Frameworks are a bit similar to Kubernetes Operators, but they are needed even for basic tasks.</blockquote><p>However with such approach, every developer would have to learn multiple different APIs, how to write the job definitions, write the deploy scripts, learn the depths of each Framework (health checks, upgrade strategies, restart policies, how to define and route HTTP requests to ports, notify crashing apps, …) over and over. Reinventing the wheel at its purest form.</p><p>That’s why we decided to create a facade/wrapper on top of Mesos and Marathon/Singularity frameworks to simplify everyday routine work for all our teams. Such wrapper acted as a middle-man between the developers, GitLab, and the container runtime itself.</p><h4>How it works</h4><p>The facade does many things, not only the mentioned build and deploy process but also everything related to the lifecycle of the microservice. It is deeply inspired by <a href="http://heroku.com/">Heroku</a> and their <a href="https://12factor.net/">12 Factor manifest</a>.</p><ul><li>Facade exists as a web application with an interactive GUI, there is also CLI version to support terminal-oriented developers</li><li>Takes care of all projects and their environments (dev/staging/prod)</li><li>Defines units for each environment (a unit can be a cron, long-running app or a batch job)</li><li>Manages connection strings and variables in one place (so we easily know which DB is used by which service in case of maintenance/upgrade)</li><li>Handles builds: creates jobs, that transform source code into a runnable app, apply buildpack(s), add required runtimes nodejs/java/python, … install dependencies, run tests, and create runnable container images.</li><li>Manages deploys: creates entities in Mesos/Kubernetes based on the built images. As mentioned before, you can have multiple units (workers/crons/web apps/batch jobs) from the same release (image) and developers are shielded from manually configuring each type and knowing the internals of the configuration.</li><li>When a deployment is made, the whole environment of a microservice is updated. All units (apps/crons, …) always run from the same codebase, use the same environment variables, and are deployed as a bulk.</li><li>Subscribes to live events in the cluster and notifies teams or SLA duties about crashes, health check fails and other important updates</li><li>The facade also listens for git hooks (branch/tag push) to support automatic build and deploy to have a seamless CI/CD.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2V1lbWRTx_1xSro-kmeJdg.png" /><figcaption>Configuration of the microservice’s environment</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gF5MHnTmEET6N2WAo-R1bg.png" /><figcaption>Unit configuration</figcaption></figure><h4>Networking</h4><p>An important feature we introduced years ago is an automatic DNS for each (http-enabled) microservice. When you require a port for your unit, a unique DNS domain name is created, and incoming traffic is routed to the unit (if the unit has multiple instances, it is load-balanced). Such an address can look likehttp://web-production-example-api-tmp.mesos-prod.us-w2.aws.internal and is available both inside the cluster as well as from the outside (anywhere on the VPN). If such service needs to be exposed to the outer world (i.e. for clients), then DevOps and some approval process is required, of course.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U3s1pu9vHB8nijL7nSnJVw.png" /><figcaption>Unit detail</figcaption></figure><h4>Before Migration</h4><p>Unfortunately, back in 2020, Mesos development started to fade away. The last release published in Nov 2020 made us thinking about replacing this tool. At that time, we had approximately 800 long-running units scaled at 2–40 instances = a total of +- 2500 processes. In addition to that, 250 crons, ticking anywhere between “every 5 mins” and “once a week”.</p><p>Luckily, we didn’t have to make any tough decisions about what the replacement was going to be. Kubernetes has already grown to an industry standard so the choice was obvious. As we are using AWS as a cloud provider, we went for their managed solution <a href="https://aws.amazon.com/eks/">EKS</a>. They provide the master nodes (control plane), you just manage your EC2 worker nodes, and the cluster is ready.</p><p>First, we had to identify all the equivalent entities and how to translate the workload from Mesos to Kubernetes. One of the complex parts was the networking. Because of AWS EKS network limitations at the time of migration we decided to use different than the native CNI plugin for networking. It led to a situation, where application ports were not exposed outside of the cluster and we needed to route traffic to them via an ingress proxy. We decided to use <a href="https://doc.traefik.io/traefik/providers/kubernetes-crd/">Traefik</a> to route the traffic into the cluster from the outside.</p><p>For load balancing, Mesos (Marathon) had a tool, that consumed the cluster’s state and generated HAProxy config file based on running tasks. When an application was deployed/restarted/scaled, the config file updated and the HAProxy reloaded, that way the automatic routing and balancing were achieved.</p><p>However, in Kubernetes, each pod allocates its own “cluster-private” IP address, so the networking comes with a bit more effort. First, you need to deploy the application (typically a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Deployment</a>). Then define a <a href="https://kubernetes.io/docs/concepts/services-networking/service/">Service</a>, connect it with deployment’s pods, and finally define an <a href="https://doc.traefik.io/traefik/providers/kubernetes-crd/">IngressRoute</a> (Traefik’s CRD) that is bound to the previously created Service. When you are writing the Kubernetes yamls manually, you quickly run into naming issues, it is hard to enforce (not only) naming policies or best practices for writing all the individual entities (CronJobs, Jobs, …).</p><p>It was at this moment we realized how helpful our facade was. Each deployment (and we are always updating all units of the microservice’s environment) results in a minimum of three different calls to the Kubernetes API per unit. What&#39;s more, every developer would need specific knowledge of how to configure a Deployment, Service or IngressRoute. We are soon going to adopt <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">HPA</a>, which would result in occasional updates to that entity as well.</p><h4>Container Images</h4><p>As previously mentioned, we are building the images themselves on the Mesos/Kubernetes platform as a one-off batch jobs. Those images are completely self-sufficient and can be run pretty much anywhere. The images contain a base system (Ubuntu) with some essential packages and our internal SSL certificates. On top of that, we clone the code and apply all required Heroku buildpacks to transform the source code into a runnable piece. Also, environment variable needed to run the application are preserved inside the image. That means, you can take the image and run it on Mesos, Kubernetes, or even on your local computer without any necessary adaptations or changes. As Kubernetes<a href="https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/"> dropped support for Docker</a> some time ago and building an image inside a container is not completely straightforward, we started using <a href="https://buildah.io/">buildah.io</a> to pack and upload the images to <a href="https://aws.amazon.com/ecr/">ECR</a>. Having this setup (not relying on any runtime-specific features) also helped us with the migration.</p><h4>Migration</h4><p>As the facade was the only place where deplyments to Mesos were made it was pretty straightforward where to begin. We decided to go for a gradual migration. Slowly start one microservice at a time to Kubernetes, check everything works, transfer public DNS records, monitor performance and if everything is fine, stop the original Mesos instances. Developers already could choose into which cluster the environment will be deployed (dev or prod). So we simply added options representing the new Kubernetes clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZ89GPZdteDrNEqZnnfhCA.png" /><figcaption>Temporary allowing deployments to both Mesos and Kubernetes clusters</figcaption></figure><p>The facade also listens on the Kubernetes event stream to read and handle live updates/changes in the cluster. This way, developers are notified about crashes (sent into Slack or via SMS) and other important updates. Mesos provided such stream and so does Kubernetes (kubectl get pods --watch)</p><p>Obviously, we had to implement the whole Kubernetes API calls, how to submit deployments, cron jobs, services, etc… So we coded all the communication — and it wasn’t difficult task at all: we had everything done for Mesos, just the “translation” to Kubernetes entities had to be done.</p><p>Important thing is cleaning up old entities. Our apps can (and often do) have multiple ports — i.e. one for API endpoints and another for web. That means one unit (= Deployment) is bound to multiple Services and IngressRoutes. Sometimes the unit configuration changes and the port count decreases — so we need to remove some of those entities. Or when the unit is completely removed, all the network related entities must be deleted too.</p><p>After everything was prepared and tested, developers were able to migrate their apps on their own. They just had to change the target cluster and our facade took care of everything.</p><p>There was one thing that needed to be adopted and modified by many applications: The environment variables. Some units are logging custom formatted log messages into Kibana, storing custom metrics into Prometheus, and to describe and identify each process, they rely on the runtime&#39;s provided process metadata. In Mesos, each task had a unique variable MESOS_TASK_ID. In Kubernetes, simmilar variable is called POD_NAME. So some of the services had to be updated so they would read the correct variables and function properly even in the new runtime. But luckily, this was one of very few (if not the only one) issues, that had to be addressed during this migration.</p><h4>Conclusion</h4><p>Creating the facade has proven as a great idea multiple times. This migration was just the tip of the iceberg. Approximately 100 developers didn&#39;t have to waste their valuable time on learning new APIs, re-discover how to deploy a cron, how to define ports, external traffic routing for their apps and million other things.</p><p>The facade also saves significant time during regular usage. After initial setup developers don&#39;t need to interact with it at all, instead they focus on what is important: writing the code. We also don&#39;t need our developers to cross-team communicate with DevOps teams (or someone like “Release Managers”) to handle the app upgrades. Developers themselves manage the application updates, which saves us both time and money.</p><p>Since the facade is the central place to communicate with Kubernetes, if we want to modify some internal setting of the Deployment, Pod or any other entity, we simple edit how the manifest is submitted to the cluster. No need to bulk update tens or hundreds of yaml files across all the repositories. We aim for decentralized work, not infrastructure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21c993949fb2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/socialbakers-engineering/how-in-house-facade-helped-migrating-from-mesos-to-kubernetes-21c993949fb2">How in-house facade helped migrating from Mesos to Kubernetes</a> was originally published in <a href="https://medium.com/socialbakers-engineering">Emplifi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Love your CLI a bit more]]></title>
            <link>https://medium.com/socialbakers-engineering/love-your-cli-a-bit-more-50f07c06228a?source=rss----881fbb2b4996---4</link>
            <guid isPermaLink="false">https://medium.com/p/50f07c06228a</guid>
            <category><![CDATA[cli]]></category>
            <category><![CDATA[tools]]></category>
            <dc:creator><![CDATA[Petr Miko]]></dc:creator>
            <pubDate>Fri, 23 Aug 2024 10:51:34 GMT</pubDate>
            <atom:updated>2024-08-23T10:51:34.279Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s face it — as developers we spend a significant amount of time in the terminal. Some people love it, some people hate it, but no matter which group you belong to, I am nearly certain you can try a few recommendations from this article to make your time in the terminal a better experience 🙂</p><p><em>Disclaimer before we start:</em></p><ul><li><em>the steps are my personal favorites and your preferred selection of tools may vary</em></li><li><em>while I tried to target all major operating systems and a broader audience, you can probably tell I am a Node.js developer working in zsh</em></li></ul><h3>0. Prepare a package manager</h3><p>Linux user? Great, you’re all set and can jump to the next section. But for the Windows and MacOS users installing new applications via the terminal can seem a bit intimidating at first. Sure, for the non-juniors among you, this is obvious, but let’s agree that package managers are great as they do the following:</p><ul><li>Provide a universal interface to operate with the application lifecycle. Find, get info, update, remove… All there. So you can even write a script to set up your machine.</li><li>Usually edit your environment variables like <em>$PATH</em>, prepare symlinks, so you can use the app without any more tinkering. If not, it tells you what to do.</li></ul><p>So what are the options here?</p><ul><li>🪟 winget (<a href="https://apps.microsoft.com/detail/9nblggh4nns1">MS Store</a> or follow the <a href="https://learn.microsoft.com/en-us/windows/package-manager/winget/">official article</a>)<br>- Microsoft’s proprietary solution<br>- you can also use <a href="https://learn.microsoft.com/en-us/windows/wsl/install">WSL</a> → welcome to the Linux world</li><li>🍎 Homebrew (<a href="https://brew.sh/">official site</a>)</li><li>🐧 Homebrew is also an option. Or use your OS’ package manager like dnf.</li></ul><h3>1. Choose you CLI application</h3><p>Ok, so you are not using raw Linux terminal with tmux, right. Right?… While there are some default terminal applications in our operating systems and the Linux ones are kinda OK, the Windows and MacOS default terminal apps feel suboptimal. Ugly, old, not offering many shortcuts and/or customizations. Yep, they mostly suck. So the first step is selecting a better application.</p><ul><li>🪟 Windows Terminal (<a href="https://apps.microsoft.com/detail/9n0dx20hk701">MS store</a> or <em>winget install -e — id Microsoft.WindowsTerminal</em>)<br>- should be default in Windows 11 by some update, but you can switch yourself in Win10/11 if you’re still stuck with cmd</li><li>🍎 iTerm2 (<a href="https://iterm2.com/">official site</a> or <em>brew install cask iterm2</em>)</li><li>🐧 KiTTY (<a href="https://sw.kovidgoyal.net/kitty/">official site</a> or your package manager of choice)</li></ul><h3>2. Install fonts with icons and ligatures support</h3><p>Following steps in this article will expect fonts that can show you fancy ligatures and icons in the directory list or next to the name of your current git branch. What is a ligature? See an example in the next picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*pmoRF17IIHNYPtD_OdTnuw.png" /><figcaption>Example of font ligature</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qAXPp9ani22MgiQPAGFJgw.png" /><figcaption>Icons in eza output</figcaption></figure><p>Head out to <a href="https://www.nerdfonts.com/font-downloads">Nerd Fonts</a>, make your pick and install some font(s)! My favourite pick is FiraCode Nerd Font.</p><h3>3. Shell themes and plugins</h3><p>By now you have your fancy terminal app loaded with great fonts, only the terminal looks are still not tweaked. So it is time to select a Theme manager based on your shell and install it!</p><ul><li><strong>powershell</strong> — Oh My Posh (<a href="https://ohmyposh.dev/">official site</a>, <em>winget install -e — id JanDeDobbeleer.OhMyPosh</em>, <em>brew install oh-my-posh</em> )</li><li><strong>zsh</strong> — Oh My ZSH (script from <a href="https://ohmyz.sh/">official site</a> is the preferred way)</li></ul><p>For both, there are plenty of available themes you can pick from, and both rely on storing the configuration in the shell profile/resource file. Follow their respective tutorials closely. Or if you want to just read the article for now, I will provide my example Powershell profile file and zshell configs by the end of it.</p><p>But wait! Did I mention plugins? Yes, but only for Oh My ZSH. You can extend its behavior based on any of <a href="https://github.com/ohmyzsh/ohmyzsh/wiki/Plugins">community available plugins</a> to alter your shell behavior even further. My favorite plugin is <em>history-substring-search </em>— type a few letters that are present in any location of some command you’ve run in the past, press the up arrow button, and bam! You are browsing through history in CTRL+R style with ease!</p><h3>4. Install all the applications!</h3><p>Congratulations! You are entering the last yet never-ending task of finding the right CLI application you will use for your daily tasks 🎉. But what are the options? For Unix-like systems, I can recommend looking into <a href="https://github.com/ibraheemdev/modern-unix">Modern Unix list</a>. I am sure you will find some that will catch your eye. But what about Windows? I got some tips even for you.</p><h3>4.1 Eza</h3><p><a href="https://github.com/eza-community/eza">Eza</a> is a Rust-based successor of the no longer maintained <em>exa</em>, so it is basically an <em>ls</em> alternative on steroids. And since it is written in Rust, not only is it blazing fast™, but also cross-platform. Now is the time to list your files with icons and all that Git status information!</p><ul><li>🪟 <em>winget install eza-community.eza</em></li></ul><pre># add into Powershell profile file<br>function EzaAliasBasic { [alias(&#39;ls&#39;)] param([string]$Path=&#39;.&#39;)<br>    eza.exe --group-directories-first --icons $Path @Args<br>}<br><br>function EzaAliasList { [alias(&#39;ll&#39;)] param([string]$Path=&#39;.&#39;)<br>    ls -l --git $Path @Args<br>}</pre><ul><li>🍎/🐧 <em>brew install eza</em></li></ul><pre># .zshrc<br>alias ls=&quot;eza --icons --git --group-directories-first&quot;</pre><h3>4.2 Zoxide</h3><p>It has been a few years since a colleague of mine introduced me to the convenience and ease of use of an application named <a href="https://github.com/wting/autojump">autojump</a>. It learns paths you visit by the usual change dir <em>cd</em> traversal, and then you can later auto-jump to that location by typing <em>j</em> plus a few letters in the path…</p><pre>cd ~/Development/analytics-unified<br># vs<br>j uni</pre><p>Seems like no big deal, but the time savings! Especially if you need to switch across many repositories. So why am I writing about <a href="https://github.com/ajeetdsouza/zoxide">zoxide</a> in the section heading and then describing autojump? Because zoxide is autojump but <strong>better</strong>.</p><p>Unlike autojump, written in Python, zoxide is written in Rust, thus it does not require any additional software to run as you are installing a binary. You can change the alias from <em>z</em> to <em>j</em> so you can continue with your autojump-grown muscle memory and avoid qwerty/qwertz issues that non-native speakers, like I often face when switching layouts. And you are not losing your history when making the switch to zoxide — you can import autojump’s history. It cannot get any better, right? Well… It can!</p><p>If you install <a href="https://github.com/junegunn/fzf">fuzzy finder</a> <em>fzf</em> , you can use interactive mode <em>zi</em> (or <em>ji</em> if you set up the alias) to easily search in paths in case of similar result paths for the same input. Love it.</p><ul><li>🪟 <em>winget install -e — id ajeetdsouza.zoxide + winget install -e — id junegunn.fzf</em></li></ul><pre># add into Powershell profile file<br>Invoke-Expression (&amp; { (zoxide init powershell --cmd j | Out-String) })</pre><ul><li>🍎/🐧 <em>brew install zoxide fzf</em></li></ul><pre># .zshrc<br>eval &quot;$(zoxide init --cmd j zsh)&quot;</pre><h3>4.3 Fast Node Manager</h3><p>Remember the time savings I mentioned in the previous section? Here is some more for Node.js developers! I am pretty sure that most of us know about <a href="https://github.com/nvm-sh/nvm">Node Version Manager</a> aka <em>nvm</em>. It serves to switch between various versions of Node.js without much overhead, except for it not making the switch automatically when you open a project directory. And as I mentioned above, I switch between directories. A lot.</p><p><a href="https://github.com/Schniz/fnm">Fast Node Manager</a> <em>fnm</em> registers your <em>.nvmrc</em> files by default so when you enter a folder, it switches to the correct version of Node.js. And if you use the auto-switch on change dir, you can stop thinking about Node.js versions for most of the time. By the way, just by switching from <em>autojump</em> and zsh-scripted <em>nvm</em> autoswitch to <em>zoxide</em>+<em>fnm</em>, I am finally no longer facing VS Code warnings about slow terminal session init. What more can you ask for? Install steps, right?</p><ul><li>🪟 <em>winget install -e — id Schniz.fnm</em></li></ul><pre># add into Powershell profile file<br>fnm env --use-on-cd | Out-String | Invoke-Expression</pre><ul><li>🍎 / 🐧 <em>brew install fnm</em></li></ul><pre># .zshrc<br>eval &quot;$(fnm env --use-on-cd)&quot;</pre><p>As for <a href="https://volta.sh/">Volta</a> — yes, it is a great tool which even switches to the right version after pulling the updated package.json from remote, which FNM cannot do, but it does not provide a prebuilt binary for Asahi Linux. I heard there are people who do not like that. Like me. So the “official reason” or made-up excuse for my recommendation of FNM is, of course, that our company is already invested in <em>.nvmrc</em> files.</p><h3>4.4 Fastfetch</h3><p>Life is not only about productivity. Sometimes we need some things just for the sake of liking them. Like this system overview list in your terminal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dKaUxp5GdCyDJ1Id0gObPg.png" /></figure><p>“Hey, that is Neofetch!”. Nope, <a href="https://github.com/fastfetch-cli/fastfetch">Fastfetch</a>. Neofetch was recently <a href="https://github.com/dylanaraps/neofetch">archived</a>, so Fastfetch is now the maintained way to get cool looking overviews of your setup.</p><ul><li>🪟 <em>winget install fastfetch</em> or <a href="https://github.com/fastfetch-cli/fastfetch/releases/">releases page</a></li><li>🍎 / 🐧 <em>brew install fastfetch</em></li></ul><h3>5. Is there more?</h3><p>For sure there is. I could recommend you <em>tig</em> for easy work with git logs and changes in commits, but that is not easily installable cross-platform. There are apps in the <a href="https://github.com/ibraheemdev/modern-unix">Modern Unix list</a>. And certainly there are some new fresh applications I have never heard of yet. Or I could tell you that you can use <em>export EDITOR=”code -w -n”</em> to use VS Code for editing commits instead of that “infamous” Vim.</p><p>But at least I hope, you learned a thing or two about the possibilities and in case you knew them all already, you at least enjoyed this article! Feel free to add comments and here are the promised <a href="https://gist.github.com/petrmiko/62b1ee39a3cef26370f3222dce28e8b1">Powershell profile</a> and <a href="https://gist.github.com/petrmiko/67fac1735d0d18191f828455a8ba6e5d#file-zshrc">ZSH</a> files.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50f07c06228a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/socialbakers-engineering/love-your-cli-a-bit-more-50f07c06228a">Love your CLI a bit more</a> was originally published in <a href="https://medium.com/socialbakers-engineering">Emplifi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text Generation in Customer Service (Part 2)]]></title>
            <link>https://medium.com/socialbakers-engineering/text-generation-in-customer-service-part-2-7c6c64103877?source=rss----881fbb2b4996---4</link>
            <guid isPermaLink="false">https://medium.com/p/7c6c64103877</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[crm]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[text-generation]]></category>
            <dc:creator><![CDATA[Renhao Cui]]></dc:creator>
            <pubDate>Mon, 17 Apr 2023 07:50:43 GMT</pubDate>
            <atom:updated>2023-04-17T07:50:43.660Z</atom:updated>
            <content:encoded><![CDATA[<h3>Before you read</h3><p>In Part 1 of the article, we introduced our Retrieval Based Response Letter Generation solution for Customer Service. In Part 2, we continue this topic to show the evaluations of our solution with real industry data, using both standard NLP evaluation metrics and human evaluation. The full test dataset consists of 1690 samples, and out of it 910 samples were randomly selected for human evaluation.</p><h3>Standard Evaluations</h3><p>In this section, we use standard NLP evaluation metrics to report the performance of the three modules of our framework: 1. Retrieval and 2. Core Response Generation and 3. Paraphrasing of templates. Additionally, we also look into several other aspects to better understand the model performance.</p><h4>Retrieval Performance</h4><p>A higher similarity between retrieved and reference response is a prerequisite for better generation and hence indicates a better retrieval. Our analysis finds that for 46% of test queries, our retrieval model fetches at least one reference-like (similarity &gt; 0.9) historical response within the top-10 candidates of each retrieval. Furthermore, our manual evaluation on randomly sampled retrievals finds 49% of the retrieved responses suitable for generation and an additional 20% are somewhat relevant. With meaningful thresholds, there are enough retrieved historical cases that can be used to generate better responses.</p><h4>Generation Quality (Standard Scores)</h4><p>The automatic evaluations of all the methods is conducted on the full test set with the optimal hyper-parameter setting and the results are shown in Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RiZTYReM2QoyC6i5" /></figure><p>Figure 1: Test set results of the proposed response generation model on DC and TT dataset. Automatic scoring metrics are: AverageSentenceSimilarity (S), BLEU-4 (B), NIST (N), METEOR (M), ROUGE-L (R) and CIDEr (C).</p><p>For the <strong><em>Retrieve Only</em></strong> method we consider only the fetched historical response (without refinement) as a hypothesis. For <strong><em>baseline</em></strong>, <strong><em>RetRef </em></strong>and <strong><em>Hybrid</em></strong> methods, we consider hypotheses that are produced using (top-k,top-p with temperature) decoding setup, as its corpus-level score was better than other combinations. In ranking enforced versions of <strong><em>RetRef</em></strong> and <strong><em>Hybrid</em></strong>, instead of new generation, for each query we picked the hypothesis with the highest rank score for evaluation.</p><p>To assess our model, we utilize commonly used metrics such as Average Sentence Similarity, BLEU, NIST, METEOR, ROUGE-L and CIDEr. The average sentence similarity score is a measure of semantic similarity between reference response and hypothesis. We use sentence-BERT [1], a trained Siamese BERT-networks to encode a reference and a hypothesis and then calculate the cosine similarity of the resulting embeddings. The final similarity score is the mean value over the test set.</p><p>For both datasets, our proposed retrieval-based response generation model (<strong><em>RetRef+Rank</em></strong>) outperforms all other baselines in all the metrics (Figure 1). Specifically, it achieves an average improvement of 16% for <em>DC</em> and 7% for <em>TT</em> dataset in all metrics over the fine tuned GPT-2 baseline without retrieval. Understandably, knowledge retrieval plays a key role in this improvement. On the other hand, without refinement, the<strong><em> Retrieve Only</em></strong> approach yields the worst scores. The <strong><em>Hybrid</em></strong> version can switch between <strong><em>baseline</em></strong> and <strong><em>RetRef</em></strong> based on the availability of suitable retrieved responses and is evaluated including such test cases. Nevertheless, it outperformed the <strong><em>baseline</em></strong> model by a significant margin across all metrics and datasets.</p><p>To measure the extent to which model incorporates a retrieved knowledge in generation, following previous work, we measure word overlap between generated and retrieved responses. The result shows that our <strong><em>RetRef+Rank</em></strong> model retained more than 70% words from retrieved information in 51% and 57% of the test generation of <em>DC</em> and <em>TT</em> dataset, respectively. This is a clear improvement over <strong><em>baseline</em></strong> and the basic <strong><em>RetRef</em></strong> model which shows such overlap less frequently.</p><h4>Qualitative Analysis</h4><p>We also conduct a manual inspection to assess the relevance and informativeness of small-scale randomly selected hypotheses by 3 experts in this field. Relevance measures if a generated response is based on the corresponding product and reason, whereas informativeness checks for its information consistency with respect to the reference response (both scored out of 5). The result shows that responses produced by our <strong><em>RetRef+Rank</em></strong> model yield roughly a 9% higher relevance (4.05 for DC, 4.49 for TT) and a 12% better informativeness (3.75 for DC, 4.24 for TT) score than the baseline model for both datasets.</p><h4>Ablation Study</h4><p>Apart from the inclusion of retrieved knowledge, two other notable contributors to the performance of the framework are data augmentation and response ranking. Our experiments reveal that the creation of more training instances with multiple candidate responses increases the automatic score by 12% in BLEU-4, 6% in CIDEr and roughly 2% in other metrics. The role of ranking is also evident from the significant raise of <strong><em>RetRef+Rank</em></strong> and <strong><em>Hybrid+Rank</em></strong> model scores from their base version as shown in figure 1. This can be attributed to the ranker’s policy to penalize irrelevant generation while favoring the one that integrates quality retrieval.</p><h4>Examples and Discussion</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*JOhAI3WfsZaDpCpbQL4c8A.png" /></figure><p>Table 1: Sample response generation using our <strong><em>RetRef+Rank</em></strong> model</p><p>Table 1 provides a randomly selected generation. It suggests that our model’s response is aligned with the customer letter type. The customer inquiry is responded accordingly with clarification, appreciation and information. In addition, having historical knowledge, our model is not only capable of producing an informed response but also refines that according to the query. A few limitations of our model include its inability to verify time-sensitive historical information and handling multiple questions from a single message. Additionally, the way it offers a coupon or commits a follow-up to a customer poses a risk. To resolve these issues, a risk or confidence measuring system can be introduced which will seek human inspection before dispatching a risky response. We leave this as future work.</p><h4>Paraphrasing Outcome</h4><p>We obtained 2826 unique new ending templates that were not in the original input ending templates. After sorting them in the ascending order of their perplexity, lexical and semantic similarity, we conduct a manual evaluation. It shows that 90% of the first 100 candidates are good to use, whereas this fraction drops to 36% for the last 100 candidates.</p><h3>Human Evaluations</h3><p>Automatic evaluation is not enough to measure the true quality of a text generation system, so we also conduct human evaluation on randomly selected 910 test samples using trained raters. For any example case, each human rater is shown a customer query, generated response (hypothesis), retrieved query-response pair, and reference response. Then they were asked to respond to the four following multiple choice questions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*FANbjZuI3j8_9HTBstg8Mw.png" /></figure><p>Table 2: Question and options settings for our human evaluation task</p><p>For each question, there are four choices and they are ordered from bad to good. The first two questions (a and b) are focused to evaluate the quality of generated response (hypothesis) from two perspectives: 1. Relevance of response without knowing the actual information and 2. Informativeness of the response with the access to true information. The last two questions (c and d) are set to assess the performance and impact of the retrieval system respectively. Each case is rated by 3 different raters and the final score is average of their ratings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*WR42ceMlofFFzXH3" /></figure><p>Figure 2: Distribution of the options selected by the annotators</p><p>Based on the Fleiss’ Kappa score, the raters’ agreement for the choices of question a) and c) was <em>fair </em>and for question b) and d), it was <em>moderate </em>and <em>good </em>respectively. In around 75% cases of both question c) and d) the rater chose <em>‘option 4’</em>. This indicates for those cases raters found that the retrieved query is very similar to the corresponding customer query and the generated responses are aligned with the retrieved information.</p><p>The evaluation also shows that for around 59% of the cases, the raters found the generated response is plausible or relevant without reference. With respect to the reference, the raters considered the models’ response at least equivalent to the reference in 61% cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*tMMvtaaimzcngCDggQLFEg.png" /></figure><p>Table 3: Comparison of distributions between copy and non-copy cases</p><p>In 75% cases the model’s generated responses are partly or fully copy of the retrieved response. Out of these cases, 39% generated responses lexically match with the corresponding reference responses. According to human evaluation, in 65% of the cases, the copy-driven generated responses are equivalent or preferable to reference response. These human supported 65% cases include almost all the machine-evaluated matched cases (38% out of aforementioned 39%). Furthermore, the human and machine evaluated label of a generated response (whether equivalent to the reference response) has co-occurred 71.98% times of all the copy-based generations.</p><p>On the other hand, in around 25% cases the generated responses are found to be lexically different (not copied) from the corresponding retrieved responses. 18% of such generated responses lexically match with the reference response whereas human evaluation considers 54% of them are equivalent or preferable to reference response. The human and machine evaluation of the generated responses matches for 80.99% cases.</p><h4>Identification of applicable case</h4><p>While in many cases our model is able to produce plausible responses, there are some cases where it struggles. Based on the human evaluation, we want to identify cases that our model is likely to generate human-favored responses. Our manual observation suggests that there are three scenarios when the model struggles:</p><ol><li>Retrieved query is not a good match to the current query.</li><li>Retrieved response is very loosely connected to the current query.</li><li>Complex, infrequent (less experienced) current query (same holds for less generic, complex retrieved response).</li></ol><p>In the above scenarios, the model tends to craft responses instead of following retrieved responses and in the process often generates generic responses that it is frequently exposed to during training for the corresponding reason code. On the other hand, in the absence of the above three scenarios, the model is often seen to follow the retrieved response which typically results in human-favored response.</p><p>Therefore, we want to capture the above three scenario with the following measures:</p><ol><li>Retrieval score.</li><li>Training an entailment model between query and response.</li><li>Complexity has been seen in larger queries, so query length can be a feature to measure query complexity. Sometimes simple queries from infrequent reason code also make the mode hallucinate. For retrieved responses, a larger length won’t prevent generating a faithful response if it is a generic one.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*YTQrFmHK-foZLkjhvoTX6A.png" /></figure><p>Table 4: Relation between the generation quality and query-response similarities</p><p>Based on the previous three observations about when the model typically succeeds, we check to see if the data supports that. In table 4, generated response quality is defined by sum of raters rate, semantic and lexical similarity between reference and the generated response. And each row gives the correlation and p-val (level of significance) between items in the first column and generated response quality. We notice that the correlation of retrieval score (Query-Retrieved Query similarity) is highly significant (p-val &lt;&lt; 0.05) in determining model’s generation quality. The other significant correlation between generation quality and retrieved response length indicates the model is typically right when it copies large generic retrieved responses.</p><p>Query-Ret. response similarity, is not exactly entailment and its correlation is not significant as well. Finally, query length is used as a heuristic of complexity. However, its correlation to response quality is also not strong.</p><p>To further improve the predictability as to when the model can be useful we apply three entailment models on the human rated dataset. A cross-encoder model is utilized to predict the human-like rating as to whether the entailment of a query-retrieved response pair or a query-hypothesis pair is likely to make a good final output.</p><p>We compared the results of the three cross encoder models for both pairs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*yDpzq-8wwA4EGMB8gUKK5Q.png" /></figure><p>Table 5: Performance of entailment models on query-retrieved response and query-hypothesis pairs</p><p>Table 5 lists the accuracies of the three entailment models. The result shows our BERT-based cross-encoder improves the accuracy by around 100% over the other two models. It indicates we can effectively estimate the generation’s usefulness in around 70% cases</p><h3>Conclusion</h3><p>We propose a neural response generation framework to reduce human labor in a customer care setting. Considering a real-world scenario where a structured knowledge base is scarce, our framework extracts knowledge from history and utilizes that for informative response generation. Our evaluation shows the efficacy of the ranking system and provides evidence for the applicability of the framework in a real-life business operation. We plan to facilitate the framework with a response validation module for further improvement.</p><h3>Reference</h3><p>[1] Reimers, Nils, and Iryna Gurevych. “Sentence-bert: Sentence embeddings using siamese bert-networks.” <em>arXiv preprint arXiv:1908.10084</em> (2019).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c6c64103877" width="1" height="1" alt=""><hr><p><a href="https://medium.com/socialbakers-engineering/text-generation-in-customer-service-part-2-7c6c64103877">Text Generation in Customer Service (Part 2)</a> was originally published in <a href="https://medium.com/socialbakers-engineering">Emplifi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text Generation in Customer Service (Part 1)]]></title>
            <link>https://medium.com/socialbakers-engineering/text-generation-in-customer-service-1-80c1114ca98e?source=rss----881fbb2b4996---4</link>
            <guid isPermaLink="false">https://medium.com/p/80c1114ca98e</guid>
            <category><![CDATA[customer-service]]></category>
            <category><![CDATA[text-generation]]></category>
            <category><![CDATA[crm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Renhao Cui]]></dc:creator>
            <pubDate>Tue, 14 Feb 2023 08:27:55 GMT</pubDate>
            <atom:updated>2023-02-14T08:27:55.846Z</atom:updated>
            <content:encoded><![CDATA[<h3>Before you read</h3><p>The article will be easier to follow if you have some understanding of generative models in the text domain. It would be even better if you are familiar with Transformer or GPT models. For more technical details, take a look at our NAACL paper <a href="https://aclanthology.org/2022.naacl-industry.20.pdf"><em>Retrieval Based Response Letter Generation For a Customer Care Setting</em></a>.</p><h3>Background</h3><p>Most modern businesses operate a customer care service to support customers with product information and to address their concerns. Letter-like communication (such as emails) serves as a major channel for such customer relationship management. These exchanges are initiated by requests from customers and then responded to by the organization within the same channel. This support service plays a vital role in ensuring a good customer experience and is a key factor in developing goodwill.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z113ceUXo4FXdDQq" /><figcaption>Figure 1: Example of a customer-agent email exchange</figcaption></figure><p>Traditionally, a trained human acts as a customer care representative or agent and provides customer support through live conversation or via email exchanges (Figure 1). Doing this manually at scale demands an enormous human effort, as the task involves understanding each customer’s query and then replying appropriately. Given a massive volume of customer queries, the process becomes very time-consuming.</p><p>Automating the response generation process can go a long way towards solving this problem. That said, while the rule-based systems that exist today can handle very specific requirements, they often struggle to capture the linguistic complexity of real-world communications. To address these issues with advanced technology, we explore an AI-driven approach that is robust, efficient and pragmatic with respect to available resources.</p><h4>Use cases</h4><p>The major use cases covered in our task focus on the automatic generation of response letters in customer service. Typically, this step is handled by human agents with the help of rule-based or template-aided systems. In contrast, we plan to build a system that can generate appropriate, consistent, and diverse responses.</p><p>The benefits of such a system are:</p><ol><li><strong>Reduction of human labor and increase of support volume</strong></li><li><strong>Diversity in agent responses, giving customers the sense of a personalized reply</strong></li><li><strong>Query analytics can help to understand customers’ inconveniences and expectations</strong></li><li><strong>Consistency across customer service cases</strong></li></ol><h4>Targets and goals</h4><p>A complete automatic generation system is always desired, however, there are certain concerns when applying it to a real customer-facing process. That’s why we decided to give human agents the option to adopt the final text generated by the system. By having human agents verify and make necessary modifications to the generations, we can ensure improved accuracy and customer experience.</p><h4>Challenges</h4><p>Recent advancement in language modeling has shown significant success in generating fluent text. In goal-oriented response generation systems, user utterances are usually fact-finding queries. To learn to understand such queries, existing solutions use datasets annotated with slot value pairs (e.g. ‘Find a park near area51’, destination: ‘park’, close_to: ‘area51’). Corresponding framework first maps the values to slots and then uses the slot-value pairs to retrieve facts from a knowledge base to reply.</p><p>Unfortunately, in reality, both the slot-value annotation and structured knowledge base are often unavailable and difficult to manage. Besides, in a customer care setting, a user prompt may not be limited to just an inquiry about facts, but may also include a complaint, suggestion, compliment, request, etc.</p><p>In this work, we factor in these challenges and present a response generation framework that automatically produces and ranks response letters addressing customers’ queries or feedback, and which doesn’t require fine grained annotation.</p><h3>Our Approaches</h3><h4>Explore the data</h4><p>In the experiments, we use two proprietary datasets from the restaurant and adhesive tape industries, naming them <strong>DC</strong> and <strong>TT</strong>. They consist of 8,448 and 14,938 unique email exchanges respectively between customers and agents. Each exchange contains a case ID, product and reason code of the service, as well as the customer query letter and the human agent’s response letter. The reason code stands for the type of customer query. A simple example of a customer service case in the dataset is listed below.</p><blockquote>Customer Query: I just signed up for X-org this morning and have not received my coupon for free pancakes yet. When will I receive it?</blockquote><blockquote>Agent Response: After signing up, it may take up to 24 hours to receive your initial offer.</blockquote><blockquote>Product Code: SERVICE RELATED, Reason Code: GC — PROMO — I</blockquote><p><strong>Data Preparation</strong></p><p>We mask specific personally identifiable or proprietary information elements such as names, email addresses, phone numbers, prices, franchise names, and dates in our dataset with corresponding generic tokens (“X-email”, “X-phone”, etc.). This serves two purposes. On one hand, this anonymization protects the privacy of the customers and the organization. Second, it forces the model to learn from and generate generic tags instead of considering noise in the form of irrelevant details such as specific names and values.</p><p>Note that a typical response may be split into two parts: the <strong><em>Core </em></strong>and the <strong><em>Template</em></strong>.</p><p><strong>Core responses</strong> directly answer or otherwise address the customer’s query and essentially has a causal relationship with it. The responses involve a diverse set of products and reasons.</p><p><strong>Template responses</strong> express gratitude and provide contact information for further communication. These remarks are independent of the nature of the customer letter, and hence they have very little diversity across the dataset. We postulate that expansion of this set will add useful diversity to the response letter.</p><p>Given this categorization (of <em>Core </em>vs. <em>Template</em>), we split each response in our dataset into these two categories with a view to treating them separately. To locate the splitting position, we consider the start of the latest sentence that contains the frequently used template phrases (e.g. bi-grams or trigrams). After this separation, each dataset is randomly divided into training (~60%), validation (~20%) and test (~20%) sets.</p><h4>Design the workflow</h4><p>We next present an end-to-end framework for generating context-aware formal responses to customer letters. As discussed earlier, a formal response letter consists of a core response and closing remarks. To handle these tasks separately, we split our framework into 2 modules: (1) Core Response Generator, and (2) Closing Remarks Diversifier.</p><p>A conditional text generation model is trained for the core response generator. For closing remarks, we create additional templates from the existing ones, by fine-tuning a separate language model for diversifying the responses through the use of paraphrasing. Following this, a closing remark is selected from the pool of expanded ending templates and concatenated to the generated core response to form a formal response letter.</p><h4>Core Response Generation</h4><p>In this section, we first describe a potential <strong>baseline</strong> approach for response generation and then put forward the retrieval guided response generation framework. In both approaches, we use a pre-trained causal language model, <a href="https://openai.com/blog/better-language-models/">GPT-2</a> [1] to train our baseline generation model. GPT-2 is largely built upon the decoder block of the initial transformer architecture [2] and employs a stack of masked self-attention layers where each token is allowed to attend to tokens from the past (left context) but is blocked from upcoming (right context) ones. The auto-regressive nature of the model makes it the preferred choice for conditional text generation. Furthermore, because it has been pre-trained on a massive web-text dataset, the language model is known to produce fluent responses when given a prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/0*RbzkDqKRzvapr877" /><figcaption>Figure 2: Core response generation steps</figcaption></figure><p><strong>Baseline generation models</strong></p><p>We approached our goal of producing a response for a customer query as a conditional text generation task and hence adopted the pre-trained GPT-2 model to tune the parameters. Given a customer query letter we want to generate a response that is conditioned on the query.</p><h4>History Guided Generation</h4><p>In the aforementioned approach, the model lacks access to factual information while responding to a query and tends to make up a safe or hallucinated reply. For instance, in response to a customer’s question regarding a restaurant’s service availability, the baseline model is seen to generate “don’t know” or “open” although the dataset indicates its closure.</p><p>To address this issue, in this task, a <a href="https://aclanthology.org/W18-5713/"><strong><em>Retrieve and Refine</em></strong></a><strong><em> (RetRef)</em></strong> [3] mechanism is employed. The idea is to retrieve valid responses for similar queries used in the recent past and utilize those responses, in addition to the query, to generate a refined (coherent) response.</p><p>We split the whole task into three steps: 1. Knowledge Retrieval, 2. Response Generation and 3. Response Ranking. The framework is depicted in Figure 3 and detailed in the following subsections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FAWavFXMdJsFQ7YP" /><figcaption>Figure 3: Retrieval-guided core response generation framework</figcaption></figure><p><strong>1. Search for similar historical cases</strong></p><p>Given a current customer query (q𝒸), knowledge can be extracted from agents’ past responses (rₚ) to a similar query (qₚ). To find such cases, we assign a candidate score, c=sim(q𝒸, qₚ)+bleu(q𝒸, qₚ) to each of historical query-response pairs where bleu is BLEU-1 score between the corresponding queries and sim(q𝒸, qₚ)=cos(Eᵩ𝒸,Eᵩ𝒸 ), is cosine similarity between the embedding of the current and of the past query, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/0*rtgsll6FGqVKbAyk" /><figcaption>Figure 4: Retrieval steps</figcaption></figure><p>The embeddings are obtained using a sentence-transformer and can be pre-computed to make the retrieval fast. While training, the similarity between the corresponding responses, sim(r𝒸, rₚ) is also added with the weight 𝛾&lt;1 to ensure that the model finds a relation between them to transfer knowledge. But this response similarity is not used during testing, as the reference response is unknown then. Furthermore, for training, we choose all the responses from candidate pairs that have c&gt;𝜏, where 𝜏 is a hyperparameter. More potential candidate responses essentially augment the training instances and we explain its use in the next section.</p><p><strong>2. Generate the response using historical cases as the reference</strong></p><p>We adopt the same GPT-2 used in our baseline model for retrieval-based training and generation. However, as input, a retrieved response is prepended to the current query. The objective is to teach the model to generate the reference response utilizing the retrieved knowledge in addition to the query.</p><p>Since only one retrieved response is used at a time, having more than one above the threshold (c&gt;𝜏) allows us to create more training instances with the same query-reference response pair. Conversely, in the absence of suitable retrievals, the reference itself is used as a retrieved response to make the model mirror the fetched response. This technique essentially works like <strong>teacher forcing</strong> and is intended to avoid ignoring the retrievals. However, in the event of such a scenario during testing, we resort to the <strong><em>baseline</em> </strong>model for generation and call this mix-model setting a <strong><em>hybrid </em></strong>generation.</p><p>Moving on to the prompt formation, it has three segments: the retrieved responses from the agent, the current query from the customer, followed by the reference response from the agent. Despite separating them by special token, a model will have little idea of the author of a token. To address this, we add a segment embedding to the token embedding as well. The way positional embedding helps the model understand the relative position of the tokens, a segment embedding of the corresponding author is likely to add more meaning to the model.</p><p><strong>3. Response Ranking</strong></p><p>Although transformer-based models produce better contextual representation by capturing long-term dependencies of input text, much depends on the decoding technique at the inference stage when it comes to generating context-aware, informative responses. Unfortunately, even with state-of-the-art decoding mechanisms, neural text generation is known to suffer from blandness or inconsistency. Hence, we generate multiple responses using different sampling methods (e.g. top-k, nucleus, etc.) and employ a ranking mechanism to measure the context-awareness of generated responses (by evaluating them as hypotheses of the source query).</p><p>Such a hypothesis would indicate stronger correspondence when it’s probability of producing the query, i.e. <em>p(query|hypothesis)</em> is higher and loss is low. However, after we built the “backward” model, we compared the original model loss (loss(h,q𝒸)) with the inverse model loss for each query (q𝒸)-hypothesis(h) pair and (interestingly) found a very high correlation. Thus we were able to avoid training an additional inverse model and performed the ranking process using forward loss only.</p><p>The final rank score of a hypothesis is computed using the formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/0*mCBg3O4aGJAgPOWt" /><figcaption>Figure 5: Explanation of the response ranking formula</figcaption></figure><p>Here, sim(h,q𝒸) indicates similarity between query and hypothesis. sim(q𝒸,qₚ)*sim(h,rₚ) takes into account the correspondence between retrieved response(rₚ) and hypothesis(h) weighted by the query similarity (sim(q𝒸,qₚ)). The rationale behind the last product is: a generation that retains knowledge from a good retrieval is likely to offer better response. Consequently, a higher rank score is expected to indicate better hypothesis quality.</p><h4><em>Hybrid Model</em></h4><p>Now not all queries have a usable similar query in the past and thus it becomes difficult for the model to handle all the situations using the past response. Instead we can use a hybrid model where we would use past history if found with certain confidence (similarity score), otherwise, we would use the structured mode of the model for generation.</p><h4>Template generation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/0*7M5Gjob-9ZBeq2ac" /><figcaption>Figure 6: Paraphrasing Model for closing remarks</figcaption></figure><p>One way we can enhance the ending template set is by paraphrasing them to produce new ones. To this end, a GPT-2 model is fine-tuned with ParaNMT-5M [4], a dataset of more than 50 million English-English sentential paraphrase pairs. We then utilize the trained paraphrasing model to paraphrase the currently available ending templates. From the paraphrased ones, we first filter out those that have a grammatical acceptability score less than 0.9 according to a BERT-based classification model trained with CoLA dataset [5]. To ensure diversity without sacrificing semantics, we further extract those that have BLEU-1 &lt; 0.7 and cosine similarity &gt; 0.8. Finally, we check if a paraphrased version retains all the delexicalized tokens such as email address, phone number, etc.</p><h3>Coming up next</h3><p>In the second part of the article, we will focus on the evaluation of the solution, in a real-world industry customer service setting. We will discuss the results on both standard evaluation metrics as well as human evaluations.</p><h3>References</h3><p>[1] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., &amp; Sutskever, I. (2019). Language models are unsupervised multitask learners. <em>OpenAI blog</em>, <em>1</em>(8), 9.</p><p>[2] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. “Attention is all you need.” <em>Advances in neural information processing systems</em> 30 (2017).</p><p>[3] Weston, Jason, Emily Dinan, and Alexander H. Miller. Retrieve and refine: Improved sequence generation models for dialogue. <em>arXiv preprint arXiv:1808.04776</em> (2018).</p><p>[4] John Wieting and Kevin Gimpel. 2018. <a href="https://aclanthology.org/P18-1042">ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations</a>. In <em>Proceedings of ACL</em>, pages 451–462.</p><p>[5] Warstadt, Alex, Amanpreet Singh, and Samuel R. Bowman. “Neural network acceptability judgments.” <em>Transactions of the Association for Computational Linguistics</em> 7 (2019): 625–641.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80c1114ca98e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/socialbakers-engineering/text-generation-in-customer-service-1-80c1114ca98e">Text Generation in Customer Service (Part 1)</a> was originally published in <a href="https://medium.com/socialbakers-engineering">Emplifi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trump versus Biden: what Twitter data reveals about the supporters of presidential candidates]]></title>
            <link>https://medium.com/socialbakers-engineering/trump-versus-biden-what-twitter-data-reveals-about-the-supporters-of-presidential-candidates-c1ff8cebebf1?source=rss----881fbb2b4996---4</link>
            <guid isPermaLink="false">https://medium.com/p/c1ff8cebebf1</guid>
            <category><![CDATA[k-means-clustering]]></category>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Jitka Baslova]]></dc:creator>
            <pubDate>Mon, 27 Jul 2020 07:51:01 GMT</pubDate>
            <atom:updated>2020-07-28T11:31:33.900Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Biden supporters are twice as likely to follow Starbucks or Netflix while Trump&#39;s crowd rather takes a sip of Black Rifle Coffee. What can we tell about the supporters of each candidate based on Twitter data?</strong></h4><p>We explored Twitter users who actively engaged and expressed their opinion in the Twitter debate with the main US presidential candidates and discovered affinities of different opinion groups and their behavior on Twitter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vbAAIheq7AF5Sk-D1S-SCg.png" /></figure><h3>Methodology and sample</h3><p>We’ve looked into US Twitter profiles that <strong>most recently mentioned Donald Trump or Joe Biden</strong> in their Tweets, and on whose mentions the <a href="https://www.socialbakers.com/"><strong>Socialbakers</strong></a><strong> </strong><a href="https://medium.com/socialbakers-engineering/sentiment-analysis-at-socialbakers-ca10f567eed9"><strong>algorithms detected positive or negative sentiment</strong></a><strong>.</strong></p><p>When gathering the sample data, we excluded mentions that contained more than one @handle, spam messages, messages with undetected sentiment, profiles of media or brands, and profiles that didn’t have stated US location. Profiles following less than 20 or more than 5,000 profiles were excluded too.</p><p>After clearing the data, we got 15k + 15k profiles mentioning each candidate. and narrowed the sample to 20k random unique Twitter profiles. We <strong>gathered lists of all profiles they follow</strong> (Twitter API calls them “friends”)<em> </em>and linked these lists to the <a href="https://www.socialbakers.com/"><strong>Socialbakers</strong></a> database. The <strong>database includes inhouse data enrichments, </strong>such as profile categories, and it allowed us to look at the sample data in a broader perspective.</p><p>To get the groups of supporters of each candidate, we used a common clustering k-means algorithm with the help of tf-idf and t-SNE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*1T5E-9_lhKSNvmFJ" /><figcaption>t-SNE visualization of users who mentioned one of the candidates</figcaption></figure><p>First, we created a matrix of all profiles (~ 1.5M) found in the gathered lists of friends and applied <strong>binary tf-idf </strong>to detect distinguishing profiles across the sample.</p><p>Due to computing reasons, we ignored profiles followed by more than 50% of and less than 2.5% twitter accounts in our user sample. In the end, only 3,047 profiles from the lists of friends remained and we worked with those.</p><p>To see if there are any clusters in the data, <strong>t-SNE </strong>was applied. It revealed that while there were user dots clearly grouped into a few clusters, there were also dots spread across the visualization.</p><p>Subsequently, we performed<strong> k-means</strong> on the tf-idf matrix and colored the t-SNE output by the k-means cluster labels. It showed that the k-means provided sufficient results, but we might need to run it multiple times on the biggest cluster (spread dots) to get more complex results.</p><h3>Interests and affinities behind the dots</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*8uXGeuNKAb1QPw6Hwdkfgg.png" /><figcaption>t-SNE visualization colored according to k-means labels</figcaption></figure><p>The clustering algorithm revealed 3 clusters. <strong>Two of them clearly defined strong supporters of each candidate<em> </em></strong>(green for Trump, blue for Biden). The third one (pink) at first didn&#39;t seem to incline to any of the candidates even though the dots appeared to be located closer to the Biden’s cluster.</p><p>However, we further explored the pink cluster and found <strong>Bernie Sanders’ supporters</strong>, <strong>Democratic-leaning profiles with weaker affiliation to Joe Biden and other Democrats,</strong> and a group of profiles that didn&#39;t show any particular political affinities nor strong interest in politics.</p><p>Interestingly, k-means showed that the sentiment of mentions themselves wasn’t a very good indicator for determining each candidate’s supporters. Overall, the conversation is rather negative on both sides <em>(neutral mentions were excluded during the sample collection)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zbTpDFytYAKd6pSfxmJ1Mw.png" /><figcaption>The mentions were gathered during the period between March 1 and May 29 and arranged in descending order, prioritizing recent ones.</figcaption></figure><h4>Trump&#39;s vs. Biden&#39;s supporters</h4><p>Both Trump&#39;s and Biden&#39;s core supporters show a stronger interest in society than the other profiles in the sample. Trump’s base has a weaker affinity towards celebrities and stronger towards media and community pages<em> </em>(community category include, among others, unofficial pages of political parties, religion pages).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lDQuqYR6ibz9w2xl" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D_9J1jXVLNSya8Vi" /><figcaption>Both groups are highly interested in politics on Twitter. While Biden’s fans tend to follow more NGOs, Trump’s fans follow more government-owned profiles.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CXMS_MV-CIos7l03" /><figcaption>Biden’s base in general follows more writers<em> (the category includes journalists)</em> and actors. Trump&#39;s fans are following rather sports stars, singers, and broadcast stars.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lUZQEByPGgW1mUsWVmnXEw.png" /><figcaption>Both Trump&#39;s and Biden&#39;s followers are way more active on Twitter than users with less strong political affiliation. They both follow and are followed by more profiles and publish more tweets. However, Trump&#39;s fans are significantly newer to Twitter; they joined approximately 4.5 years ago, which is roughly the time of the first Trump&#39;s campaign. All values in the chart are medians.</figcaption></figure><p>After analyzing words in profile descriptions of each group, we knew that the clusters contained strong supporters of each candidate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*69-mf0INHhCpe3AGZjSwUw.png" /><figcaption>Biden’s supporters (on the left) include in their description an anti-Trump hashtag “resist” and other pro-Democratic keywords, such as bluewave, liberal, biden, voteblue. The “biden” keyword is a bit overshadowed by the “trump” keyword — the cause could be that these profiles position themselves more as “anti-Trump” rather than pro-Biden. Trump’s supporters describe themselves as conservative, patriots and often include #MAGA or #KAG.</figcaption></figure><h4>Profiles dividing the groups the most</h4><p>Trump’s supporters show clear interest in the current administrative (@whitehouse), while Biden’s supporters still follow profiles of Obama’s administration (@vp44, @obamawhitehouse) and, for instance, are more interested in WHO, which was criticized by Trump during the COVID-19 pandemic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hYRV3rbAadXjRnuM" /></figure><p>Interestingly, there are more Joe Biden’s supporters following Adam Schiff or Barack Obama than the actual Joe Biden&#39;s profile. Trump’s supporters show a stronger affinity to their candidate, nearly 97.5% of them follow the @realdonaldtrump account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vnSb_MQQCzGTvqmPexKuUQ.png" /></figure><p>The clustering algorithm revealed not only differences among the profile topics, political parties and politicians the supporters usually follow, but also proved that their diverse opinions are reinforced by highly different lists of specific media outlets from which the supporters gather information. Their sources include TV shows, celebrities, or even NGOs, with only a minor flow of information coming in from the opposite side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VcYOrUyg3gi4S0KwHm8DmA.png" /><figcaption>The biggest differences in the media followed by the two groups are present in Trump promoting One America News and Breitbart News, while Joe Biden’s supporters follow left-leaning news outlets, such as MSNBC, The Washington Post or The New York Times more often.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*72r2iCEpzRsR1RIHpR5Zew.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OeZh25XmugN7dBmCBehhnA.png" /></figure><p>When it comes to NGOs, celebrities, and other segments, the political polarization projects into the differences shown in the following chart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Cpc7x1WFFDf2yJmc2eAdQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1CGQk6FSR3-rtoNXXq7FKA.png" /></figure><p>There are some differences among sports teams and organizations the supporters follow, even though their affinity towards sports isn’t particularly strong. Biden’s supporters follow <strong>women’s soccer,</strong> while Trump’s are more likely to follow <strong>NASCAR</strong> or, not surprisingly as Texas is traditionally a Republican-leaning state, <strong>Dallas Cowboys.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ABmsfidYoOQ-Ywvi" /></figure><p>Although brands are not commonly followed by any of the analyzed groups, there are still some tendencies we can see.</p><p>For example, Biden&#39;s supporters are twice as likely to follow <strong>Starbucks</strong> or <strong>Netflix</strong>. Only about 1% of Trump&#39;s followers in the sample follow <strong>Nike,</strong> compared to almost 5% of Biden&#39;s supporters, even though Trump&#39;s fans seem to be slightly more interested in sports (as seen previously). This may be the effect of the 2019 dispute in which Trump urged to boycott the brand. Not surprisingly, the most followed brand among Trump supporters is <strong>The Trump Organization</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SElJmBuFm3cVSqVX" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GxKLZb9Ompq7qRfW" /></figure><h4>Less opinionated profiles</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*oIKonfE2PRpeRdo7n93TVg.png" /></figure><p>As stated at the beginning, we also explored the pink blurry cluster of the initial k-means result as the visualization was suggesting there might be at least one clear cluster among the people who didn’t show any strong affiliation with Trump nor Biden.</p><p>We performed a second round of k-means on the remaining profiles which led to the discovery of supporters of the former<strong> Democratic candidate Bernie Sanders</strong> (red).</p><p>The blue cluster gathers <strong>Democratic-leaning voters,</strong> and the yellow cluster groups people who are generally <strong>not interested in politics</strong> on Twitter (and they prefer sports and celebrities) and don’t show clear affiliation with any party.</p><p>When it comes to the activity of Twitter users who are not Trump’s or Biden’s supporters, the least active is the yellow, “Unknown Preference cluster” (if we can call the spread dots a cluster). On the opposite, Bernie Sanders’ supporters turned out to be the most active ones, although they don’t reach the numbers of Trump’s nor Biden’s supporters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ipH7elRGefSq7g_dSyC2pQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/0*3MPZOd6GqeqMho_J" /><figcaption>The word cloud made of words in the profile descriptions of Sanders’ supporters confirms the affinity towards Sanders. Bernie, socialist, progressive, or #NotMeUs are among the most distinguishing keywords<em> (TF-IDF method)</em>.</figcaption></figure><p>As seen from the clustering visualization, the two remaining clusters are a bit blurry. Even though we later discovered some distinguishing features, the word clouds don’t provide much information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fPlzomo1dLqBU0Z4RMZVhg.png" /><figcaption>On the left we see words from the Democratic-leaning voters cluster while on the right there are keywords coming from people who don’t seem to be interested in politics as much as other profiles in the sample.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_gvvn_tchqW7Zn2z" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bhKhX56snarsl2lk" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UmqACsfFsfFWiubk" /></figure><p>Among <strong>Bernie Sanders</strong>’<strong> followers,</strong> we don’t see a strong affiliation with the winning Democratic candidate Joe Biden. However, among the 10 most followed politicians we can see another candidate, Elizabeth Warren.</p><p>It’s interesting that among Sanders’ supporters the most followed politician is Alexandria Ocasio Cortez, a strong supporter of Sanders, but not Sanders himself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8Anatdk6jZGV705r" /></figure><p>On the other hand, among the <strong>Democratic-leaning profiles,</strong> we see Joe Biden on top together with other famous Democrats. The list gathers current or former Democratic presidential candidates. However, the affinity towards Democrats is not to be taken for granted as the 10th most followed politician is Donald Trump.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3JtdY2pMff8xogjR" /></figure><p>The last political ranking shows people who can’t be affiliated to any of the political parties nor candidates. The visualization of the clusters showed that the remaining, yellow profiles are spread across the chart, meaning they don’t show many common features.</p><p>Their overall interest in politicians is weaker (smaller percentages) than among other clusters. When looking at the politicians they follow, the top two are Donald Trump and Barack Obama. As they are overall one of the most followed Twitter profiles, we can’t draw any conclusions based on that.</p><p>However, compared to the Democratic-leaning profiles, we can spot that the difference between the percentage of users following Donald Trump and Barack Obama is radically smaller here and in favor of Trump.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t3NFC8OZDZuSuRyx" /></figure><h3>Summary</h3><p>Based on public data from Twitter we defined two main groups of users, supporters Donald Trump and supporters of Joe Biden. We analyzed the data and discovered that Trump&#39;s supporters show a strong affinity towards their candidate as almost all of them follow the @realDonaldTrump Twitter profile and many follow profiles of the cabinet. The most popular brand among Trump&#39;s supporters is The Trump Organisation.</p><p>Supporters of Joe Biden are slightly less connected to their candidate on Twitter as 85.25% of them follow @joebiden, but actually even more of them follow Barack Obama (93.24%) or Adam Schiff (86.28%). Yet, we also discovered some other potential voters of the Democratic candidate who haven&#39;t shown that strong connection to Joe Biden.</p><p>Based on the data we worked with, crucial for the election results might be supporters of Bernie Sanders, who haven’t found a way to either of the nominees quite yet. Only 17.85% of Bernie Sanders supports follow Joe Biden, while 27.55% follow Donald Trump.</p><h4>Disclaimer</h4><p>All the data we worked with (mentions of the candidates and information about the profiles who post them) is publicly available via Twitter API. We didn&#39;t use any personal information and we didn&#39;t provide the list of profiles to any third party.</p><p>Sounds interesting? Check out more of our <a href="https://medium.com/socialbakers-engineering">stories</a> &amp; don’t forget <a href="https://www.socialbakers.com/careers/open-positions?utm_source=mediumblog&amp;utm_medium=.&amp;utm_campaign=Trump%20vs.%20Biden">we’re hiring</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1ff8cebebf1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/socialbakers-engineering/trump-versus-biden-what-twitter-data-reveals-about-the-supporters-of-presidential-candidates-c1ff8cebebf1">Trump versus Biden: what Twitter data reveals about the supporters of presidential candidates</a> was originally published in <a href="https://medium.com/socialbakers-engineering">Emplifi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sentiment Analysis at Socialbakers]]></title>
            <link>https://medium.com/socialbakers-engineering/sentiment-analysis-at-socialbakers-c98cf7c908e?source=rss----881fbb2b4996---4</link>
            <guid isPermaLink="false">https://medium.com/p/c98cf7c908e</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jan Rus]]></dc:creator>
            <pubDate>Mon, 23 Mar 2020 09:56:10 GMT</pubDate>
            <atom:updated>2020-03-23T15:39:12.150Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mQqnGJj0iKQaD6SpFeiARA.png" /></figure><p>Our Innovations team at <a href="https://www.socialbakers.com/">Socialbakers</a> was working on a very interesting task last year, <em>sentiment analysis</em> for social media. In the context of our work, sentiment analysis is an automated process of text analysis leading to a prediction of how an audience feels towards any given subject from texts of social media messages.</p><p>The result of this process is a <em>classification</em> of the texts into 3 classes: positive, neutral and negative.</p><p>Such text analysis can be rented as a service these days, but our goal was a system that can analyze 150,000,000 messages in 5–10 different languages every day. We researched this option and found out that it would be an extremely expensive service to rent even from the cheapest providers on the market in 2019.</p><p>Moreover, most text analysis tools and services are not fine-tuned to social media and don’t support all languages we are interested in such as Czech or Arabic.</p><p>So, we decided to build the solution in-house. We shared the architecture of our deep learning solution (recurrent neural network) and the necessary theoretical background in the previous post: <a href="https://medium.com/socialbakers-engineering/sentiment-analysis-at-socialbakers-ca10f567eed9">Theoretical Introduction and Architecture</a>¹.</p><p>But as we wrote at the end of the blog post, we think that the largest building block of any successful deep learning project is the data. If the neural network is a student then the dataset is a textbook.</p><h3>Data for Deep Learning POC</h3><p>In the case of courses, trainings or competitions deep learning practitioners often start their work with given training and test datasets. They can invest their effort into algorithms and tuning of their parameters from the very beginning.</p><blockquote><strong>In practice, the work starts with getting the dataset first.</strong></blockquote><p>Ideally, research teams can find an open-source dataset that is large, high quality and matches the goal (including domain) of the deep learning project but it is a very rare situation. The research teams usually have three options: 1) to build the perfect manually annotated dataset from scratch in-house which is an expensive and time-consuming process, 2) to find and combine the right but smaller datasets from different sources with lower and varying quality that at least partially overlap with the goal of the project or 3) to find a way how to create a dataset with labels derived from a different but related information. The 2) and 3) may lead to larger but noisy datasets or even a necessity of <em>transfer learning</em>².</p><h4>The First Dataset</h4><p>After initial research, we realized that it will be hard to find publicly available datasets with texts and sentiment labels that would fit our needs. Datasets we found were:</p><ul><li><strong>small</strong>, having only 500–1,000 samples is currently unsuitable for such deep learning project,</li><li><strong>using only 2 classes</strong> (positive and negative), while we need 3 classes,</li><li><strong>from a different domain</strong>, for example, long movie reviews,</li><li><strong>having wrong properties</strong> such as too long texts missing slang or emojis,</li><li><strong>licensed</strong> in a way that prevents us from using them even for experiments.</li></ul><p>Simply, nothing close to a large dataset of short social media messages in a number of different languages and sentiment labels with 3 classes. We, therefore, started to evaluate the more laborious options:</p><ul><li><strong>Amazon Mechanical Turk</strong>: unnecessarily expensive and without decent control over the quality of the annotations. We would need multiple annotations per text to keep the quality high and so we estimated the cost to at least $15,000 just for the English training set of 200,000–300,000 texts annotated by 3–5 annotators.</li><li><strong>Client sentiment labels</strong>: it is possible to label sentiment in our Socialbakers Suite manually, so we have a large number of such labels in our databases. Here, we worried (and experimentally confirmed) the quality of annotations would suffer as well because the clients are not trained to do the job, they are also biased and they also use the labels for different purposes in some cases.</li><li><strong>Customer product reviews</strong>: these datasets are mostly pairs of textual reviews and up to 5 stars assigned to the reviewed product, for example, <em>Amazon reviews</em>³. The number of stars can be used as a noisy sentiment label (the more starts the more positive sentiment). We researched a few such datasets and concluded that a) the number of stars doesn’t correlate well with the sentiment of the textual review and b) a customer review sentiment is different from the common conversation sentiment. Consider that the typical “positive” phrases in these datasets are phrases like <em>low price</em>, <em>quick delivery</em> or <em>100% leather</em>. The last argument against such datasets is that they are mostly not available for commercial use.</li></ul><p>We experimented with even more options that didn&#39;t work for us at the end, but the last option in the list above was closest to our idea of how we may get the first data for our deep learning proof of concept.</p><p>We intended to use a widespread noisy (not always correct) labels that could be quickly and automatically gathered on a large scale and correlated with the sentiment in the related texts.</p><p>Social media messages are full of emojis and we decided to use the emojis as the noisy automatically gathered sentiment labels. We focused on English first, because it is a very frequent language with a higher chance to find some available datasets and competitive algorithms for benchmarking. But in general, this approach would enable us to build datasets for any language.</p><p>Our approach was to download a sample of Facebook messages written in English (we used PyCLD⁴ for language detection) and for each message:</p><ul><li>extract all emojis,</li><li>map each emoji to a sentiment class,</li><li>use the most frequent sentiment class as a label for the original message.</li></ul><p>If there was no emoji found in the message or if there wasn’t a single most frequent sentiment class assigned to the emojis (e.g. the same amount of positive and neutral emojis appeared in the message), the message was removed from the dataset, see example in image 1 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*694QNpje5ySrvdMwqspIQg.png" /><figcaption>Image 1: Mapping emojis to sentiment label.</figcaption></figure><p>The mapping between emojis and sentiment was made in-house but it is a very small amount of work in comparison to the manual annotation of all the downloaded messages. With this approach, we quickly prepared a balanced dataset of 38,000 social media messages written in English. This was our source for the first training and validation datasets.</p><p>Next, we created a small balanced dataset consisting of 500 Facebook messages (not all of them contained emojis) which were labeled manually by our Innovations team. This was our first test dataset. The reason to choose this approach was that we knew that we are not able to build a large<br>hand-labeled dataset in a short time but we are able to quickly build at least a test set this way and have a well-defined target to aim on.</p><p><em>The messages we use in this project are publicly available user comments provided by Facebook and other social networks via their APIs. They are anonymized.</em></p><h4>Performance Benchmark</h4><p>To be able to measure our progress during the project, we needed to set up an accuracy benchmark. We had no KPIs in place at this phase but we used a well-known sentiment analysis library called <em>VADER</em>⁵ as a benchmark. The VADER is a very lightweight model based on the valence lexicon and a set of linguistic rules, it was the cheapest solution and also the fastest to deliver we were aware of. A tempting solution for quick prototyping in the commercial sphere. The only (and significant) drawback is the dictionary itself that exists only for English and we would need an expert in linguistics to be able to extend it to other languages. We wanted to overcome this drawback and prove that a machine learning solution will be not just easier to extend and maintain but also better performing on our use case. Overall, a better approach in the long term. The VADER became the lower threshold for the benchmark.</p><p>Next, we chose the <em>inter-annotator agreement</em>⁶ to be the upper threshold for the benchmark. If you ask a group of people to label the sentiment of the given set of texts, you will find out that there will be some examples where they will not agree on the sentiment, i.e. they will not agree in 100% cases. It would be therefore naive to expect that a machine learning algorithm would achieve better results. It has been experimentally proven that the typical inter-annotator agreement in sentiment analysis is approximately 80%⁶. This means that if an algorithm reaches 80% accuracy, it has approximately a human-level performance.</p><p>Last, we picked 2 services (APIs) providing sentiment analysis as a reference: <em>MeaningCloud</em> and <em>Amazon Comprehend</em>. We analyzed the first test set with these two services and the VADER, measured their accuracy and constructed the benchmark. See chart 1 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KSsfkGY8mDX6u7bzG6agig.png" /><figcaption>Chart 1: Accuracy benchmark for our POC.</figcaption></figure><h4>The First Experiment</h4><p>During the first series of experiments, we tuned, trained and validated our neural network on the dataset of 38,000 emoji-labeled Facebook messages. Then, we run a test against the 500 Facebook messages labeled by our Innovations team.</p><blockquote><strong>We failed really badly.</strong></blockquote><p>We achieved accuracy better than random (which is 33%) but much worse than VADER which we intended to beat, see the accuracy measurements in table 1 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-tLcxEP6XR68LUF9muDvrg.png" /><figcaption>Table 1: Accuracy measurements for a model trained on the emoji-labeled dataset.</figcaption></figure><p>We suspected the problem is in the training dataset immediately. VADER was working worse on the emoji-labeled dataset than on the manually labeled dataset. On the other hand, the neural network learned to work quite well on the emoji-labeled dataset and then totally failed on the manually labeled dataset. So the neural network can learn to predict emojis from a text but it doesn’t help it to predict sentiment from a text. It’s not that surprising in retrospective.</p><p>In contrast to for example stars and customer reviews, the emojis are an integral part of the messages. If we remove the emojis from the text and use them as a sentiment label, we may completely change the meaning of the message (especially if it’s a short message) and build a very noisy labeled dataset. If we keep the emoji in the message, the network will likely learn to predict the sentiment mostly based on the emojis (remember the 96% accuracy on the validation set above). We experimented with both but the results on the test dataset were more or less the same.</p><h4>The Second Dataset and Experiment</h4><p>Due to the tight time constraints and lessons taught during the previous experiments, we avoided using any hints to a sentiment of a text hidden in the text itself (emojis, hashtags, etc.) in the future and built the next dataset with real sentiment labels exploiting VADER and two more third-party services. We managed to do this practically at no cost. We kept only the messages where the sentiment analysis algorithms agreed as sufficiently reliable for our training set (approximately 80% of cases). This way we obtained a balanced dataset of 147,000 English social media messages with sentiment labels.</p><p>We repeated the previous experiment with training and validating on one dataset and testing on the manually labeled dataset and we finally succeeded, see table 2 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pcnkVcSvsFblrY1Bl5UBCw.png" /><figcaption>Table 2: Accuracy measurements for a model trained on the automatically-labeled dataset.</figcaption></figure><p>The results may be a bit misleading because the test set is very small (only 500 messages) and the training/validation set is probably not very challenging (a subset of the original dataset where multiple algorithms agreed) but it was a good signal and reliable enough to get support from the Product Management team. Until now, it was just a proof of concept created by 2 team members in 3 weeks in parallel with other higher-priority projects.</p><h3>In-house Sentiment Dataset</h3><p>After a discussion of the results with the stakeholders, we decided to continue in the project, to invest effort into building the infrastructure and further experiments with our own machine learning solution. We also agreed that we can start cooperating with our internal tagging team on building a manually labeled dataset for this sentiment analysis project. The Taggers proved to be a tremendous help on many occasions since then. In the beginning, we wanted to use the created dataset only as a test set. But with a larger volume of data, we could start to mix a fraction of the dataset into training and validation sets and ideally end up with a dataset large enough to train, validate and test our models only on this dataset itself.</p><h4>Crash Course for Annotators</h4><p>The Taggers team underwent a crash course on the sentiment analysis and how to label the datasets. Then, we selected 5 of them, asked them to label a set of approximately 400 texts and calculated their inter-annotator agreement. They reached almost 90% agreement which is higher than the claimed typical inter-annotator agreement (80%) but it’s likely due to the fact that the Taggers are trained and they are trained to work as a team (a single annotator). Such a high agreement encouraged us to let the team to label each text of the target dataset by only one annotator (typically multiple untrained annotators label every single text to achieve a higher quality of the labels) and progress faster to the first milestone that was set to 100,000 labeled English Facebook messages.</p><p>Once the team produced a dataset of 15,000 labeled texts, we proceeded to a test of alignment to be able to identify and fix eventual small discrepancies in time. We used the previously trained model and predicted sentiment for the original small dataset of 500 Facebook messages labeled by our Innovations team. Next, we used the same model to predict the sentiment for the larger new (unbalanced) dataset of 15,000 Facebook messages labeled by the team of annotators and compared the achieved accuracies.</p><blockquote><strong>We failed badly again.</strong></blockquote><p>We expected the accuracy will change with a larger and more robust test dataset but we didn’t expect it to drop by 14%, from 75% to <strong>61%</strong> failing to beat the VADER again.</p><h4>What is Sentiment</h4><p>Such a suspiciously big drop in performance motivated us to start researching the cause immediately. Luckily, we monitored which annotator labeled which text. We filtered the messages where labels created by the Taggers team were different from predictions of the model and looked for outliers: taggers with a significantly worse agreement with the model than others. We didn’t find them. Not just the amount, but also the distribution of positive, neutral and negative labels were similar for all annotators. It looked like a systematic error. The team of annotators (true labels) tends to a wider range of what is neutral than the model (predictions), see chart 2 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vIlZcPrba6NpegxFYGHTfA.png" /><figcaption>Chart 2: Distribution of true labels and predictions with the old labeling manual.</figcaption></figure><p>We decided to relabel a small subset of these different messages only in the Innovations team and we inclined (also systematically) to the results of the model, not the Taggers team. In fact, it was good news. Systematic errors are easier to uncover, understand and fix than random errors.</p><p>We organized another training session with the Taggers team, shared our findings and followed with a discussion about the possible cause of the systematic discrepancy between labels provided by the Innovations team, the model and the Taggers team. The outcome was quite surprising. <strong>Their high-level understanding of what is sentiment was different from ours.</strong> They understood sentiment as to <strong>how an author feels</strong> when writing the given text, for us it was <strong>how a reader feels</strong> when reading the given text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Y3oRTsrW68yNbgilfGi5g.png" /><figcaption>Image 2: This sample text was labeled as neutral by Taggers because it’s an ad (the author feels neutral) and positive by Innovations because we had a positive feeling while reading the text.</figcaption></figure><p>It’s a crucial principle yet something we didn’t think of before and didn’t cover with examples in the labeling manual. What the Taggers did wasn’t wrong. It was an interesting finding, another point of view and we needed to know which one is the right one. We prepared a questionnaire with a description of sentiment analysis, the issue we were facing and a set of 20 examples of texts with questionable sentiment labels. This document was shared with selected Socialbakers employees and clients and collected feedback uncovered that the <strong>how a reader feels</strong> point of view is the more desirable one. We updated the labeling manual with this enlightening discovery, ask Taggers to relabel affected part of the manually labeled dataset and then continue in pursuing the 100,000 dataset milestone.</p><p>Shortly thereafter, when the Taggers team reached 20,000 manually labeled texts, we repeated the alignment test and were happy with the result. You can see in the chart 3 below that the distribution of the predicted and labeled sentiment became better defined, the positive and negative classes were rarely confused with the neutral class and the (balanced and unbalanced) accuracy achieved by the model on this dataset raised to 77%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bvLzgUCc7_UN5_rZJiGUEQ.png" /><figcaption>Chart 3: Distribution of true labels and predictions with the new labeling manual.</figcaption></figure><p>With the growing and more robust test set, the accuracy of our solution slowly dropped to 70% after some time. We kept searching for and experimenting with new sources of sentiment labels and mixing them together to counter fight the drop in the performance. But once the team of annotators reached another milestone of 200,000 labeled messages, we started to use this dataset for training and validation as well and pushed the performance of our sentiment analysis model even further.</p><h3>Summary</h3><p>You can see that it is not a straightforward task to build a dataset for a machine learning project. Even for businesses. But having the (bug-fixed) labeling manual, a trained team of annotators and a large amount of data from the right domain allowed us to build a large high-quality dataset of English social media messages with sentiment labels.</p><p>This is the end of this episode. This is where many machine learning projects just begin.</p><p>Sounds interesting? Check out more of our <a href="https://medium.com/socialbakers-engineering">stories</a> &amp; don’t forget <a href="https://www.socialbakers.com/careers?utm_source=mediumblog&amp;utm_medium=.&amp;utm_campaign=Sentiment_Analysis_2">we’re hiring</a>!</p><h3>Notes and References</h3><p>[1]: Sentiment Analysis: Theoretical introduction and Architecture<br><a href="https://medium.com/socialbakers-engineering/sentiment-analysis-at-socialbakers-ca10f567eed9">https://medium.com/socialbakers-engineering/sentiment-analysis-at-socialbakers-ca10f567eed9</a></p><p>[2]: Transfer learning, exploiting a knowledge gained while solving a different but related problem, see a brief description here:<br><a href="https://en.wikipedia.org/wiki/Transfer_learning">https://en.wikipedia.org/wiki/Transfer_learning</a></p><p>[3]: Amazon Customer Reviews dataset<br><a href="https://s3.amazonaws.com/amazon-reviews-pds/readme.html">https://s3.amazonaws.com/amazon-reviews-pds/readme.html</a></p><p>[4]: PyCLD, Python bindings for the Compact Language Detect 2<br><a href="https://github.com/aboSamoor/pycld2">https://github.com/aboSamoor/pycld2</a></p><p>[5]: VADER, a lexicon and rule-based sentiment analysis tool<br><a href="https://github.com/cjhutto/vaderSentiment">https://github.com/cjhutto/vaderSentiment</a></p><p>[6]: How Companies Can Use Sentiment Analysis to Improve Their …, 2012<br><a href="https://mashable.com/2010/04/19/sentiment-analysis">https://mashable.com/2010/04/19/sentiment-analysis</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c98cf7c908e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/socialbakers-engineering/sentiment-analysis-at-socialbakers-c98cf7c908e">Sentiment Analysis at Socialbakers</a> was originally published in <a href="https://medium.com/socialbakers-engineering">Emplifi</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>