<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeff Carpenter on Medium]]></title>
        <description><![CDATA[Stories by Jeff Carpenter on Medium]]></description>
        <link>https://medium.com/@jscarp?source=rss-343198ddcf60------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5AYcAO8y3l_MO2B_SnKMHQ.png</url>
            <title>Stories by Jeff Carpenter on Medium</title>
            <link>https://medium.com/@jscarp?source=rss-343198ddcf60------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:23:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jscarp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My next book: Working with Cloud-Native Data on Kubernetes]]></title>
            <link>https://jscarp.medium.com/my-next-book-working-with-cloud-native-data-on-kubernetes-b613fb936a1?source=rss-343198ddcf60------2</link>
            <guid isPermaLink="false">https://medium.com/p/b613fb936a1</guid>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Jeff Carpenter]]></dc:creator>
            <pubDate>Wed, 19 May 2021 20:55:36 GMT</pubDate>
            <atom:updated>2021-05-19T20:55:36.091Z</atom:updated>
            <content:encoded><![CDATA[<p>Since the beginning of 2021, I’ve been super-focused on the idea of running databases on Kubernetes, and specifically Cassandra and the <a href="https://k8ssandra.io/">K8ssandra</a> open source project. Over the past few months, I’ve formed a strong opinion: we are now at a point as an industry at which stateful workloads are not just “ok” to run on Kubernetes, but we actually need to move aggressively toward running all our cloud workloads on Kubernetes. Standardizing on Kubernetes as a platform will be an accelerator, enabling us to worry less about infrastructure and more about new capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sac8A76xVxzWvZZhYRBoeQ.jpeg" /></figure><p>As you can probably guess from the headline, I’ve already doubled down on this position, but let’s take a step back to see how I got there. My journey can be traced in a few key articles which I’ll summarize below.</p><h4>Cassandra on Kubernetes? Skeptic to Convert</h4><p>Chris Bradford’s <a href="https://thenewstack.io/a-case-for-databases-on-kubernetes-from-a-former-skeptic/">A Case for Databases on Kubernetes from a Former Skeptic</a> is a great description of his evolving perspective on running databases first in containers, and then orchestrating those containers in Kubernetes. Over time, his point of view changed from “no way!” to “yes, definitely!”</p><p>This mirrors my own path: when I wrote the 2nd edition of <a href="https://learning.oreilly.com/library/view/cassandra-the-definitive/9781098115159/">Cassandra: The Definitive Guide</a> in 2015, I echoed the prevailing wisdom of the Cassandra community at the time regarding deploying Cassandra in Docker — “don’t try this in production”. By the time I was working on the 3rd edition in early 2020, I was ready to say “OK” to Cassandra in Docker, but the operators for managing clusters at scale in K8s weren’t yet mature.</p><h4>Combining efforts on a unified Cassandra Operator</h4><p>Speaking of operator maturity, the Cassandra community took a major step forward by taking the best ideas and code from several parallel Kubernetes operator projects, as Rahul Singh shared in his article for <a href="https://containerjournal.com/topics/cassandra-kubernetes-sig-picks-cass-operator-for-k8s/">Container Journal</a>. This unified effort was a precursor to more collaboration to come.</p><h4>Building a full production data layer</h4><p>As the operator development was maturing, John Sanda and others started talking about bringing together the best Cassandra tooling alongside <a href="https://github.com/k8ssandra/cass-operator">cass-operator</a> to produce a production-grade distribution of Cassandra known as <a href="https://k8ssandra.io/">K8ssandra</a>. At this point, I was fully committed, and helped explain the rationale for this project in my post <a href="https://k8ssandra.io/blog/articles/why-k8ssandra/">Why K8ssandra?</a></p><h4>Joining a community of communities</h4><p>Around this time my horizons began to broaden as I was exposed to a broader ecosystem of people and organizations working toward this common goal of stateful workloads on Kubernetes. I started participating in the <a href="https://dok.community/">Data on Kubernetes Community</a> and was pleasantly surprised to discover it had already been running for months. You can read about the vision for the DoKC in <a href="https://dok.community/blog/a-call-for-collaboration-announcing-the-data-on-kubernetes-community/">A Call for Collaboration: Announcing the Data On Kubernetes Community</a>.</p><h4>Defining cloud-native for databases</h4><p>The conversations in the DoKC got me thinking about what it meant for a database to be “cloud-native” — that is, to truly embody the characteristics required to maximize the advantages of the cloud. Further discussions with Cedrick Lunven led to a couple of posts. In the <a href="https://developereconomics.com/blog/the-search-for-a-cloud-native-database">The Search for a Cloud-Native Database</a> we proposed some definitions for cloud-native data concepts, and then built on those definitions in <a href="https://containerjournal.com/topics/a-maturity-model-for-cloud-native-databases/">A Maturity Model for Cloud-Native Databases</a> to discuss steps along the path to becoming fully cloud-native.</p><h4>A Line in the Sand</h4><p>Ultimately, the time came for me to admit I was “all in” on this idea of databases and other stateful workloads running on Kubernetes. I started noticing that many of the design patterns and principles embodied in K8s were really advantageous for databases seeking this “cloud nativeWhile the headline <a href="https://containerjournal.com/kubeconcnc/why-a-cloud-native-database-must-run-on-k8s/">Why a Cloud-Native Database Must Run on K8s</a> is maybe a <em>bit</em> stronger than I put it in the article, it’s not too much of an exaggeration.</p><h4>Doubling down on data on K8s</h4><p>Now, it’s time to put some weight behind my opinions and help push this conversation forward. I’m excited to share that Patrick McFadin and I have contracted with O’Reilly to write a new book which we’re calling “Managing Cloud-Native Data on Kubernetes”.</p><p>In this book, we plan to explore how the cloud commodities of compute, networking and storage can be combined to produce new cloud-native infrastructure for databases, streaming, and analytics, enabling new use cases that will drive the next decade of innovation.</p><h4>How you can help</h4><p>We are looking for input from the developers, operators, and data professionals that this book is for, so the <a href="https://github.com/data-on-k8s-book/outline">book outline</a> is available on GitHub so that you can see what will be included. Are we missing key topics? Just submit an issue or pull request.</p><p>We also intend to include several case studies in the book, with inputs from experts in a few key areas, so stay tuned as we develop those opportunities.</p><p>While the book isn’t scheduled to publish until next year, look for chapters to start appearing in the second half of 2021.</p><p>Thanks for reading this far, and for all your support of communities like Cassandra, K8ssandra and Data on Kubernetes. Take care!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b613fb936a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hands-on Learning for Cassandra]]></title>
            <link>https://jscarp.medium.com/hands-on-learning-for-cassandra-8e8ff38f6990?source=rss-343198ddcf60------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e8ff38f6990</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[learning-to-code]]></category>
            <category><![CDATA[katacoda]]></category>
            <category><![CDATA[cassandra]]></category>
            <dc:creator><![CDATA[Jeff Carpenter]]></dc:creator>
            <pubDate>Sat, 29 Aug 2020 22:54:37 GMT</pubDate>
            <atom:updated>2020-08-29T22:54:37.431Z</atom:updated>
            <content:encoded><![CDATA[<h4>Katacoda scenarios now available on O’Reilly Learning site</h4><p>It’s been almost 4 months since the release of <a href="https://learning.oreilly.com/library/view/cassandra-the-definitive/9781098115159/">Cassandra: The Definitive Guide, 3rd edition</a>. The response to the book has been great and I’m really grateful to my employer, DataStax, for sponsoring the book and making a <a href="https://www.datastax.com/resources/ebook/oreilly-cassandra-definitive-guide">digital copy available for free</a>.</p><p>I’m writing to share about a new companion project I’ve had the chance to create. If you’ve read the book, you’ve seen that it contains quite a few code samples in CQL, Java, and more. It’s always been my desire to make it simple for readers to leverage these code samples without having to type out each individual command. To this end, for the 2nd edition of the book, I put together some scripts and made them <a href="https://github.com/jeffreyscarpenter/cassandra-guide">available on GitHub</a>, but for the 3rd edition there’s an exciting new option available — <a href="http://katacoda.com/">Katacoda</a>. Here’s an example of what these scenarios look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ab8g4ESC_bW4etk96wL7rA.png" /><figcaption>The Get to Know CQL scenario teaches you basic syntax in a few quick steps</figcaption></figure><p>In all, there are 17 different scenarios that I’ve developed for the <a href="https://learning.oreilly.com/">O’Reilly Learning site</a>, grouped into four short courses:</p><ul><li>Getting Started with Cassandra — this string is based on Chapter 3, Installing Cassandra, and Chapter 4, The Cassandra Query Language</li><li>Data Modeling with Cassandra — This string is based on Chapter 5, Cassandra Data Modeling, and also brings in some of the advanced topics from Chapter 7, Application Development with Cassandra</li><li>Application Development with DataStax Java Driver — This string is based on Chapter 8, Application Development with Drivers</li><li>Apache Cassandra Integrations — This string is based on Chapter 15, Migrating and Integrating</li></ul><h4>The development process</h4><p>Writing the scenarios was a fun experience, as I really enjoyed the process of creating “learning as code”, carrying on the same toolset and process I shared about previously in my post <a href="https://medium.com/@jscarp/books-as-code-f988133ea41e">Books as Code</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g1NJ8yX_X53dlJleJ7sIeQ.png" /><figcaption>I write Katacoda scenarios with my go-to dev toolset — IntelliJ and Git</figcaption></figure><h4>How to find the scenarios</h4><p>The quickest way to access these scenarios within the site is to use <a href="https://learning.oreilly.com/playlists/746f36eb-60f8-4884-abbc-ef36b9305dc5/">this playlist</a> that walks you through the flow of the chapters and scenarios mentioned above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ZuSNx3CMsEOvGARYmLq2Q.png" /><figcaption>The playlist mixes text from the book with scenarios that allow you to execute the sample code</figcaption></figure><p>While the site does require a subscription, I’m happy to share that O’Reilly has made it possible for users to get a 30-day free trial that will allow you to experience these Cassandra scenarios and a wealth of other content that you’ll want to try out. Use <a href="https://learning.oreilly.com/get-learning/?code=CASSANDRA3E20">this link</a> to create your account (it will auto-populate the discount code CASSANDRA3E20).</p><p>Personally I’ve found learning.oreilly.com to be a super valuable collection of resources that I’ve made frequent use of in my day job, and as an O’Reilly author. As an example, check out this <a href="https://learning.oreilly.com/playlists/d883e7ad-3a6f-4ccf-b18f-62abda2ff6e6/">playlist on NoSQL Databases</a> that I helped curate.</p><h4>Upcoming hands-on workshops</h4><p>I should also note that I’ll be making use of the Katacoda scenarios in two upcoming workshops on O’Reilly Learning:</p><p><a href="https://learning.oreilly.com/live-training/courses/building-applications-with-apache-cassandra/0636920415954/">Building Applications with Apache Cassandra</a> (Sept 9)— I’ve been doing this course for about three years but have now had the opportunity to rework it so that we use Katacoda scenarios for course exercises.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WKAedf3d_FYlCycsUbRR6g.png" /></figure><p><a href="https://learning.oreilly.com/live-training/courses/building-data-pipelines-with-cassandra-kafka-and-spark/0636920416739/">Building Data Pipelines with Cassandra, Kafka, and Spark</a> (Sept 30)— this is a new course that was originally intended for Strata London this past Spring. I’m really excited about the opportunity to share this content via live training using Katacoda instead!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SSVlUHjzpKg1oYQQ2cCf8A.png" /></figure><p>If you’re reading this later and have already missed the workshops, just check the <a href="https://learning.oreilly.com/live-training/">O’Reilly Live Training page</a> for future dates.</p><h4>Innovation in learning at DataStax</h4><p>We’re also leveraging Katacoda at DataStax. Last month I had the privilege to <a href="https://www.datastax.com/blog/2020/07/learning-without-fear-datastaxcomdev">announce the release of our new developer portal</a>. <a href="https://datastax.com/dev">datastax.com/dev</a> features lots of hands on learning for Apache Cassandra and <a href="https://astra.datastax.com/">DataStax Astra</a>, our Cassandra-as-a-service offering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUt-6y8IY7btHPCyAtg-1g.png" /><figcaption>datastax.com/dev features tons of free, hands-on exercises you can do at your own pace</figcaption></figure><p>I’m really excited about the emergence of technologies like Katacoda that aid in creating learning experiences directly alongside code, and cloud IDEs like <a href="https://www.gitpod.io/">Gitpod</a> that lower the barriers to setting up a development environment. I’m also looking forward to the opportunity we have to combine these technologies with personal interaction — whether online, or hopefully soon in person — and promote a culture of learning together.</p><p>Please reach out and let me know what you think of these tools and other ideas you have for improving the learning experience. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e8ff38f6990" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Books as code]]></title>
            <link>https://jscarp.medium.com/books-as-code-f988133ea41e?source=rss-343198ddcf60------2</link>
            <guid isPermaLink="false">https://medium.com/p/f988133ea41e</guid>
            <category><![CDATA[intellij]]></category>
            <category><![CDATA[asciidoc]]></category>
            <category><![CDATA[writing]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Jeff Carpenter]]></dc:creator>
            <pubDate>Tue, 14 Jan 2020 03:42:13 GMT</pubDate>
            <atom:updated>2020-01-14T03:42:13.893Z</atom:updated>
            <content:encoded><![CDATA[<h4>My toolset for Cassandra: The Definitive Guide, 3rd Ed</h4><p>Last night I completed the first draft of the final chapter of the 3rd edition of <a href="http://shop.oreilly.com/product/0636920299837.do">Cassandra: The Definitive Guide</a> for O’Reilly (early access chapters available at the link). That might explain where all my writing energy has been invested lately :).</p><h3>Jeff Carpenter on Twitter</h3><p>Just pushed my last edits on the first draft of @Cassandra The Definitive Guide, 3rd edition. Lots of editing work to come but the hardest part is done!</p><p>Now that I get a short pause while my reviewers take aim at the text, I thought it was about time to get back to blogging. I received a question about the image in that tweet above which inspired me to write this post:</p><blockquote>What tool do you use for writing the book?</blockquote><p>Great question! Here we go…</p><h3>The old toolset</h3><p>When I started work on the 2nd edition a couple of years back, I did most of my editing in O’Reilly’s Atlas tool, which uses Git for source control and provides a very nice WYSIWYG / HTML interface. The only thing I didn’t like about it actually was having to switch back and forth between the text editor and raw HTML views (it was occasionally necessary to dive down into HTML to fix cross references and paragraph breaks).</p><p>But enough about the 2nd edition, that’s ancient history.</p><h3>A new toolset for writing books as code</h3><p>When it came time to start work on the 3rd edition, I decided to go for a “books as code” type of experience. I had a small number of preferences I was hoping to satisfy:</p><ul><li>ability to use my preferred IDE (IntelliJ)</li><li>using a markdown-like format for editing</li><li>side-by-side view of source code and rendered versions</li><li>ability to edit my own image files according to O’Reilly style guide</li></ul><p>Thankfully, I was able to achieve these goals. O’Reilly was kind enough to convert the 2nd edition source code from HTML to <a href="http://asciidoc.org/">Asciidoc</a>, which was similar enough to Markdown for me to be comfortable pretty quickly.</p><p>So the components of my toolset were:</p><ul><li>IntelliJ IDEA Ultimate Edition — my preferred IDE, used both for book text and code for the <a href="https://github.com/jeffreyscarpenter/reservation-service">Reservation Service</a> example used throughout the book</li><li>Asciidoctor plugin — provides syntax highlighting, side-by-side rendered view, and PDF/HTML generation</li><li>Git — using familiar version control was a big comfort factor</li><li>LucidChart — provides great UX for editing drawings and made it easy to conform with standard O’Reilly conventions and fonts</li></ul><p>This picture gives a good view of my typical editing environment (dark mode FTW!):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UG5XBy4D6jjlc8Fm5ang8Q.png" /><figcaption>Side-by-side preview in IntelliJ using Asciidoctor plugin</figcaption></figure><p>Even though it took me a while to learn the Asciidoc syntax, I was usually able to find an example of the format I needed elsewhere in the text and copy it. I really only had to resort to the <a href="https://asciidoctor.org/docs/user-manual/">documentation</a> when it came time to update a couple of math formulas in LaTEX.</p><p>There were a couple of benefits of this approach that I discovered along the way:</p><h4>Managing TODOs</h4><p>Because Asciidoc supports comments, I was able to leverage IntelliJ’s TODO feature to keep track of action items for myself as well as reviewer comments I needed to address:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*GlcrE2TvQSq-XGfXLieRYA.png" /><figcaption>Example TODO item</figcaption></figure><p>Then I could use the TODO window to iterate through my various work items and burn them down one at a time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*ChLu0rJQp4jCAsYL-YwsdQ.png" /></figure><h4>Previewing Rendered Content</h4><p>It brings a great comfort factor to be able to read the rendered content side by side with the original source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*LI0GurRSzFbaO9Kc5ajlOw.png" /><figcaption>Previewing rendered content including images</figcaption></figure><p>Being able to review the rendered content immediately kept me from needing to continually print out paper copies to review and mark up, which was my practice when working on the 2nd edition. Using less paper is a big plus!</p><p>The Asciidoctor plugin does support HTML and PDF generation, which is a super nice feature. There are a couple small dialect differences between the plugin and O’Reilly’s Atlas, but nothing too distracting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B1I1k-gpxF3K3CBUZMey9A.png" /><figcaption>Example PDF preview generated by Asciidoctor plugin</figcaption></figure><p>I mainly used the PDF generation as a quick way to determine how long a given chapter was, as I was trying to keep each chapter to 30 pages or less. Even though the layout used for O’Reilly print editions is quite different than the Asciidoctor generated PDFs (compare the images above and below), they typically result in about the same number of pages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wDzvnVYIr02zromPxbucNQ.png" /><figcaption>Example PDF generated by O’Reilly Atlas</figcaption></figure><h4>Refactoring</h4><p>Another great discovery I made while working on the book was how easy it was to refactor content when working in IntelliJ. This took two main forms: moving content and renumbering images.</p><p>In working on the 3rd edition, I was really trying to improve on the flow of the book by telling a consistent story from start to finish. To accomplish this I sometimes needed to reorder content within a chapter, or move content between chapters. Sometimes I even moved content to a “parking lot”. This was useful when I wanted to get some content out of the way but wasn’t ready to make a final decision on whether to keep or delete. The “structure” view was also super helpful for making sure I had my outline levels correct.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*hKTiAWHU6BueOti5R5K_3g.png" /><figcaption>The parking lot as an actual source file</figcaption></figure><p>The second form of refactoring was image renumbering. In working on the 2nd edition it was a real pain whenever I decided I wanted to introduce a new image into the flow of a chapter, because O’Reilly uses a convention of storing images in numbered order by chapter.</p><p>For the 3rd edition this was so much easier. I just used IntelliJ’s name refactoring to rename image files, which would also update references in the text:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*gT1URLEgn3Dv2bQiOdPTBg.png" /><figcaption>Right click on an image file to bring up the “rename” refactor option</figcaption></figure><p>One other thing I would love to have automated was the process of updating images. I’ve been managing all my own artwork using LucidChart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*UYb_mLXGTEU_35V17WNDWA.png" /></figure><p>I love LucidChart as a drawing tool but I wasn’t able to coerce the filenames it uses on generated image files, so I had to do a lot of manualmv commands to move images from my Downloads folder into my Git repo.</p><h4>Do I really need revision history?</h4><p>One interesting thing I found that didn’t match my original expectation. I thought I would make use of revision history to compare previous versions. In actual practice, I found that beyond using a simple git diff to compare my current changes with the last saved version, I never actually needed to compare versions. However, I will say that it felt great to do a git commit with comment each time I finished writing a new section. For some reason or force of habit I always seem to jump out to command line for my commits:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*koQSDKRt_qiYsk-26XaVIg.png" /></figure><h3>Wrapping up</h3><p>So there it is, my toolset for writing a book as code. Hope you find it interesting and/or helpful, and I can’t wait to have a completed book to share with you. I’m pushing hard on edits so we can have have paper copies in hand in time for <a href="https://www.datastax.com/accelerate">DataStax Accelerate</a> in May, we will see what happens…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f988133ea41e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My first week with DataStax Apollo]]></title>
            <link>https://jscarp.medium.com/my-first-week-with-datastax-apollo-9db982f4cf50?source=rss-343198ddcf60------2</link>
            <guid isPermaLink="false">https://medium.com/p/9db982f4cf50</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[datastax]]></category>
            <dc:creator><![CDATA[Jeff Carpenter]]></dc:creator>
            <pubDate>Tue, 29 Oct 2019 21:55:11 GMT</pubDate>
            <atom:updated>2019-10-31T17:59:10.008Z</atom:updated>
            <content:encoded><![CDATA[<h3>My First Week with DataStax Apollo</h3><h4>Porting a Java microservice to use an Apache Cassandra DB as a Service</h4><p>In case you haven’t heard, <a href="https://www.datastax.com/blog/2019/10/simpler-approach-data-driven-application-development-and-database-deployment-datastax">DataStax has announced a free Beta</a> for our Apache Cassandra as a Service offering, known as <a href="https://apollo.datastax.com/">Apollo</a>. Apollo is the first service to be released on our cloud platform known as Constellation.</p><h3>Jeff Carpenter on Twitter</h3><p>One of the reasons I joined @DataStax was the vision to provide scalable database services in the cloud based on Cassandra. I&#39;m super excited that the reality is here. Check out our free beta at https://t.co/6la2XCsN7x. I&#39;d love to hear your feedback!</p><p>I wanted to share a few of my first impressions and tips on using Apollo. I’ll provide a quick disclaimer up front that since this is a Beta, some of the details and links I share will be subject to change.</p><p>As part of the Beta you can create clusters in both Constellation (deployed on Amazon Web Services) and Google Cloud Platform. To expand on that last sentence — the AWS offering is part of our Constellation cloud platform, which will eventually expand to other public clouds. In this environment, the billing for production databases (when you create non-development clusters after the Beta period) will be through DataStax. On the Google side, you can provision clusters through the Google Cloud Marketplace which gives you the advantage of billing through your GCP account.</p><p>During the Beta period, documentation for running clusters is found on a temporary docs site: <a href="https://docs.datastax.com/en/apollo/aws/doc/index.html">AWS docs</a> / <a href="https://docs.datastax.com/en/apollo/gcp/doc/index.html">GCP docs</a>.</p><h3>Creating a Database on Apollo</h3><p>It’s really straightforward to create an account and then start databases. My colleague Adron Hall has a great post on <a href="https://compositecode.blog/2019/10/11/apollo-beta-released/">getting started with Apollo</a>, so I won’t repeat the details here. The best thing about the experience was how, well, <em>boring </em>it was. I specified a database name, keyspace name, username and password and started a database. A few minutes later I got an email saying it was up — no big deal.</p><p>Each cluster includes access via DataStax Studio, which you can quickly use to configure schema, load sample data, and so on.</p><p>One quick thing to note is that databases are terminated automatically 72 hours after start during the Beta period, so don’t get too attached, these are just for testing purposes at this point.</p><h3>Porting the Reservation Service</h3><p>Once I had a database set up and poked around a bit, I got down to business with development. I decided to port my <a href="https://github.com/jeffreyscarpenter/reservation-service/tree/apollo">Reservation Service</a>, a sample microservice I created in conjunction with Cassandra: The Definitive Guide.</p><p>You can see the design of the service in the figure below. It’s a basic Spring Boot service that provides a RESTful API and persists data to Cassandra using the DataStax Java Driver.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Emv0Nno-Op6dNLwFoFjw9g.png" /><figcaption>Reservation Service design</figcaption></figure><p>My colleague Cedrick Lunven (<a href="https://twitter.com/clunven">Twitter</a>, <a href="https://github.com/clun">GitHub</a>) and I recently collaborated on some updates to the service including upgrading to the <a href="https://docs.datastax.com/en/developer/java-driver/4.2/">4.2 version of the Java Driver</a> and adding some automated tests. As part of that work, Cedrick added code to create the reservation schema programmatically.</p><p>Here’s how it works:</p><ul><li>The Reservation Service automatically creates the schema each time it starts up.</li><li>The code uses the CQLIF NOT EXISTS semantics so that the statements don’t error if the keyspace and tables already exist.</li></ul><p>We also changed the driver initialization logic to take advantage of the <a href="https://docs.datastax.com/en/developer/java-driver/4.2/manual/core/configuration/">file-based configuration</a> feature introduced in the 4.0 release.</p><h4>Flexible Session Creation</h4><p>In order to update the Reservation Service to work with Apollo, I followed the instructions on the <a href="https://docs.datastax.com/en/apollo/aws/doc/index.html">DataStax Docs site</a>. There’s a very helpful page devoted to exactly the topic I needed — <a href="https://docs.datastax.com/en/apollo/aws/doc/dscloud/apollo/dscloudMigrateJavaDriver.html">migrating Java applications to work with Apollo</a>.</p><p>One of my goals for this update was to see how much of the logic of connecting to a Cassandra cluster I could externalize from the Reservation Service code, so that, if possible, I could point the Reservation Service at any Cassandra cluster just by changing the configuration file with no changes.</p><p>The good news is that I was able to achieve exactly this goal! Here is theapplication.conf <a href="https://github.com/jeffreyscarpenter/reservation-service/blob/apollo/src/main/resources/application.conf">configuration file</a> that I use connect to my Apollo cluster:</p><pre>datastax-java-driver {<br><br>  basic {<br><em>    </em>cloud.secure-connect-bundle = &lt;INSERT PATH TO BUNDLE&gt;<br>    request.consistency = LOCAL_QUORUM<br>  }<br><br>  advanced.auth-provider {<br>    class = PlainTextAuthProvider<br>    username = &lt;INSERT USERNAME&gt;<br>    password = &lt;INSERT PASSWORD&gt;<br>  }<br>}</pre><p>To run the Reservation Service yourself, you’ll need to create an Apollo database and replace the &lt;INSERT ...&gt; elements based on your configuration. The key elements here are:</p><ul><li>The path to the secure connect bundle — the bundle is a zip file containing connection points and encryption keys the application needs to connect to the cluster securely. It also includes the keyspace, which is important for reasons you’ll see below</li><li>The default consistency level — for Apollo, you’ll want to default to run all queries at the LOCAL_QUORUM consistency level (<a href="https://docs.datastax.com/en/apollo/aws/doc/dscloud/apollo/dscloudDatabaseConditions.html">it’s required for writes</a>)</li><li>The username and password for the database user.</li></ul><p>For contrast, file-based configuration for a self-managed cluster typically looks something like this (excerpt from the <a href="https://docs.datastax.com/en/developer/java-driver/4.2/manual/core/">Java Driver documentation page</a>):</p><pre>datastax-java-driver {<br>  basic {<br>    contact-points = [ &quot;1.2.3.4:9042&quot;, &quot;5.6.7.8:9042&quot; ]<br>    load-balancing-policy.local-datacenter = datacenter1<br>  }<br>}</pre><p>This highlights two primary differences of this self-managed configuration vs. the Apollo configuration:</p><ol><li>In the Apollo configuration, the secure connect bundle abstracts the contact points</li><li>Apollo itself determines the load balancing policy, so you don’t need to specify one</li></ol><h4>The tricky part — programmatic schema management</h4><p>While the configuration updates were simple, the harder part was updating the code that performed schema management.</p><p>At least for the present time, Apollo limits databases to a single keyspace which is defined when the cluster is created. The name of the keyspace is also defined as part of the secure connect bundle so that you don’t have to specify it as a separate configuration parameter.</p><p>This led to a situation that I needed to work around in order to update the schema management code. The Reservation Service previously provided schema management via an operation on the ReservationRepository class which was invoked as part of the constructor using adropSchema boolean argument. If the argument was set, as it was in test configurations, requesting a repository object would result in the reservation keyspace being dropped and rebuilt.</p><p>However, given the single keyspace per database restriction in Apollo, dropping and adding keyspaces are disallowed operations. This is implemented via Cassandra’s access control mechanisms — the database user you specify at database creation time is not given permissions to add or drop keyspaces.</p><blockquote>Aside: If you haven’t worked before with Cassandra’s authorization model, now is the time you’ll want to bring yourself up to date. The permissions allocated to your database user are <a href="https://docs.datastax.com/en/apollo/aws/doc/dscloud/apollo/dscloudCreateUsers.html">described in the Apollo documentation</a>.</blockquote><p>My first attempt was to modify the dropSchema logic to drop tables instead of the entire keyspace. However after thinking about things more the solution I decided to implement was to leverage the CQL TRUNCATE operation instead of dropping and rebuilding schema. My solution included the following:</p><ul><li>Refactoring schema management operations out or the ReservationRepository into a new ReservationSchemaManager class</li><li>Separating keyspace creation and table creation into separate operations on ReservationSchemaManager for flexibility, and ensuring these operations use IF NOT EXIST semantics so that no errors are generated if tables already exist</li><li>Invoking the schema creation operations in the ReservationRepository constructor to ensure that schema exists. When run against a new Apollo, the keyspace will already exist, but the tables might not.</li><li>Adding an operation to ReservationSchemaManager to truncate the tables in the reservationkeyspace.</li><li>Updating the test code to truncate tables between each test case instead of dropping and rebuilding the keyspace.</li></ul><h4>An example you can reference</h4><p>You can see the results of my refactoring on the apollo branch of the Reservation Service at: <a href="https://github.com/jeffreyscarpenter/reservation-service/tree/apollo">https://github.com/jeffreyscarpenter/reservation-service/tree/apollo</a>.</p><p>To summarize, here are my top 3 recommendations in for porting an application to run on Apollo:</p><ol><li>Upgrade to the latest DataStax Java driver and maximize your usage of file-based configuration</li><li>Check your application and data model against the guardrails including default consistency level</li><li>Update your schema creation approach, making sure any programmatic schema creation is flexible enough to work around Apollo’s restrictions on manipulating keyspaces.</li></ol><h3>What’s next?</h3><p>There’s still more of Apollo to explore. I’m looking forward to trying out the GCP version, as well as accessing Apollo clusters via the new <a href="https://helpdocs.datastax.com/aws/dscloud/apollo/dscloudConnectcqlsh.html">standalone cqlsh download</a>.</p><p>During this free Beta period we’d love to get feedback from as many people as possible. You can go to <a href="https://apollo.datastax.com/">https://apollo.datastax.com</a> to get started.</p><p>Please drop me a note to let me know you’ve joined the trial. I’d love to hear what you think!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9db982f4cf50" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[After the MVP, development continues]]></title>
            <link>https://jscarp.medium.com/after-the-mvp-development-continues-b89c486d0608?source=rss-343198ddcf60------2</link>
            <guid isPermaLink="false">https://medium.com/p/b89c486d0608</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[datastax]]></category>
            <category><![CDATA[application-development]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Jeff Carpenter]]></dc:creator>
            <pubDate>Wed, 28 Aug 2019 02:17:14 GMT</pubDate>
            <atom:updated>2019-08-28T02:17:14.773Z</atom:updated>
            <content:encoded><![CDATA[<h4>Continuing upgrades on KillrVideo Python</h4><p>In a <a href="https://medium.com/datadriveninvestor/killrvideo-python-pt-1-the-backstory-5c38191fb330">previous series</a> I wrapped up a couple of months ago, I shared how I had created a microservice implementation in Python for the KillrVideo application, a reference application my team built to demonstrate best practices for developing with Apache Cassandra and DataStax Enterprise.</p><p>The response to this series has been great, as I published it first on my <a href="https://medium.com/@jscarp">Medium blog</a> and then crossposted to the <a href="https://datastax.com/dev/blog">DataStax Developer Blog</a>. While the whole series has been well received, “<a href="https://medium.com/@jscarp/cassandra-with-python-simple-to-complex-44298391867d">Cassandra with Python: Simple to Complex</a>” has become my most popular post.</p><p>In the final post in the series, I described how I had rushed to complete the application in time for my talk at DataStax Accelerate (slides below). I also shared observations about the experience in “<a href="https://medium.com/@jscarp/lessons-learned-on-my-first-python-app-34b276698467?source=your_stories_page---------------------------">Lessons Learned on my first Python app</a>”.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Fembed_code%2Fkey%2Fl2Sf6wD9rv8U4r&amp;url=https%3A%2F%2Fwww.slideshare.net%2FJeffreyCarpenter%2Fcreating-a-python-microservice-tier-in-four-sprints-with-cassandra-kafka-and-dse-graph-147264915&amp;image=https%3A%2F%2Fcdn.slidesharecdn.com%2Fss_thumbnails%2Fdatastaxaccelerate2019pythonmicroservices-190523133745-thumbnail-4.jpg%3Fcb%3D1558618732&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=slideshare" width="600" height="500" frameborder="0" scrolling="no"><a href="https://medium.com/media/e64cd7e4b18230473a332d4cde971ac4/href">https://medium.com/media/e64cd7e4b18230473a332d4cde971ac4/href</a></iframe><p>One of the things I highlighted in my Accelerate talk was that there was some remaining work in the Python services:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hm8-opJsUvH407L9ZaoC8w.png" /></figure><p>I’m happy to report that I’ve now completed those issues and roadmap items! Here are some quick highlights of the development experience:</p><h3>Upgrade to Python 3</h3><p>After investigating the various changes I needed to make, I planned out my order of attack and started right in on upgrading the services to Python 3.</p><p>After struggling a bit with trying to get Python 3 installed, I quickly discovered the value of using <a href="https://docs.python-guide.org/dev/virtualenvs/">Python virtual environments</a> and created a <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/requirements.txt">requirements.txt</a> file to capture my library dependencies and make it easier for other developers to start working with the project.</p><p>Most of the work of updating the application to Python 3 was simple syntax changes, for example: interacting with lists. The trickiest part was updating the code that performed serialization/deserialization of the Cassandra paging state between hex and string formats, as the Python 3 codecs API is a different approach than I was used to in Python 2.</p><h3>Running Python Services in Docker</h3><p>Prior to Accelerate, I hadn’t yet built a Docker image for the Python services. I had only run the services on my local machine outside of Docker, with the rest of the application components running in Docker.</p><p>I was glad to be working with a Python 3 image as the start point in my <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/Dockerfile">Dockerfile</a> as it made things quite simple:</p><pre># Dockerfile<br>FROM python:3.7</pre><pre>...</pre><p>Once I had created the Dockerfile, I also created a .travis.yml file and enabled the killrvideo-python repo in our Travis CI account for automated builds. You can now find the Docker container for KillrVideo Python services in our DockerHub: <a href="https://hub.docker.com/r/killrvideo/killrvideo-python">https://hub.docker.com/r/killrvideo/killrvideo-python</a>.</p><h3>Integration with killrvideo-all-in-one</h3><p>While I was working on the KillrVideo Python services, my colleague Aleks (<a href="https://medium.com/u/3d11563f662">Aleksandr Volochnev</a>) was working in parallel on some improvements to how we deployed the overall <a href="http://github.com/KillrVideo">KillrVideo application</a>. His primary goal was to create the simplest possible starting point for someone wanting to try to deploy the KillrVideo application with a simple docker-compose.yaml file. You can find the results in the “<strong>all-in-one</strong>” repo: <a href="https://github.com/killrvideo/killrvideo-all-in-one">https://github.com/killrvideo/killrvideo-all-in-one</a>. A secondary goal was to demonstrate how easy it is to deploy KillrVideo to Kubernetes.</p><h4>Removing dependence on etcd for service discovery</h4><p>One of the key simplifications in the all-in-one approach was to remove etcd from the KillrVideo system. As I discussed in <a href="https://medium.com/@jscarp/advertising-python-services-via-etcd-aa1eca6d8ecc?source=your_stories_page---------------------------">Advertising Python services via etcd</a>, we were using etcd as a service registry, enabling the various components to locate each other. The new approach is to use well known names within the docker network as described in our docker-compose configuration:</p><ul><li>dse</li><li>backend</li><li>kafka</li></ul><p>and so on…</p><p>As part of this change, I converted most of the configurable values in the Python services to be read from environment variables instead of the configuration file I was using previously. In keeping with <a href="https://dantehranian.wordpress.com/2015/03/25/how-should-i-get-application-configuration-into-my-docker-containers/">best practices</a> for containerized applications, using environment variables makes these parameters much easier to configure. I use the os.getenv() operation to access these values and provide default values when appropriate, as shown in the <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/__init__.py">main function</a> of the Python app:</p><pre>dse_username = os.getenv(&#39;KILLRVIDEO_DSE_USERNAME&#39;) # no default<br>dse_password = os.getenv(&#39;KILLRVIDEO_DSE_PASSWORD&#39;) # no default<br>contact_points = os.getenv(&#39;KILLRVIDEO_DSE_CONTACT_POINTS&#39;,&#39;dse&#39;)<br>  .split(&#39;,&#39;)<br>service_port = os.getenv(&#39;KILLRVIDEO_SERVICE_PORT&#39;, &#39;50101&#39;)</pre><p>The most challenging part of the etcd removal and migration to well known names was learning how to configure the Kafka listeners correctly so that the services could correctly connect to the Kafka broker. Fortunately <a href="https://rmoff.net/2018/08/02/kafka-listeners-explained/">an excellent blog post from Robin Moffat</a> helped me with the configuration in my docker-compose.yaml:</p><pre># Start a Kafka Broker for Killrvideo<br>  kafka:<br>    image: wurstmeister/kafka<br>    depends_on:<br>      - zookeeper<br>    environment:<br><strong>      KAFKA_LISTENERS: PLAINTEXT://:9092<br>      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092</strong><br>      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181<br>      KAFKA_CREATE_TOPICS: &quot;topic-kv-videoCreation:1:1,topic-kv-userCreation:1:1,topic-kv-videoRating:1:1,topic-kv-commentCreation:1:1&quot;<br>    ports:<br>      - 9092:9092</pre><h4>A more composable configuration for development</h4><p>I really enjoyed the simplicity of the all-in-one configuration, so of course I wanted to add things to it such as metrics collection and visualization using the new DSE metrics collector, Prometheus, and Grafana — irony intended ;).</p><p>After a short team discussion I realized that adding additional components would defeat the purpose of all-in-one being simple. Instead, I reworked the <a href="https://github.com/KillrVideo/killrvideo-docker-common">killrvideo-docker-common</a> repo to be a “one stop shop” for building composable development configurations of KillrVideo, including:</p><ul><li>Switching between Python, Java, Node.js service implementations</li><li>Adding OpsCenter and/or metrics collection infrastructure</li><li>Running against an externally managed database cluster</li><li>Configuring database authentication</li></ul><p>Since this repo isn’t just about the Python services, it’s probably worthy of its own separate blog post, but for now the <a href="https://github.com/KillrVideo/killrvideo-docker-common/blob/master/README.md">README</a> is a great place to start.</p><h3>Authenticating to DSE</h3><p>I’ll admit some regret that it took so long to finish the security related features of the KillrVideo Python services. Specifically, we’ve made provision in the KillrVideo architecture for securing access to the database. The primary features are for Authentication and Authorization.</p><p>We had already built in the capability to enable auth/auth for DSE running in Docker by overriding the dse.yaml configuration file, now I just needed to configure the Python services to authenticate to DSE using username/password provided via environment variables (which were read as shown above).</p><p>To perform this authentication, I configured the AuthProvider used to initialize the DataStax Python driver:</p><pre>auth_provider = PlainTextAuthProvider(username=dse_username, password=dse_password)</pre><pre>...</pre><pre>session = Cluster(contact_points=dse_contact_points,<br>                  execution_profiles=execution_profiles,<br>                  auth_provider = auth_provider)<br>                    .connect(&quot;killrvideo&quot;)</pre><h3>Graph Recommender Optimization (ok, Bug Fix)</h3><p>As I described in <a href="https://medium.com/@jscarp/the-long-sprint-building-a-graph-recommender-cb264a094655?source=your_stories_page---------------------------">The long sprint — building a Graph Recommender</a>, my original strategy for implementing the graph recommender for the KillrVideo Python services was to rely heavily on the graph traversals that David Gilardi had written first in Gremlin and then translated into the Java implementation of the KillrVideo services. There were two graph traversals implemented in the <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/suggested_videos/suggested_videos_service.py">Suggested Videos Service</a> — one used to generate recommendations based on a user, and the other to generate recommendations based on a video.</p><p>As I tested the “recommendations by user” traversal I had translated into <a href="http://tinkerpop.apache.org/docs/current/tutorials/gremlin-language-variants/">gremlin-python</a> syntax, I encountered an error in the traversal on a step that omitted videos that the user had already watched. At the time I was able to get somewhat satisfactory recommendation results by commenting out this step temporarily. I figured that if your list of recommended videos contained an occasional video you had already watched, it wouldn’t be too much of a hassle.</p><p>After the MVP, I recalled this “shortcut” and began debugging the problem with David. As we were able to reproduce the problem in Java code, we sought out DSE Graph expert Daniel Kuppitz, who was able to help us rewrite the traversal to fix the error and at the same time make it more performant. Bonus!</p><h3>Feedback to docs/driver teams</h3><p>One of the privileges (and responsibilities) of being an advocate at DataStax is the access that I have to the engineering teams that build and document our technology. We often refer to ourselves as “User 0” and try to provide feedback early and often on products and features both new and old.</p><p>While I did keep extensive notes during my development work, I found it difficult to pause to provide feedback . As a result, I ended up consolidating my notes to provide my feedback on the Python driver and documentation in one batch.</p><p>The upside of this approach is that it likely eliminated some noise from my feedback, while the downside was the delay in delivering the feedback. If you’ve ever had the privilege of working with our DataStax Driver team you’ll not be surprised to hear the feedback was really well received and has resulted in improvements to the driver documentation and a couple of minor bug fixes.</p><h3>Yes, there’s more to come</h3><p>Over the past weeks we (the DataStax Developer Relations team) have been working on an updated version of the <a href="https://www.datastax.com/accelerate/datastax-accelerate-bootcamp">Application Development Bootcamp</a> that we presented at the <a href="https://www.datastax.com/accelerate">DataStax Accelerate</a> conference, which included development exercises based on KillrVideo. We’re super excited about taking the bootcamp on the road this fall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KKRe4UIzOR8G8k7Vkyl2vw.png" /><figcaption>DataStax Accelerate Bootcamp — May 2019</figcaption></figure><p>Now I don’t want to reveal too much, but one of the improvements we’re working on is the ability to support service implementations in multiple languages, so bootcamp attendees can choose whether to do their development exercises in Java, Node.js, or Python.</p><p>That’s right, KillrVideo Python’s coming to bootcamp! And hopefully bootcamp will be coming to a city near you soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b89c486d0608" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons Learned on my first Python app]]></title>
            <link>https://jscarp.medium.com/lessons-learned-on-my-first-python-app-34b276698467?source=rss-343198ddcf60------2</link>
            <guid isPermaLink="false">https://medium.com/p/34b276698467</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[lessons-learned]]></category>
            <dc:creator><![CDATA[Jeff Carpenter]]></dc:creator>
            <pubDate>Thu, 20 Jun 2019 19:20:25 GMT</pubDate>
            <atom:updated>2019-06-20T19:20:25.263Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 9 — concluding a series on the KillrVideo Python implementation</h4><p>In this series I’ve been sharing how I built my first Python application with Apache Cassandra — a <a href="https://github.com/KillrVideo/killrvideo-python">Python implementation</a> of the <a href="http://killrvideo.github.io/">KillrVideo</a> microservice tier. As I reached the end of the project, I was able to share at the recent <a href="https://datastax.com/accelerate">DataStax Accelerate</a> conference some lessons learned. Here are my slides from that presentation:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Fembed_code%2Fkey%2Fl2Sf6wD9rv8U4r&amp;url=https%3A%2F%2Fwww.slideshare.net%2FJeffreyCarpenter%2Fcreating-a-python-microservice-tier-in-four-sprints-with-cassandra-kafka-and-dse-graph-147264915&amp;image=https%3A%2F%2Fcdn.slidesharecdn.com%2Fss_thumbnails%2Fdatastaxaccelerate2019pythonmicroservices-190523133745-thumbnail-4.jpg%3Fcb%3D1558618732&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=slideshare" width="600" height="500" frameborder="0" scrolling="no"><a href="https://medium.com/media/e64cd7e4b18230473a332d4cde971ac4/href">https://medium.com/media/e64cd7e4b18230473a332d4cde971ac4/href</a></iframe><p>You can access the presentation recording (slides and audio) via the <a href="https://www.datastax.com/accelerate/sessions">Accelerate website</a> (registration required) — my talk is part of the “Building Modern Applications” track.</p><p>For the final post in this series, I thought I would summarize some of the lessons learned that I shared in that presentation. I’ll divide these up into some technology-specific learnings as well as some things that I learned about my own personal software development process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*eGXquMUgD-RQiICV" /></figure><h3>Apache Cassandra and DataStax Enterprise Learnings</h3><p>Even though I have experience using these technologies as part of my job, there are a few lessons I learned (or relearned) as part of this process.</p><ul><li>The Cassandra development methodology works!</li></ul><p>The traditional guidance for developing with Cassandra has been to use a data modeling approach based on denormalization, that is, to design tables around the queries your application will need to perform. I found it to be quite easy to develop the data access layer of my Python microservices because of having Cassandra tables already defined.</p><p>You might argue that I also benefitted from having a complete microservice design complete with <a href="https://medium.com/@jscarp/python-app-dev-with-protobuf-and-grpc-e5bff779783d">GRPC service interfaces</a>, and that of course it would seem easy because I was simply reproducing an existing design in a new language. I’d concede that point but still maintain that query-first design is an effective approach for development with Cassandra.</p><ul><li>Use multiple methods of driver interaction</li></ul><p>For my development of data access portions of the service using the <a href="https://docs.datastax.com/en/developer/python-dse-driver/2.9/">DataStax Python Driver</a>, I decided to use the cqlengine object mapper where it made sense for <a href="https://medium.com/@jscarp/when-data-access-is-the-easiest-part-of-a-microservice-c7a90dee701a">simple data access</a> and then use PreparedStatements for <a href="https://medium.com/@jscarp/cassandra-with-python-simple-to-complex-44298391867d">more complex data access</a>, such as cases where I needed paging. This approach of using the most convenient, straightforward API for each query saved a lot of time in my development process.</p><ul><li>Search is a powerful thing</li></ul><p>I was surprised by how easy it was to access DSE Search features from the Python Driver. I inherited CQL scripts as part of the KillrVideo infrastructure that created the search indexes, so all I had to do was to incorporate the correctsolr_query syntax as part of my search queries. This was one of the simplest parts of the implementation.</p><ul><li>Graph way of thinking</li></ul><p>Creating the <a href="https://www.twitch.tv/datastaxdevs">graph recommender</a> was the portion of the project that took the longest time, partly because my travel schedule was ramping up, and because I started doing a lot of the development work via live coding on <a href="https://www.twitch.tv/datastaxdevs">Twitch</a> (more on that below). My biggest lesson here was that there really is a Graph way of thinking that you need to learn to get the most of this kind of database. My rate of progress increased when I first prototyped my traversals in DataStax Studio to get the Gremlin syntax correct, then translated them into Python in the shell step by step, then moved the code into the application.</p><h3>Personal Software Process Learnings</h3><p>The bulk of my learnings were really about my own personal software development practices. There were definitely some things I did well and some areas in which I found I could improve. Let’s start with a happy lesson:</p><ul><li>Decouple all the things</li></ul><p>For the Python implementation of the KillrVideo services, I deliberately copied the approach taken by the <a href="https://github.com/KillrVideo/killrvideo-java-v2">Java implementation</a>, separating out the different infrastructure technologies as much as possible, similar to what I show <a href="https://www.slideshare.net/JeffreyCarpenter/patterns-for-persistence-and-streaming-in-microservice-architectures">here</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*g0mfJMREsEJhR7E-" /><figcaption>Isolating APIs, messaging interfaces from business logic and data access to minimize the cost of change</figcaption></figure><p>For each service, I created separate classes for the GRPC API implementation and any Kafka messaging, isolating those interface-specific details from the the business logic and data access code. This will make it easier to change interface styles or the message bus in the future.</p><ul><li>Multiple service interface styles 👍</li></ul><p>At various points in the implementation, I struggled with use of GRPC. There were a couple of occasions where I mistakenly assumed there was a problem with GRPC that actually turned out to be in my code. One helpful thing was that in many cases, errors that occurred in the Python services that I was not catching actually propagated up to the clients of those services via GRPC, which enabled me to locate and debug issues.</p><p>Stepping back, I think there is definitely a place for different API styles in microservice architectures. I think that using something like GRPC is highly appropriate for services that sit pretty low in the stack, like the KillrVideo microservice tier, in which many of the services are basic data storage / CRUD (create, read, update, delete) style services. As you get closer to the user interface, Graph QL is a great way to aggregate data from multiple lower level services. We’re actually working on a new implementation of the KillrVideo web application that uses Graph QL for this purpose. Not to be left out, I think REST APIs are still a very effective way to expose services outside of your organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*5Qhi-XFzpoNJrdiY" /><figcaption>Pros and cons of different service interface approaches</figcaption></figure><ul><li>Consider monorepo for new work</li></ul><p>Some of the challenges in working on the KillrVideo Python services came from having the different components of the architecture stored in different GitHub repositories. Specifically, there are a couple of cases of “shared” assets such as <a href="https://github.com/KillrVideo/killrvideo-service-protos">interface definitions</a>, <a href="https://github.com/KillrVideo/killrvideo-data">database schema</a>, and <a href="https://github.com/KillrVideo/killrvideo-docker-common">Docker configurations</a> that we define in common repositories and then incorporate where needed in other repositories via git subtree.</p><p>This has proved to be more of a hassle than a benefit, and the consensus of our Developer Advocate team is that if we were starting over, we’d develop KillrVideo in a single repository — a “monorepo” — to eliminate duplication and having to remember to copy code between projects.</p><ul><li>The levels of testing blur as services get smaller</li></ul><p>I took the semi-controversial approach of not creating unit tests for my Python service implementations and instead relying on a set of <a href="https://github.com/KillrVideo/killrvideo-integration-tests">integration tests</a> that had been created to test implementations of the microservice tier in other languages. I still feel pretty good about this approach, as I only discovered one additional bug via the web application after passing all of the integration tests. If you are truly implementing microservices that perform a single responsibility, the distinctions between unit, component and integration tests really begin to lose their meaning.</p><ul><li>Vertical slices FTW!</li></ul><p>Another benefit that came from having the client web application already complete was that I could immediately test the end-to-end behavior of the entire application as I implemented each function. Granted, this is not the case in most development efforts, but it reminded me of the value of implementing complete vertical slices of functionality from user interface all the way through to data access.</p><ul><li>Friction logging as a time capsule</li></ul><p>I learned a lot in this project from working with new technology, and familiar technology in new programming languages. One of the best improvements I made to my personal software process was keeping a log of my development progress, including stack traces, error messages, documentation sites that helped, and so on. It was fun to go back through that log and trace the course of my learning, see where I had wrong assumptions, and so on.</p><p>At the same time, I identified a few areas in which I could have made better use of this <a href="https://devrel.net/developer-experience/an-introduction-to-friction-logging">friction log</a>. I wish that I had surfaced areas where I had challenges with the driver API and documentation to the driver team as I encountered them, rather than waiting until the end of the process. A second thing I would improve is to record my TODOs in the source code rather than a separate file — it was harder to keep track of them that way.</p><ul><li>Example code — buyer beware!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/0*TscoPzFEQ-I_EtQE.jpg" /></figure><p>I know there’s a running joke about copying and pasting code from the internet, but sometimes those code samples are just what you need to get pointed in the right direction. I recounted one instance where I got burned in the post on <a href="https://medium.com/@jscarp/kafka-cassandra-like-peanut-butter-and-chocolate-267815b7ba22">introducing Kafka</a> into the architecture and didn’t understand the implications of multiprocessing vs. multithreading in Python.</p><ul><li>One learning curve at a time</li></ul><p>I alluded to this above in the discussion about learning the graph way of thinking, but it’s worth repeating here. Anything you can do to separate out your learning curves is beneficial. I had the most success and fastest process when I focused on just learning Gremlin syntax before trying to do Gremlin in Python.</p><ul><li>New language, new process?</li></ul><p>This point builds on the previous one. When I started this project, I brought my standard habits of Java development. It took me a while to realize that I could prototype bits of Python code at the console. The quick feedback loop of trying out various statements at the command line really saved me a lot of time once I realized this was an option. I also learned that when you’re learning a new programming language or API, it’s important to make a small investment to learn the basics. Even using a simple “cheat sheet” can be extremely helpful. Googling “python for loop” is a bit embarrassing, after all.</p><ul><li>Live coding is like pair programming++</li></ul><p>As mentioned above, switching to live coding for the final sprint of this project was a big reason why the graph recommender took so much longer than the other parts of this project. However, there were also a number of benefits that came from doing a good portion of the development work on a live stream, the most obvious of which was the ability to get additional eyes on the code and helpful suggestions from those participating or viewing the stream. Another was that by explaining what I was doing out loud, I was able to detect some flaws in my own thought processes more quickly. Finally, doing regular streams also ensured that I had dedicated time to code in my schedule that couldn’t be overtaken by other meetings.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FjK1J4sb6qP0%3Fstart%3D94%26feature%3Doembed%26start%3D94&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DjK1J4sb6qP0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FjK1J4sb6qP0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6308d490e5bd572c83c9328891f9de5d/href">https://medium.com/media/6308d490e5bd572c83c9328891f9de5d/href</a></iframe><h3>Wrapping up an MVP</h3><p>One of the hard things for me was determining when to call the project “done” and attach a 1.0 version number. Now that I have completed an upgrade of the codebase to Python 3, I’m basically at that point, although I am working on the last couple of minor items to make a better developer experience:</p><ul><li>Adding in the option to use authentication to secure communication between the services and DataStax Enterprise.</li><li>Creating an automated build via Travis.ci to create a Docker image for the KillrVideo Python services.</li><li>Incorporating an option to swap in the Python service implementation as part of the <a href="https://github.com/KillrVideo/killrvideo-all-in-one">KillrVideo all-in-one</a> repository, which provides a great starting point for running the entire KillrVideo application on your desktop in Docker containers</li></ul><p>Thanks to all my readers who have followed this series and provided feedback and support! Let me know if you have other ideas on how to improve KillrVideo or an idea for another fun project with Apache Cassandra and DataStax Enterprise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34b276698467" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The long sprint — building a Graph Recommender]]></title>
            <link>https://jscarp.medium.com/the-long-sprint-building-a-graph-recommender-cb264a094655?source=rss-343198ddcf60------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb264a094655</guid>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[tinkerpop]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Jeff Carpenter]]></dc:creator>
            <pubDate>Wed, 29 May 2019 15:36:22 GMT</pubDate>
            <atom:updated>2019-08-27T22:37:33.132Z</atom:updated>
            <content:encoded><![CDATA[<h3>The long sprint — building a Graph Recommender</h3><h4>Part 8 of a series on the KillrVideo Python implementation</h4><p>Over the past couple of months I’ve been blogging about building a Python application with Apache Cassandra — specifically a Python implementation of the <a href="http://killrvideo.github.io/">KillrVideo</a> microservice tier — and we’re getting close to the end!</p><p>In the first few posts I shared about <a href="https://medium.com/datadriveninvestor/killrvideo-python-pt-1-the-backstory-5c38191fb330">why I started this project</a>, how I incorporated <a href="https://medium.com/@jscarp/python-app-dev-with-protobuf-and-grpc-e5bff779783d">GRPC</a> and <a href="https://medium.com/@jscarp/advertising-python-services-via-etcd-aa1eca6d8ecc">Etcd</a>, and the <a href="https://medium.com/@jscarp/who-needs-unit-tests-im-building-microservices-4c8fe40d7095">testing approach</a>. Then we looked at implementing the KillrVideo Python services, including <a href="https://medium.com/@jscarp/when-data-access-is-the-easiest-part-of-a-microservice-c7a90dee701a">simple</a> and <a href="https://medium.com/@jscarp/cassandra-with-python-simple-to-complex-44298391867d">more complex</a> data access using various features of the DataStax Python Driver. In the most recent post I shared how I <a href="https://medium.com/@jscarp/kafka-cassandra-like-peanut-butter-and-chocolate-267815b7ba22">introduced Kafka</a> into the architecture in order to pass events between services. Now we’re entering the final stage — building a graph recommendation engine using DataStax Graph in order to implement the SuggestedVideosService.</p><h3>Live Coding on Twitch</h3><p>For this stage of the project, I decided it would be helpful to get some graph expertise from my colleague David Gilardi (@SonicDMG on Twitter), so we started live streaming the development work on our <a href="https://www.twitch.tv/datastaxdevs">DataStax Developers</a> channel on Twitch. You can see some of our past streams on our <a href="https://www.youtube.com/datastaxdevs">YouTube channel</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FjK1J4sb6qP0%3Fstart%3D94%26feature%3Doembed%26start%3D94&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DjK1J4sb6qP0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FjK1J4sb6qP0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6308d490e5bd572c83c9328891f9de5d/href">https://medium.com/media/6308d490e5bd572c83c9328891f9de5d/href</a></iframe><p>The great thing about live coding approach was how much I was able to learn from David based on his past experience with DataStax Graph. With the contributions of others on the Twitch stream, it was kind of like pair programming ++.</p><p>However, since I started doing a significant amount of the development work on Twitch streams, and minimal work outside of those streams, my pace of progress slowed considerably. I was now only coding a couple of hours a week on the KillrVideo Python project. That made this final part of the project the longest sprint by far — about 3 months instead of two weeks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2E_VlNXvAmsiZajRAQt7gg.png" /><figcaption>The graph recommender “Sprint 4” took a bit longer than the other sprints due to live coding and travel</figcaption></figure><h3>The Approach — Why Graph?</h3><p>There are a number of possible approaches we could have taken to implementing a recommender for our KillrVideo Python services. There are libraries available in Python for machine learning tasks including recommendations. For the purpose of this implementation I decided to emulate the approach we took in other language implementations of KillrVideo, which was to implement a recommender using DataStax Enterprise Graph.</p><p>The basic approach of this implementation is as follows:</p><ul><li>Graph population: the SuggestedVideosService consumes events published to Kafka topics by other services and uses that information to populate a graph stored in DSE Graph. The information consumed includes new users, video uploads, and user ratings of videos to populate a graph.</li><li>Recommendation: when the web application requests suggested videos for a particular user, generate recommendations for that user by performing a query on the graph that leverages the relationships between the users and videos such as uploading and rating videos. We’ll discuss the specifics of the algorithm below.</li></ul><h3>Creating a Graph Schema</h3><p>First, we’ll need a schema to describe the structure of our graph. The information we need to capture includes entities such as users and videos, and relationships, such as which users uploaded or rated which videos. The entities become vertices in our graph, and the relationships become edges. This schema is shown graphically in the figure below, which was captured from the schema viewer built into DataStax Studio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zos3JFMwuM8EQRLgRv76Fw.png" /><figcaption>Graph schema used for our KillrVideo recommendation algorithm</figcaption></figure><p>Users and videos are modeled as vertices in the graph, with relationships between users and videos to capture user ratings of videos and to identify which user uploaded each video. Note that in addition to users and videos, we’ve also elected to model tags as vertices in the graph. This was done in order to support the ability to navigate relationships in the graph between movies with the same tag. We experimented with various recommendation algorithms, including some that leveraged movies with the same tag. As a starting point, I elected to use the same algorithm used by the Java implementation.</p><p>For the purposes of our application, the schema is actually written and communicated to DSE using a graph traversal language known as Gremlin, which is part of the <a href="http://tinkerpop.apache.org/">Apache TinkerPop</a> project. Gremlin is widely used in graph databases. Here’s the key portions of the schema that define the vertex and edge types and properties used in the graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*l8SVts334Fggi2DvXZ9-Kw.png" /><figcaption>Property, Vertex, and Edge labels used in the KillrVideo recommendation graph</figcaption></figure><p>Check out the KillrVideo GitHub for the <a href="https://github.com/KillrVideo/killrvideo-data/blob/master/graph/killrvideo_video_recommendations_schema.groovy">full schema</a> and the <a href="https://github.com/KillrVideo/killrvideo-dse-config">code that loads this schema</a> into DataStax Enterprise.</p><h3>Adding Data to the Graph</h3><p>Before we can generate recommendations, we need to write the code that populates the graph with data. In the <a href="https://medium.com/@jscarp/kafka-cassandra-like-peanut-butter-and-chocolate-267815b7ba22">previous post in this series</a> I introduced the code that processes events from Kafka , which you can find in <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/suggested_videos/suggested_videos_events_kafka.py">suggested_videos_events_kafka.py</a>. That event processing code makes calls into the implementation of the Suggested Videos Service (<a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/suggested_videos/suggested_videos_service.py">suggested_videos_service.py</a>), specifically the functions handle_user_created(), handle_youtube_video_added(), and handle_user_rated_video().</p><p>For example, the code to add a new user to the graph is quite simple:</p><pre>def handle_user_created(self, user_id, first_name, last_name, email, timestamp):<br><br>    self.graph.addV(&#39;user&#39;) \<br>        .property(&#39;userId&#39;, user_id).property(&#39;email&#39;, email) \<br>        .property(&#39;added_date&#39;, timestamp).next()</pre><p>The graph object is what is known as a “traversal source” to which we add operations in the traversal such as addV() to add a vertex and property() to specify the properties of the vertex. The next() operation is what is known as a terminal operator which indicates to the graph engine our desire to execute the traversal. (By the way, the best reference I’ve found to learn these Gremlin operations is the <a href="http://tinkerpop.apache.org/docs/current/reference/">Apache TinkerPop documentation</a>.)</p><p>The constructor for the class is where we instantiated the graph traversal source, providing a DseSession object and the name of the graph:</p><pre>def __init__(self, session):<br>    self.session = session<br>    <strong>self.graph = DseGraph.traversal_source(session=self.session, graph_name=&#39;killrvideo_video_recommendations&#39;)</strong><br>    self.suggested_videos_consumer = SuggestedVideosConsumer(self)</pre><h3>Accessing DataStax Graph in Python</h3><p>Despite the apparent simplicity of the sample above, I initially struggled with writing this code to insert the data into the graph. To explain why, let me take a step back. It’s helpful to understand that the Python driver supports different API styles for interacting with graphs, as described in the <a href="https://docs.datastax.com/en/developer/python-dse-graph/1.7/api/dse_graph/">DataStax documentation</a>:</p><ul><li>The DseSession.execute_graph() operation allows you to provide a Gremlin traversal as a string in the standard Groovy syntax that you might use in the gremlin-console or DataStax Studio. For example, the code to select a user might look like session.execute_graph(&quot;g.V().has(&#39;user&#39;, ‘userId&#39;, “ + user_id + “)”)This can be handy if you have simple traversals, but as you can see, building up the query strings can be a bit more unwieldy if you need to create larger traversals referencing multiple variables.</li><li>The Graph fluent API allows you to create a traversal source that is accessible directly in Python with a syntax that is quite similar to Groovy, but with some occasional differences, for example the code to select a user might look like: g.V().has(&#39;user&#39;, &#39;userId&#39;, user_id)</li><li>If you use the fluent API, you also have the option of developing a Domain-Specific Language (DSL) to create more natural-sounding graph traversals that might be more easily understood by those familiar with the application domain. For example, we might have something like g.user(&#39;Fred&#39;).rateVideo(&#39;video_id&#39;, 5), where operations like user() and rateVideo() operate on a traversal to add steps.</li></ul><p>The Java implementation of the KillrVideo graph recommender uses a DSL, so I considered using the same approach for the Python implementation. However, I felt the approach that would help me to learn DataStax Graph most effectively would be to use the fluent API first, and then consider factoring some of the traversal details into a DSL as a follow-up step.</p><h4>Leveraging the Python Console</h4><p>With that context, I can better explain the challenges I experienced in using the fluent API. First, I started out with a pretty naive development process:</p><ul><li>Study the Python fluent API syntax to locate the desired operations to build my traversal</li><li>Implement calls using the fluent API in suggested_videos_service.py</li><li>Restart the Python services</li><li>Run generator to create test data</li><li>Debug</li><li>Repeat</li></ul><p>This approach felt pretty slow. As a fairly new Python developer, I had failed to realize the value of testing out my syntax using the console. Once I realized the value of this, I quickly created for myself a quick list of Python commands that I could use to initialize the DataStax Python Driver and start executing traversals:</p><pre>from dse.cluster import Cluster, EXEC_PROFILE_GRAPH_DEFAULT<br>from dse_graph import DseGraph<br>ep = DseGraph.create_execution_profile(&#39;graph_name&#39;)<br>cluster = Cluster(execution_profiles={EXEC_PROFILE_GRAPH_DEFAULT: ep}, contact_points=[&#39;10.0.75.1&#39;])<br>session = cluster.connect(&quot;killrvideo&quot;)<br>graph = DseGraph.traversal_source(session=session, graph_name=&#39;killrvideo_video_recommendations&#39;)</pre><p>These few lines of code would give me a working graph traversal source that I could use at the console. This allowed me to get a tighter feedback loop on syntax errors in my usage of the fluent API.</p><h4>Leveraging DataStax Studio</h4><p>Using the Python console was definitely helpful, but I soon realized that another reason for my slow progress was that I was trying to tackle two learning curves at once. It wasn’t just that my Python syntax was wrong, it was that my Gremlin traversals had errors as well. So I took another step back and used DataStax Studio to write my traversals in Groovy to make sure I got them correct. Again, this shortened my development cycle.</p><p>I soon arrived at a a process I’d recommend, at least for newer graph users:</p><ul><li>Develop Gremlin traversals in Groovy using DataStax Studio or gremlin-console</li><li>Translate traversals into the fluent API in Python (or other chosen language), testing them out at the console (where available)</li><li>Move working traversals into application code</li></ul><h4>Gremlin Python API Caveats</h4><p>The step of translating traversals into the Python fluent API has a couple of tricks to it:</p><ul><li>It’s not immediately clear how to locate the correct package for each operation you might want to perform in a traversal.</li><li>There are some reserved words in Python such as not, as, or, and others. In these cases, the operator names have an underscore appended (not_, as_, or_, etc.)</li></ul><p>Fortunately I was able to find the answers to these questions in the <a href="http://tinkerpop.apache.org/docs/current/reference/#gremlin-python">gremlin-python documentation</a> from the Apache TinkerPop project.</p><h3>More complex data insertion</h3><p>My improved development process served me well for adding new videos to the graph. As you may recall from the graph schema shown above, videos are represented as vertexes in the graph, with edges to users and tags. Adding a video to the graph actually involves creation of multiple edges and vertexes:</p><ol><li>Creating the vertex for the video</li><li>Locating the vertex representing the user and creating an “uploaded” edge to the video vertex</li><li>Locating or creating the vertexes representing each tag for the video, and creating “taggedWith” edges from the video to each tag</li></ol><p>My initial implementation approach for adding a video was to create a separate graph traversal for each stage of the process. However, as I looked more at the equivalent code in the Java implementation, I realized that the “Gremlin way” to approach this was as a single traversal. It actually turned out to be easier to develop this way (although I did take advantage of the fluent API to build things up a bit at a time, it is a single traversal). Part of this was due to the ability to locate particular vertexes in the graph and then use the as_() operator to give them names that could be referenced later:</p><pre>def handle_youtube_video_added(self, video_id, user_id, name, <br>                               description, location, <br>                               preview_image_location,<br>                               tags, added_date, timestamp):</pre><pre>    <em># make sure tags are unique (no duplicates)<br>    </em>unique_tags = set(tags)<br><em><br>    # locate user vertex<br>    </em>traversal = self.graph.V() \<br>                    .has(&#39;user&#39;, &#39;userId&#39;, user_id).<strong>as_(&#39;^user&#39;)</strong><br><br>    <em># add video vertex<br>    </em>traversal = traversal.addV(&#39;video&#39;) \<br>        .property(&#39;videoId&#39;, video_id)\<br>        .property(&#39;added_date&#39;, added_date) \<br>        .property(&#39;description&#39;, description) \<br>        .property(&#39;name&#39;, name) \<br>        .property(&#39;preview_image_location&#39;, preview_image_location)\<br>        <strong>.as_(&#39;^video&#39;)</strong><br><br>    <em># add edge from user to video vertex<br>    </em>traversal = traversal.addE(&#39;uploaded&#39;) \<br>        <strong>.from_(&#39;^user&#39;).to(&#39;^video&#39;)</strong> \<br>        .property(&#39;added_date&#39;, added_date)<br><br>    <em># find vertices for tags and add edges from video vertex<br>    </em>for tag in unique_tags:<br>        traversal = traversal.addE(&#39;taggedWith&#39;) \<br>            <strong>.from_(&#39;^video&#39;)</strong>.to(__.coalesce(<br>                __.V().has(&#39;tag&#39;, &#39;name&#39;, tag),<br>                __.addV(&#39;tag&#39;).property(&#39;name&#39;, tag) \    <br>                              .property(&#39;tagged_date&#39;, added_date)))<br><br>    <em># execute the traversal<br>    </em>traversal.iterate()</pre><p>Another very interesting section of this traversal is the use of the coalesce() operation, which is an operation that takes multiple traversals as input and returns the first traversal that generates a result. In this case, we first attempt to locate a given tag by name (__.V().has(‘tag’, ‘name’, tag)), and if the tag does not exist, we create it (__.addV(‘tag’)...).</p><p>On this occasion, we use iterate() as our terminal operator to cause the traversal to execute.</p><p>Finally, you may have noticed the unusual looking operator __. This is known as an anonymous traverser and is what we use as the starting point for new traversals we might need to create within a larger traversal.</p><h3>Generating recommendations</h3><p>Once the code was implemented to insert users, videos, and tags into the graph, it was finally time to generate recommendations from the contents of the graph. The first operation to implement is get_suggested_for_user, which provides video recommendations for a user given their user_id.</p><p>Fortunately, I did not have to invent a new recommendation algorithm. In fact, I was able to reuse a DataStax Studio notebook that David gave me containing a couple of different traversals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Te349GcfsMkc1eUC80AO2g.png" /><figcaption>DataStax Studio notebook containing various graph-based recommendation algorithms</figcaption></figure><p>The basic idea of the algorithm I chose is as follows:</p><ul><li>locate the user vertex for the provided ID</li><li>select videos that user gave high ratings (in this case, 4 or 5)</li><li>select “similar users” that also rated the same videos highly</li><li>sample other videos that were highly rated by those similar users, sorting by those with the highest average rating</li></ul><p>As it turns out, the task of translating the algorithm into Python was fairly simple:</p><pre>def get_suggested_for_user(self, user_id, page_size, paging_state):<br>    <em><br>    </em>traversal = self.graph.V().has(&#39;user&#39;, &#39;userId&#39;, user_id).as_(&#39;^user&#39;) \<br>        .map(__.out(&#39;rated&#39;).dedup().fold()).as_(&#39;^watchedVideos&#39;) \<br>        .select(&#39;^user&#39;) \<br>        .outE(&#39;rated&#39;).has(&#39;rating&#39;, gte(MIN_RATING)).inV() \<br>        .inE(&#39;rated&#39;).has(&#39;rating&#39;, gte(MIN_RATING)) \<br>        .sample(NUM_RATINGS_TO_SAMPLE).by(&#39;rating&#39;).outV() \<br>        .where(neq(&#39;^user&#39;)) \<br>        .local(__.outE(&#39;rated&#39;).has(&#39;rating&#39;, gte(MIN_RATING)).limit(LOCAL_USER_RATINGS_TO_SAMPLE)) \<br>        .sack(Operator.assign).by(&#39;rating&#39;).inV() \<br>        .filter(__.in_(&#39;uploaded&#39;).hasLabel(&#39;user&#39;)) \<br>        .group().by().by(__.sack().sum()) \<br>        .order(Scope.local).by(Column.values, Order.decr) \<br>        .limit(Scope.local, NUM_RECOMMENDATIONS).select(Column.keys).unfold() \<br>        .project(&#39;video_id&#39;, &#39;added_date&#39;, &#39;name&#39;, &#39;preview_image_location&#39;, &#39;user_id&#39;) \<br>        .by(&#39;videoId&#39;).by(&#39;added_date&#39;).by(&#39;name&#39;).by(&#39;preview_image_location&#39;).by(__.in_(&#39;uploaded&#39;).values(&#39;userId&#39;))<br><br>    results = traversal.toList()</pre><p>Rather than explaining the entire traversal here, I’ll refer you to the <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/suggested_videos/suggested_videos_service.py#L97">detailed comments in the source code</a>.</p><p>There’s a second recommendation operation in the Suggested Videos Service, get_related_videos(). This operation is similar, but the starting point is a specific video. I diverged from emulating the Java implementation at this point, which used DSE Search, and chose to use a graph-based algorithm similar to the one for recommending videos for a user.</p><h3>Evaluating results and next steps</h3><p>Finally, I had a working graph recommender in KillrVideo Python!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*gxYynbK5Usw5QKqZ7Zthvg.png" /><figcaption>Output from the graph recommender shows up under “Recommended for You” in the KillrVideo home screen</figcaption></figure><p>Results are great, but this algorithm isn’t perfect. One of the observations that we had is that there needs to be a reasonable amount of data (users, videos, ratings) in the graph for the algorithm to generate meaningful results. Thankfully we have a <a href="https://github.com/KillrVideo/killrvideo-generator">sample data generator</a> that we frequently run as part of our testing, but it does take a few minutes to generate a good amount of data and doesn’t really help us pass the <a href="https://github.com/KillrVideo/killrvideo-integration-tests">integration tests</a> for the Suggested Videos Service.</p><p>Another issue: the algorithm also won’t generate any results for a user that hasn’t rated any videos, which is a problem for new users. An improvement I’m considering is to have a very simple fallback algorithm that always generates results, for example just recommending popular videos that are rated highly by users.</p><p>Lastly, some of the graph experts we’ve shared this with have offered suggestions on how we can improve the performance of the traversal. Given all this, let’s consider this implementation a minimum viable product (MVP)!</p><h3>Next Time</h3><p>With the completion of the graph recommender, I’ve now got a full implementation of the basic capability of the KillrVideo Python services. There are a couple of small things left like adding authentication from the services to DataStax Enterprise. In the final post in this series I’ll take a step back and summarize lessons learned from this effort.</p><p><strong>Next post in this series: </strong><a href="https://medium.com/@jscarp/lessons-learned-on-my-first-python-app-34b276698467?source=your_stories_page---------------------------"><strong>Lessons Learned on my first Python app</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb264a094655" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka + Cassandra — like peanut butter and chocolate?]]></title>
            <link>https://jscarp.medium.com/kafka-cassandra-like-peanut-butter-and-chocolate-267815b7ba22?source=rss-343198ddcf60------2</link>
            <guid isPermaLink="false">https://medium.com/p/267815b7ba22</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[app-development]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Jeff Carpenter]]></dc:creator>
            <pubDate>Mon, 18 Mar 2019 22:44:38 GMT</pubDate>
            <atom:updated>2019-08-27T22:36:33.897Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kafka + Cassandra — like peanut butter and chocolate?</h3><h4>Part 7 of a series on the KillrVideo Python implementation</h4><p>This is the latest installment of a series about building a Python application with Apache Cassandra — specifically a Python implementation of the <a href="http://killrvideo.github.io/">KillrVideo</a> microservice tier. In the first few posts I shared about <a href="https://medium.com/datadriveninvestor/killrvideo-python-pt-1-the-backstory-5c38191fb330">why I started this project</a>, how I incorporated <a href="https://medium.com/@jscarp/python-app-dev-with-protobuf-and-grpc-e5bff779783d">GRPC</a> and <a href="https://medium.com/@jscarp/advertising-python-services-via-etcd-aa1eca6d8ecc">Etcd</a>, and the <a href="https://medium.com/@jscarp/who-needs-unit-tests-im-building-microservices-4c8fe40d7095">testing approach</a>. In the most recent two posts I’ve taken you on a guided tour of the data access code for most of the KillrVideo Python services, including <a href="https://medium.com/@jscarp/when-data-access-is-the-easiest-part-of-a-microservice-c7a90dee701a">simple</a> examples using the cqlengine mapper, and <a href="https://medium.com/@jscarp/cassandra-with-python-simple-to-complex-44298391867d">more complex</a> Cassandra data access using other features of the DataStax Python Driver.</p><p>The final service to implement is the Suggested Videos Service, which includes a recommendation engine built using DataStax Enterprise Graph. In this post, we’ll focus about how we share data between services using Kafka in order to populate data into the graph used for recommendations.</p><h3>Why Kafka for KillrVideo?</h3><p>The first question to address is why I chose to use Kafka. In fact, a previous version of the KillrVideo <a href="http://killrvideo.github.io/">project documentation</a> actually suggested that an in-memory messaging implementation would be preferable. Of course, this statement was predicated on the idea that the KillrVideo services would all be run in the same process. Since I harbor a not-so-secret agenda to have flexible deployment for KillrVideo, where we can do cool things like combine service implementations from various languages and scale each service independently, an in-memory solution is not necessarily the best long term approach.</p><p>I chose Kafka as the messaging solution because it embodies many of the same distributed systems values as Cassandra — for example, scalability and high availability, and because Cassandra and Kafka are technologies that complement each other well. An additional reason is that I hadn’t worked with Kafka much and wanted to get my hands dirty!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*CKoK5MImGYlSMSObEwftUQ.png" /><figcaption>Cassandra and Kafka — Like Chocolate and Peanut Butter!</figcaption></figure><h4>Patterns for integrating Kafka and Cassandra</h4><p>Let’s take a quick pause and expand on this point of how Cassandra and Kafka can complement each other. I discussed a few patterns for combining these two technologies with <a href="https://twitter.com/tlberglund">Tim Berglund</a> on a recent episode of the Distributed Data Show:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F3e5GrJBy4ik%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D3e5GrJBy4ik&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F3e5GrJBy4ik%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d59b4de4b5639382d9e899401466261/href">https://medium.com/media/9d59b4de4b5639382d9e899401466261/href</a></iframe><p>To summarize, these are the basic patterns we discussed:</p><ul><li>Using Kafka as an event fabric between microservices. In this approach, a service consumes events from a Kafka stream, performs computation on the events, and produces new Kafka events and/or writes data to Cassandra. The service may optionally use data stored in Cassandra as part of the event processing.</li><li>Using Cassandra as a sink for Kafka. In this approach, data from Kafka streams is ingested into Cassandra, for example using the DataStax Kafka Connector. This is useful when you want to leverage Cassandra’s multi-DC replication or when you need to query across multiple events rather than processing each in sequence.</li><li>Using Kafka for Change Data Capture from Cassandra. This approach leverages the Kafka Connect framework to perform CDC from Cassandra or other databases via plugins.</li></ul><p>The first pattern is the one used for the recommendation engine in KillrVideo.</p><h3>Integrating Kafka into KillrVideo</h3><p>So, let’s talk through the process of how we integrated Kafka into KillrVideo. The architecture calls for some of the services to emit events when specific actions occur:</p><ul><li>The Video Catalog Service produces a YouTubeVideoAdded event when a video is added to the system</li><li>The User Management Service produces a UserCreated event when a new user is registered</li><li>The Ratings Service produces a UserRatedVideo event when a user rates a video</li><li>The Comments Service produces a UserCommentAdded event when a user comments on a video</li></ul><p>The recommendation algorithm used in the Java implementation of the Suggested Videos Service leverages the first three events to make recommendations based on users, videos and ratings. The algorithm currently doesn’t use comments, but could potentially do that in the future. For example, perhaps we could incorporate sentiment analysis of user comments in our recommendations. Something fun to explore!</p><p>Here’s a quick overview of the architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J0U5hGFXFTXVMLiPz7fZXA.png" /></figure><h4>Running Kafka in Docker</h4><p>First, let’s get Kafka running. For a minimal Kafka deployment, we’ll need to run an instance of Zookeeper and a single Kafka node. We’ll leverage the docker containers built by <a href="https://github.com/wurstmeister">Wurstmeister</a> and add containers to our <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/docker-compose.yaml">docker-compose.yaml</a> file (note: we’re in the process of moving the KillrVideo Docker scripts to a new “<a href="https://github.com/KillrVideo/killrvideo-all-in-one">all-in-one</a>” repository, so the link to the compose file may soon break):</p><pre><em># Ensure consistency in Kafka World<br></em><strong>zookeeper:<br>  image: </strong>wurstmeister/zookeeper<br>  <strong>ports:<br>    </strong>- 2181:2181<br><br><em># Start the Kafka Engine for Killrvideo<br></em><strong>kafka:<br>  image: </strong>wurstmeister/kafka<br>  <strong>depends_on:<br>    </strong>- zookeeper<br>  <strong>environment:<br>    KAFKA_ADVERTISED_HOST_NAME: </strong>localhost<br>    <strong>KAFKA_ADVERTISED_PORT: </strong>9092<br>    <strong>KAFKA_ZOOKEEPER_CONNECT: </strong>zookeeper:2181<br>    <strong>KAFKA_CREATE_TOPICS: &quot;topic-kv-videoCreation:1:1,topic-kv-userCreation:1:1,topic-kv-videoRating:1:1,topic-kv-commentCreation:1:1&quot;<br>  ports:<br>    </strong>- 9092:9092</pre><p>Note the use of the KAFKA_CREATE_TOPICS environment variable in the definition of the kafka container. This is a handy shortcut for us to create the topics that the KillrVideo services will be using.</p><h4>My first Kafka error</h4><p>One funny note is that when I first started using this method of topic creation, I cut-and-paste some code from another project and defined the topics using the compact flag:</p><pre>KAFKA_CREATE_TOPICS: &quot;topic-kv-videoCreation:1:1:<strong>compact</strong>,topic-kv-userCreation:1:1:<strong>compact</strong>,topic-kv-videoRating:1:1:<strong>compact</strong>&quot;</pre><p>Because I wasn’t familiar with the notation, I failed to realize that Kafka supports optional key-value semantics for events. Setting the compact flag tells Kafka that you are requesting it to only keep the latest value for a given key. Since the KillrVideo event design doesn’t use the concept of a key, only a value, I started receiving error messages on my first attempt to publish events:</p><pre>DEBUG:kafka.producer.record_accumulator:Produced messages to topic-partition TopicPartition(topic=&#39;topic-kv-videoRating&#39;, partition=0) with base offset -1 and error &lt;class &#39;kafka.errors.CorruptRecordException&#39;&gt;.</pre><p>The CorruptRecordException is caused by not including a key when publishing an event to a topic with the compact option set. Changing the topic definition to not use the compact option fixed this problem.</p><h4>Installing the Kafka Client</h4><p>To get started using the Kafka client, install the Python package:</p><pre>pip install kafka-python</pre><h4>Defining Event Payloads</h4><p>We’ll need to define some common event payloads that can be exchanged between services. Thankfully, we already have Protobuf event definitions as part of the <a href="https://github.com/KillrVideo/killrvideo-service-protos">killrvideo-service-protos</a> repository. Let’s look at the event defined in the <a href="https://github.com/KillrVideo/killrvideo-service-protos/blob/master/src/ratings/ratings_events.proto">ratings_events.proto</a> file for example:</p><pre>message <strong>UserRatedVideo</strong> {<br>  killrvideo.common.Uuid <strong>video_id</strong> = 1;<br>  killrvideo.common.Uuid <strong>user_id</strong> = 2;<br>  int32 <strong>rating</strong> = 3;<br>  google.protobuf.Timestamp <strong>rating_timestamp</strong> = 4;<br>}</pre><p>I described in a <a href="https://medium.com/@jscarp/python-app-dev-with-protobuf-and-grpc-e5bff779783d">previous post</a> how we used the grpcio generator to create stubs for our Python services. This same process also generated Python classes representing the messages in our Protobuf event files. For example, you can find the Python representation for the UserRatedVideo event defined above in the file <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/ratings/ratings_events_pb2.py">ratings_events_pb2.py</a>.</p><p>By the way, I found this article to be a helpful reference on using Protobuf events with Kafka: <a href="https://medium.com/@brynmathias/kafka-and-google-protobuf-a-match-made-in-python-a1bc3381da1a">https://medium.com/@brynmathias/kafka-and-google-protobuf-a-match-made-in-python-a1bc3381da1a</a>.</p><h4>Producing Events</h4><p>As mentioned above, there are several services in KillrVideo that publish events. I created separate classes for event publishing with Kafka in each of these services. For example, you can find the RatingPublisher class in the file <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/ratings/ratings_events_kafka.py">ratings_events_kafka.py</a>. The constructor for this class creates a KafkaProducer. Since we’re only using a single Kafka node in this simple deployment, we pass its address as the bootstrap server:</p><pre><strong>class </strong>RatingPublisher(object):<br><em><br>    </em><strong>def </strong>__init__(self):<br>        self.producer =KafkaProducer(<br>              bootstrap_servers=<strong>&#39;10.0.75.1:9092&#39;</strong>,<br>              client_id=<strong>&#39;killrvideo-python:RatingsService&#39;</strong>)</pre><p>The primary method of this class is responsible. This method is called by the main service implementation in <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/ratings/ratings_service.py">ratings_service.py</a> after it completes adding a new user rating. This method creates a Protobuf representation of the event using the parameters provided, <a href="https://www.datadoghq.com/blog/engineering/protobuf-parsing-in-python/">serializes the event contents to a string</a>, and sends a message on the UserRatedVideo topic:</p><pre><strong>def </strong>publish_user_rated_video_event(self, video_id, user_id, <br>                                   rating, timestamp):<br>    event = UserRatedVideo(video_id=UUID_to_grpc(video_id), <br>                           user_id=UUID_to_grpc(user_id),<br>                           rating=rating,<br>                           rating_timestamp=datetime_to_Timestamp(<br>                               timestamp))<br><br>    serialized_event = event.SerializeToString()<br><br>    self.producer.send(topic=USER_RATED_VIDEO_TOPIC,                        <br>                       value=serialized_event)</pre><p>The process of producing the other events is quite similar. Once we have events being produced successfully, it’s helpful to be able to see the event contents. I recommend using something like <a href="http://www.kafkatool.com/index.html">Kafka Tool</a> in order to verify the events are flowing into topics as you expect.</p><h4>Consuming Events</h4><p>Now, let’s look at the other side of things — consuming events. The Suggested Videos Service is responsible for receiving events and populating data in a graph which can later be used for producing recommendations. For now we’ll focus just on the event processing. The SuggestedVideoConsumer class is found in the file <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/suggested_videos/suggested_videos_events_kafka.py">suggested_videos_events_kafka.py</a>.</p><pre><strong>class </strong>SuggestedVideosConsumer(object):<br><em><br>    </em><strong>def </strong>__init__(self, suggested_videos_service):<br>        self.suggested_videos_service = suggested_videos_service<br>        self.consumer = SuggestedVideoKafkaConsumer(self)<br>        self.consumer.start()</pre><p>The constructor for this class takes in a reference to the SuggestedVideosService implementation and creates a SuggestedVideoKafkaConsumer to which it can delegate the job of handling events. The SuggestedVideoKafkaConsumer contains our event processing loop. Based on <a href="https://github.com/dpkp/kafka-python/blob/master/example.py">a useful example in the </a><a href="https://github.com/dpkp/kafka-python/blob/master/example.py">kafka-python library documentation</a>, I decided to create a separate event handling loop, with the exception that I used <a href="https://medium.com/@nbosco/multithreading-vs-multiprocessing-in-python-c7dc88b50b5b">multithreading instead of multiprocessing</a> [1]:</p><pre><strong>class </strong>SuggestedVideoKafkaConsumer(threading.Thread):<br>    <strong>def </strong>__init__(self, suggested_videos_consumer):<br>        threading.Thread.__init__(self)<br>        self.suggested_videos_consumer = suggested_videos_consumer</pre><p>The run() operation of this thread consumes events from all three topics of interest and delegates the processing to the SuggestedVideoConsumer class:</p><pre><strong>def </strong>run(self):<br>    consumer = KafkaConsumer(bootstrap_servers=BOOTSTRAP_SERVERS,<br>              client_id=<strong>&#39;killrvideo-python:SuggestedVideosService&#39;</strong>,<br>              fetch_max_wait_ms=10000) <em><br><br>    </em>consumer.subscribe([USER_CREATED_TOPIC, <br>       USER_RATED_VIDEO_TOPIC, YOUTUBE_VIDEO_ADDED_TOPIC])<br><br>    <strong>for </strong>event <strong>in </strong>consumer:<br>        logging.debug(event)<br>        <strong>if </strong>event.topic == USER_CREATED_TOPIC:<br>            self.suggested_videos_consumer.process_user_created(<br>                event.value)<br>        <strong>elif </strong>event.topic == USER_RATED_VIDEO_TOPIC:<br>            self.suggested_videos_consumer.process_user_rated_video(<br>                event.value)<br>        <strong>elif </strong>event.topic == YOUTUBE_VIDEO_ADDED_TOPIC:<br>         self.suggested_videos_consumer.process_youtube_video_added(<br>                event.value)</pre><p>The process_* methods deserialize the message payloads into the original Prototbuf types and call operations on the SuggestedVideosService to perform the actual business logic.</p><h3>Next time: building a graph recommender</h3><p>That’s quite enough to cover for one post, but I’m sure you’re waiting for the payoff — the graph recommender. Next time, I’ll discuss how we use data from the events in order to populate a graph, which we can then use to implement a recommendation engine in the Suggested Videos Service.</p><p><strong>Next post in this series: </strong><a href="https://medium.com/@jscarp/the-long-sprint-building-a-graph-recommender-cb264a094655?source=your_stories_page---------------------------"><strong>The long sprint — building a Graph Recommender</strong></a></p><p>[1]: This article was corrected on 3/22/2019 to reflect that I updated the code to usethreading.Thread instead of multiprocessing.Process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=267815b7ba22" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cassandra with Python: Simple to Complex]]></title>
            <link>https://jscarp.medium.com/cassandra-with-python-simple-to-complex-44298391867d?source=rss-343198ddcf60------2</link>
            <guid isPermaLink="false">https://medium.com/p/44298391867d</guid>
            <category><![CDATA[search]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[app-development]]></category>
            <category><![CDATA[killrvideo]]></category>
            <category><![CDATA[cassandra]]></category>
            <dc:creator><![CDATA[Jeff Carpenter]]></dc:creator>
            <pubDate>Fri, 01 Mar 2019 03:32:28 GMT</pubDate>
            <atom:updated>2019-08-27T22:35:34.531Z</atom:updated>
            <content:encoded><![CDATA[<h4>Pt. 6 of a series on the KillrVideo Python project</h4><p>This is the latest installment of a series about building a Python application with Apache Cassandra — specifically a Python implementation of the <a href="http://killrvideo.github.io/">KillrVideo</a> microservice tier. In previous posts I shared <a href="https://medium.com/datadriveninvestor/killrvideo-python-pt-1-the-backstory-5c38191fb330">what motivated this project</a>, how I started with infrastructure including <a href="https://medium.com/@jscarp/python-app-dev-with-protobuf-and-grpc-e5bff779783d">GRPC</a> and <a href="https://medium.com/@jscarp/advertising-python-services-via-etcd-aa1eca6d8ecc">Etcd</a>, the <a href="https://medium.com/@jscarp/who-needs-unit-tests-im-building-microservices-4c8fe40d7095">testing approach</a>, and most recently, how I began implementing <a href="https://medium.com/@jscarp/when-data-access-is-the-easiest-part-of-a-microservice-c7a90dee701a">data access using Cassandra</a>.</p><p>In this post we’ll look at some additional examples of data access using the DataStax Python Driver, ranging from the simple to the complex. (I’ll be making reference some driver / mapper concepts from the previous post so you may want to review that if you haven’t already.)</p><h3>Keeping it simple</h3><p>In the last post I shared how easy it was to implement the data access for the User Management Service using Apache Cassandra and the DataStax Python Driver. Using the <a href="https://datastax.github.io/python-driver/object_mapper.html">cqlengine</a> mapper was a big factor in my productivity. Because of that productivity boost, when implementing the other services I started with the mapper and only used other approaches when the complexity of the data access required it.</p><p>In fact, a couple additional services were implemented entirely using the mapper: the Statistics Service and the Ratings Service. These demonstrate Cassandra features including counters and batches.</p><h4>Simple example 1: Counters in the Statistics Service</h4><p>The Statistics Service (<a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/statistics/statistics_service.py">statistics_service.py</a>) stores counts of how many time each video has been viewed. Counting statistics is one of the relatively few use cases for which the <a href="https://docs.datastax.com/en/cql/3.3/cql/cql_reference/counter_type.html">Cassandra counter type</a> is a good fit, because and therefore a good example of how to manipulate counters with the mapper. The model class we used to track statistics is shown here:</p><pre><strong>class </strong>VideoPlaybackStatsModel(Model):<br>    <em>&quot;&quot;&quot;Model class that maps to the video_playback_stats table&quot;&quot;&quot;<br>    </em>__table_name__ = <strong>&#39;video_playback_stats&#39;<br>    </strong>video_id = columns.UUID(primary_key=True, db_field=<strong>&#39;videoid&#39;</strong>)<br>    views = columns.Counter()</pre><p>Note the use of the columns.Counter() type for the views column.</p><p>Incrementing the count of views in the record_playback_started() operation is very simple:</p><pre>VideoPlaybackStatsModel(video_id=video_id).update(views=1)</pre><p>Note that the value views=1 represents a single view of the video.</p><h4>Simple example 2: Batch Writes in the Ratings Service</h4><p>The Ratings Service (<a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/ratings/ratings_service.py">ratings_service.py</a>) stores user ratings of videos and allows retrieval of the ratings either by user or by video. One interesting aspect of this service is the use of the mapper as part of a <a href="https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlBatch.html">Cassandra batch</a> when storing ratings in order to support writes to two denormalized tables supporting the “by user” and “by video” queries mentioned above. The code block below shows the data access code from the rate_video() operation:</p><pre>now = datetime.utcnow()<br><br><em># create and execute batch statement to insert into multiple tables<br></em>batch_query = BatchQuery(timestamp=now)</pre><pre>VideoRatingsByUserModel.batch(batch_query)\<br>    .create(video_id=video_id, user_id=user_id, rating=rating)</pre><pre><em># updating counter columns rating_counter and rating_total <br># values are interpreted as amount to increment<br></em>VideoRatingsModel(video_id=video_id).\<br>    update(rating_counter=1, rating_total=rating).batch(batch_query)<br><br>batch_query.execute()</pre><p>In this example, we use two different mapper classes to write to two different tables. Note the use of each mapper’s batch() operation to add a statement to the batch_query, which we can then execute. The first statement created is a CQLINSERT into the killrvideo.video_ratings_by_user via the VideoRatingsByUserModel.</p><p>If you’re looking closely may you have also noticed something about that second write — we’re using the VideoRatingsModel, which goes to the killrvideo.video_ratings table to do a CQLUPDATE. This table is another interesting use of counters. In this case, two counter columns are used. The rating_counter tracks the number of times a video has been rated, while the rating_total tracks the sum of all ratings for the video. The average rating can then be calculated by the client via simple division (rating_total/rating_counter).</p><h3>When simple doesn’t cut it</h3><p>There were a few other services that involved more complex data access where the mapper couldn’t fully address my needs. These included the Video Catalog Service, the Comments Service, and the Search Service.</p><h4>Complex example 1: Paging in the Comments Service</h4><p>Although it’s not “YouTube scale”, the KillrVideo application is designed to support a very large number of videos, users, ratings, and other data types, in order to demonstrate best practices for data modeling and driver usage found in real-world applications.</p><p>With this in mind, imagine a user being presented with screens of videos and comments in a web browser on a client device. It would be infeasible and unnecessary to return the entire video catalog or the entire comment history for a popular video to the client at once, so of course some form of paging is required. However, paging is a feature that isn’t supported by the cqlengine mapper provided with the DataStax Python Driver. So, we need to fall back to other methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*vLsuvnQUSnBZc5gsdSx6HA.png" /><figcaption>Paging in the KillrVideo Comments Service is used by the web application</figcaption></figure><p>A good example of the approach is found in the Comments Service (<a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/comments/comments_service.py">comments_service.py</a>). Since paging only comes into effect on reads, we can use the mapper for writes, and then use regular CQL statements on the reads to get control over paging.</p><p>As an example, to access comments by user, we first create prepared statements in the constructor for the class:</p><pre><em># Prepared statements for get_user_comments()<br></em>self.userComments_startingPointPrepared = \<br>    session.prepare(<strong>&#39;SELECT </strong><em>*</em><strong> FROM comments_by_user WHERE userid = ? AND (commentid) &lt;= (?)&#39;</strong>)<br>self.userComments_noStartingPointPrepared = \<br>    session.prepare(<strong>&#39;SELECT </strong><em>*</em><strong> FROM comments_by_user WHERE userid = ?&#39;</strong>)</pre><p>Then, in the get_user_comments() operation, we use one of the prepared statements to create a bound statement which we can then execute:</p><pre>bound_statement = None<br><br><strong>if </strong>starting_comment_id:<br>    bound_statement = self.userComments_startingPointPrepared<br>        .bind([user_id, starting_comment_id])<br><strong>else</strong>:<br>    bound_statement = self.userComments_noStartingPointPrepared<br>        .bind([user_id])<br><br>bound_statement.fetch_size = page_size<br>result_set = None<br><br><strong>if </strong>paging_state:<br>    <em># see below where we encode paging state to hex before returning<br>    </em>result_set = self.session.execute(bound_statement, <br>        paging_state=paging_state.decode(<strong>&#39;hex&#39;</strong>))<br><strong>else</strong>:<br>    result_set = self.session.execute(bound_statement)</pre><p>Note where the fetch_size of the batch statement is set to the page_size requested by the client. The code after this iterates over the rows in the result_set to build up a list of comments to return to the client. If page_size rows were returned, then the Cassandra paging state is extracted from the result set and returned to the client:</p><pre><strong>if </strong>len(results) == page_size:<br>    <em># Use hex encoding since paging state is raw bytes that won&#39;t encode to UTF-8<br>    </em>next_page_state = result_set.paging_state.encode(<strong>&#39;hex&#39;</strong>)</pre><p>This allows the client to pass back the paging state on a subsequent call to get_user_comments() to retrieve the next page. Note that we encode/decode the paging state to a hex string, which works well over the Protobuf message format we’re using on our service interfaces.</p><p>The Video Catalog Service follows a similar paging approach, with an interesting twist. To read about this and some other nuances of paging in KillrVideo, check out my <a href="https://academy.datastax.com/content/paging-killrvideo">previous post</a> on this topic. Although that post uses examples from the Java implementation of the microservice tier, the basic algorithms described in that post are also used in the Python implementation.</p><h4>Complex example 2: Full-text Search in the Search Service</h4><p>The Search Service presents a different sort of problem. If you’re familiar with Cassandra data modeling practices, you’ll be aware that Cassandra doesn’t support arbitrary searches, and the secondary index implementation that comes with Cassandra is known to perform poorly over large data sets.</p><p>Instead, the best practice is to design a table per query, with primary keys based on the attributes you will specify in each query. However, your options when you don’t know the exact key you are looking for, with limited support for range queries and no support for text search features like prefix/postfix or fuzzy matching.</p><p>However, we do have a couple of cases where we need text search features in the Search Service, in both the get_query_suggestions() operation, which performs a typeahead search for common search terms, and the search_videos() operation, which is used to search for videos containing a search term in the title or description.</p><p>Following the pattern set in the other language implementations of the KillrVideo services, the Python implementation of the Search Service uses DataStax Enterprise Search. First, we need a search index on the videos table (extracted from the CQL for the <a href="https://github.com/KillrVideo/killrvideo-data/blob/master/search/videos_search.cql">search schema</a>):</p><pre>CREATE SEARCH INDEX IF NOT EXISTS on videos;</pre><p>Then, we create a prepared statement in the constructor of the Search Service:</p><pre>self.search_videos_prepared = \<br>    session.prepare(<strong>&#39;SELECT </strong><em>*</em><strong> FROM videos WHERE solr_query = ?&#39;</strong>)</pre><p>Finally, in the search_videos() operation, we use the solr_query syntax supported by DSE Search to create a bound statement containing our desired CQL query:</p><pre>solr_query = <strong>&#39;{&quot;q&quot;:&quot;name:(&#39; </strong>+ query + <strong>&#39;)^4 OR tags:(&#39; </strong>+ query + \<br>    <strong>&#39;)^2 OR description:(&#39; </strong>+ query + <strong>&#39;)&quot;, &quot;paging&quot;:&quot;driver&quot;}&#39;<br></strong>bound_statement = self.search_videos_prepared.bind([solr_query])</pre><p>We can then execute the bound statement and iterate over the results (not shown).</p><p>The solr_query defined above searches for the provided search term (query) in the name, tags, or description columns. The solr_query places additional weight on the name and description columns so that movies with the query appearing in the name column will appear the highest in search results, followed by those that have the search term in the description column.</p><h3>And now for something completely different</h3><p>In the past two posts I’ve taken you on a guided tour of the data access code for most of the KillrVideo Python services. The remaining service we haven’t discussed yet is the Suggested Videos Service. As you may have guessed, this involves building a recommender, which we’re doing using DataStax Enterprise Graph since that is built on top of Cassandra.</p><p>In upcoming posts I’ll share about this recommender, starting with how we shared data between services using Kafka in order to populate data into a graph used for recommendations.</p><p><strong>Next post in this series: </strong><a href="https://medium.com/@jscarp/kafka-cassandra-like-peanut-butter-and-chocolate-267815b7ba22?source=your_stories_page---------------------------"><strong>Kafka + Cassandra — like peanut butter and chocolate?</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44298391867d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When data access is the easiest part of a microservice]]></title>
            <link>https://medium.com/quick-code/when-data-access-is-the-easiest-part-of-a-microservice-c7a90dee701a?source=rss-343198ddcf60------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7a90dee701a</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[killrvideo]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[cassandra]]></category>
            <dc:creator><![CDATA[Jeff Carpenter]]></dc:creator>
            <pubDate>Thu, 21 Feb 2019 19:46:01 GMT</pubDate>
            <atom:updated>2019-08-27T22:34:53.009Z</atom:updated>
            <content:encoded><![CDATA[<h4>Pt. 5 of a series on the KillrVideo Python project</h4><p>Over the first few posts of this series, I’ve been sharing about my experience building a Python implementation of the <a href="http://killrvideo.github.io/">KillrVideo</a> microservice tier. In the previous posts I shared <a href="https://medium.com/datadriveninvestor/killrvideo-python-pt-1-the-backstory-5c38191fb330">why I started this project</a>, about <a href="https://medium.com/@jscarp/python-app-dev-with-protobuf-and-grpc-e5bff779783d">building GRPC service stubs</a>, advertising the endpoints in etcd, and <a href="https://medium.com/@jscarp/who-needs-unit-tests-im-building-microservices-4c8fe40d7095">setting up integration tests</a> to exercise the service APIs.</p><p>So it took four posts to describe all of the setup prior to implementing any business logic or data access code! All that work made me wonder if maybe the rest would be easy in comparison.</p><p>This post is all about writing the business logic and reading and writing data to DataStax Enterprise (DSE) using the Cassandra Query Language (CQL).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*JVWEq-xpNw5LyIFZI6IH4A.png" /></figure><h3>Business logic? What business logic?</h3><p>As it turns out, there’s really not a lot of business logic to speak of in the KillrVideo service tier. A previous version of the system that allowed uploading of actual video files via an Uploads Service certainly had more interesting business logic.</p><p>However, in the current situation, most of the business logic is validation code. There are a couple of locations where a minor amount of work is done with the provided inputs, for example:</p><ul><li>the AddYouTubeVideo operation in the Video Catalog Service infers the location of the video preview image from the URL of a YouTube video being added to the system.</li><li>converting a password to a md5 hashed value, and comparing hashed password strings to support user login, as in the User Management Service</li></ul><h3>Data access code — no sweat! (mostly)</h3><p>As I mentioned in the previous post, the process I followed for the Python services was to implement service operations in the order implied by the <a href="https://github.com/KillrVideo/killrvideo-integration-tests">killrvideo-integration-tests</a>, starting with the User Management Service. I’ll walk you through how the User Management Service was implemented, as the same basic pattern was followed for the other services.</p><h4>Installing the driver</h4><p>The first step is to install the DataStax Enterprise Python Driver:</p><pre>pip install dse-driver</pre><p>If we were using the DataStax Python Driver for Cassandra (not DSE), we would install cassandra-driver. However, for the purposes of KillrVideo, we know that we’re intending to use DSE features for a couple of the services, specifically DSE Search for the Search Service and DSE Graph for the Suggested Videos Service. There’s a <a href="https://killrvideo.github.io/docs/guides/feature-matrix/">feature matrix on the KillrVideo documentation site</a> that describes usage of DSE features by implementation language in detail.</p><h4>Initializing Cluster and Session objects</h4><p>In order to read and write data to Cassandra (DSE), we’ll need to initialize the driver establish connections to the cluster. We do this through the Cluster and Session classes provided by the driver, as you can read about on the <a href="https://datastax.github.io/python-driver/getting_started.html">Getting Started page</a> of the documentation.</p><p>Since I’ve chosen to implement all of the KillrVideo services within a single application, I put the connection logic in the <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/__init__.py">__init__.py</a> file for the application:</p><pre><strong>from </strong>dse.cluster <strong>import </strong>Cluster, ExecutionProfile, EXEC_PROFILE_DEFAULT<br><strong>from </strong>dse <strong>import </strong>ConsistencyLevel<br><strong>import </strong>dse.cqlengine.connection</pre><pre><strong>def </strong>serve():<br><br>    file = open(<strong>&#39;config.json&#39;</strong>, <strong>&#39;r&#39;</strong>)<br>    config = json.load(file)</pre><pre>    contact_points = config[<strong>&#39;CONTACT_POINTS&#39;</strong>]<br>    default_consistency_level = config[<strong>&#39;DEFAULT_CONSISTENCY_LEVEL&#39;</strong>]<br><br>    <em># Wait for Cassandra (DSE) to be up, aka registered in etcd</em><br><br>    <em># Initialize Cassandra Driver and Mapper<br>    </em>profile = ExecutionProfile(consistency_level = <br>        ConsistencyLevel.name_to_value[default_consistency_level])<br>    cluster = Cluster(contact_points=contact_points, <br>                execution_profiles={EXEC_PROFILE_DEFAULT: profile})<br>    session = cluster.connect(<strong>&quot;killrvideo&quot;</strong>)<br>    dse.cqlengine.connection.set_session(session)</pre><p>(Note: code not directly associated with initializing the driver has been omitted for brevity.)</p><p>As you may have noticed above, I’ve chosen to put configuration values associated with the driver in a JSON configuration file (based on a <a href="https://hackernoon.com/4-ways-to-manage-the-configuration-in-python-4623049e841b">helpful article on various ways to approach configuration in Python</a>). The JSON config file is very simple, looking something like this:</p><pre>{<br>  <strong>&quot;CONTACT_POINTS&quot;</strong>: [<strong>&quot;10.0.75.1&quot;</strong>],<br>  <strong>&quot;DEFAULT_CONSISTENCY_LEVEL&quot;</strong>: <strong>&quot;LOCAL_QUORUM&quot;<br></strong>}</pre><p>Unpacking this connection code a bit, the first statement creates an <a href="https://datastax.github.io/python-driver/execution_profiles.html">ExecutionProfile</a> where we can configure settings such as retry and load balancing policies. Here I’m using an ExecutionProfile to set a default consistency level for queries in the KillrVideo application:</p><pre>profile = ExecutionProfile(consistency_level = <br>                ConsistencyLevel[default_consistency_level])</pre><p>The next step is to create a <a href="https://datastax.github.io/python-driver/api/cassandra/cluster.html">Cluster</a> object using the ExecutionProfile:</p><pre>cluster = Cluster(contact_points=contact_points, <br>                execution_profiles={EXEC_PROFILE_DEFAULT: profile})</pre><p>Then we create a Session object bound to our killrvideo keyspace. This works because all of the tables used by the KillrVideo services are in the same keyspace:</p><pre>session = cluster.connect(<strong>&quot;killrvideo&quot;</strong>)</pre><p>Finally, we initialize the cqlengine, the mapper that is provided as part of the DataStax Python driver. As we will see, the cqlengine is not suitable for every data access pattern we will need in our application, but is sufficient in many cases and I’ve used it where possible for the simplicity it provides.</p><h4>Creating mapper classes</h4><p>In order to use the mapper, we’re going to need some classes that define the types that will be mapped to our Cassandra tables. In keeping with the principle of <a href="https://www.oreilly.com/ideas/cassandra-data-modeling">modeling Cassandra tables around our application queries</a>, there are two tables used to store data, which have been designed to support queries we’ll discuss more below.</p><p>The entity classes that model our Cassandra tables are defined in the file <a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/user_management/user_management_service.py">user_management_service.py</a>:</p><pre>from dse.cqlengine import columns<br>from dse.cqlengine.models import Model</pre><pre><strong>class </strong>UserModel(Model):<br>    <em>&quot;&quot;&quot;Model class that maps to the user table&quot;&quot;&quot;<br>    </em>__table_name__ = <strong>&#39;users&#39;<br>    </strong>user_id = columns.UUID(db_field=<strong>&#39;userid&#39;</strong>, primary_key=True)<br>    first_name = columns.Text(db_field=<strong>&#39;firstname&#39;</strong>)<br>    last_name = columns.Text(db_field=<strong>&#39;lastname&#39;</strong>)<br>    email = columns.Text()<br>    created_date = columns.Date()<br><br><br><strong>class </strong>UserCredentialsModel(Model):<br>    <em>&quot;&quot;&quot;Model class that maps to the user_credentials table&quot;&quot;&quot;<br>    </em>__table_name__ = <strong>&#39;user_credentials&#39;<br>    </strong>email = columns.Text(primary_key=True)<br>    user_id = columns.UUID(db_field=<strong>&#39;userid&#39;</strong>)<br>    password = columns.Text()</pre><p>Each entity class extends the dse.cqlengine.models.Model class. You’ll note the use of the __table_name__ attribute to specify the Cassandra table we’re using (we’ve already specified the keyspace the mapper is using above). We define an attribute of the class for each table column by referencing the appropriate type from thedse.cqlengine.columns module. Note the use of the primary_key designation to identify Cassandra primary key columns. These tables don’t involve any clustering keys but the clustering_key designation is available to describe any of those as well. The final item to note is the use of the db_field designation, used to allow us to have an attribute name that conforms to Python naming conventions but differs slightly from the CQL column name.</p><h4>Inserting data using the mapper</h4><p>Following the integration test order, the first operation to implement in the User Management Service is create_user. Note that we need to insert into two different tables to support our access patterns:</p><pre><em># insert into user_credentials table first so we can ensure uniqueness with LWT<br></em><strong>try</strong>:<br>    UserCredentialsModel.if_not_exists().create(user_id=user_id, email=email, password=hashed_password)<br><strong>except </strong>LWTException:<br>    <em># Exact string in this message is expected by integration test<br>    </em><strong>raise </strong>ValueError(<strong>&#39;Exception creating user because it already exists for &#39; </strong>+ email)<br><br><em># insert into users table<br></em>UserModel.create(user_id=user_id, first_name=first_name, last_name=last_name, email=email)</pre><p>One way to approach this insert would be to use Cassandra’s batch feature. However, there is an additional concern that comes into play here, that of ensuring that we enforce uniqueness and do not allow multiple user accounts for the same email. The the logic we use here is to first attempt to create a record for the user in the user_credentials table using a lightweight transaction — that is what the if_not_exists() operation is doing. If the account creation is for a new email address, the insert will succeed and we can proceed to inserting into the users table. Otherwise, an error is raised.</p><p>The actual notation of each insert is quite simple, using the create() operation inherited from the Model class and specifying a value for each of the columns by name.</p><h4>Retrieving data using the mapper</h4><p>Next it’s time to implement the methods for retrieving user account data. The first query our application requires is used to support login in the verify_credentials operation:</p><pre><em># retrieve the credentials for provided email from user_credentials table<br></em>user_credentials = UserCredentialsModel.get(email=email)</pre><p>This is a very simple query — we’re retrieving a record from the user_credentials table based on the partition key for that table — the email address. We can then hash the provided password and compare it to the hashed password retrieved from the database to enforce a secure login.</p><p>The second query is used to obtain information about one or more user accounts. Because the API of the service allows lookup of multiple accounts by user ID, we’ve used the <a href="# see: https://datastax.github.io/python-driver/cqlengine/queryset.html#retrieving-objects-with-filters">filter()</a> operation of the UserModel class instead of a get():</p><pre><em># filter().all() returns a ModelQuerySet, we iterate over the query set to get the Model instances<br></em>user_results = UserModel.filter(user_id__in=user_ids).all()<br>users = list()<br><strong>for </strong>user <strong>in </strong>user_results:<br>    users.append(user)<br><strong>return </strong>users</pre><p>Note the use of the __in notation appended to the column name to indicate our desire to use a CQL IN clause. The mapper will convert this into a query that looks like this:</p><pre>SELECT * FROM killrvideo.users WHERE user_id IN &lt;id1&gt;, &lt;id2&gt;...</pre><p>We can then iterate over the results to build a list of user account information to return to the client.</p><p>Note that we could also have iterated over the provided user IDs and performed individual get() operations, but by using an in operation, we’re able to query across multiple partition keys at once.</p><p>Because the KillrVideo application is not configured to request a large number of user accounts at once, our approach should perform well, more quickly than iterating and performing individual queries. However, you should definitely read more in <a href="https://lostechies.com/ryansvihla/2014/09/22/cassandra-query-patterns-not-using-the-in-query-for-multiple-partitions/">this classic blog post</a> about the limitations of using in for queries involving multiple partitions, and a workaround involving asynchronous programming.</p><h3>Next time: more complex data access cases</h3><p>So, that was pretty easy, right? If you’re suspicious, you’re right, there are some cases where the “easy way” doesn’t work.</p><p>In the next post, I’ll show some more complex examples including cases where the cqlengine is not a good fit, as well as some interesting details I discovered about implementing queries using DSE Search in the Search Service.</p><p><strong>Next pos in this series: </strong><a href="https://medium.com/@jscarp/cassandra-with-python-simple-to-complex-44298391867d?source=your_stories_page---------------------------"><strong>Cassandra with Python: Simple to Complex</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7a90dee701a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quick-code/when-data-access-is-the-easiest-part-of-a-microservice-c7a90dee701a">When data access is the easiest part of a microservice</a> was originally published in <a href="https://medium.com/quick-code">Quick Code</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>