<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Travis Crawford on Medium]]></title>
        <description><![CDATA[Stories by Travis Crawford on Medium]]></description>
        <link>https://medium.com/@tc?source=rss-bdfe87fcc7b0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4W7H1dXcL2ZjL60WTv5o9g.jpeg</url>
            <title>Stories by Travis Crawford on Medium</title>
            <link>https://medium.com/@tc?source=rss-bdfe87fcc7b0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 12:46:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Medium’s DynamoDB Data Source for Apache Spark]]></title>
            <link>https://medium.engineering/mediums-dynamodb-data-source-for-apache-spark-62c6599a6dfd?source=rss-bdfe87fcc7b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/62c6599a6dfd</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[dynamodb]]></category>
            <dc:creator><![CDATA[Travis Crawford]]></dc:creator>
            <pubDate>Fri, 07 Oct 2016 17:38:00 GMT</pubDate>
            <atom:updated>2016-10-07T17:38:00.558Z</atom:updated>
            <content:encoded><![CDATA[<p>Medium’s <a href="https://medium.engineering/the-stack-that-helped-medium-drive-2-6-millennia-of-reading-time-e56801f7c492#.hdsm9abzo">tech stack</a> includes DynamoDB for our production data, and Apache Spark for backend data processing. We need a fast and reliable integration between these systems to support our data warehouse. Talking with data engineers at other companies, this is a common pairing and we all have somewhat similar internal tools making these systems play well together.</p><p>We’ve open-sourced our <a href="https://github.com/traviscrawford/spark-dynamodb">DynamoDB Data Source for Apache Spark</a> with the goal of making DynamoDB &amp; Spark easy to use together for everyone.</p><p>Key features:</p><ul><li>Load a DynamoDB table as a DataFrame or RDD.</li><li>Scans the table in parallel per <a href="http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScanGuidelines.html">Amazon’s recommendations</a>.</li><li>Rate limiting support.</li></ul><p>Scanning DynamoDB tables in parallel is particularly beneficial because they are partitioned, requiring parallel scans to take advantage of all your provisioned read capacity. We have observed significant scan time improvements when switching to the parallel scanner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C4sSauT635uST1FaJUf1FA.png" /><figcaption>75% improvement with parallel scanner</figcaption></figure><h3>RDD Integration</h3><p>Our primary use-case at Medium is using Spark to backup DynamoDB tables to S3. Separate Spark jobs restore the backups — JSON files on S3 which Spark has great support for — into a variety of systems, including Redshift and <a href="https://databricks.com/">Databricks</a>. We chose this approach (rather than Data Pipeline) so our ETL and analysis jobs have a common look &amp; feel, keeping things simple for our users and developers.</p><p>We start our backup jobs with a command similar to:</p><pre>spark-submit \<br>  --class com.github.traviscrawford.spark.dynamodb.DynamoBackupJob \<br>  --packages com.github.traviscrawford:spark-dynamodb:0.0.5 &#39;&#39; \<br>  -table users -output s3://myBucket/users \<br>  -totalSegments 4 -rateLimit 100</pre><p>DynamoDB table scans work by defining a set of <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/document/spec/ScanSpec.html">ScanSpec</a>’s, or description of what data to scan, including which portion of the table to scan, and which projections and/or filters to apply.</p><p>This data source creates ScanSpec’s in the Spark driver based on the provided configuration, then distributes them to cluster workers where the table is scanned in parallel.</p><p>As a convenience, we provide a backup job that writes the data set to any supported filesystem. For more control, you can use the <a href="https://github.com/traviscrawford/spark-dynamodb/blob/master/src/main/scala/com/github/traviscrawford/spark/dynamodb/DynamoScanner.scala">table scanner</a> as a library.</p><h3>DataFrame Integration</h3><p>As a NoSQL database, DynamoDB does not strictly enforce schemas. However, if all records in a table share a schema it may be useful to load the table as a DataFrame.</p><p>Here’s an example running SQL on a DynamoDB table. Notice how no schema was provided — it’s inferred by sampling some records in the table. You can optionally specify the schema if needed (we generate ours from protobufs).</p><pre>import com.github.traviscrawford.spark.dynamodb._<br><br>val users = sqlContext.read.dynamodb(&quot;users&quot;).cache()<br>users.registerTempTable(&quot;users&quot;)<br>val data = sqlContext.sql(<br>  &quot;select username from users where username = &#39;tc&#39;&quot;)</pre><p>You can perform any DataFrame operation on the data that was scanned from the DynamoDB table, such as continued analysis, or saving to S3/Redshift.</p><blockquote><em>Interested in helping move Medium forward? </em><a href="https://jobs.medium.com/"><em>We’re hiring!</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62c6599a6dfd" width="1" height="1" alt=""><hr><p><a href="https://medium.engineering/mediums-dynamodb-data-source-for-apache-spark-62c6599a6dfd">Medium’s DynamoDB Data Source for Apache Spark</a> was originally published in <a href="https://medium.engineering/">Medium Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analytics Engineer]]></title>
            <link>https://medium.com/@tc/analytics-engineer-cefea925aecd?source=rss-bdfe87fcc7b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/cefea925aecd</guid>
            <category><![CDATA[medium]]></category>
            <category><![CDATA[jobs]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Travis Crawford]]></dc:creator>
            <pubDate>Fri, 16 Oct 2015 18:01:13 GMT</pubDate>
            <atom:updated>2016-11-22T00:13:51.778Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FDxNNXnl9LmxvyQR2Er-Ww.jpeg" /></figure><p>The Product Science team is looking for our next Analytics Engineer, responsible for <strong>building our analytics system and data products</strong>. What others might call a growth team, analytics team, or data team, we call Product Science — emphasis on product. We believe that understanding and applying our data is one of the ways we can make Medium better for our users.</p><h4>Building our analytics system</h4><p>The goal of our analytics system is to make all of the data we have accessible and usable. Analytics data is used for everything from testing product hypotheses to debugging new API’s, so it’s critical that it’s reliable and flexible.</p><p>Analytics Engineers build systems to collect, store, &amp; process our data sets, ensuring the right data are available in the right places at the right times. We champion data quality, ensuring its completeness &amp; correctness.</p><p>You are passionate about data, both making it available and using it effectively. You feel deep ownership over the usefulness and accuracy of the tools we build. We don’t filter for technology keywords. Instead, you should love solving problems creatively and be familiar with different analytics tools and methods — SQL, MapReduce, ETL, etc.</p><h4>Data Products</h4><p>Building tools to collect data isn’t enough, you should also be passionate about using data to improve the product.</p><p>Analytics Engineers work closely with our <a href="https://medium.com/@sall/product-scientist-ffd1ae846172">Product Scientists</a>, turning insights into data products like the <a href="https://medium.com/the-story/taking-medium-for-ios-to-the-next-level-d318192a2a5c">Search and Explore</a> backend, and <a href="https://medium.com/data-lab/how-we-used-data-to-suggest-tags-for-your-story-a120076d0bb6">tag suggestions</a>.</p><p>If you’ve built significant data products in the past and welcome the opportunity to help Medium level up the data and monetization products we ship to our users, we want to hear from you.</p><blockquote><a href="https://jobs.lever.co/medium/31d4b6c8-c9d2-4ce6-8038-6df0aa0182bd/apply">If you’ve been nodding along and want to join us, apply!</a></blockquote><p><em>If you’re curious what it’s like to work at Medium, </em><a href="https://medium.com/jobs/b7d6d31b4cd0"><em>check out this story from behind the scenes</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cefea925aecd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>