<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tim on Medium]]></title>
        <description><![CDATA[Stories by Tim on Medium]]></description>
        <link>https://medium.com/@ttaschke?source=rss-5b841c2a7840------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*WkasB_0onEDGEhMQL4QaRA.jpeg</url>
            <title>Stories by Tim on Medium</title>
            <link>https://medium.com/@ttaschke?source=rss-5b841c2a7840------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:14:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ttaschke/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AWS Glue feature overview]]></title>
            <link>https://medium.com/@ttaschke/aws-glue-feature-overview-fc6e09b85623?source=rss-5b841c2a7840------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc6e09b85623</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Tim]]></dc:creator>
            <pubDate>Wed, 16 Aug 2017 07:49:54 GMT</pubDate>
            <atom:updated>2017-08-16T08:02:31.231Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eRz92s5k0XVw_r85S5lyeA.jpeg" /></figure><p>Glue is a fully-managed ETL service on AWS. Provides <strong>crawlers</strong> to index data from files in S3 or relational databases and infers schema using provided or custom <strong>classifiers</strong>. Indexed metadata is stored in <strong>Data Catalog</strong>, which can be used as Hive metadata store. <strong>Jobs</strong> written in PySpark and scheduled either time-based or event-based, transform the data on a fully-managed Spark execution engine. Pyspark code can be tested in Zeppelin notebooks or REPL shell that connects to Glue service.</p><h3>Crawlers</h3><h4><strong>Indexing</strong></h4><ul><li>only indexes data; does not actually process the data</li><li>can to detect both new data and changes to existing data</li><li>automatically adds new tables, new partitions, new versions of table definitions</li><li>automatic schema discovery via classifiers</li><li>semi-structured data (mixed data with single columns, nested columns etc.) can be automatically converted to structured relational data</li></ul><h4>Crawlable Datastores</h4><ul><li>S3 (Several tables can be created automatically from one bucket)</li><li>Redshift</li><li>Amazon Relational Database Service (MySQL, PostgreSQL, Aurora, and MariaDB)</li><li>JDBC (accessible from VPC)</li></ul><h3>Classifiers</h3><p><em>A crawler uses a prioritized list of classifiers and chooses the first matching classifier to infer schema / parse data</em></p><h4>Supported formats</h4><ul><li>CSV, JSON, Avro, Parquet, …</li><li>Predefined data formats: apache log, linux kernel log, ruby log, …</li><li>Relational database schemas</li><li>Supported compression codecs: ZIP, BZIP, GZIP, LZ4, Snappy (not Hadoop Snappy)<br><em>Note: If file is compressed it needs to be downloaded to be processed</em></li></ul><h4>Custom classifier</h4><ul><li>defined as grok pattern</li><li>Example: %{PATTERN:field-name:data-type}</li><li>Documentation: <a href="https://www.elastic.co/guide/en/logstash/5.2/plugins-filters-grok.html">https://www.elastic.co/guide/en/logstash/5.2/plugins-filters-grok.html</a></li><li>Tools: <a href="http://grokconstructor.appspot.com/">http://grokconstructor.appspot.com/</a> and <a href="http://grokdebug.herokuapp.com/">http://grokdebug.herokuapp.com/</a></li></ul><h3>Data Catalog</h3><h4>Metastore</h4><ul><li>tables and databases are only metadata objects in the Data catalog, but do not contain the actual data</li><li>can be used as drop-in replacement to Hive metastore in EMR</li><li>metadata sources: glue crawlers, HIVE DDL, bulk import from Hive metastore</li></ul><h4>History per table</h4><ul><li>job execution times</li><li>number of added rows</li><li>run time</li><li>schema version history</li></ul><h4>Partitions</h4><ul><li>only partitioned tables created by crawlers can be used</li><li>S3 partitions need to have the same file schema, file format and compression format</li></ul><h4>Table metadata example</h4><pre>{<br>“StorageDescriptor”: {<br>“cols”: {<br> “FieldSchema”: [<br> {<br> “name”: “primary-1”,<br> “type”: “CHAR”,<br> “comment”: “”<br> },<br> {<br> “name”: “second “,<br> “type”: “STRING”,<br> “comment”: “”<br> }<br> ]<br>},<br>“location”: “s3://aws-logs-111122223333-us-east-1”,<br>“inputFormat”: “”,<br>“outputFormat”: “org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat”,<br>“compressed”: “false”,<br>“numBuckets”: “0”,<br>“SerDeInfo”: {<br> “name”: “”,<br> “serializationLib”: “org.apache.hadoop.hive.serde2.OpenCSVSerde”,<br> “parameters”: {<br> “separatorChar”: “|”<br> }<br>},<br>“bucketCols”: [],<br>“sortCols”: [],<br>“parameters”: {},<br>“SkewedInfo”: {},<br>“storedAsSubDirectories”: “false”<br>},<br>“parameters”: {<br>“classification”: “csv”<br>}<br>}</pre><h3>ETL</h3><h4>Transform script</h4><ul><li>PySpark code</li><li>AWS Glue library provided <br><a href="https://github.com/awslabs/aws-glue-libs/tree/master/awsglue">https://github.com/awslabs/aws-glue-libs/tree/master/awsglue</a></li><li>Library contents: GlueContext, DynamicFrame, Transforms</li><li>cutom PySpark libraries can be imported</li></ul><h4>Development Environment</h4><ul><li>Developer Endpoints to connect to Glue service</li><li>can be tested in Zeppelin notebook (hosted on EC2) or REPL shell</li><li>can be connected to IDE</li></ul><h4>Error handling</h4><ul><li>retries 3 times before sending error notification</li><li>statistics and errors are send to Cloudwatch</li><li>errors / success notification can also be used to trigger Lambda functions</li><li>filtering bad data is handled automatically</li></ul><h3>Scheduler</h3><h4>Trigger</h4><ul><li>time-based (cron)</li><li>event-based</li><li>on-demand</li></ul><h4>Execution</h4><ul><li>smallest possible execution frequency 5min, for streaming use Kinesis etc.</li><li>handles dataset dependencies automatically (trigger next step in pipeline on completion etc.)</li><li>automatic retry</li><li>Bookmarks to store which partitions have been processed</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc6e09b85623" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>