<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Mostly AI - Medium]]></title>
        <description><![CDATA[Mostly AI Blog - Medium]]></description>
        <link>https://medium.com/mostly-ai?source=rss----e95d2c58ff19---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Mostly AI - Medium</title>
            <link>https://medium.com/mostly-ai?source=rss----e95d2c58ff19---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:28:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/mostly-ai" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Being Open in the Era of Privacy]]></title>
            <link>https://medium.com/mostly-ai/being-open-in-the-era-of-privacy-8916912f726a?source=rss----e95d2c58ff19---4</link>
            <guid isPermaLink="false">https://medium.com/p/8916912f726a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[data-security]]></category>
            <category><![CDATA[privacy]]></category>
            <dc:creator><![CDATA[Michael Platzer]]></dc:creator>
            <pubDate>Mon, 18 Mar 2019 11:52:11 GMT</pubDate>
            <atom:updated>2019-03-18T13:17:58.033Z</atom:updated>
            <content:encoded><![CDATA[<p>When Netflix announced a 1 million dollar challenge for improving their recommendation engine, together with releasing 100 million movie ratings in 2006, they knew little what would happen next. Netflix has been working on a recommender systems for years by then, were recognized for their business innovation as well as technical excellence, managed to hire the smartest data engineers and machine learning experts alike, and certainly knew all the ins and outs on movies, genres and popular actors. Still, they were eager to learn more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NJURwmKkHc8SQgQDq35eGw.jpeg" /></figure><p>The competition began on October 2nd, 2006. Within a mere six days a first contestant succeeded in beating their existing solution. Six days! From getting access to the data and taking a first look, to building a movie recommender algorithm from scratch, all the way to making more accurate predictions for over a million ratings than anyone before. Six days, from zero to world class.</p><p>It didn’t stop there. Within a year, over 40,000 teams from 186 countries entered the competition, all trying to improve on Netflix’ algorithms. And contestants increasingly started collaborating, sharing their learnings and taking lesson from others, formed bigger teams and ensembled ever more powerful models. Hardly ever before was there such a rich &amp; big dataset on consumer behavior openly available. Together with a clearly stated and measurable objective, it provided a challenging, yet safe and fair <em>sandbox</em> for a worldwide community of intellectually driven engineers. All working together on advancing science, while, as a more than welcome side effect, also helping Netflix improve its core algorithm.</p><p><strong>The Case for #OpenBigData</strong> - Being open enables to pick the brain of a wider group, to bring in fresh perspectives to existing challenges, to build upon the creative minds of the many. And with data being the <em>lingua franca</em> of today’s business world, being the common denominator across departments, across corporations, across industries, being open is really about sharing data, about sharing granular level data at scale!</p><p><em>Openly sharing your customer data at scale in 2006 was a bold move.</em> But it was no coincidence that it’s been done by Netflix. Already in their early years they successfully established a culture of excellence, curiosity as well as courage, well documented in “one of Silicon’s valley’s most important PowerPoint decks” [<a href="https://hbr.org/2014/01/how-netflix-reinvented-hr"><em>1</em></a>]. Sharing data broadly takes courage, but is even more so a sign of curiosity and a thrive for excellence. No holding back, no hiding out, no making excuses. Netflix was never afraid of their competitors. They were afraid of stopping to strive for the best.</p><p><em>Openly sharing your customer data at scale in 2019 is an (unl)awful move.</em> Over the past years, the explosion in data volumes met a poorly regulated market, with little sanctions being imposed, and thus allowing excessive misuse of personal data. The tide though has turned, both the regulators as well as corporates are acknowledging privacy as a fundamental human right, one that is to be defended [<a href="https://www.weforum.org/agenda/2019/01/privacy-is-a-human-right-we-need-a-gdpr-for-the-world-microsoft-ceo/">2</a>,<a href="https://www.securitynow.com/author.asp?section_id=613&amp;doc_id=747058">3</a>]. This is indeed a new era of privacy.</p><p>Unfortunately, this plays into the hands of modern-day corporate gatekeepers. Those decision makers, who’ve never really been fond of being transparent and being challenged, and thus reluctant to share “their” data in the first place. <strong>It turns out they found a new ally in defending their corporate data silos: privacy.</strong></p><p><strong>The Case for #SyntheticData </strong>- This is the point where one needs to tell the lesser known part of the Netflix Prize story: As successful the competition was for the company overall, they also had to pay their prize at court. In fact, they were forced to cancel their second machine learning challenge, that was planned for 2011 [<a href="https://www.wired.com/2010/03/netflix-cancels-contest/">3</a>]. Netflix had misjudged their anonymization measures they’ve had put in place [<a href="https://www.netflixprize.com/faq.html">4</a>]. Even though they limited data to movie ratings and their dates, merely linked to a scrambled user ID, it proved inefficient to prevent re-identification. It had taken only 16 days, after the data was released for an outsider (with no superhuman hacking skills) to link these user IDs with freely available public data — with enough time left at hand to write up a whole paper on de-anonymization [<a href="https://arxiv.org/abs/cs/0610105v1">5</a>]. Netflix had unintentionally exposed the full movie history for parts of their customer base, with no chance of making that privacy infringement undone. A decade later Facebook had to learn that same painful lesson. Once the data is out and you failed to properly anonymize, no matter how good your intentions might have been, you will have a hard time to undo your actions.</p><p>This risk of re-identification in large-scale data is by now well understood by privacy and security experts [<a href="https://www.nature.com/articles/srep01376">6</a>], and yet still widely under-estimated by the corporate world. That’s why these experts face a challenging role within organizations, as they need to educate their colleagues, that most anonymization attempts for big data in fact fail to provide safety for their customers. And these experts are forced to give a NO more often than a YES to a new initiative, to a new innovation project, in order to keep privacy safe and secure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vt9zPC2EMYivDc3g8CAMeg.png" /><figcaption>the under-estimated risk of re-identification</figcaption></figure><p>So, this is the big quest of our time: <em>How to be open, while being private at the same time? </em>How to put big data to good use, while still protecting each and everyone’s right for privacy? How to foster data-driven, people-centric and innovative societies and organizations, all at the same time, while not giving up an inch on safeguarding privacy?</p><p>We, at Mostly AI, set out to solve this challenge, and developed a one-of-its-kind technical solution to this long-standing problem: an <a href="https://mostly.ai/synthetic-data-engine.html">AI-based synthetic data generator</a>. One that learns based on actual behavioral data, to generate statistical representative synthetic personas and their data. Synthetic data, that can be broadly shared, internally as well as externally, without exposing any individuals. It’s all the value of the original data, but without the privacy risk.</p><p>This is 2019. It’s time to protect privacy, as well as to embrace the power of open again. <strong>It’s time to #GoSynthetic!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8916912f726a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mostly-ai/being-open-in-the-era-of-privacy-8916912f726a">Being Open in the Era of Privacy</a> was originally published in <a href="https://medium.com/mostly-ai">Mostly AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Synthetic Data Diamonds]]></title>
            <link>https://medium.com/mostly-ai/synthetic-data-diamonds-5e42a4269978?source=rss----e95d2c58ff19---4</link>
            <guid isPermaLink="false">https://medium.com/p/5e42a4269978</guid>
            <category><![CDATA[data-security]]></category>
            <category><![CDATA[synthetic-data]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Michael Platzer]]></dc:creator>
            <pubDate>Mon, 18 Mar 2019 11:49:45 GMT</pubDate>
            <atom:updated>2019-03-18T11:51:13.130Z</atom:updated>
            <content:encoded><![CDATA[<p>My colleague Mitch likens synthetic data to synthetic diamonds when pitching our value proposition to potential customers. And it’s indeed a fitting analogy. Synthetic diamonds are near indistinguishable from actual diamonds to the human eye, they have the same structure, and they bear they same valuable characteristics, like hardness, purity and thermal conductivity. Yet, they take 3 billion years less to form, come at a fraction of cost and are considered more ethical in terms of sourcing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*zIT6nCrePENvSjDm3Rbr2w.jpeg" /></figure><p>Along the same lines, synthetic data can as well be near <em>indistinguishable</em> from actual data, can have the <em>same structure</em>, and can retain all the same <em>valuable properties</em> (=statistical information) of actual data. Yet, machines can generate synthetic data in unlimited quantities, and more importantly, synthetic data allows big data to be utilized and shared without putting anyone’s privacy at risk. Synthetic data has the potential to become the new risk-free &amp; ethical norm to leverage customer data at scale. Finally, there is a solution for <strong>big data privacy!</strong></p><p>However, just like with diamonds, the process of generating high quality synthetic data is anything but trivial. At <a href="https://mostly.ai/">Mostly AI</a> we’ve mastered the automated generation of synthetic structured data over the past 2 years by leveraging state-of-the-art generative AI, and are confident to claim that we offer the world’s most advanced <a href="https://mostly.ai/synthetic-data-engine.html">synthetic data engine</a>.</p><p>As a simple demonstration, as part of this first post, let’s apply our solution to a <a href="https://ggplot2.tidyverse.org/reference/diamonds.html">publicly available diamonds dataset</a> with 53’940 records, and 10 attributes. As can be seen, the attributes are a mix of categorical as well as numerical variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bXZzjEEjGa5IiecN9Xk8WA.png" /><figcaption>first 8 records of the original diamonds dataset</figcaption></figure><p>Generating an equally sized, structural identical, yet synthetic version of this dataset is, thanks to the flexibility of our engine, as simple as:</p><pre>&gt; mostly train diamonds.csv <br>&gt; mostly generate -n 53940</pre><p>This will create a new data file with 53’940 records, where none of the generated records have any direct relationship anymore to the records in the original dataset. Hence, the information and thus privacy of any individual diamond is being protected (in case they care :)), while the structure of the population is retained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*84T8L2VpRkBOft1rt7i6jA.png" /><figcaption>first 8 records of a synthetically generated diamonds dataset</figcaption></figure><p>Further the statistical properties of the various attributes are successfully retained. Here are side-by-side comparison of the frequency of the three categorical variables clarity, cut and color. All these univariate distributions are matched to near perfection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l5wUn17TWZIchuYwk64lEg.png" /><figcaption>side-by-side comparison of distributions of categorical variables</figcaption></figure><p>Similarly, the engine is also able to retain the distributions of the numerical variables, and nicely captures location, skew, the tails as well as the multiple modes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FJFYkkahOCRJtaWF5bQPWw.png" /><figcaption>side-by-side comparison of price percentiles</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_HMv1h0BF3O9cJgfo6So0A.png" /><figcaption>side-by-side comparison of histograms for price, x, yand z</figcaption></figure><p>More importantly, the (non-linear) interdependencies between the various variables is being properly captured. These statistical multivariate relationships are the key <strong>value</strong> of any dataset, as they help to gain a deeper understanding of the underlying domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hc7RofEKuOV98HfMbUmbQQ.png" /></figure><p>As can be seen, the patterns of the synthetic dataset mimic the patterns found in the original dataset to a very high degree. By analyzing the synthetic data we can gain various insights in terms of the diamond market. Whether that’s the relationship between market prices and carats, or the dominance of “Ideal” cut for diamonds with clarity “IF”. Even unexpected patterns, like the lack of “just-little-less-than-2-carat” diamonds is perfectly captured as well, if you take a close look at the upper chart.</p><p>Apart from descriptive statistics the synthetic data also turns out to be of immense value for training machine learning algorithms and predictive tasks. This opens up a whole range of opportunities to power next-generation AI, as more data can be gathered across different sources, across borders and industries, without infringing individual’s privacy.</p><p>Let’s now take a subset of 40k actual records, and generate 40k synthetic records based on that. We will need the remaining 13’940 actual records to have a fair holdout set available to benchmark our two models for diamond prices: one model trained on actual data, and another one on synthetic data. As expected, there is a loss in accuracy, as the data synthesis results in an information loss (to protect privacy), but the loss is within a rather small margin. In terms of mean absolute error we acchieve an error of 350$ vs 296$. Still, the 350$ is a far cry from a naive model, that would in this case result in an error of $3036 when having no data available at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jMfIIwBhi1-OGb05jtw2cw.png" /><figcaption>R code for benchmarking models trained on actual vs. synthetic data</figcaption></figure><p>In an upcoming post we will dive deeper into an example of synthetic <em>sequential</em> data, which actually turns out to be the prevalent form when it comes to privacy-sensitive user data. But this is a story for another day. For now, let’s simply generate us a million more synthetic data diamonds and wallow in these for the time being.</p><pre>&gt; mostly generate -n 1000000 <br>&gt; mostly wallow :)</pre><p><em>Originally published at </em><a href="https://blog.mostly.ai/2019/02/21/synthetic-data-diamonds/"><em>blog.mostly.ai</em></a><em> on February 21, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e42a4269978" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mostly-ai/synthetic-data-diamonds-5e42a4269978">Synthetic Data Diamonds</a> was originally published in <a href="https://medium.com/mostly-ai">Mostly AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tensorflow Records? What they are and how to use them]]></title>
            <link>https://medium.com/mostly-ai/tensorflow-records-what-they-are-and-how-to-use-them-c46bc4bbb564?source=rss----e95d2c58ff19---4</link>
            <guid isPermaLink="false">https://medium.com/p/c46bc4bbb564</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Thomas Gamauf]]></dc:creator>
            <pubDate>Tue, 20 Mar 2018 07:54:41 GMT</pubDate>
            <atom:updated>2018-10-02T12:10:42.063Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LSxfK4mp06r-p69a3-m_zw.png" /></figure><p>Interest in Tensorflow has increased steadily since its introduction in November 2015. A lesser-known component of Tensorflow is the <a href="https://www.tensorflow.org/api_guides/python/python_io">TFRecord file format</a>, Tensorflow’s own binary storage format.</p><p>If you are working with large datasets, using a binary file format for storage of your data can have a significant impact on the performance of your import pipeline and as a consequence on the training time of your model. Binary data takes up less space on disk, takes less time to copy and can be read much more efficiently from disk. This is especially true if your data is stored on spinning disks, due to the much lower read/write performance in comparison with SSDs.</p><p>However, pure performance isn’t the only advantage of the TFRecord file format. It is optimized for use with Tensorflow in multiple ways. To start with, it makes it easy to combine multiple datasets and integrates seamlessly with the data import and preprocessing functionality provided by the library. Especially for datasets that are too large to be stored fully in memory this is an advantage as only the data that is required at the time (e.g. a batch) is loaded from disk and then processed. Another major advantage of TFRecords is that it is possible to store sequence data — for instance, a time series or word encodings — in a way that allows for very efficient and (from a coding perspective) convenient import of this type of data. Check out the <a href="https://www.tensorflow.org/api_guides/python/reading_data">Reading Data</a> guide to learn more about reading TFRecord files.</p><p>So, there are a lot of advantages to using TFRecords. But <a href="https://www.goodreads.com/quotes/456951-where-there-is-light-there-must-be-shadow-where-there">where there is light, there must be shadow</a> and in the case of TFRecords the downside is that you have to convert your data to this format in the first place and only limited documentation is available on how to do that. An <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/how_tos/reading_data/convert_to_records.py">official tutorial</a> and a number of articles about writing TFRecords exist, but I found that they only got me covered part of the way to solve my challenge.</p><p>In this post I will explain the components required to structure and write a TFRecord file, and explain in detail how to write different types of data. This will help you get started to tackle your own challenges.</p><h3>Structuring TFRecords</h3><p>A TFRecord file stores your data as a sequence of binary strings. This means you need to specify the structure of your data before you write it to the file. Tensorflow provides two components for this purpose: <a href="https://www.tensorflow.org/api_docs/python/tf/train/Example">tf.train.Example</a> and <a href="https://www.tensorflow.org/api_docs/python/tf/train/SequenceExample">tf.train.SequenceExample</a>. You have to store each sample of your data in one of these structures, then serialize it and use a <a href="https://www.tensorflow.org/api_docs/python/tf/python_io/TFRecordWriter">tf.python_io.TFRecordWriter</a> to write it to disk.</p><blockquote>tf.train.Example isn’t a normal Python class, but a <a href="https://en.wikipedia.org/wiki/Protocol_Buffers">protocol buffer</a>.</blockquote><p>As a software developer, the main problem I had at the beginning was that many of the components in the Tensorflow API don’t have a description of the attributes or methods of the class. For instance, for tf.train.Example only a “.proto” file with cryptic structures called “message” is provided, along with examples in pseudocode. The reason for this is that tf.train.Example isn’t a normal Python class, but a <a href="https://en.wikipedia.org/wiki/Protocol_Buffers">protocol buffer</a>. A protocol buffer is a method developed by Google to serialize structured data in an efficient way. I will now discuss the two main ways to structure Tensorflow TFRecords, give an overview of the components from a developers view and provide a detailed example of how to use tf.train.Example and tf.train.SequenceExample.</p><h3>Movie recommendations using tf.train.Example</h3><blockquote>If your dataset consist of features, where each feature is a list of values of the same type, tf.train.Example is the right component to use.</blockquote><p>Let’s use the movie recommendation application from the <a href="https://github.com/tensorflow/tensorflow/blob/r1.5/tensorflow/core/example/example.proto">Tensorflow documentation</a> as an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*At3Y8UvzwAK1bfl5EjRQiA.jpeg" /></figure><p>We have a number of features, each being a list where every entry has the same data type. In order to store these features in a TFRecord, we fist need to create the lists that constitute the features.</p><p><a href="https://www.tensorflow.org/api_docs/python/tf/train/BytesList"><strong>tf.train.BytesList</strong></a>, <a href="https://www.tensorflow.org/api_docs/python/tf/train/FloatList"><strong>tf.train.FloatList</strong></a>, and <a href="https://www.tensorflow.org/api_docs/python/tf/train/Int64List"><strong>tf.train.Int64List</strong></a> are at the core of a tf.train.Feature. All three have a single attribute <em>value</em>, which expects a list of respective bytes, float, and int.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22ec5fe7ad2cd7ea4a5cb503fc94a432/href">https://medium.com/media/22ec5fe7ad2cd7ea4a5cb503fc94a432/href</a></iframe><p>Python strings need to be converted to bytes, <em>(</em>e.g. <em>my_string.encode(‘utf-8’))</em> before they are stored in a tf.train.BytesList.</p><p><a href="https://www.tensorflow.org/api_docs/python/tf/train/Feature"><strong>tf.train.Feature</strong></a> wraps a list of data of a specific type so Tensorflow can understand it. It has a single attribute, which is a <a href="https://en.wikipedia.org/wiki/Union_type">union</a> of <em>bytes_list/float_list/int64_list</em>. Being a union, the stored list can be of type tf.train.BytesList (attribute name <em>bytes_list</em>), tf.train.FloatList (attribute name <em>float_list</em>), or tf.train.Int64List (attribute name <em>int64_list</em>).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3c3f4d54ac75a351f8bcf27913ae3e8/href">https://medium.com/media/a3c3f4d54ac75a351f8bcf27913ae3e8/href</a></iframe><p><a href="https://www.tensorflow.org/api_docs/python/tf/train/Features"><strong>tf.train.Features</strong></a> is a collection of named features. It has a single attribute <em>feature</em> that expects a dictionary where the key is the name of the features and the value a tf.train.Feature.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5245ed6cd8a51932cdffd861dafb442d/href">https://medium.com/media/5245ed6cd8a51932cdffd861dafb442d/href</a></iframe><p><a href="https://www.tensorflow.org/api_docs/python/tf/train/Example"><strong>tf.train.Example</strong></a> is one of the main components for structuring a TFRecord. An tf.train.Example stores features in a single attribute <em>features</em> of type tf.train.Features.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb7c66feb89297823b6813672ee362df/href">https://medium.com/media/bb7c66feb89297823b6813672ee362df/href</a></iframe><p>In contrast to the previous components, <a href="https://www.tensorflow.org/api_docs/python/tf/python_io/TFRecordWriter"><strong>tf.python_io.TFRecordWriter</strong></a> actually is a Python class. It accepts a file path in its <em>path</em> attribute and creates a writer object that works just like any other file object. The TFRecordWriter class offers <em>write</em>, <em>flush</em> and <em>close</em> methods. The method <em>write</em> accepts a string as parameter and writes it to disk, meaning that structured data must be serialized first. To this end, tf.train.Example and tf.train.SequenceExample provide <em>SerializeToString </em>methods:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff42b501f17c9a18dbfc04ffba50ff0e/href">https://medium.com/media/ff42b501f17c9a18dbfc04ffba50ff0e/href</a></iframe><p>In our example, each TFRecord represents the movie ratings and corresponding suggestions of a single user (<em>a single sample</em>). Writing recommendations for all users in the dataset follows the same process. It is important that the type of a feature (e.g. float for the movie rating) is the same across all samples in the dataset. This conformance criterion and others are defined in the <a href="https://github.com/tensorflow/tensorflow/blob/r1.5/tensorflow/core/example/example.proto">protocol buffer definition</a> of tf.train.Example.</p><p>Here’s a complete example that writes the features to a TFRecord file, then reads the file back in and prints the parsed features.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/17dd40734e2ff04c35fb84d19c3cedea/href">https://medium.com/media/17dd40734e2ff04c35fb84d19c3cedea/href</a></iframe><p>Now that we’ve covered the structure of TFRecords, the process of reading them is straightforward:</p><ol><li>Read the TFRecord using a <a href="https://www.tensorflow.org/api_docs/python/tf/TFRecordReader">tf.TFRecordReader</a>.</li><li>Define the features you expect in the TFRecord by using <a href="https://www.tensorflow.org/api_docs/python/tf/FixedLenFeature">tf.FixedLenFeature</a> and <a href="https://www.tensorflow.org/api_docs/python/tf/VarLenFeature">tf.VarLenFeature</a>, depending on what has been defined during the definition of tf.train.Example.</li><li>Parse one tf.train.Example (one file) a time using <a href="https://www.tensorflow.org/api_docs/python/tf/parse_single_example">tf.parse_single_example</a>.</li></ol><h3>Movie recommendations using tf.train.SequenceExample</h3><blockquote>tf.train.SequenceExample is the right choice if you have features that consist of lists of identically typed data and maybe some contextual data.</blockquote><p>Now, let’s take a slightly different set of data, again from the <a href="https://github.com/tensorflow/tensorflow/blob/r1.5/tensorflow/core/example/example.proto">Tensorflow documentation</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sDDlsCCY8hOPj6HFjRA3Tw.jpeg" /></figure><p>We have a number of context features — <em>Locale, Age, and Favorites</em> — that are user specific and a list of movie recommendations of the user, which consist of <em>Movie Name, Movie Rating</em>, and <em>Actors</em>.</p><p>The data looks very similar — in the previous example we had a set of features, where each feature consisted of a single list. Each entry in the list represented the same information for a different movie, for instance the movie rating. This didn’t change, but now we also have <em>Actors</em>, which is a list of the actors with a role in the movie. This type of data cannot be stored in a tf.train.Example. We need a different type of structure for this kind of data, and Tensorflow provides it in the form of tf.train.SequenceExample. In contrast to tf.train.Example, it does not store a list of bytes, floats or int64s, but a *list of lists* of bytes, floats or int64s, and is thus suited for our dataset.</p><p>More formally, tf.train.SequenceExample has two attributes:</p><ul><li><em>context </em>of type tf.train.Features</li><li><em>features_lists</em> of type tf.train.FeatureLists</li></ul><p>The data from table “Context” is stored in <em>context </em>as tf.train.Features, just like we did for tf.train.Example. The data from table “Data” — <em>Movie Name, Movie Rating, and Actors —</em> are each stored in a separate tf.train.FeatureList each.</p><p><a href="https://www.tensorflow.org/api_docs/python/tf/train/FeatureList"><strong>tf.train.FeatureList</strong></a> has a single attribute <em>feature </em>that expects a list with entries of type tf.train.Feature. At first, this might look similar to tf.train.Features, which also contains multiple entries of type tf.train.Feature, but there are two big differences. First, all of the features in the list must have the same internal list type. Second, while tf.train.Features is a dictionary containing (unordered) named features, tf.train.FeatureList is a list containing ordered unnamed features.</p><p>A typical example of data stored in a tf.train.FeatureList would be a time series where each tf.train.Feature in the list is a time step of the sequence, or the list of actors for several differnt movies.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5fb9d7498b9bf002182cd7236537386d/href">https://medium.com/media/5fb9d7498b9bf002182cd7236537386d/href</a></iframe><p><a href="https://www.tensorflow.org/api_docs/python/tf/train/FeatureLists"><strong>tf.train.FeatureLists</strong></a> is a collection of named instances of tf.train.FeatureList. This component has a single attribute <em>feature_list</em> that expects a dict. The key of the dict is the name of the tf.train.FeatureList, while the value is the tf.train.FeatureList itself.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ecee4a2d28a0e748385ae6350edcc0c/href">https://medium.com/media/1ecee4a2d28a0e748385ae6350edcc0c/href</a></iframe><p><a href="https://www.tensorflow.org/api_docs/python/tf/train/SequenceExample"><strong>tf.train.SequenceExample</strong></a>, just like tf.train.Example, is one of the main components for structuring a TFRecord. In contrast to tf.train.Example, it has two attributes:</p><ol><li><em>context</em>: this attribute expects type tf.train.Features. It contains information that is relevant for each of the features in the <em>feature_list</em> attribute. The behavior of context is identical to the <em>features </em>attribute of tf.train.Example.</li><li><em>feature_lists</em>: the type of this attribute is tf.train.FeatureLists. It contains lists of features, where each feature again is some kind of sequential data (e.g. a time series, or frames).</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb935bb7481138a8e460ede27ba8f132/href">https://medium.com/media/cb935bb7481138a8e460ede27ba8f132/href</a></iframe><p>You can find more information about tf.train.SequenceExample in the <a href="https://github.com/tensorflow/tensorflow/blob/r1.5/tensorflow/core/example/example.proto">protocol buffer definition</a>. As a side note, while conformance criteria exist, they are not necessarily enforced — e.g. <em>feature_list_invalid</em> from the FeatureList example won’t throw a exception.</p><p>Here again is a full example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d97640b66e33edf475917d3b4c4f6cc8/href">https://medium.com/media/d97640b66e33edf475917d3b4c4f6cc8/href</a></iframe><p>Reading TFRecords based on tf.train.SequenceExample works the same as for tf.train.Examples. The only difference is that instead of just a single set of features, we need to define two: context and sequence features. Context features work exactly the same as shown before. Sequence features must be of type <a href="https://www.tensorflow.org/api_docs/python/tf/VarLenFeature">tf.VarLenFeature</a> or <a href="https://www.tensorflow.org/api_docs/python/tf/FixedLenSequenceFeature">tf.FixedLenSequenceFeature</a> and are parsed using <a href="https://www.tensorflow.org/api_docs/python/tf/parse_single_sequence_example">tf.parse_single_sequence_example</a>.</p><h3>Conclusion</h3><p>Using Tensorflow TFRecords is a convenient way to get your data into your machine learning pipeline, but understanding all the bits and pieces can be daunting at the beginning. The examples in this post should clarify the whole process and get you started.</p><p>You find all of the code snippets I used and more on <a href="https://gist.github.com/eega">Github </a>— feel free to copy and use them in any way you like.</p><p>I used a number of resources to create this story:</p><ul><li><a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/how_tos/reading_data/convert_to_records.py">The official tutorial</a> shows a coding example of how to write image data to a TFRecord using tf.train.Example.</li><li>This tutorial from <a href="http://machinelearninguru.com/deep_learning/tensorflow/basics/tfrecord/tfrecord.html">Machine Learning Guru</a> uses the example from the official tutorial and goes into more depth.</li><li>This <a href="https://stackoverflow.com/questions/45634450/what-are-the-advantages-of-using-tf-train-sequenceexample-over-tf-train-example">StackOverflow post</a> shows how to use tf.train.SequenceExample.</li></ul><h4>If you enjoyed this story then please click the 👏 button and share to help others find it! Follow me on <a href="https://twitter.com/thomas_gamauf">Twitter</a>/<a href="https://www.linkedin.com/in/thomas-gamauf-15aa96105/">LinkedIn</a>, or follow <a href="https://medium.com/@mostly_ai">Mostly AI</a>, if you want to read more from me.</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c46bc4bbb564" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mostly-ai/tensorflow-records-what-they-are-and-how-to-use-them-c46bc4bbb564">Tensorflow Records? What they are and how to use them</a> was originally published in <a href="https://medium.com/mostly-ai">Mostly AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>