<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sam Tarakajian on Medium]]></title>
        <description><![CDATA[Stories by Sam Tarakajian on Medium]]></description>
        <link>https://medium.com/@starakaj?source=rss-a5f942c18ae1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6xWcF5sz3Wn6C6euAOoJIg.jpeg</url>
            <title>Stories by Sam Tarakajian on Medium</title>
            <link>https://medium.com/@starakaj?source=rss-a5f942c18ae1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:22:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@starakaj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An Open Letter to the Armenian National Committee of America]]></title>
            <link>https://medium.com/@starakaj/an-open-letter-to-the-armenian-national-committee-of-america-ee9fafc05ec6?source=rss-a5f942c18ae1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee9fafc05ec6</guid>
            <category><![CDATA[open-letter]]></category>
            <category><![CDATA[armenia]]></category>
            <category><![CDATA[reparations]]></category>
            <category><![CDATA[blacklivesmatter]]></category>
            <dc:creator><![CDATA[Sam Tarakajian]]></dc:creator>
            <pubDate>Fri, 05 Jun 2020 14:09:39 GMT</pubDate>
            <atom:updated>2020-06-05T14:11:52.889Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P7eN8UzLsJz058MccieL4w.png" /><figcaption>Poster by <a href="https://www.aratherat.com/">Ara the Rat</a></figcaption></figure><p>Hi, my name is Sam Tarakajian. As you can probably guess from my last name, my father is Armenian. His father’s father came to America in the early part of the twentieth century, escaping the region around Diyarbakır in modern-day Turkey.</p><p>I’m calling on the <a href="https://anca.org/">Armenian National Committee of America</a> to make a strong statement of support for the Black Lives Matter movement generally, and further to support a policy of reparations for Black Americans in the United States.</p><p>As you well know, Armenians have the unfortunate distinction of being the target of systematic persecution and extermination — genocide — in the first part of the twentieth century. The Turkish courts-martial of 1919–20 found the massacres to be war crimes, and sentenced the perpetrators to death. The very next year, the resurgence of the Turkish National Movement granted amnesty to those found guilty. Under Mustafa Kemal Atatürk, the Turkish<strong> </strong>government adopted a policy of denial that <a href="https://web.archive.org/web/20170726153405/http://www.qgazette.com/weekinreview/75979">continues to this day</a>. Thus Armenians live with the double insult of both horrific suffering as well as denial of that suffering.</p><p>Today, Black Americans live with a similar burden. Their pain dates to before the founding of this country, in fact this country was built on the unpaid labor — enslavement — of Black people. Since their emancipation,<strong> </strong>they have been dogged by systemic discrimination, oppression, and violence: through <a href="https://www.theatlantic.com/business/archive/2014/05/the-racist-housing-policy-that-made-your-neighborhood/371439/">racist housing policies</a>, <a href="https://www.pnas.org/content/114/41/10870.full">discriminatory hiring practices</a>, segregated schooling, and outright lynching. To this day, Black Americans are <a href="https://www.naacp.org/criminal-justice-fact-sheet/">more likely to find themselves imprisoned than White Americans</a>, <a href="https://www.pnas.org/content/116/34/16793">more likely to be attacked by the police than Whites</a>, and <a href="https://ucr.fbi.gov/hate-crime/2018/topic-pages/victims">more likely to be the victims of hate crimes than Whites</a>. Still, Americans live in denial. We deny that systemic racism exists. We deny that White poverty and Black poverty are different. We deny state violence for what it is, and we deny the humanity of people protesting against it.</p><p>Armenians know too well how painful denial can be, and how vital it is for the perpetrator<strong>s</strong> of racism to begin actively undoing the evils of the past. We Armenians today can look at two countries, Turkey and Germany, and recognize instantly the difference between <a href="https://qz.com/1680558/for-slavery-reparations-the-us-can-look-to-post-holocaust-germany/">a country that has engaged in a determined and persistent policy of reconciliation</a>, and one that <a href="https://www.dw.com/en/erdogan-turkey-will-never-accept-genocide-charges/a-19307115">still denies the truth</a>. We know that real, lasting change takes deliberate, top-down work — work that is often unpopular from the outset. We must stand by Black Americans and demand that the United States begin a lasting tradition of making amends, of presenting for honest, unflinching scrutiny the crimes of the past.</p><p>Writing for <a href="https://www.theatlantic.com/magazine/archive/2014/06/the-case-for-reparations/361631/">The Atlantic</a>, Ta-Nehisi Coates says that reparations can not “make up for murder,” but that they begin a reckoning, one that could ultimately reveal “how a great civilization might make itself worthy of the name.”</p><p>I urge you strongly to rally Armenian Americans around this issue. I feel it is our duty to make sure that the countries of the present do not repeat the sins of the past.</p><p>Thank you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee9fafc05ec6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why data is easier than gardening: using Kibana to understand art]]></title>
            <link>https://medium.com/barnes-foundation/why-data-is-easier-than-gardening-using-kibana-to-understand-art-785a2f3690d?source=rss-a5f942c18ae1------2</link>
            <guid isPermaLink="false">https://medium.com/p/785a2f3690d</guid>
            <category><![CDATA[barnescollection]]></category>
            <category><![CDATA[kibana]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[musetech]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Sam Tarakajian]]></dc:creator>
            <pubDate>Wed, 24 May 2017 15:01:17 GMT</pubDate>
            <atom:updated>2017-05-24T15:01:39.820Z</atom:updated>
            <content:encoded><![CDATA[<p>For my birthday last year, a friend bought me an herb garden.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*-GL1i7MpEBsoB3s3HN7bBg.jpeg" /><figcaption>Not as nice as this one</figcaption></figure><p>Fresh herbs! Tired of going to the supermarket to buy a gallon of basil when you only need a tablespoon? Just reach over to your own personal herb garden and take what you need.</p><p>There’s just one small problem: I don’t cook. I’d like to, but instead I buy the occasional cookbook, promise myself I’m going to cook more and then break my promise by ordering dumplings on the internet. And so the herb garden, instead of transporting me to rapturous heights of culinary excellence, becomes an increasingly dry and brown reminder of my inability to start cooking.</p><p>In this series, we’re writing about working with the Barnes Foundation to build <a href="https://medium.com/barnes-foundation/rethinking-the-museum-collection-online-e3b864d8bb39">a new way to explore their collection online</a>. Previously we’ve talked about getting our data <a href="http://medium.com/@starakaj/breaking-away-from-tms-84c28e059e33">out of TMS</a> and pushing that data into <a href="https://medium.com/@starakaj/elasticsurf-is-not-a-made-up-word-4c26aec0b4a0">Elasticsearch</a>. But now that it’s in Elasticsearch, what do we do with it? Or is the database doomed to become like my herb garden, an amazing resource that we don’t have the time to learn how to use?</p><p>Fortunately there’s <a href="https://www.elastic.co/products/kibana">Kibana</a>. One of the main reasons that we decided to go with Elasticsearch is that the plugins and support tools are so great. Chief among those tools is Kibana, useful for monitoring, visualizing, and basically getting a sense of what our data actually looks like.</p><h4>Painting by Queries</h4><p>Once you’ve got your data indexed, the first thing you’ll want to do is ask some questions about it. “Who are all the painters in the collection?” for example, is something I wanted to know. Kibana has a tab called “Discover” that lets us explore the data in the index by sending search queries and adding filters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9PrkoPoUK81Cty0ek3gosQ.png" /></figure><p>To the left of this view there’s a pane titled “Available Fields,” which lists all of the fields in the Elasticsearch index. I’m interested in paintings, so I can click on the “classification” field and then add a filter for paintings by clicking on the magnifying class with the plus sign in it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7MVuagtSZnvuL7W-NvDHDQ.png" /></figure><p>With this filter added, the number of hits, displayed in the top-left of the window, drops from 2217 to 961—the number of items in the collection classified as paintings. I may want to use this search later, to create an aggregation for example, so I’ll click on “Save” at the top of the window and save this search as “Paintings.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H1pZR8aSNRFQJ8L1JIJc0A.png" /></figure><p>Now I’d like to create a visualization, so I can group paintings by the person who painted them. I click on “Visualize” in the lefthand navigation menu, and then on the blue plus button to create a new visualization. There’s a lot of choices for the type of visualization to create here. I’ve heard somewhere that every chart has a purpose, except the pie chart which is always wrong. Whatever, I’m picking the pie chart. In the next view, I can click on the “Paintings” search that I saved before to create a pie chart from that search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJEvK8JUN2aVZpN-bmIAaA.png" /></figure><p>And lo, welcome to the least informative pie chart of all time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YnrDN6hr1kOMWSvJ9KRZeA.png" /><figcaption>It doesn’t even look very tasty</figcaption></figure><p>If we want to make this pie chart a bit more useful, then we have to click on “Split Slices” to create some pie slices. Now under the word Aggregation, there’s a drop-down that lets us pick what we’d like to aggregate over. I’ll pick “Terms”, since I want to aggregate over the maker of the painting. Now the “Field” drop-down appears, and I can select “people.keyword.” Finally, I can press the blue “play” button at the top of the view to load the data into the pie. And then I get something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nWlnAMhB3U_noonMJv23wA.png" /></figure><p>Nice! Never has it been more clear how strongly the Barnes Collection favors Renoir. One thing I’d like to change, however, is the presence of “Unidentified artist” as an artist. Fortunately, it’s super easy to add a filter. We can just go back to the saved “Paintings” search, click on “people,” and then on the magnifying glass with the “minus” sign in it to add a filter that excludes this value for the “people” field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mkvwgjlc6oikI7H4mAPbrw.png" /></figure><p>Now when I save the search and return to the visualization, the pie updates automatically. Charles Demuth has joined the party.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OSVLaX2hMSO-xJOQSdRwMg.png" /></figure><p>Alright, so that’s pretty cool, but let’s get a little more advanced. What if I want to see a breakdown of how many paintings by each painter appear in each room? It’s time for a sub-aggregation, or an aggregation within an aggregation. This time, I’ll pick a bar chart, and I’ll start with the same saved “Paintings” search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zjtcdXcweLos_kREAu_4aw.png" /><figcaption>Informative</figcaption></figure><p>Again, this bar chart is not currently very useful. All it tells us is that all 900 or so paintings are indeed paintings. Let’s start by adding a bucket along the x-axis, aggregating on the “room” field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C9QXS0td06BtlJl9Vz0dkg.png" /></figure><p>Alright, so now this is showing us that Room 17 has the most paintings, with almost twice as many paintings as Room 13. But now for the really cool part. Press the button that says “Add sub buckets,” and then click on “Split Series.” Let’s add a sub-aggregation over the “people.keyword” field, limiting it to the top 5 painters per room.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*51_vY2nLOF2QyfGTEyJ_Ow.png" /></figure><p>How cool is that? We can see that even though Renoir has the most paintings in the collection, he’s totally absent from Room 17, which has more paintings than any other room. The king of Room 17 is actually our old friend Demuth. Let’s have a Demuth to celebrate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ng4OKkSjxd_N8WKqg7bwiQ.jpeg" /><figcaption>Charles Demuth. Count Muffat Discovers Nana with the Marquis de Chouard. 1915. BF1071 — How I feel using MySQL.</figcaption></figure><p>The important takeaway here is that I was able to make these quick visualizations without having to know anything about Elasticsearch. I didn’t have to learn the query syntax, or write code for an Elasticsearch client. All I had to do was load up Kibana, click a few buttons, and I was able to make an interactive graph that taught me something interesting about the structure of the Barnes Collection.</p><p>If only using my herb garden were so easy.</p><p>The Barnes Foundation collection online project is funded by the Knight Foundation and our code is <a href="https://github.com/BarnesFoundation/CollectionWebsite">open source</a>. Follow the <a href="https://medium.com/barnes-foundation">Barnes Foundation</a> on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=785a2f3690d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/barnes-foundation/why-data-is-easier-than-gardening-using-kibana-to-understand-art-785a2f3690d">Why data is easier than gardening: using Kibana to understand art</a> was originally published in <a href="https://medium.com/barnes-foundation">Barnes Foundation</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elasticsurf is not a Made Up Word]]></title>
            <link>https://medium.com/barnes-foundation/elasticsurf-is-not-a-made-up-word-4c26aec0b4a0?source=rss-a5f942c18ae1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c26aec0b4a0</guid>
            <category><![CDATA[surfing]]></category>
            <category><![CDATA[metadata]]></category>
            <category><![CDATA[barnescollection]]></category>
            <category><![CDATA[musetech]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Sam Tarakajian]]></dc:creator>
            <pubDate>Thu, 11 May 2017 14:53:04 GMT</pubDate>
            <atom:updated>2017-05-18T15:26:56.872Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p>I want you to imagine something. Imagine you’re out on the ocean, surfing. You catch a big, beautiful wave, and then, riding it to shore, with the sunlight glinting gold off the water and the taste of sea salt in the air, you wonder, “What would the exact opposite of this experience be?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*6BhLc-4RTfOlCSCdquY07Q.jpeg" /><figcaption>Here’s a hint</figcaption></figure><p><a href="https://medium.com/barnes-foundation/rethinking-the-museum-collection-online-e3b864d8bb39">Remember</a>, we’re working with the Barnes to bring the experience of visiting their collection in person to the browser. To do that, we want the collection to feel less like a spreadsheet and more like a big, smooth ocean. We want to help users to get into a state of art flow—picking a direction, surfing through the collection, finding new things without having to think too much about how the data is organized. In other words, a painless search. A fun search.<a href="https://www.elastic.co/"> Elasticsearch</a>.</p><p><a href="https://medium.com/barnes-foundation/breaking-away-from-tms-84c28e059e33">In a previous post</a>, I talked quite a bit about why <a href="http://www.gallerysystems.com/products-and-services/tms/">TMS</a> wasn’t that great for our purposes. But even assuming we’re all in agreement about that, what makes Elasticsearch so great? We think there are four main reasons.</p><h4>I’m in Love with the Shape of our Data</h4><p>Maybe you’ve worked with a Relational Database Management System (RDMBS) before, something like <a href="https://www.mysql.com/">MySQL</a>. One of the challenges with a system like this is adding new data, especially when the shape of the data changes from one iteration to the next. Suppose we’d already created a collection database using something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e5fc27b7d53b2981ad5529090b9131b/href">https://medium.com/media/9e5fc27b7d53b2981ad5529090b9131b/href</a></iframe><p>But now, we decide that we want to add a new item to the database, only this one not only has an id, name, and description, but also a maker.</p><p>&lt;sarcasm&gt;Great.&lt;/sarcasm&gt;</p><p>Now we need to add a new column (what’s the syntax for that again?), pick a default value for that column (should it be null? an empty string? how many characters should it be?) and then finally we can insert our new entry into the collection.</p><p>The way I see it, the problem is that we have to take our data, which is already nice, structured JSON, and translate it into SQL queries in order to update the database. Allow me to contrast this with how you add new data to an Elasticsearch index (“index” means “database” in Elasticsearch).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6eb8302a0844d748f37939329e1df5a2/href">https://medium.com/media/6eb8302a0844d748f37939329e1df5a2/href</a></iframe><p>That’s it. No worrying about types or varchar lengths, primary keys or indexes, or whether anything is in the store already. Like many people, we may not know right from the start exactly how our data should be structured. As we work, typically we make a guess, find a bug in our prototype and then update our model. Having to organize a data migration every single time would be at best a headache and at worst a massive time-suck.</p><h4>It’s Fast</h4><p>We’re interested in two basic things here: how fast can we build a new index from our CSV data files, and how fast can we execute queries? Well, we’ve got about 5000 rows of data, and the whole thing takes about 5 seconds to import into Elasticsearch. Mind you, that includes tokenizing and stemming all the text (more on that in a second) and building an inverted index on all our words.</p><p>As for queries, anecdotally I can tell you they’re really fast too. Of course, 5000 rows is quite small for a database, so perhaps much larger collections would have to employ some optimization.</p><p>For more on how fast Elasticsearch is, check out more <a href="https://www.elastic.co/blog/elasticsearch-queries-or-term-queries-are-really-fast">writing on the subject</a>.</p><h4>It Scales</h4><p>Elasticsearch has been, in my personal experience, almost <em>too </em>eager to scale up. The first time I started Elasticsearch I accidentally split my data across 5 shards (Elasticsearch distributes its data across stores called “shards”), each of which had its own backup shard. Not particularly useful when I’m just trying to run search on my laptop, but it does mean that wherever we choose to deploy Elasticsearch, it’s going to be able to scale up to whatever size we need.</p><h4>Full-Text Search</h4><p>I like beards. Who doesn’t?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*2hDmQWudY0P3mtBg86WXRA.jpeg" /><figcaption>Portrait of a Man with Beard and Ruff Collar. Judith Leyster</figcaption></figure><p>Say you wanted to find all of the paintings in the Barnes collection that contained a beard. Luckily for you, each object in the collection has a description. So you could quite easily fire off a query like this one:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/24b5565907fb700efdc96383d2f88946/href">https://medium.com/media/24b5565907fb700efdc96383d2f88946/href</a></iframe><p>There’s just one problem: what if a collection description includes the word <em>beards</em>? Or <em>bearded</em>? Or <em>beardelicious</em>? Unsurprisingly, our search for descriptions containing the word “beard” will miss this one. Luckily, it’s extremely easy to configure Elasticsearch to create a field for us that is <em>stemmed</em>, meaning Elasticsearch will do the work of converting the word “bearded” to its stem, “beard,” before storing the word. This is done by creating an Elasticsearch <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html">mapping</a>. First, you need to create an analyzer that Elasticsearch can run on the text before indexing it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca25139adbf78419b9e91173e3855346/href">https://medium.com/media/ca25139adbf78419b9e91173e3855346/href</a></iframe><p>Later, in the mapping for the field you’d like stemmed and tokenized, you apply this analyzer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90759a177d0b2b7fac69cd722fa26c39/href">https://medium.com/media/90759a177d0b2b7fac69cd722fa26c39/href</a></iframe><p>With this mapping in place, the Elasticsearch server will automatically analyze the description of each object, extracting the stems of each word according to the English language stemmer. By the way, this stemmer wasn’t something we wrote or installed or even configured—it comes prepackaged with Elasticsearch along with stemmers for French, German, Armenian and a whole bunch of other languages as well.</p><p>As a result, we can search for the string beard and return results that contain the string bearded. In the case that the original search query was the string bearded, we can even search for the stem beard but prioritize the full string bearded, since we’ve stored both the stemmed as well as the unstemmed description:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e32e64d90dadcc792d3af55289f91feb/href">https://medium.com/media/e32e64d90dadcc792d3af55289f91feb/href</a></iframe><p>Using this query, I was able to find this highly bearded painting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*usGW6c2chiH1X2YWHfXMtg.jpeg" /><figcaption>The Postman (Joseph-Étienne Roulin). 1889. Oil on canvas. 25 7/8 x 21 3/4 in. (65.7 x 55.2 cm). BF37</figcaption></figure><p>In short, Elasticsearch is making it easy and, dare I say, fun, to play with the data in the Barnes collection. Analyzing, comparing and discovering relationships in our data is feeling less like staring at a spreadsheet and more like surfing. Surfing on an ocean of art.</p><p>If you’re curious, why not try <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html">setting up Elasticsearch yourself</a>? Next time we’ll talk about Kibana, and some of the fun stuff that we’re doing with our data on the back end.</p><p>The Barnes Foundation collection online project is funded by the Knight Foundation and our code is <a href="https://github.com/BarnesFoundation/CollectionWebsite">open source</a>. Follow the <a href="https://medium.com/barnes-foundation">Barnes Foundation</a> on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c26aec0b4a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/barnes-foundation/elasticsurf-is-not-a-made-up-word-4c26aec0b4a0">Elasticsurf is not a Made Up Word</a> was originally published in <a href="https://medium.com/barnes-foundation">Barnes Foundation</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Breaking away from TMS]]></title>
            <link>https://medium.com/barnes-foundation/breaking-away-from-tms-84c28e059e33?source=rss-a5f942c18ae1------2</link>
            <guid isPermaLink="false">https://medium.com/p/84c28e059e33</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[musetech]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[barnescollection]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Sam Tarakajian]]></dc:creator>
            <pubDate>Thu, 13 Apr 2017 13:20:51 GMT</pubDate>
            <atom:updated>2017-05-24T15:01:20.441Z</atom:updated>
            <content:encoded><![CDATA[<p>A visit to the Barnes means discovering unexpected and exciting contrasts between works from different eras, places and cultures. At <a href="http://girlfriends.site/">Girlfriends Labs</a>, we want to bring that same experience to the collection online. We’re building a tool that imitates Barnes’s unique way of seeing art, through the visual relationships between pieces (You can read more about Albert Barnes’s approach to curation in our <a href="https://medium.com/@shell7/rethinking-the-museum-collection-online-e3b864d8bb39">first post on this project</a>).</p><p>Before I dive into TMS and our backend architecture, let’s look at some art.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*mlyZrPVbhvcp5EPtxQRUbA.jpeg" /><figcaption>I think art is very healing. Don’t you?</figcaption></figure><h4>Backend Requirements</h4><p>Suppose we want know which oil paintings from French makers appear on west-facing walls of the Barnes Museum. If that’s what it takes to build the online collection, then we need a datastore that can process complex queries. Also, we want to extract information about the visual relationships between artworks in the collection. Therefore, we need to build an extensible system, one to which we can add image processing modules as we go. There’s an added constraint as well: we need to do all of this without upsetting TMS, the system that people at the museum are using right now to manage the collection (more on that in a minute).</p><p>We decided that the best way to have the tools that we wanted given our constraints was to write a collection of scripts that would automatically pull data from TMS into Elasticsearch, organized so that we could easily add image processing later as we worked out how to do it. We’ll be talking about some of these components in a bit more detail in upcoming blog posts, but for now here’s a high level view of all of the components and how they fit together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kV0K04-6_yK3BlyJevEOHA.png" /><figcaption>Fitting it all together</figcaption></figure><h4>Emigrating from TMS</h4><p><a href="http://www.gallerysystems.com/products-and-services/tms/">The Museum System</a> (TMS) is the world’s leading collection management software. Museums all over the world use it to store their collections, exhibitions and more. The Barnes Foundation has been using TMS for years to hold its collection data, and so you might well ask, why do all this work to move the data to Elasticsearch, when it’s already sitting in TMS ready to be used? Well, for one, TMS won’t let us execute structured queries. If we wanted to know which French oil paintings are on a west-facing wall, it’s just not possible to access that information with TMS alone.</p><p>However, even if TMS did support structured queries, we’d still want to find a way to move away from it. We’ve already encountered our share of unpleasant quirks (we lost about a day learning that TMS seems to store text not in utf-8, not in cp-1252, but in cp-1252 <em>encoded </em>utf-8), and we’ve found that people who work with TMS on a daily basis have had even worse experiences. Changing TMS is clunky and slow—after you’ve added new collection objects to TMS you can’t actually look at your data until it’s finished a mysterious process known as “re-indexing.” Also, while TMS purports to have an API, it’s only available after purchasing the whole eMuseum product suite. That’s a lot to ask if all you want to do is use the TMS API.</p><p>The core problem with TMS is that makes it hard to see our own data. Which is exactly the opposite of what we want from our datastore.</p><h4>Pulling from the API</h4><p>In order to pull data from TMS, we wrote a Node.js script to automatically iterate through the collection and pull JSON data for each object. If you look at <a href="https://github.com/BarnesFoundation/CollectionWebsite">our repository</a>, you’ll see in config/export.json we specify the URL that is currently serving our TMS API</p><pre>// from config/export.json<br>{<br> &quot;TMS&quot;: {<br>  &quot;export&quot;: {<br>   &quot;apiURL&quot;: https://the.barnes.emuseum.api,<br>   &quot;outputDirectory&quot;: &quot;src/dashboard/public/output&quot;,<br>...</pre><p>To read from the API, we create an instance of TMSURLReader (which you can find in src/tms_csv/src/script/tmsURLReader.js) using the API URL. That object has a function called _urlForObjectWithId that we use to get the URL corresponding to a particular object in the collection.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8348483b91f7754568074f8164a141b6/href">https://medium.com/media/8348483b91f7754568074f8164a141b6/href</a></iframe><p>To export from TMS to a CSV file, we simply call TMSURLExporter.next until the API stops returning new objects. To write to a CSV file, we use a Node package called fast-csv, which makes everything ridiculously easy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/56412a5986abb2625a0dccbbfb52bd16/href">https://medium.com/media/56412a5986abb2625a0dccbbfb52bd16/href</a></iframe><p>If you want to try running the TMS to CSV export script yourself, against your own TMS API, just modify config/export.json and run npm run tms-export from the repository root.</p><h4>The Value of Comma Separated Values</h4><p>Rather than export data directly from TMS into Elasticsearch, we decided to export a CSV (comma separated value) as an intermediate step. Why? First and foremost, a CSV file is readable. It’s about as readable as a non-prose file can be. It’s hard to overstate how nice it is to be able to look at a file with your own, human eyes and say, “Indeed, this is my data.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*25Zjj9FCq0awhb0xb-GBGA.png" /><figcaption>Very reassuring</figcaption></figure><p>Rather than having our data floating around in TMS, with a CSV file we have a database-agnostic data blob that we can pass around conveniently. Later we may build an Elasticsearch index with caches and views and nested data types and other such trickery for speeding up queries or performing advanced searches, but the CSV is just Big Dumb Data. And sometimes that’s nice.</p><h4>An Elasticsearch Appetizer</h4><p>Elasticsearch is a flexible, modern search and analytics engine that can also be used as a datastore. With our data in Elasticsearch, the world is our oyster. Remember before when I wanted to know about west-facing French oil paintings? With Elasticsearch getting the answer couldn’t be simpler (you just have to remember that west-facing works are on the east wall).</p><pre>GET collection/object/_search<br>{<br>  &quot;_source&quot;: &quot;title&quot;,<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;must&quot;: [<br>        { &quot;match&quot;: {<br>          &quot;locations&quot;: {<br>            &quot;query&quot;: &quot;East Wall&quot;,<br>            &quot;type&quot;: &quot;phrase&quot;<br>          }<br>        } },<br>        { &quot;match&quot;: { &quot;culture&quot;: &quot;French&quot; } },<br>        { &quot;match&quot;: { &quot;classification&quot;: &quot;paintings&quot; } },<br>        { &quot;match&quot;: { &quot;medium&quot;: &quot;oil&quot; } }<br>      ]<br>    }<br>  }<br>}</pre><p>Using this query, I can tell you that there are in fact two west-facing French oil paintings in the Barnes Collection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*MAOZbFJ63gblAslOhYpA_g.jpeg" /><figcaption>Child Holding Fruit. c. 1840. Oil on canvas. 21 3/4 x 18 1/4 in. (55.2 x 46.4 cm) BF554</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*Fy2v0IVkQn5ermTnzR4ILQ.jpeg" /><figcaption>Bishop Saint, Saint Roch, and Saint Sebastian. c. 1460–1480. Oil on panel. 30 1/2 x 28 7/8 in. (77.5 x 73.3 cm) BF418</figcaption></figure><h4>More to come</h4><p>We’ve only just scratched the surface of some of the cool possibilities that Elasticsearch opens up for visualizing and playing with your data. In our next few posts, we’ll be diving deeper into our Elasticsearch configuration, exploring monitoring and discovery with Kibana, and taking a look at tying our backend together with Seneca.js. Until then!</p><p>The Barnes Foundation collection online project is funded by the Knight Foundation and our code is <a href="https://github.com/BarnesFoundation/CollectionWebsite">open source</a>. Follow the <a href="https://medium.com/barnes-foundation">Barnes Foundation</a> on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84c28e059e33" width="1" height="1" alt=""><hr><p><a href="https://medium.com/barnes-foundation/breaking-away-from-tms-84c28e059e33">Breaking away from TMS</a> was originally published in <a href="https://medium.com/barnes-foundation">Barnes Foundation</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>