<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Namara - Medium]]></title>
        <description><![CDATA[Namara is the data management platform created, designed, and maintained by ThinkData Works, a Toronto-based startup that’s easing access to external data for everyone from civic hackers to business leaders. - Medium]]></description>
        <link>https://medium.com/namara?source=rss----7fe2f9f63a30---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Namara - Medium</title>
            <link>https://medium.com/namara?source=rss----7fe2f9f63a30---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:26:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/namara" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The data every city should release, and why]]></title>
            <link>https://medium.com/namara/the-data-every-city-should-release-and-why-2fc088268ad5?source=rss----7fe2f9f63a30---4</link>
            <guid isPermaLink="false">https://medium.com/p/2fc088268ad5</guid>
            <category><![CDATA[government]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Lewis Wynne-Jones]]></dc:creator>
            <pubDate>Tue, 15 Aug 2017 13:42:11 GMT</pubDate>
            <atom:updated>2017-08-15T13:42:11.027Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*NYh7ZfimltvEj0fJCSe9Ww.png" /></figure><p>After spending the majority of every day looking at thousands of open data sets from hundreds of municipalities across North America, I came to some conclusions about the data that everyone was releasing. My main realization was this:</p><p>No one is doing the same thing.</p><p>That’s great, on one hand, because it means that governments are <em>trying things out. </em>I like this. It’s great to see a government try new things. On the other hand, it’s really profoundly difficult to work on any project involving public data if you can’t reasonably predict what’s going to be available on any given data portal.</p><p>Want to track spending? Very difficult to do if you can’t get city financials from half of the jurisdictions you’re looking at. Want to build an app that tells you how the commute’s going to be? Better hope Peel and Mississauga are releasing the same traffic data as Toronto.</p><p>Up to this point, the open data success stories we’ve seen follow a similar pattern, which generally looks like this: Someone used <em>their city’s</em> data to build something cool for <em>their city</em>.</p><p>That’s great. Building a killer transit app is great. The problem is, it’s small potatoes.</p><p>Now what if someone built the same app with national coverage? What if city officials could easily check their performance against neighbouring communities? What if cancer researchers in Barcelona had access to the research being done in Chennai? The more we tear down the silos that separate one city’s data from another the more problems we’ll be able to solve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*nLVm2oryqFHFsSeG8lEwnQ.png" /></figure><p>This is the sort of thing you start to think about when you’re looking at hundreds of thousands of open data sets. It’s a big opportunity. The whole thing has to start with the cities themselves, and it absolutely has to start with a baseline of common data.</p><p>We were thinking about this for a while, so we decided to put together a list of data sets that we felt every city should release. The data sets we decided on were by no means definitive, but they’re a good place to start. These are the data sets that most cities already have access to, and they’re the ones that drive immediate value, both to citizens and businesses.</p><p>After plotting these on a map we realized that they all seemed to emanate from 6 major themes, so we built a wheel that showed how the data sets might relate to each other. After thinking about it for a while, we realized that these data sets were really the elemental ones; the ones that you could mix together to create totally new data, derive different information, and see different use cases. When we realized <em>that</em>, we started calling it the Open Data Big Bang.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*dv-S6Gsg_TsZo6Yfk0Q3lw.jpeg" /></figure><p>An open data strategy doesn’t have to be written in stone. It should, in fact, be agile enough to transform as the uses of public data become better understood. But every open data strategy, whether it’s for New York City or Moose Jaw, can start with these elemental data sets. You’ll be amazed at what they’ll create.</p><p><a href="http://namara.io/"><em>Namara</em></a><em> is the data management platform created and maintained by ThinkData Works. Sign up today to start using all the open data in North America.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fc088268ad5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namara/the-data-every-city-should-release-and-why-2fc088268ad5">The data every city should release, and why</a> was originally published in <a href="https://medium.com/namara">Namara</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unity Case Study: Standardizing and Aggregating Address Points]]></title>
            <link>https://medium.com/namara/unity-case-study-standardizing-and-aggregating-address-points-90b12343c0f8?source=rss----7fe2f9f63a30---4</link>
            <guid isPermaLink="false">https://medium.com/p/90b12343c0f8</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Lewis Wynne-Jones]]></dc:creator>
            <pubDate>Tue, 15 Aug 2017 13:41:04 GMT</pubDate>
            <atom:updated>2017-08-15T13:44:07.173Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*96AAG3E876OAQYJWEud-jw.png" /></figure><h4>The Data</h4><p>Address points are one of the most critically important data sets that any city can release on their open data portals. They provide an invaluable geospatial base layer to virtually any app.</p><p>A quick search for “Address Points” on <a href="https://namara.io/#/search/open?query=address%20points&amp;order=relevance&amp;page=1&amp;states=imported">Namara</a> tells me that there are over 700 address point data sets available on the platform. These range in size from the <a href="https://namara.io/#/data_sets/d19dea19-b86e-4de4-a03c-570b9b0aaf49">Kodiak Island Borough</a> in Alaska (7764 rows) to <a href="https://namara.io/#/data_sets/762515aa-2cd9-4b16-90e0-7a1cda9e8e06">New York City</a> (942,321 rows), and while some of these data sets are statewide/provincial (way to go, Utah!) and a number of them are by county, the vast majority of them are by city. Since they’re all released in different places, collected by different agencies, and published in different ways, it’s hard to find any two among the 700 that look the same.</p><h4>The Problem</h4><p>Since addresses are usually <em>maintained</em> at the municipal level, that means they’re usually <em>released</em> at the municipal level.</p><p>This is fine if you want to build an app that doesn’t extend past your city limits, but if you’ve set your mind on something bigger, you’re going to have to get your hands dirty.</p><p>Take these three address point data sets from the Waterloo, Toronto, and Ottawa open data portals:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8mD2AV4dFtSqOzvzAnEWsg.png" /></figure><p>A quick look at each tells me that there’s a lot of variation in what’s offered in a standard address point data set. Ultimately, though, it boils down to an address and a coordinate.</p><h4>The Solution: Aggregating in Unity</h4><p>To merge these data sets in Unity, I need to define a schema, my <em>ideal output.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*Xt3MYzxLlOagGDUbf32ziw.png" /><figcaption>note the default value in “province”</figcaption></figure><p>The above contains the requisite address information, the metadata from the source (so I can keep track of when the data sets have updated), and geometry, so I can plot the aggregate on a map.</p><p>After I’ve defined the schema, I need to bring the graphs into Unity and map their attributes to the schema that I’ve already set up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*U7BikVCaEnqU42YbNGh1-A.png" /><figcaption>Ottawa Addresses mapped to the ideal schema</figcaption></figure><p>The Ottawa address data set maps pretty cleanly to the ideal schema. Despite lacking ward information and some differences in naming conventions (<em>addrnum </em>on theirs, <em>street_number </em>on mine, etc), the only thing I need to actually add<em> </em>is a static node specifying that for this graph the default value in <em>city </em>should be “Ottawa”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*FI5op62iiuUDbfaBuZ3V3A.png" /><figcaption>Toronto Addresses to the ideal schema</figcaption></figure><p>The Toronto Address graph is also relatively straightforward. Since I prefer my <em>street_name </em>and <em>ward_name </em>uppercased, I’ve added that transformation. Looking at the Toronto data set, I can see that a lot of the values in <em>fcode_des </em>are marked as unknown. Since neither of the other data sets acknowledge unknown values I added a replace node to look for instances of the word “Unknown” and replace them with null.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/1*Sh76GojinmUV9IknzvnwGA.png" /></figure><p>Finally, since the original graph’s <em>ward_name </em>provides ward name and number in the same column, I added two extract nodes and plugged in a regular expression that pulls out the name and number separately. It took two minutes, and it means that later I’ll be able to query the data better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*XU4k7ktE1EkmikYQC899nQ.png" /><figcaption>Waterloo Addresses to the ideal schema</figcaption></figure><p>Waterloo’s address point data set doesn’t have a column that maps cleanly to <em>street_number</em>, but I can see that the data in <em>civic_addr </em>is always recorded with a space between the street number and the street name, so I split the value on the first space and plugged that value into <em>street_number. </em>I want to make sure I’m only pulling numbers, though, so I add in a “to_integer” node to make sure I’m only grabbing what I want.</p><p>The rest of the graph lines up well, with the exception of <em>ward. </em>In the original data set, Waterloo only provides the ward number as a data point. I could easily map that and leave <em>ward_name</em> blank, but I would prefer to have both values in my final data set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*LjI0V4rS-Wv04u1Oz2IbQw.png" /></figure><p>Since Waterloo has pretty great open data, I decided to see if they released a ward map data set. Turns out they do.</p><p>Using the Namara node on Unity, I can point towards the ward data set on Namara, match on <em>ward_no, </em>and output the ward <em>name </em>(which on <a href="https://namara.io/#/data_sets/2db045bd-a7d0-4096-bcf8-bcde13140408">the data set</a> is called <em>ward_2014</em>).</p><p>So by matching the values in one data set with the corresponding value in a second data set, I can add a totally new attribute to my final aggregate.</p><p>Once the data sets have all been mapped to the schema, I can kick off a Unity run and then upload the data to Namara, where it will update automatically.</p><p>Check out the final product <a href="https://namara.io/#/data_sets/42925673-3ba2-428f-8e2f-6321e19d0bf5">here</a>.</p><p><em>A note regarding Unity screenshots: Unity is currently an internal feature and the GUI is in prototype. There are UX and UI components that are in the design phase and cannot be displayed.</em></p><p><em>For more Unity case studies, or to try it out for yourself, please don’t hesitate to </em><a href="/service/mailto:info@thinkdataworks.com"><em>get in touch</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90b12343c0f8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namara/unity-case-study-standardizing-and-aggregating-address-points-90b12343c0f8">Unity Case Study: Standardizing and Aggregating Address Points</a> was originally published in <a href="https://medium.com/namara">Namara</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Data Play Nice]]></title>
            <link>https://medium.com/namara/making-data-play-nice-ad8f678d3e7e?source=rss----7fe2f9f63a30---4</link>
            <guid isPermaLink="false">https://medium.com/p/ad8f678d3e7e</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[government]]></category>
            <dc:creator><![CDATA[Lewis Wynne-Jones]]></dc:creator>
            <pubDate>Tue, 15 Aug 2017 13:40:18 GMT</pubDate>
            <atom:updated>2017-08-16T20:21:48.381Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to find value in a mess of open data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*UCjHTNnuzTKrMjWJUfhvJw.png" /></figure><p>When we started working on Namara a few years ago the idea propelling it forward was clear.</p><blockquote>In order to live up to its potential, all the open data in the world needs to be made available on a single platform</blockquote><p>It was a pretty easy problem to diagnose, all things considered. Governments of every size were starting to release open data, but they were doing it all over the place. Not only did you need to know where to look for the data, once you found it you had to know how to start working with one of the dozens of different formats it came in. Open data was supposed to be the fuel for innovative thinking, but in reality it was like a hundred people in different rooms trying to solve a problem together.</p><p><a href="http://namara.io/">Namara</a> was our solution. It would sit on top of every data portal, standardize to a common format, update and monitor changes, and in so doing create a standard space for all open data to exist. Ultimately, we thought it would be everything open data needed in order to finally live up to its potential. Two years later, we launched the platform and waited for the world to change.</p><p>When it didn’t, we looked a little harder.</p><p>To be fair, we’d solved one problem very effectively. We’d tracked down hundreds of data sources and brought their data (over a hundred thousand data sets) onto a single platform. We’d done what we set out to do. We’d put all the problem-solvers in the same room.</p><p>Turns out none of them spoke the same language.</p><p>Not only is the data that’s offered on these portals wildly different from source to source (I’ve written about this problem <a href="https://medium.com/p/2fc088268ad5/">here</a>), when you <em>do</em> find similar data sets there’s never an easy way to tie them together. We’d aggregated all the data and standardized it to a common format, but now we were confronted by a different problem:</p><p>The data <em>sets</em> were as idiosyncratic as the data <em>sources</em>.</p><p>To solve this new problem, we realized we had to build a tool that was capable of merging, refining, and enriching open data to create data sets more valuable than the sum of its parts.</p><p>The feature that came out of this realization is called Unity, and we’ve been amazed at the kinds of insight it’s provided. We’ll be making Unity available on Namara in the coming months, but before we do we wanted to showcase a few of the ways that we’ve used it.</p><p>Case Study: <a href="https://blog.namara.io/unity-case-study-standardizing-and-aggregating-address-points-90b12343c0f8">Standardizing and Aggregating Address Points</a></p><p>Case Study: <a href="https://blog.namara.io/unity-case-study-geocoding-traffic-accidents-e8b4e724c163">Geocoding Quebec Accidents</a></p><p>Case Study: <a href="https://blog.namara.io/unity-case-study-basement-flooding-watermain-breaks-14f819307e48">Basement Flooding &amp; Watermain Breaks</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad8f678d3e7e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namara/making-data-play-nice-ad8f678d3e7e">Making Data Play Nice</a> was originally published in <a href="https://medium.com/namara">Namara</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unity Case Study: Geocoding Traffic Accidents]]></title>
            <link>https://medium.com/namara/unity-case-study-geocoding-traffic-accidents-e8b4e724c163?source=rss----7fe2f9f63a30---4</link>
            <guid isPermaLink="false">https://medium.com/p/e8b4e724c163</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Lewis Wynne-Jones]]></dc:creator>
            <pubDate>Tue, 15 Aug 2017 13:39:34 GMT</pubDate>
            <atom:updated>2017-08-15T13:44:22.248Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*jayBbjVpJ3f6NLgow7dg5g.png" /></figure><h4>The Data</h4><p>Police departments are starting to launch open data portals (<a href="https://namara.io/#/search/open?order=relevance&amp;page=1&amp;states=imported&amp;source=5790d4d5b4bf8f00050008cd">Toronto</a>, <a href="https://namara.io/#/search/open?order=relevance&amp;page=1&amp;states=imported&amp;source=58b487d5fdac1e000a000061">Fort Lauderdale</a>, <a href="https://namara.io/#/search/open?order=relevance&amp;page=1&amp;states=imported&amp;source=5926e0e1e5248d000a000de6">South Bend</a>, and most recently, <a href="https://namara.io/#/data_sets/64ce9500-119d-4f7b-8fc6-98f136a8a372">Minneapolis</a>, to name a few…). This is excellent news. Not only does it open a window to a historically opaque part of the criminal justice system, it also has the potential to provide both the government and an engaged citizenry with the information they need to start solving real problems facing real people.</p><p>Take the <a href="https://namara.io/#/data_sets/61bb67f9-477c-4523-8ed4-a11d206e811c">traffic accident report</a> published on Québec’s provincial <a href="https://www.donneesquebec.ca/recherche/fr/dataset/rapports-d-accident">open data portal</a>. The report provides detailed information on the day the accident occurred, what kind of road it was on, who was involved (cyclists, pedestrians, etc), how many were injured and how severely, etc. What it doesn’t provide are the coordinates at which the accident happened.</p><h4>The Problem</h4><p>This is one of those situations where data visualization is important. I can easily query the data to see how many of the accidents happened on days when it was raining, or how many accidents happened between midnight and 4:00am, but until I can plot the data on a map, I’m only seeing part of the picture.</p><p>Anyone with a bit of time can geocode physical addresses, but in the case of the accident report the physical address that’s provided isn’t a route, street name, unit, city, and postal code. In fact, the address information doesn’t initially make a whole lot of sense.</p><p>There’s a field for the civic address of the closest landmark, but it’s <strong>62% empty</strong>. Since these are traffic accidents, there’s always a road name, but they’re frequently recorded as (and I’m translating here) “close to the Burger King on Demasse st”, or something similar. Next to the route title, there’s a column named <em>tp_reprr_accdn</em> which contains the values 0, 1, and 2.</p><p>None of this helps me out if I want to plot these accidents on a map.</p><h4>The Solution: Refinement and Geocoding with Unity</h4><p>Metadata is critical. The SAAQ (Société de l’assurance automobile du Québec) maintains very good information to go along with the accident report, but unless I want to have a pdf explaining what I’m looking at while I’m checking out the data itself, I need to find a way to plug the information from the metadata into the data set itself.</p><p>From looking at the metadata (and a few years of French immersion) I can tell that the column <em>cd_etat_surfc</em> signifies road surface quality. The values in this column are the numbers 11–20 and 99. Using the extract node on Unity, I can look for those numbers in the original data set and change them to their corresponding description in the output data (“14” becomes “Sand/Gravel”, etc). Since these values are used internally at SAAQ, I’m confident that every time the data updates they’ll use the same set of values, which means I won’t have to worry about validating these descriptions every time the data set updates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*kMlqiytERf4GcApFb5b05Q.png" /></figure><p>Looking through the metadata, I also learn that the column named <em>tp_reprr_accdn</em> relates to the type of reference point on the road. “0” means “not specified”, “2” means “other”, and “1” means intersection.</p><p>This is important information for me to have. I didn’t want to geocode based on a street name alone, because I couldn’t be sure where the pin drop would go. But knowing which accidents took place at intersections means that I can geocode based on &lt;route name 1&gt; &amp; &lt;route name 2&gt;, &lt;municipality&gt;, &lt;province&gt;.</p><p>To do this, I have to perform a number of different transformations on unity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*BsLqp1vsumAUZzB93OO7TQ.png" /></figure><p>Here’s the simple breakdown of the transformations I’ve added:</p><ul><li>The if_then_else node looks for the value “1”, signifying intersection, in the landmark (<em>tp_reprr_accdn</em>) column.</li><li>When it finds a “1” it pulls the values from route 1 (<em>rue_accdn</em>) and route 2 (<em>accdn_pres_de</em>). I’ve concatenated these two values and separated them by an ampersand, which is how they’ll be geocoded.</li><li>These, in turn, get concatenated with the municipality name (<em>mrc</em>) and a static node with the value “Quebec”, because I want to make sure I limit my results to the right place. This gives me the string &lt;route name 1&gt; &amp; &lt;route name 2&gt;, &lt;municipality&gt;, &lt;province&gt;.</li><li>Since I already know that not all of the data is ideally structured (recall the “close to the Burger King on Demasse st”), I’ll toggle the geocoding node to disallow partial matches. This will limit the number of successes I get, but it will ensure that the matches on the output data are accurate.</li><li>That output is then plugged into the geocoding node</li><li>The latitude and longitude are pulled from the geocoding node into a to_geo_point node which formats the latitude and longitude based on our preferred WKT format.</li><li>The to_geo_point node output connects to the geometry column</li></ul><p>After I’ve made these and a few other transformations, I get a data set that looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*km0aW39sT0ZPAj54bDt1pA.png" /></figure><p>By asking a bit more from the data, we can start pulling out points that can help us start understanding accident trends. While the success rate for this kind of geocoding is currently limited by the quality of the data, it’s through seeing<em> </em>the data presented this way that we can start to see the value of plotting it on a map.</p><p><em>A note regarding Unity screenshots: Unity is currently an internal feature and the GUI is in prototype. There are UX and UI components that are in the design phase and cannot be displayed.</em></p><p><em>Click </em><a href="https://namara.io/#/data_sets/51a1a8ea-622c-4002-ba76-3f29ebc1c85e"><em>here</em></a><em> to view a sample of the refined data set. For clarity, I’ve limited the columns and translated all French in the data set to English. For more Unity case studies, or to try it out for yourself, please don’t hesitate to </em><a href="/service/mailto:info@thinkdataworks.com"><em>get in touch</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8b4e724c163" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namara/unity-case-study-geocoding-traffic-accidents-e8b4e724c163">Unity Case Study: Geocoding Traffic Accidents</a> was originally published in <a href="https://medium.com/namara">Namara</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unity Case Study: Basement Flooding & Watermain Breaks]]></title>
            <link>https://medium.com/namara/unity-case-study-basement-flooding-watermain-breaks-14f819307e48?source=rss----7fe2f9f63a30---4</link>
            <guid isPermaLink="false">https://medium.com/p/14f819307e48</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Lewis Wynne-Jones]]></dc:creator>
            <pubDate>Tue, 15 Aug 2017 13:38:33 GMT</pubDate>
            <atom:updated>2017-08-15T13:44:41.721Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*51BfIad6fYlL4Hfzfzd3WA.png" /></figure><h4>The Data</h4><p>The City of Toronto has some excellent open data. Not only that, they’re constantly iterating on their portal and their strategy. This kind of agility is absolutely necessary if we want open data to succeed in the coming years. It’s critical to get the data out the door, sure, but governments at every level also need to be thinking about how they’re going to <a href="https://blog.namara.io/bridging-the-digital-divide-52f6f8192eae">bridge the divide</a> between data producers and data consumers, how they’ll optimize their data to make it <a href="https://blog.namara.io/is-your-data-ready-for-ai-b19eb4f99c23">business compatible</a>, and how they’ll <a href="https://blog.namara.io/where-were-heading-ab9e480d6a17">plan for the future</a>.</p><p>As with any open data, though, there are a few data sets on Toronto’s portal that leave something to be desired. Take <a href="https://namara.io/#/data_sets/a2e2225e-512f-4791-b00e-c841cc4ff68e">this</a> data set, for example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZvQ3Yiift-iRjWMbb3kDw.png" /></figure><h4>The Problem</h4><p>Despite proudly calling Toronto my home, I am not (yet) able to conjure up the geospatial coordinates of each of the city’s wards from memory. Lucky for me, the city publishes a <a href="https://namara.io/#/data_sets/13364600-a54a-49bf-8a55-c84694a7d52d">data set</a> on their portal that gives me, among other things, Toronto ward name and coordinates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-3QOjXDRdIa3he6IoUQh0Q.png" /></figure><h4>The Solution: Joining and Enriching with Unity</h4><p>Using Unity, I can join these two data sets on the one value they have in common, <em>ward </em>in one, and <em>lcode_name </em>in the other. In Unity, the graph looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qbe6s-dpN2wr9OQ-r9ffxQ.png" /></figure><p>Since ward numbers 1–9 in the second graph are preceded by an unnecessary “0”, I transform the entire column into integers before matching it to the first graph. Then, using the Namara node, I specify the data set id of other graph I want (“basement flooding by ward”) and then specify each of the values I want to pull from the one data set into the other. After I run it, I can see the data set plotted on a map.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ovmee84hm8O5LKI9FA9CA.png" /></figure><p>Now I can query the data in a way that makes a bit more sense, and I can immediately click through to see which wards have a higher rate of basement flooding.</p><p>Once I was looking at the data set, though, I started wondering about the other factors that might go into a high incident rate for flooding. Looking through Toronto’s open data portal, I found a data set that shows <a href="https://namara.io/#/data_sets/eaf9f80d-abe6-4ec0-a917-a58ad64ba170">watermain breaks</a> in Toronto</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5AOyk_P3Q3FFk2LyRhhrRA.png" /></figure><p>By layering that data set on top of the other one, I can start to see if there’s a correlation between the two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*07J9E0mao6fs6i5ELr8YHQ.png" /></figure><p>Now I’ve got something that can potentially help the city understand areas in need of an infrastructure overhaul. If I’m a homeowner, I can use this data to predict the likelihood of my basement flooding. If I’m a guy selling flood insurance… well you get the picture.</p><p><em>A note regarding Unity screenshots: Unity is currently an internal feature and the GUI is in prototype. There are UX and UI components that are in the design phase and cannot be displayed.</em></p><p><em>For more Unity case studies, or to try it out for yourself, please don’t hesitate to </em><a href="/service/mailto:info@thinkdataworks.com"><em>get in touch</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14f819307e48" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namara/unity-case-study-basement-flooding-watermain-breaks-14f819307e48">Unity Case Study: Basement Flooding &amp; Watermain Breaks</a> was originally published in <a href="https://medium.com/namara">Namara</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is There a Difference Between Open Data and Public Data?]]></title>
            <link>https://medium.com/namara/is-there-a-difference-between-open-data-and-public-data-b0e7f3a874c4?source=rss----7fe2f9f63a30---4</link>
            <guid isPermaLink="false">https://medium.com/p/b0e7f3a874c4</guid>
            <category><![CDATA[transparency]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[government]]></category>
            <dc:creator><![CDATA[Lewis Wynne-Jones]]></dc:creator>
            <pubDate>Wed, 02 Aug 2017 16:17:38 GMT</pubDate>
            <atom:updated>2017-08-02T16:17:38.225Z</atom:updated>
            <content:encoded><![CDATA[<h4>Yep. And it’s a big one.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*xbrxQrE1yoO2YiHFGQzV9w.png" /></figure><p>There is a general consensus that when we talk about open data we are referring to any piece of data or content that is free to access, use, reuse, and redistribute. Due to the way most governments have rolled out their open data portals, however, it would be easy to assume that the data available on these sites is the <em>only </em>data that’s available for public consumption.</p><p>This isn’t true.</p><p>Although data sets that receive a governmental stamp of openness receive a lot more publicity, they actually only represent a fraction of the public data that exists on the web.</p><p><strong>So what’s the difference between “public” data and “open” data?</strong></p><h3>Open Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*XmIw2RGBkTVObIeyLoJrow.gif" /></figure><p>Generally speaking, “open data” is the information that has been published on government-sanctioned portals. In the best case, this data is structured, machine-readable, open licensed, and well maintained.</p><h3>Public Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*SHfzT2EbUE71KT4ItZtPGg.gif" /></figure><p>Public data is the data that exists everywhere else. This is information that’s freely available (but not really accessible) on the web. It is frequently unstructured and unruly, and its usage requirements are often vague.</p><blockquote>Only 10% of government data is published as open data</blockquote><p>What does this mean?</p><p>Well, for starters, it means that there’s a discrepancy between the open data that exists in government portals and public data in general. This is an important distinction to make, because while there’s a lot of excitement surrounding open data initiatives and their potential to transform modern society, the data that this premise rests on — open data — is only a fraction of what’s needed in order for this potential to be realized.</p><p>The fact is this: the majority of useful government data is either still proprietary or stowed away in a filing cabinet somewhere, and the stuff that <em>is</em> available is being released haphazardly.</p><h4>What’s in a name?</h4><p>Does it actually <em>matter </em>that there’s a distinction between these two kinds of data? Well… yes.</p><p>Open data, because it represents such a small portion of what’s available, hasn’t lived up to its potential. People, like me, who have very high hopes for the open data movement haven’t yet seen the ROI (economically or socially) that we were supposed to. The reason<em> </em>we haven’t is manyfold, but this distinction is part of the problem.</p><p>In order for open data to be as effective as predicted, the line that demarcates open and public data needs to be erased, and governments need to start making a lot more of their public information open data. After all, we’re the ones paying for it.</p><p><em>ThinkData Works proudly works with governments to help them release high-value data, and their data management platform </em><a href="https://namara.io/#/"><em>Namara</em></a><em> connects users to every open data portal in North America (and a lot of public data too).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0e7f3a874c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namara/is-there-a-difference-between-open-data-and-public-data-b0e7f3a874c4">Is There a Difference Between Open Data and Public Data?</a> was originally published in <a href="https://medium.com/namara">Namara</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yukon — Optimized File Deduplication and Merge]]></title>
            <link>https://medium.com/namara/yukon-optimized-file-deduplication-and-merge-cb921069cb3b?source=rss----7fe2f9f63a30---4</link>
            <guid isPermaLink="false">https://medium.com/p/cb921069cb3b</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Jason Piao]]></dc:creator>
            <pubDate>Wed, 26 Jul 2017 19:07:26 GMT</pubDate>
            <atom:updated>2017-07-27T14:36:51.955Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*2LwlFWbUJ8Smf5JHkClFPw.jpeg" /></figure><p>Between 1896 and 1899 a stampede of over 100,000 prospectors made their way north, leaving San Francisco and Seattle behind to search for gold along the Klondike. Of those that went, fewer than a third made it to the Yukon, and those that did were constantly challenged by unforgiving conditions. The gold rushes of the 19th century are characterized by this kind of atmosphere. Gold is a tricky thing to mine — it takes intelligence, grit, and patience to sift through the piles of dirt you don’t want to uncover the chunk of something you do.</p><p>This was on our mind when we first discussed developing a strategy for removing and merging duplicated data points in a data set. Building a tool that could uncover nuggets of value in mountains of data sounded incredibly valuable. It also sounded incredibly difficult. We like things that are difficult. We thought we’d call it Yukon.</p><h4>How it Works</h4><p>To identify duplicate rows, we first have to specify which columns Yukon will look at. In the sample below, we’ve selected columns 1 and 5 (FirstName and Company) as the composite key, or <em>what defines a unique record</em>. The entire row will be used if no columns are specified.</p><p>Let’s say we take this example data set:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2DkM1wcr6e0LvGKmQ6R41A.png" /><figcaption>sample data set</figcaption></figure><p>By specifying columns 1 and 5, Yukon will produce this deduplicated and merged dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xfnV2IE-99ylyZriLzijPQ.png" /><figcaption>deduplicated and merged data set</figcaption></figure><p>In theory, deduplication on small data sets like these sounds relatively simple. However, the data sets on <a href="http://namara.io/"><em>Namara</em></a><em> </em>that need to undergo this deduplication and merging process can be greater than 100GB. To put the size of this file into perspective, a 75 page essay with 23,000 words is only about 1MB, 100,000 times smaller.</p><p>Most computers will have difficulty opening and operating on 1GB files — using programs such as Microsoft Word or Excel — due to the heavy usage on computer memory (RAM). As you could imagine, a file over 100GB simply can not be loaded into a standard computer’s memory without causing it to crash.</p><p>Though RAM presented us with limitations, we were able to solve this problem by utilizing disk memory. Disk memory is abundant on modern computers and is relatively cheap to upgrade in comparison to RAM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*-lBGWoNx_GGxReHrLsOS3A.jpeg" /><figcaption>Holy Guacamole! Look at all the implicit value in that data set!</figcaption></figure><h4>Approach 1</h4><p>Our first approach resembled how a <a href="https://en.wikipedia.org/wiki/Bloom_filter">Bloom Filter</a> is implemented:</p><ul><li>Initialize a hash file using <em>x</em> buckets (dependent on data set size)- where a bucket is a string of bytes of a fixed size</li><li>Obtain hash of each row in the data set, and remember the current byte offset of the row</li><li>Using this hash, locate the corresponding bucket in the hash file and read the byte string for that bucket</li><li>If the byte string points to some offset in the original data set, then we’ve detected a collision. Otherwise, record the current byte offset in the bucket</li></ul><p>This meant having to check whether the rows that caused a collision were unique or true duplicates. This would determine whether or not to merge them into a single row.</p><h4><strong>Problems with Approach 1</strong></h4><p>We ran into issues regarding the size of the hash we were considering (7 bits of a 32 bit hash). The size of the hash file would grow exponentially with the portion of the hash we would use. However, using a smaller portion of the hash would also increase the number of collisions. This meant leaving a lot of dirt in with the gold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*iRPBnabD8C5louZbUqNC0w.jpeg" /><figcaption>“Aw man look at all that dirt” :(</figcaption></figure><h4>Approach 2</h4><p>We wanted to see what would happen if we got rid of the hash file and relied only on memory (RAM). We started by calculating the largest row in the data set, and giving each row padding to match that length. This allowed us to easily seek through rows in the file using an offset multiplied by a constant row length, allowing us to re-write and merge quickly. This also cut down hashing collisions since we could use all 32 bits of the hash.</p><h4>Problems with Approach 2</h4><p>This approach had us reading the data set twice — first for the reading and then for the padding, before doing any actual operations. The frequent file seeking also took a toll on runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*9zb_SHz8omEXrkUoHOP9yg.jpeg" /><figcaption>Yeehaw</figcaption></figure><h4>Approach 3</h4><p>We wanted to utilize both RAM and disk memory to gain speed without sacrificing storage. Keeping tack of the duplicated rows themselves (and their offsets) in RAM allowed us to directly access the duplicate rows and merge them, instead of seeking through disk memory.</p><p>Deduplicate:</p><ul><li>Write all “prime” rows (the initial row that duplicates would be merged into) to an intermediary file, which keeping track, in memory, of the row offset and length of their first occurring duplicates</li><li>When a duplicate row is detected, merge with the original, while storing both offsets. Now we have references to (a) all rows that need to be merged (RAM), and (b) all unique rows and precedent duplicate rows (disk memory)</li></ul><p>Merge:</p><ul><li>Read the intermediary file in, row by row</li><li>If the row we are examining is unique, we won’t have an offset to match a merged row, and it can be written to an output file. Otherwise, the offset should correspond to a merged row that we are already storing in memory. We can merge these and write them to our output file.</li></ul><p>This process gives us the desired data set with all rows deduplicated and merged. No more dirt in the gold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8bZkoxG9u-gF8x6TG9Ggbw.png" /><figcaption>Yippee!</figcaption></figure><h4>Approach 3 Conclusions</h4><p>This approach gave us no collisions and had reasonable memory usage. We were also satisfied with the speed (about ~35 seconds on a 5 million row data set, running on our development machines).</p><p>Benchmarks are given below:</p><h4>Comparisons against approach 1 and approach 3.</h4><p><strong>Red Line: Approach 1</strong></p><p><strong>Blue Line: Approach 3</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aWJsyuaLdX3cfZ-lp71IRw.png" /><figcaption>File size vs memory usage</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T5rPG5hEo4jgZzKiCRVw5g.png" /><figcaption>File size vs time</figcaption></figure><h4>Conclusions</h4><blockquote>Algorithm growth, in this case, means how much slower / more memory the algorithm uses as the file size increases. When one algorithm grows slower than another in terms of speed, it means the ratio of speed and file size of the faster algorithm is smaller than the ratio of the slower algorithm. The smaller the ratio, the better the algorithm scales for large files.</blockquote><p>Given the benchmarks, Approach 3 is the clear winner. Though the memory usage is greater, both file size and memory usage grow at linear rates. The amount of memory required is about 35% of the original data set — proportional to the number of rows, rather than the size of the file. That is to say, a 100GB file with 50 million rows would take less memory than a 40GB file with 100 million rows.</p><p>The difference in processing time is significant. Not only is Approach 3 significantly faster, but also grows at a slower rate. In our case, the extra memory usage is well worth the performance boost.</p><blockquote>If you would like to know more about how Namara’s making external data easier to use, please get in touch!</blockquote><blockquote>Jason is a developer intern at ThinkdataWorks, keep track of what he gets up to on his <a href="http://www.github.com/JPiao">Github</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb921069cb3b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namara/yukon-optimized-file-deduplication-and-merge-cb921069cb3b">Yukon — Optimized File Deduplication and Merge</a> was originally published in <a href="https://medium.com/namara">Namara</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is your data ready for AI?]]></title>
            <link>https://medium.com/namara/is-your-data-ready-for-ai-b19eb4f99c23?source=rss----7fe2f9f63a30---4</link>
            <guid isPermaLink="false">https://medium.com/p/b19eb4f99c23</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Bryan Smith]]></dc:creator>
            <pubDate>Wed, 24 May 2017 19:26:23 GMT</pubDate>
            <atom:updated>2017-05-24T20:44:27.208Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Companies are champing at the bit to introduce any solution that promises Artificial Intelligence and Machine Learning. But hasty adoption is leaving one important question unanswered: Is your data ready to power this new machine-driven revolution?</strong></h4><h4>For most companies, the answer is no.</h4><p>Industry leaders in every sector are racing to adopt sophisticated systems that promise the introduction of AI and ML into their organizations. Executives often get sold on the concept of a technology that will give them the ability to see things that humans can’t, making suggestions or changes to business that can unlock new opportunities and drive major efficiencies.</p><p>Although the opportunities are definitely there, and in some cases so is the technology, there is a major disconnect between the possibilities that AI and ML are presenting compared to the operational readiness of most organization’s data environments.</p><blockquote>“AI is like sex in high school. Everyone says they’re doing it, very few actually are, and those who are, are doing it wrong.”</blockquote><p>What we’re seeing in our discussions with decision makers at many of the market leading companies across diverse sectors (finance, insurance, real estate, etc) is a pretty solid understanding of data warehousing combined with a very frothy desire for a rich application layer that can start solving their business needs.</p><p>The reality, though, is that a data warehouse and an application layer represent the beginning and end of a process, and what’s usually missing in the middle of this process is a pretty sophisticated stack that transforms raw data at one end, into usable information at the other. The problem is, not only is this “in-between” layer a difficult problem to solve, it’s also a major unknown for many decision makers setting IT strategy, budget, and timelines.</p><blockquote>“If Watson can win Jeopardy, it must have the cognitive ability to wade through my data and develop insight, right?”</blockquote><p>Often, organizations have a data warehousing strategy or a fully deployed data warehouse solution that focuses on getting data into one place. Appended to this strategy is usually a laundry list of already purchased applications or solution roadmaps using new technology (AI, ML, Automation, Analytics, Visualization). From a high level, it would seem as though the problem is solved:</p><p>Get all the data together — Check.</p><p>Purchase applications that turn raw data into insight — Check.</p><p>The concept here is that once we get all the data to a common place, we can plug it into whatever we want. Simple. The problem is, the data coming out of the warehouse(s) lack a common standard, and the applications sitting on top of the data can’t be fully implemented because…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dILqq1TUIezHveKRHTspog.png" /></figure><p>…there’s a massive refinement layer sitting in between the data (lake or no lake) and the enterprise apps that hasn’t been defined or implemented. To solve this problem, a system needs to be introduced that has the power to transform data from its native tongue into a centralized standard that any app can learn to understand. Not only that, but it has to be able to do this on a consistent and go forward basis and be able to scale to an infinite number of new and existing data sources.</p><h4>Even the best software needs good data.</h4><p>At the end of the day, any application, automation, or AI/ML technology needs to be fuelled by refined, standardized data in order to function to its greatest capacity. Without standard data, the game changing tech you wish to implement that promises to revolutionize your business will be limited to the lowest common denominator of standard data it can gain access to. Usually, that means an extremely limited amount of information even if it is now all coming from the same place.</p><p>Data needs to be refined into fuel and delivered in needed quantities to the applications it is tasked with powering. A system that does this, or better yet, a data refinery process that checks all of the required boxes, looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X2TuUR2yMhgN1ER6fSC5Iw.png" /></figure><p>Or, for those interested in what we do for some of our clients with our platform, Namara, it looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9mjbXCv1nP6oc7E6ywi2eg.png" /></figure><p>A lot of the really successful showcases of AI and ML technology is launched on the back of extremely clean and standardized libraries of data. But in most organizations, data is still quite segregated. Bringing data together into a “lake” or onto an enterprise cloud gets data in a common place, but it is often still lacking standardization among the data itself. This layer of data standardization is still a huge hurdle that many organizations have yet to get over. And the complexity of the problem only compounds with the addition of new, external data sources that forward thinking organizations are beginning to value and leverage.</p><h4><strong>A variety of data means a variety of problems.</strong></h4><p>The problem we’re trying to solve is that of data variety. Proper warehousing helps us solve volume and velocity issues, but a world with fully unlocked enterprise apps requires us to answer the question of how we transform similar data coming from multiple places into a common standard on a consistent basis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TlhBYATcWjO5X0ukH5B0Gg.png" /></figure><p>In many organizations, the answer is, “that’s the job of our data scientists”. But if we really think about it, should it be? Data scientists are highly skilled, well-paid individuals whose time and effort should be focused on helping the business grow in a data-driven way. If that’s the case, then why is so much of their time focused on finding, cleaning, standardizing, and monitoring data? The short answer is because it has to be, given most organizations don’t have the right data processing pipeline in place to generate ideal data in a way that allows their apps and their scientists to work as efficiently as possible.</p><p>This turns most data scientists into data janitors, and it’s also is the reason most enterprise apps will never fully reach enterprise deployment.</p><blockquote>“The problem isn’t lack of direction or ability. It’s lack of refined data.”</blockquote><p>The idea of a data lake or cloud is a good one and a necessary concept. Data needs to be pooled into places so that it can be accessed by multiple functions for multiple reasons. A lot of good things became possible because people started realizing the benefit of knowing where their data was and what it was actually made of. Now, as we look to the next phase of business intelligence, one that brings with it automation, artificial intelligence and machine learning, we must think less about data sitting in a lake, and more about how that data needs to flow throughout an organization in a standard way. We’re evolving beyond the need for a container that can hold our data and we’re moving towards the need for a vascular system that can flow data to where it needs to be in the shape and form that is required.</p><p>This may seem like a huge leap forward, but in reality, it’s very much attainable. In fact, many of the pieces required are laid out in the above graphics. On your own, it’s not a quick or easy build, but it is a necessary problem to solve if you are truly thinking from an enterprise perspective.</p><p>The takeaway here is not to stop looking at new technology that can help unlock huge potential. It’s to remember that regardless how shiny and amazing new solutions are, they all require rocket fuel to help them blast off. When we talk game changing enterprise apps, the rocket fuel is standardized data. And a process to refine it needs to be implemented in parallel with the adoption of new Business Intelligence tools and applications.</p><p><em>Head over to </em><a href="http://www.namara.io/"><em>Namara</em></a><em> to learn more.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b19eb4f99c23" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namara/is-your-data-ready-for-ai-b19eb4f99c23">Is your data ready for AI?</a> was originally published in <a href="https://medium.com/namara">Namara</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Namara Improvements]]></title>
            <link>https://medium.com/namara/namara-improvements-4f63b57ad79e?source=rss----7fe2f9f63a30---4</link>
            <guid isPermaLink="false">https://medium.com/p/4f63b57ad79e</guid>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[release-notes]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Jordan Bibla]]></dc:creator>
            <pubDate>Fri, 05 May 2017 18:02:14 GMT</pubDate>
            <atom:updated>2017-05-05T18:02:14.340Z</atom:updated>
            <content:encoded><![CDATA[<h4>Release Notes</h4><p>For the past few weeks we spent some time improving the data set tables. This core feature needs to accommodate a great diversity of data and we wanted to improve the way our users can interact with the table on our platform. Here is an overview of some key improvements.</p><ul><li>Data set table and project table filters, sorts, and select distincts appear in URL for improved shareability and usability.</li><li>Data set tables now support dynamic column widths.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UeMxE1wToSUHmKLaX8_Kyg.png" /><figcaption>Dynamic column widths.</figcaption></figure><ul><li>New<strong> </strong>SQL style <em>Select Distinct</em> toggle on data set table columns.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*jbCbvdIBL_-n_mVa3qEevQ.png" /><figcaption><em>Select distinct for distinct values in the column.</em></figcaption></figure><ul><li>New <em>Null / Not Null</em> toggle for seeing or hiding cells with null values.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*yGkNqu728TMiGMx123eLYg.png" /><figcaption>Click “Null” or “Not Null” to filter the columns cells.</figcaption></figure><ul><li>Double-click on a table cell to reveal the new <em>Cell Action Menu.</em> This feature provides the ability to expand a cell to see the full value, copy the cell value, and “filter by” the selected cell.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*Tm6yj9MupJpOaHZxyPLXWQ.png" /><figcaption>Double-click a table cell to reveal the Cell Action Menu (Expand, Copy, Filter By).</figcaption></figure><ul><li>Applied filters, sorts, and distincts will appear in a <em>filter list</em> on data sets and maps to make seeing table manipulations more obvious and to make applied manipulations easier to remove.</li><li>Filter list will persist from table to map.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1PqofqwTc2-EidfcE5iYeA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6vybzNLqz0ef6uge4o0LDQ.png" /><figcaption>Filter list on table persists on map view.</figcaption></figure><ul><li><em>Reset all filters</em> button for when a user has filtered out all the data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*pmpkO2WtBgjFDR6aksbN1Q.png" /><figcaption>Having too much fun with filters? Reset all filters button.</figcaption></figure><ul><li>Project table now has a small ‘?’ for column definitions and explanations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UBwVLzk4iLXsV_cCHn4bUg.png" /><figcaption>Click the ‘?’ to learn more about project table columns.</figcaption></figure><ul><li>Imported data sets will now have a <em>Download tab</em> for easier downloading without having to add data to a project (API access still requires the data being in a project).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*P2v1Mnq7sASiBqElcT1org.png" /><figcaption>New Download tab for easier downloading with link to source.</figcaption></figure><p>Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f63b57ad79e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namara/namara-improvements-4f63b57ad79e">Namara Improvements</a> was originally published in <a href="https://medium.com/namara">Namara</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to view Namara data on your Salesforce dashboard]]></title>
            <link>https://medium.com/namara/how-to-view-namara-data-on-your-salesforce-dashboard-cc53d9b8e6d4?source=rss----7fe2f9f63a30---4</link>
            <guid isPermaLink="false">https://medium.com/p/cc53d9b8e6d4</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[namara]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[odata]]></category>
            <category><![CDATA[salesforce]]></category>
            <dc:creator><![CDATA[Chris Sandison]]></dc:creator>
            <pubDate>Mon, 27 Mar 2017 20:00:06 GMT</pubDate>
            <atom:updated>2017-03-27T20:00:05.662Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*uP-fEf820Huq-Bl3vRlE7A.png" /></figure><p>What’s <em>really</em> behind this post is the exciting news that <strong>Namara API now supports </strong><a href="http://www.odata.org/"><strong>OData Version 4.0</strong></a><strong>!</strong> Since each data set is offered as it’s own OData Service, you can use the query capabilities of any OData-compliant client to view external data in real time. Though this post will focus on getting up and running with Salesforce, you can use the service endpoints mentioned bellow for comparable .NET or SAP services.</p><h4>Step 1: Retrieve your Namara API Key</h4><p>After logging into your Namara Account, click on your user icon in the sidebar (bottom left of the window) and select <a href="https://namara.io/#/user/account"><strong>Account Settings</strong></a><strong>. </strong>Then click the <strong>API Key </strong>tab.</p><p>You will need to create an API key if you have not already done so. Otherwise, retrieve the one that you have already created and paste this somewhere where it can be accessed. You will need to use it a few times.</p><h4>Step 2: Find the data set you would like to add</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fkn7XBnBaC6UhfQw8wlYJw.png" /></figure><p>Navigate to the data set of interest. In the top bar, click on the <strong>API Info</strong> tab, to view integration information. Copy the contents of the <strong>API Endpoint</strong> field into somewhere where you can access it later.</p><h4>Step 3: Adding a new External Data Source to Salesforce</h4><p>On your Salesforce Lightning dashboard, access “Setup” in the top toolbar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*K9zQqpoXDXK4OfAPvasDAg.png" /></figure><p>Either enter “External Data Source” in the Quickfind bar, or navigate <strong>Data &gt; External Data Sources</strong> in the sidebar under the “Administration” header. Then click the “New External Data Source” button.</p><p>In the form, fill out the following fields:</p><ul><li><strong>External Data Source: </strong>give this a name to refer to the data set that you are adding a view for.</li><li><strong>Type:</strong> Salesforce Connect: OData 4.0 (<em>Please note that 2.0 is not supported by Namara</em>)</li><li><strong>URL:</strong> paste the <strong>API Endpoint</strong> url that you copied from before, then add “OData.svc/” to the end, so that it looks like this:</li></ul><pre>https://api.namara.io/v0/data_sets/:data_set_id/data/:version/OData.svc/</pre><ul><li><strong>Identity Type: </strong>Per User</li><li><strong>Authentication Protocol: </strong>Password Authentication</li><li><strong>Administration Username</strong>: Set this to whatever you want. This field will not be used for verification and will be ignored.</li><li><strong>Administration Password</strong>: Paste your <strong>API Key </strong>here.</li></ul><p>The form should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_EHcv5WJ8b9CiB0gzd0MoA.png" /></figure><p>Now click “Save”. You will be taken to a screen to review the details. If you are satisfied, click “Validate and Sync”.</p><p>Salesforce will access a specific metadata endpoint for this data set and model the external data as Salesforce objects. If successful, you will be shown a table of all External objects found. Only the table named “Data” is necessary for viewing. After selecting that, click “Sync”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*p45UWOIQFosRPccSAYzwaQ.png" /></figure><p>If you go back to the <strong>Setup</strong> sidebar and view your “External Data Objects”, you should see one called <strong>Data</strong>. <strong>DataSets</strong> and <strong>Versions</strong> only store metadata and aren’t required for viewing. You can edit this object to enable reporting. I recommend changing the label to something more descriptive like “HamiltonSchool”, but <strong><em>do not change Object Name</em></strong>.</p><p>Click “Save” when you are finished.</p><h4><strong>Step 4: Grant User Permissions for Viewing</strong></h4><p>In the sidebar, navigate to “Profiles” and find the profiles of whatever user will be accessing this data. Click “Edit” for the profile and ensure that <strong>External Object Permissions</strong> are set for this external source.</p><p>Then under <strong>Enabled External Data Source Access</strong>, click “Edit” and grant access for your external data source.</p><p>Next you need to store the access credentials for this data set so that the <strong>API Key</strong> can be used on subsequent access calls. To do this, click on your user icon in the Top Right corner and then click “Settings”. You will be taken to another page with a Sidebar &amp; Quickfind text field. Click on the tab labelled “Authentication Settings For External Systems” and then click “New”.</p><p>Fill out the form with access credentials for a user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*o61_x3Z3hm44KxsUaOlfEg.png" /></figure><h4><strong>Step 5: Create a Tab</strong></h4><p>In your Sidebar Quickfind, search for “Tabs”, then create a new <strong>Custom Object</strong> tab. Select the External Object by the label that you added in Step 3 and add an icon style. Click “Next”. Set visibility for Tabs however you would like.</p><h4>Step 6: Create a View</h4><p>Navigate to the tab that you created in Step 5. When you click “Go!” you’ll notice that these aren’t the fields you are looking for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TK9VvbLrKbZqIL9qiMkXog.png" /></figure><p><em>NOTE: </em>We do not give API-level access to a single row in a data set, so accessing Display URL will give you an error.</p><p>You will have to create a new View and use $filters and $select statements in order to pull out the data. Start by clicking the “Create New View” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*7Ix6El1QNHoGBmd1iHnq2w.png" /></figure><p>Create whatever query you would like on the data. Here, I am trying to view all high schools with enrolment over 500.</p><p>When selecting fields to display, ignore _row_id — it’s just a unique key we use so that the OData adapter can tell records apart.</p><p>Since there are a maximum 15 fields for a view, it’s more manageable to create several meaningful views rather than a single one that attempts to read the entire data set.</p><p>Now when you click “Go!” you will see the fields from your view, and be able to step through the paginated results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8rmOCGCdALkFx7FtC6VeTw.png" /></figure><p>There you have it! I apologize for switching between the old Salesforce interface and the new Lightning dashboard, but I was informed that the new lightning dashboard did not support filter views (yet).</p><p>Please leave comments if:</p><ul><li>Things in the tutorial are unclear, or you have recommendations for improving the flow of this post</li><li>You’ve been able to do some neat things using Namara &amp; OData. We would love to hear about it!</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc53d9b8e6d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/namara/how-to-view-namara-data-on-your-salesforce-dashboard-cc53d9b8e6d4">How to view Namara data on your Salesforce dashboard</a> was originally published in <a href="https://medium.com/namara">Namara</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>