This is the configuration:

• 1x Tilt/Tent Kit — Black
• 1x The Wing: Custom ErgoDox EZ Wrist Rest
• 1x ErgoDox EZ Shine Standalone — Blank / Black / Cherry MX Clear

I decided to go with the more resistant Cherry MX Clear keys, because I tend to bottom out with my current Kinesis Advantage keyboard, with the Cheery MX Brown. You can see a list of keyswitches offered here.

The similarities to Kinesis Advantage are obvious.

The entire keyboard, minus a vertical set of keys is directly from the Advantage. In fact, this vertical set of keys (left most on right split, right most on left) is a bit hard to reach with thumb without accidentally pressing nearby keys, as my initial attempts to use them found. Over the day though, I realized pressing them with my index finger rather than thumb is a better approach. I mapped them out to Ctrl and Windows keys.

You can see my current keyboard layout here. This is still a work in progress, so expect more updates as time goes — each update has an incrementing roman numeral (currently at VI).

I’ve designed the layout around Qwerty, Vim, i3-wm and Go language.

What’s so unique about Ergodox?

Ergodox is a lot smaller and more lightweight than Advantage. The splits are small enough to take along with the laptop during trips, which means Ergodox would see lot more action if you do frequent travels. This was the prime reason for me to order it, and I’d consider it the most unique selling point (USP) of the product.

The second biggest USP is the online graphical configurator. Never has remapping keys been so easy. It’s so useful that I decided to forgo the key
prints on the keycaps entirely, and use the blank edition.

The configurator really brought the advantage of layers to me. Before this, I
think I was familiar with the concept, but it just never took hold. While coding, with the typical QWERTY layout, the brackets were never convenient, either too far down, or too far up. With Ergodox, I put those into a new layer (Layer 1), right in my left home row. Not only those, but I ran a little script and found the most common letters used in the posting package of Dgraph.

It’s a work in progress, but I mapped those to layer 1 as well.

sed ‘s/\(.\)/\1\n/g’ *.go | sort | grep -v ‘[a-z A-Z]’ | uniq -c | sort -nrk 1

Also have a look at the Windows key I set, which is in a great location for all
the i3-wm keyboard actions. I still need to gain familiarity with layer 1 containing all the special letters, which I hope to do over time.

Overall, I feel this keyboard is a bit less ergonomic than Kinesis Advantage. My hands extend out a bit more than the Advantage, and they don’t sit as nicely. But the mobility and layers are definitely winning me over. I have been using Advantage since mid-2009, and I doubt that’d go anywhere, but this keyboard would be my new go to for travel (and currently doubling as my desktop keyboard).

This is it for now. As I gain more experience with the keyboard, I’ll add more things.

Update (13 Mar 2018): It’s been a few months since I got Ergodox, so I’ve had a good chance to get used to it. But, I still reach out to my Kinesis Advantage 2.0 if I’m at my desk or at my office. And Ergodox is the keyboard I bring along if I’m working from a cafe or traveling. Kinesis has a better, more ergonomic feel; while Ergodox is smaller and easy to carry along.

To give a bit of a background — Dgraph, as a graph database, is designed to excel at the weaknesses of traditional relational databases: traversing relationships and efficiently executing complex joins at scale. Relational databases are not going anywhere, and they would continue to play a huge role in tech. But, at the same time, there’s a big need for handling highly interconnected, sparse data with flexible schema, to truly understand user queries. Google is going this way, and so is the rest of the world. This is what inspired us to build Dgraph back in 2015, and this basic premise continues to shape the project.

Our investors immediately saw the value of what we were building and where the database market is headed. While it is hard as an outsider to raise funds from the valley, we got backing from some of the smartest investors. We’re thrilled to announce that we have raised $3M in seed funding, led by Salil Deshpande of Bain Capital Ventures, joined by Mike Cannon-Brookes of Atlassian, Blackbird Ventures, and AirTree Ventures.

With experience in early-stage infrastructure software and open source, Salil will make an invaluable asset to Dgraph Labs by joining the board. He’s been on Midas list of top 100 VCs, with prominent investments in Mulesoft, Lending Club and, Redis Labs. This funding round comes at the right time for us to expand our offerings to the cloud, and continue to build the most advanced graph database in the world.

For this next phase of Dgraph, we’re moving our engineering team to the SF bay area, which is where the inspiration of Dgraph came from, and where we’ll have access to an unparalleled level of knowledge, talent, and market. At the same time, we’re opening an office in Bangalore.

Dgraph version 1.0

At Dgraph, our mission is to build world’s best graph database. We’re the only graph database which inspires to become the primary database in the tech stack. This idea has been the driving factor behind the design choices of Dgraph, which we’ll explore now.

Distributed Graph

Dgraph is built to scale horizontally. Modern data sets have long grown beyond the capabilities of a single server. However, most SQL and graph databases are limited by design to store the universal data set into a single machine. This has many disadvantages, including introducing a single point of failure and being disproportionately more expensive to run.

Building a distributed graph database is a hard challenge. Graph DBs must provide a superset of relational database operations, including storing the data in a way which can execute constant time edge traversals and arbitrarily deep joins, without a view of the universal data set. Most graph databases are, therefore, either a single server architecture or a graph layer built on top of other databases, where they can get this vantage point of universal view. This leads to either lack of scalability, or lack of performance, choose at least one.

Dgraph, on the other hand, was designed keeping the problems of a fast-growing company in mind. It not only shards the data in a way where it can execute queries (joins, traversals, filters, sorts, paginations) without this universal view, it also minimizes the number of network calls and disk seeks required to execute these queries. Both of these factors become prominent in datasets spanning hundreds of servers. Not only that, as you add more servers, data automatically gets sharded and moved to fill the new ones. Dgraph automatically rebalances shards among servers to ensure that load is evenly distributed.

While being distributed, Dgraph provides lock-free ACID transactions with snapshot isolation, designed for low latency. Transactions make it a lot easier for application developers to think through database behavior, removing any data integrity issues.

Consistent Replication and High-Availability

Dgraph does away with the whole master-slave replication architecture. Early on, we decided against providing an eventual consistency model. In that model, a replica which is running behind the master would not show latest writes, and the developer needs to handle this in the application logic. We want Dgraph system to be reliable and consistent, where scalability is an operational feature, not a hurdle application developers need to code around.

Instead, Dgraph provides consistent (synchronous) replication, utilizing Raft, a consensus algorithm, allowing k servers dying in a 2k+1 replication setting, without affecting any reads or writes. Each data shard can be replicated an odd number of times, and replicas located in different datacenters, or availability zones to allow faster communication from your clients. Dgraph guarantees atomic consistency of writes, which means irrespective of which replica is hit for reading, any write done before is guaranteed to be available. This guarantee makes it a lot easier to build applications, which can then serve queries from the closest datacenter to the end-user.

Despite the focus on distribution, Dgraph performs exceptionally well as a single server. In fact, many Dgraph users run it on a single server and benefit from the amazing performance it brings. You can read more about the capabilities of Dgraph, in this recent post.

What Next

The next chapter of Dgraph is very exciting. We’re planning to provide a managed hosted service, to make it easier for adopters to use Dgraph without any of the operational overhead. We’ll also be significantly improving our user interface, to allow cluster management and advanced data exploration.

They say the true test of any distributed system is Jepsen, and in the coming year, we’re going to be working with Kyle Kingsbury to help set up the tests for Dgraph. Simultaneously, we’ll push our design to span globally, with automatic replication across geographically distributed regions, to become the Spanner of Graph Databases.

We’ll be expanding our list of languages to include Open Cypher and Gremlin. Both of these are very popular graph languages, and would make it easier for existing graph users to switch to Dgraph. We also plan to support the standard version of GraphQL.

Onwards and Upwards

We’re really excited about v1.0 release of Dgraph. Pushing it this far has been a particularly challenging road, full of experimentation. Dgraph is the only graph database which is building for native geographical distribution of data, while providing efficient arbitrarily deep joins and traversals. Those are hard problems that we have solved.

Lacking a guiding Google paper on distributed graph databases, we built a system which puts together the best of Bigtable (NoSQL) and Spanner (NewSQL). Within Dgraph, we term this NewGraph — a graph database which is horizontally scalable, multi-versional, transactional, synchronously replicated and inherently capable as a primary database. In fact, we think a paper describing the unique design of Dgraph is quite important, so we’re working towards writing one. Being engineers, we can’t think of a better way to celebrate the last over two years of hard work.

With Dgraph, we hope to break the cycle of many fast-growing companies from looking for a graph database, trying something, giving up and building their own graph system; to looking for a graph database, finding Dgraph and building applications on top.

While traditional databases only serve data, graph databases allow machine interpretation of data, which is crucial to build smart systems. We look forward to putting Dgraph into the hands of users who need the stability of v1.0, and hear about all smart use cases they are building. And we’re confident that Dgraph would make their use cases successful. We have taken giant strides in graph database world, and we promise to continue taking those, so our users can push the envelope for what’s possible and maintain the competitive edge.

Also read coverage on TechCrunch.

Originally published at blog.dgraph.io.

Amazon Neptune has been optimized to handle billions of relationships and run queries within milliseconds. Neptune supports fast-failover, point-in-time recovery and Multi-AZ deployments. And you can also encrypt data at rest.

This is very exciting news for the entire tech ecosystem. It clearly shows that graph databases are going mainstream. Already many tech companies are using existing graph solutions or building their own graph-like systems.

Many devs would be happy users of Neptune, given Amazon is running this as a managed service. People trust Amazon; they have done a great job with AWS so far. So, this is good news for adoption.

But, in this article, I want to look past the blinding light of Amazon — into the design of this new graph database. Despite being built by Amazon, Neptune’s design is obsolete right out of the gate. Let’s look through the design features of Neptune.

Licensing: Nothing is mentioned about the availability of an open source version. It’s safe to presume that Neptune is closed source.

Scalability: Neptune is vertically scalable instead of being horizontally scalable. That means if your data size increases, it must still fit on a single server, as there’s no data sharding possibility.

ACID Transactions: There’s no mention of transactions on their features page, which indicates that Neptune potentially does not support ACID transactions.

Replication: The replicas are global copies of the original data, with all writes going to one server, and the rest being read-only servers. Neptune provides eventual consistency for replication, (looks to be) built on top of Amazon Aurora, where the write server writes to this storage, and rest of the servers reading from this storage. They provide fast reads by bringing the data to RAM (“in-memory optimized architecture”).

APIs: Neptune supports Gremlin and SPARQL.

There are benefits and drawbacks to this architecture.

Benefits wise, having read replicas is great for increasing the query throughput. Aurora as storage would ensure that once written; your data would have the same guarantees (protection from disk losses, etc.). Amazon has a long record in providing hosted services, so a managed service by them is great for adoption.

It comes with a fair bit of drawbacks as well. Vertical scaling introduces a single point of failure in the system — a single server crashing could bring the entire system down. While Amazon has strict SLAs on their managed service, this is still a questionable system design from the tech giant.

Vertical scalability is also disproportionately more expensive than horizontal scalability. Having to fit the entire data into a single machine requires specialized hardware, something which gets disproportionately more expensive than just buying another server. In fact, that ship sailed a long time ago when the internet exploded in the early 2000s. Bigtable, MongoDB, Cassandra, Elastic Search, Spanner, CockroachDB, etc. — all the new technologies since the 2000s have been designed to be horizontally scalable. So, Neptune is at least 10–20 years behind regarding the needs of the market. In fact, is vertical scalability even a thing anymore?

Lack of ACID transactions is the biggest complaint from NoSQL databases. Distributed transactions are a hard but solvable problem. But, Neptune is a single server architecture, so it’s almost unbelievable that they don’t mention a single word about their support of ACID transactions. I’m going to be optimistic and presume that they do support them but just failed to mention it in the feature list. It’s commonly believed that support of transactions is what distinguishes a database from a datastore.

Why are these drawbacks important?

Let’s take a step back and look at database evolution. For a long time, we only had relational databases which ran on a single server. You could potentially have an eventually consistent replica, but all the data still had to fit into one machine.

Web exploded, and Google being the search engine for the web, had to keep up. They launched MapReduce in 2004, which among other things, helped run the indexing system at Google to process all the data. Google realized that single server databases were not able to handle the data growth and launched Bigtable in 2006. Bigtable was designed solely with the idea to build a horizontally scalable database, something which relational DBs didn’t provide. This was a remarkable feat at the time, but achieved at a cost to the developer, with lack of joins and transactions.

Bigtable kickstarted the entire NoSQL revolution, out of which MongoDB, RethinkDB, and Cassandra came. But, lack of transactions continued to remain an issue for developers. Within Google, Megastore and Percolator were born to deal with these shortcomings, but at the cost of performance.

Calling lack of transactions in Bigtable, the “biggest mistake as an engineer,” Google worked to build Spanner, a horizontally scalable, multi-version, globally-distributed, and synchronously-replicated database, with distributed transactions. Again, this kickstarted a whole new revolution, people are calling NewSQL, with CockroachDB and TiDB following Spanner’s heels.

All these show where the database market is going. A database of the 2000s is horizontally scalable, supports distributed transactions and is consistently replicated. Without these guarantees, building application on top becomes complicated, something Google’s Bigtable team and NoSQL users learned the hard way.

Given these advancements, Amazon Neptune’s design is pre-2000. Single server vertically scaled, asynchronously replicated, lack of transactions — all this screams outdated.

Open Source Alternative

Now, let’s consider an open source alternative: Dgraph, a distributed graph database.

Licensing: Dgraph is available under AGPLv3 with clients under Apache 2.0, an open source license (similar to MongoDB) allowing any one to modify it, use it and run it for free.

Scalability: This is 2017! Dgraph is horizontally scalable. As you add more servers, data automatically gets sharded and moved to fill the new ones. Also, Dgraph would automatically rebalance shards among servers to ensure that load is evenly distributed.

ACID Transactions: While Neptune doesn’t mention anything about transactions, Dgraph runs lock-free distributed ACID transactions, with snapshot isolation, designed for performance. Transactions make it a lot easier for application developers to think through database behavior, removing any data integrity issues.

Replication: Dgraph provides consistent (synchronous) replication, utilizing Raft, a consensus algorithm, to allow k servers dying in a 2k+1 replication setting. Each data shard can be replicated an odd number of times, and located in different datacenters, or availability zones to allow faster communication from your clients.

Linearizable reads: Neptune’s eventual consistency model, can cause issues where a read after write might not be visible through a replica; requiring the application to build logic around this limitation.

Dgraph avoids this problem by guaranteeing atomic consistency of writes, which means irrespective of which replica is hit for reading, any write done before is guaranteed to be available. This guarantee makes it a lot easier to build applications.

APIs: Dgraph supports a variant of GraphQL, a modern language to replace REST. Cypher and Gremlin support are in the pipeline.

Others: While Neptune is built on top of another DB (Amazon Aurora), Dgraph is a native graph database. As such, data storage is designed for constant time edge traversals. Also, Dgraph has a unique design, which uses posting lists, a concept from search engines, to make joins highly efficient. It’s designed for a real-world cluster, minimizing network calls and disk seeks, running automatic fail overs in case of communication delays, running all operations concurrently. All these innovations show in the low query latency provided by Dgraph.

Conclusion

Amazon’s Neptune is an exciting new graph database, a clear signal that graph databases are being used by companies small and large. I hope that Amazon provides more content to help developers create and switch their applications to utilize the power of graph DBs.

While Neptune comes with the convenience of a managed service, it’s design is outdated and the technology proprietary. Therefore, potential users of graph databases, who want to build smart, scalable applications and need ACID transactions, horizontally scalable design and strong replication consistency should look into the various open source graph databases available, like Dgraph.

Disclaimer: I’m the founder of Dgraph Labs. We’re building Dgraph, a distributed graph database. Our mission is to build world’s best graph database, period. I have strong opinions about technology.

People think Linux on a laptop doesn’t work. I find that funny. I have been using it since early 2016, without any issues. Here’s me giving talks at conferences, and you’ll notice my laptop isn’t a Mac.

The Hardware

So, here’s my setup. I got rid of my Macbook Pro (2013 model), and bought a Thinkpad T460. They have many other thinkpads, so you can choose a model and then customize the hardware all you want. I thought about going for Dell’s Linux laptop, but had bad experiences with hardware quality of Dell.

One could potentially run Linux on Mac itself. But, here’s the deal. Being an engineer, I type a lot. Back in 2009, I started having a bit of pain in my wrists, aka RSI. So, I switched over to using Kinesis Advantage. That keyboard is truly ergonomic and has saved my RSI from developing further.

On Mac’s keyboard, after 10–15 mins of typing, my wrists would start paining. And I couldn’t type anymore. The keyboard design causes fingers to “bottom-out” easily. Due to this, I stopped using my Mac laptop for any real work.

Thinkpad’s keyboard is mechanical [See 1 below]. Since I got it, I have been able to type emails, code, and just use it as an essential tool laptops typically are. That itself was a great benefit to me.

On top of that, I configured my Thinkpad to come with a dual battery pack. Which makes it last an entire day of non-stop work, including hangouts sessions, without having to charge. This has been very useful as you can imagine. Long flights, conferences, or just carrying laptop to office, I don’t need to have the charger with me.

Downside: I don’t use my trackpad anymore. My first laptop back in 2003 was a Thinkpad, and I’ve been a fan of the Trackpoint. I use it, and in general just avoid using mouse. My window manager enables less mouse usage (described below), so I can get away with not using the trackpad.

The Software

Mac’s entire ecosystem is designed to take money from the consumer and lock them in. You can easily find the equivalent Linux programs for almost everything, free and more user friendly. For e.g., I paid for 1Password on Mac. Instead I use pass on Linux. I paid for Omnifocus on Mac. Instead I use Neovim with Notes. All the video chatting happens via Hangouts, or Google Meet, or other browser supported softwares. Slack has a Linux app.

I use Arch Linux, a flavor of Linux with rolling releases. So, I get the latest software update automatically, without having to wait for the release cycle of Ubuntu. Arch has a small, but ardent community.

Now here’s the killer app that I vouch most of you don’t know about. It’s called i3 window manager. I came to know about it during this machine build for Eric S. Raymond. It’s so good, that it should be installed by default for any software developer out there. Everything else feels like a struggle in comparison. Window’s UI, Mac’s UI, Ubuntu’s UI, bring anything. I’m just to outright say that nothing is as productive as i3-wm. I’m not going to go into reasons here. But, you’ll find every program that you need in the right place, you’ll use your entire screen estate to its full capacity, and you’ll never have to drag and drop, minimize, maximize anything. And that multiple desktop thing that every OS has, but you don’t use because it’s cumbersome; that’d no longer be the case with i3-wm.

I know the above might sound too much. I’m practically saying all other window managers suck in comparison (your mileage may vary). So, you could say I’m a fan. In fact, my entire team is. All my team and I run Linux on Thinkpad, with i3-wm.

Overall, I shell out way lesser money since I switched over to Linux on my laptop, and my experience has actually improved.

Downsides: Some people find Linux lacks good photo and video editing tools in comparison with Mac. I don’t do that, so that’s never been a problem for me.

Conclusion

I can’t think of anything more to add about my experience using Linux on a Thinkpad. But, I’ll say this, I’ll never go back to a Mac. It’s not because I have something inherent against Apple. But, in Linux and Thinkpad, I’ve found a solid tool and medium.

Update

[1] As per Michał Krzysztof Feiler, ThinkPad’s keyboard is not mechanical. It is a scissor switch keyboard, a pretty good one though, with a much longer drop and much more cushioned stroke then what is found in modern Apple keyboards, so there is a neat feedback and no bottoming out.

I’m very excited to use this first post to talk about DGraph, what it is and why it was created.

Before I explain what’s DGraph, let’s start with a basic understanding of graphs. A graph is a mathematical structure used to model pairwise relationship between entities. A graph is thus composed of nodes connected by edges. Each node represents an entity (a person, place, thing, etc.), and each edge represents the relationship between two nodes. Some popular graphs that we all know about are the Facebook Social Graph, and the Google Knowledge Graph.

A graph database is a database that uses graph structures with nodes and edges to represent, store and serve data.

But who really uses graph databases? More teams and companies than you’d think. Google, Facebook, Twitter, eBay, LinkedIn, Amazon, Dropbox, Pinterest — pick a company you are familiar with. If they’re doing something smart, chances are they’re probably using a graph database. Even very simple web apps have much to gain from graph databases. In the past, I’ve built a graph based REST framework, and using that cut down the code in half.

So now that we understand graphs, let’s talk about DGraph.

DGraph is an open source, low latency, high throughput, native and distributed graph database.

To understand why it was created, let’s rewind a few years back to 2011.

I’d been at Google about 4+ years with the Web Search Infrastructure Group. Google had just then acquired Metaweb a year earlier in 2010. I’d been wrapping my head around the newly acquired Knowledge Graph, trying to find ways to integrate Knowledge Graph with Google Search. This is when I found a problem.

At Google, we had multiple knowledge bearing feeds called One Boxes. You know, the boxed snippets that sometimes show up at the top of the search results, for instance when you search for Tesla stock, Weather in Sydney, or Events in San Francisco.

There were multiple custom built backends, each serving a One Box. A search query hitting www.google.com would be sent iteratively through each of these One Box backends to check if any of them has a response. When one of the backends responds, the One Box data is retrieved and rendered on the top of the search results page. This is how that well formatted box with just the right information shows up below the search bar, thus saving you a few clicks.

As good as it sounded, One Boxes had several inefficiencies and missed opportunities.

For starters, each One Box was custom built by a separate team that was responsible for running and maintaining it. As a result there was no particular sharing of the framework used to build the One Box.

This also meant that there was no single standard for the data format used by the One Boxes. Each One Box kept its data in its very own data structure, and no common querying language. Thus there didn’t exist an opportunity to share data amongst the boxes, to respond to more interesting queries that required an intersection of diverse data feeds.

A good example of this would be the ability to recommend events based on the weather to a tourist exploring NYC — that couldn’t easily be done with the existing system.

This motivated me to start a project to standardize the data structures, and eventually serve them all using a single backend. Using the vast expertise of Metaweb team, we chose a data normalization structure that was also used by Knowledge Graph, the RDF Triples. By reconciling all the various entities from the different data feeds, we could start to reuse the data.

But, there was a second and more challenging part to the problem.

It was to build a system that could serve structured queries with data updating in real time. The system had to run behind Web Search, which meant that if it doesn’t respond within allocated milliseconds, Search would time out and move on. Also, this system had to tackle a major chunk of query load to Web Search, which amounts to thousands of queries per second.

We basically had to build a low latency, high throughput system to serve graph queries.

It was certainly an exciting project and held much promise. But, the harsh realities of the business environment and the attendant politics resulted in the cancellation of the project. Shortly thereafter I left Google in 2013 and didn’t give much thought to the project.

Fast forward two years, I was hanging out on the Go language’s Slack channel and Stack Overflow. I saw quite a few people complaining about a popular graph database’s performance and stability.

That’s when I realized that graph databases were starting to be used more frequently than it would appear from the surface. But a bit more digging around revealed a deeper problem.

Existing native graph databases weren’t designed to be performant or distributed.

The ones that sharded the data and distributed it across a cluster weren’t actually native graph databases. They were largely serving as a graph layer over another database. This meant having many network calls should the intermediate number of results be large, which leads to performance degradation.

For example, say you wanted to find [People living in SF who eat Sushi]. Assuming you have this data (hey Facebook!) and keeping things simple, this requires 2 steps.

First you find all the people living in SF, and then secondly, intersect that list with all the people who eat Sushi.

As you can imagine, the intermediate step here has a large fan-out, i.e. there’re over a million results. If you were to shard the data by entities (people), you’d end up broadcasting to all the servers in the cluster. Thus this query would be affected by even a single slow machine in the cluster.

Do that for every query, and it would spike the 95%-ile latency numbers up dramatically, higher latency being worse.

DGraph, on the other hand, is a native graph database in the sense that the data is handled directly by DGraph, and not given off to another database layer.

This allows us to shard and relocate the data better, to minimize the number of network calls required per query. In fact, the above query would run in 2 network calls, irrespective of the cluster size.

The number of network calls being directly proportional to the complexity of the query, not the number of intermediate or final results.

DGraph is designed to easily scale from meeting the needs of a small startup to that of Dropbox, or even Facebook. This means being able to run on a laptop as well as on a big cluster of hundreds of machines serving thousands of queries per second.

Additionally it would also have to survive machine failures and partial datacenter collapses. The data stored would have to be automatically replicated with no single point of failure, and be able to move around the cluster to better distribute traffic.

This is the big vision which led to DGraph. And I’m fortunate to have a team that believes in and shares this vision with me.

Apart from use with diverse social and knowledge graphs, DGraph can also be used to:

• build real-time recommendation engines,
• doing semantic search,
• pattern matching,
• serving relationship data, and
• serving web apps via GraphQL, a full feature graph query language by Facebook.

We’ll be reporting some performance numbers for DGraph in our next few posts, to give you an idea for what you can expect from the system.

If you or your company plan to use a graph database, we’d love to hear from you at discuss.dgraph.io and Gitter. We have big dreams for DGraph, and if deep and complex open source distributed systems interest you, note that we’re hiring! So, check out our Github repository, then find us hanging out at discuss.dgraph.io and come talk to us.

- Written by Manish R Jain. Thanks to Prashanth Koppula and the team for proof reading.

Originally published at blog.dgraph.io.

Hello, World! was originally published in Dgraph Deprecated Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Have been thinking about sharing my setup with general public, but my personal setup contained, well… some sensitive data. So, it took time to fork it all out for public consumption. And ta-da, here it is.

• mrjn/godev Repository | Docker Hub Registry - Repositories of Docker Images
• manishrjain / godev

This contains my development language of choice: Golang with my editor of choice: vim, setup for working neatly with golang. And some python, some nodejs, and some other things I need on a daily basis. Modify it to your liking, otherwise just enjoy the default setup, which works like a charm!

A bit about me: I joined Google right out of school in Singapore, and my first full time project belonged to Web Search Infrastructure group. That was 2007. By last year, 2013, I’d gone through 5 different projects all belonging to Web Search/ Knowledge infrastructure within Google. I’m a big fan of massively parallel, distributed, low latency systems. They impose very interesting challenges related to threading, network communication, asynchronous / synchronous callbacks, and of course logic. I strongly believe that if a system is to be performant, you can’t add it on top as a functionality. You have to design the system from scratch for performance, weighing out every extra bit of complexity against functionality gains.

The interviewer: While at Google, I interviewed around 50 candidates, and my favorite question was to dig into open ended design.

Over time, I established a really nice rule to figure out if the candidate did well. The rule is simple:

A: Did I have a great debate about the pros and cons of the approach taken by the candidate? Did we agree a lot? Did we disagree a lot? Did I learn something new? Overall, did I have a great conversation, the likes of which I’d want to have with him / her as my colleague?

or,

B: Did I have to spoon feed thoughts to the candidate? Was he really slow? Was it an effort to keep the conversation alive?

Case A was amazing. I loved those interviews. I felt energized when a candidate challenged me in my thinking, could think of alternative solutions, and why they would or wouldn’t work. The best of my candidates were the ones who designed something, admitted in the middle that what they designed was flawed, and they’d do it from scratch with a new methodology. I was particularly impressed by them, because their love for perfection was above their ego.

Case B were no hires. It was an effort even to write feedback for such candidates.

And then there was a whole spectrum in between.

I think a great engineer resembles a painter. He’d design something, learn what works, what doesn’t. Throw it away, and design better. Most importantly, he’d be hard to please with his own design.

The interviewee: It had been 6 years at Google, so I decided to work for a smaller company. I left Google, and started interviewing. And I was surprised!

Interviewers were asking me design questions, and I was trying to have a great conversation, a debate, thinking / rethinking pros and cons of various alternatives. But, a lot of them weren’t looking for that.

What were they looking for? Think of an algorithm question. The question is defined. The answers are well known to interviews. There’s the typical brute force solution, then there’re improvements. Then there’s the best answer, which has best space-time complexity possible. If the interviewee can come up with the best and code it, you got a happy interviewer. This is the standard.

What was surprising was, these interviewers who were asking “open ended design questions” were stuck with the same mindset. They were looking for a particular flow. They were looking for a particular answer, or a particular question that the interviewee was supposed to ask. What astounded me was how little they cared about entering a conversation, a discussion, a debate about alternatives.

Recently, I interviewed at a really well known company in Berlin. I was asked to design a key-value store. I started with explaining a design similar to Memcached. Then went onto explain the design for a persistent store, like Bigtable. Then went through the failure scenarios — if master crashes, if worker crashes, if the network goes down, if the datacenter goes down etc. I talked about replication, talked about sharding, talked about backups. I think within that hour, I’d discussed elaborately about the pros and cons of every piece and aspect of design I could think about.

The interviewer, apparently, wasn’t satisfied. He complained that I didn’t ask the “right questions.” I was rejected. I remember how we started the conversation:

Interviewer: Design a distributed key value store.

Me: Like memcache?

I: Memcache isn’t distributed.

Me: Well, it actually is. The Memcache client supports sharding by default. And uses a continuum based consistent hasing approach, which allows us to add or remove servers without affecting all the keys… You only affect 1/(N+1) keys, yada yada.

I: … nothing ..

Me: Hmm.. alright, so here’s how a design like Memcache would work..

And so on..

I can sense when the interview doesn’t go well. Even though, this particular one went really good in my opinion; I could feel that the interviewer wasn’t into it by the end of it.

How do you know if the interview went well or not? You get 5 mins at the end of interview for questions. As an interviewee, you can gauge by asking the interviewer a question about the interview process (don’t ask how the interview went). If the interviewer didn’t feel good, he’d shy away or be awkward about it.

I sensed it. After the interview, I asked myself honestly. Would it be possible that the interviewer rejects me? And my experience told me, that’d be stupid. I explained so many aspects of the system, it would be hard to ignore all that “meat”. And it very well, turned out to be very idiotic.

I think that’s deeply flawed.

And the reason this happens is lack of experience of the interviewer, and hence lack of sensible reasonable expectations for the outcome of the interview. If you’re an interviewer, asking design questions, please do everybody a favor: Enter the discussion. See if you can have a great conversation. Keep yourself open to ideas, new ways of thinking, even though they might conflict your established design. Ask the candidate lots of questions, challenge him. You might just find yourself a colleague you’d want to be around.

In any case, don’t sit quietly looking for a “particular question”. If you’re one of those guys, stick to algorithm questions, don’t venture into open ended design territory. You suck at it!

About the cover: Van Gogh wasn’t impressed with his now most well-known and admired work.

In a letter to his brother, he describes the painting:

The first four canvases are studies without the effect of a whole that the others have . . . The olives with white clouds and background of mountains, also the moonrise and the night effect, these are exaggerations from the point of view of arrangement, their lines are warped as that of old wood.

Leonardo Da Vinci held onto his painting of “Mona Lisa”, without releasing it, continuing to work on it for years, stroking a brush now and then. It was drawn on a normal size canvas, at somebody’s request. He didn’t think of it as his masterpiece. It was supposed to be a painting for a gallery, to be hung along with many others. Definitely not deserving of an entire wall, in his opinion.

Engineers can learn a lot from such remarkable painters.

Pangs of the American migration system for an India born emigrant

In the definition of the American Dream by James Truslow Adams in 1931, “life should be better and richer and fuller for everyone, with opportunity for each according to ability or achievement” regardless of social class or circumstances of birth. — Wikipedia

2006 — The Exchange Visitor

I landed in San Francisco on a Japan Airlines flight from Singapore on a lazy Saturday afternoon. In two days, I’d be working for the best tech company there was, Google. I had realized my life long dream, to live and work in the US. The land of opportunities and equality. And I believed in the American dream. I was living it.

Two days before my flight into US, I had received a J-1 visa, which allows interns from all over the world to come and learn from American advancements in technology, medicine and science. However, J-1 came with a clause.

When you agree to participate in an Exchange Visitor Program and your program falls under the conditions explained below, you will be subject to the two-year home-country physical presence (foreign residence) requirement. This means you will be required to return to your home country for two years at the end of your exchange visitor program. This requirement under immigration law is based on Section 212(e) of the Immigration and Nationality Act.

http://travel.state.gov/visa/temp/types/types_1267.html#twoyear

This special requirement put my life on a trajectory I didn’t anticipate. Even though I was a student at NTU, Singapore, and had left India 3 years back already, this clause meant I was supposed to go back to India and work there.

2007 — Singapore Permanent Residence

I once asked visiting guest to NTU, Vivian Balakrishnan, Singapore Minister from ruling party, why he was paying for my education, my living expenses and giving me extra money to enjoy my life in Singapore. He replied, “So you don’t go and work for India, China or the US. We’d rather have you working for Singapore once you graduate, and developing and innovating here.”

As soon as I graduated from NTU, Singapore, I got offered a Permanent Residence to live and work in Singapore. All I had to do was to sign a paper, and mail it in to the Singapore government, and wait a couple of weeks.

I rejected that offer. I still hadn’t achieved my American dream.

2009 — The Waiver

I was living in Zurich, Switzerland. My H1-B visa for US was denied in 2007, due to my 2-year residency requirement. But, Switzerland welcomed me. So, I was working in Google Zurich office, living a rich life in a beautiful city, with excellent public transportation.

The Swiss work permit allowed me to freely travel in Schengen region which wasn’t possible through my Indian passport, my spouse to work, and for me to receive a Permanent Residence within 5 years. For many, it was a dream come true. But, not for me.

It took me 1.5 years and 7 different ministries in 3 countries to get my 2-year residency requirement waived. 4 ministries from India were involved in this process, 2 from US, and 1 from Switzerland. But, all the trouble did mean one thing: I could now apply for an American working visa. So, the process started again, and this time I got an L-1 visa, an “intra-company transfer visa” from a foreign branch to a US based branch.

2010 — The Marriage

I was working in Google’s Mountain View office, when I fell in love with a beautiful Googler, who decided to reply to my deliberately casual email, talk to me, and finally go out on a date with me. We married within 6 months. She was working in Google India office, and after marriage, she moved with me in San Francisco. She applied for an L-2 visa, a spouse of L-1.

2011 — The H4

My 2 year L-1 visa was coming towards its expiry, and so I applied for an H1-B. This year, the H1-B lottery wasn’t too bad, and I did receive it. My spouse switched from L-2 to H-4.

Job market was doing better, and after an initial period of honeymoon, my wife was now looking to settle down in this country that both she and I had started to call our home. However, this wasn’t going to be possible.

H-4 visa holders are not eligible to get a Social Security Number and cannot be employed. — Wikipedia

It was a simple statement, yet so difficult to comprehend. This literally meant, she had no other option but to give up her career. Even though we repeated the statement over and over again to ourselves, it didn’t get better. US immigration system did not care about my spouse. She couldn’t get a personal credit card, a employer covered insurance policy, or share the burden of high cost of living in the Silicon Valley. She could no longer work.

That’s when I realized, America — the land of opportunities, isn’t really so.

2012 — The Greencard

A green card allows a person to live and work anywhere in America, and is a path to citizenship. An H-1B visa-holder is beholden to the employer who hired him or her, and can be deported unless the holder can find another H-1B sponsor.

Commonly termed as Permanent Residence all over the world, Americans call it a Greencard, because the first Resident Alien Card had green-stripes on one side. This card allows one to work indefinitely in the US, switch jobs without renewing your H1-B, and start and own innovative new companies.

The Greencard through jobs category, while aimed at picking up the best talent from all over the world for jobs that “no American was qualified to do”, has no formal points based process to pick up the most qualified people. Most of the applicants are put into 2 categories: workers (EB-3), or professionals (EB-2, holding master’s degree or above).

Having landed a job with Google even before graduating, I chose not to do masters, a decision that’s a trend in Silicon Valley, where the smartest people drop out of schools to innovate and build the next best things. If you’re scratching your head for examples, such folks include Bill Gates, Sergey Brin, Steve Jobs, and Mark Zuckerberg.

For me, Greencard was never really an option. I landed in US with a bachelor’s degree, which put me in the lowest EB-3 category. I was born in India, which put me in a long waiting list of 10 years.

If Bill Gates, Steve Jobs or Mark Zuckerberg were to be in my shoes, born in India and a desire to live and work in US, they’d be denied a working permit, i.e. H1-B, because they didn’t have even a bachelor’s degree. Sergey Brin would get an H1-B, but would have to wait 10 years to get a Greencard, like all the Indians in EB-3 category do, before being allowed to start up a company.

If I were a Mexican worker, I would get a Greencard within 2-3 years. But, I can’t change where I was born. My American dream was to provide me equal opportunities regardless of social class or circumstances of birth. That’s not really how it works though, does it?

The American dream, I finally realized, is totally fucked up!

2013 — The Reawakening

So, I put my family and life before this dream, and decided to move to Australia. It was the same me, but a different system, which recognized what I had to offer to their country. And they loved me back, granting me a Permanent Residence based upon my skills and qualifications, within a year.

My dreams I felt, were misplaced.

I wrote this post to remind everybody in my situation, choices that they have to make, whether conscious or unconscious.

In the next post, I’ll talk about why I chose to migrate to Australia over the rest of the world.

Update (May 18, 2016): Sorry that next blog post never happened. But, I doubt it’s necessary now, because this:

http://www.afr.com/technology/cannonbrookes-blackbird-bain-back-new-migrants-graph-startup-20160517-goxglw