I’ve been lucky enough to work on small and large scale projects, some more and some less critical. This is a brain dump of my thoughts on processes, constraints, excessive red tape, consequent complexity and frustration and whether (and how) there’s a way to make things better. The TL;DR is that some (“best”) practices are contextual and understanding when to use them is ultimately what gives us the title of “engineers”.

Also the charts below are purely indicative, take them with a grain of salt.

Shipping confidence

We can define “shipping confidence” as the feeling a mentally sane developer has when they know their code is about to be deployed to production (whether it can be updated over the air or not). Like with any feeling, shipping confidence is a spectrum.

A company can be placed on such spectrum as shown in this example:

If Boeing makes a mistake in their flight control software, people will most likely die.

If Tesla makes a mistake in their autopilot software, people might die.

If JP Morgan makes a mistake in their funding software they will lose $ and some people might see funky numbers in their accounts, leading them to changing bank.

If Netflix makes a mistake in their streaming software millions of people’s dates and lunch breaks will be ruined and they might not renew. They would also lose credibility with their partners.

If GitHub makes a mistake it can affect thousands of businesses but they’ll likely shrug and their DevOps team will just post “GitHub is down, nothing we can do” on some Slack channel. They might offer some credits you can use within 30 days.

If Pieter Levels (👑) makes a mistake some customer might request a refund. He’ll fix it, then go back to shitposting about threadbois.

It’s clear that shipping confidence is tightly coupled with the loss of money (even the loss of life can be quantified in $, sad truth but truth nonetheless), whether direct or as a consequence of reputational damage. This is something that is normally explained to sophomore CS/CE students in some “software engineering” or “software architecture” kind of course.

The reason it’s usually taught to sophomores is because freshmen are too busy to understand what it means (they are only, rightfully, worried about making something work no matter what). And junior/senior students have likely already experienced this through an internship or a job.

Intuitively there’s also a direct relationship between shipping confidence and developer experience. And since the latter has a direct impact on throughput, correctness and, in general, team morale, it is important to understand how they behave together.

Cannon and flies

We have seen that the two main dimensions of shipping software are confidence and financial impact. Sometimes it’s hard to estimate the latter so we’ll just name it “criticality”.

Changing the name from “financial impact” to “criticality” allows us to turn the question “how much $ would we lose if this went wrong?” to a more generic, yet easier to estimate, “what’s the worst that can happen?”.

Some examples:

What’s the financial impact of a SpaceX rocket exploding?

Too hard to estimate the financial impact. How many man hours were spent on building it? What about the materials? Was anyone injured? Does it expose the company to a lawsuit?

Instead we can ask:

What’s the worst that can happen if a SpaceX rocket explodes?

Well, we can absolutely limit the criticality of such event. Stay as far as possible from any inhabited area, make sure everything can be remotely controlled, avoid a human payload for the first N tests etc.

Obviously SpaceX already does all that but it helps to outline some of the steps because they tell us something very important: it is fundamental for SpaceX to decrease criticality as much as possible, so time spent doing so is time well spent.

But not all companies deal with this kind of criticality. In fact the four quadrants below help us determine what degree of preparation is necessary once we can answer the question “what’s the worst that can happen?”.

The ideal quadrant is obviously “Safe and fast” but not everyone is lucky enough to be able to stay there. A small project that starts there might move to “Safe but slow” (e.g. a company that scales up and can’t afford downtime) or to “Move fast and break things™️ zone” (e.g. a weekend project that gets funding and becomes a growth oriented startup).

There is absolutely no reason why any software project should stay in the bottom left quadrant. The “opportunity cost zone” should be avoided like you would with nuclear waste.

Yet so many projects end up right there. Why is that?

Premature optimization

I hate the terms “junior” and “senior” so I’ll replace them with “smart” and “grug brained”, respectively.

Most smart engineers are primarily moved by one thing: wanting to prove themselves. So they dive into best practises, elegant patterns, flamboyant versioning strategies and they often ask “what’s the best way to do X?”. How I know? Because I used to be one. And I was lucky enough to meet grug brained peers that were willing to question my choices.

Grug brained engineers start with a much simpler question “what’s the point of doing X?”. For a grug brained dev it’s truly soul crushing to work on X when the impact of X isn’t clearly defined, or at the very least estimated.

A great example of this is, lo and behold:

microservices.

Microservices are a great pattern when you have multiple teams and you want each team to have a high degree of autonomy in things like the choice of stack, deployment processes etc. As long as the edges of these services are clearly specced and documented (whether this is true is purely a matter of diligence and competence) things will be fine.

Microservices are utter crap when your team is made up of 5–10 people, everyone kinda works on everything, you’re not even sure of what you’re building yet and maybe you even rely on a monorepo because you read that Google does it so why shouldn’t you? Increased complexity and friction for no benefit whatsoever.

A smart developer might read somewhere that microservices are a great solution to complexity. What he fails to understand is that there’s no such thing as a free lunch. Employing a pattern like microservices (or whatever else) will decrease overall complexity by N% but add a fixed amount of inherent complexity due to the added constraints.

This translates into the single worst thing you can do to (figuratively) kill your engineering department by a thousand (churn) cuts: poor developer experience.

Developer experience

Developer experience is how much your (current and potential) devs want to work on a specific project. Poor developer experience is a byproduct of dissonance between perceived complexity of a project and tangible complexity of the systems and processes related to it.

Imagine taking a big, satisfying dump and then having to solve a Rubik’s cube to get some toilet paper. The perceived complexity of taking some paper to wipe your ass is (I hope we agree) low. When you’re presented with a Rubik’s cube your only thought is that you’ll never use that toilet ever again.

In the best case you end up spending minutes or hours (opportunity cost!) solving the cube and leave in frustration (but with a clean butt). In the worst case you throw away your underwear.

Now, I’m sure whoever designed the toilet had a good reason to put that Rubik’s cube there. Maybe someone kept stealing the toilet paper. Maybe it’s a test to determine whether you’re worthy of having a clean butt. Who knows, but most of all, who cares!

The only people who’ll keep enduring that kind of pain are desperate people. People who need that paycheck, people who can’t be bothered to look for something different. Unfortunately there is no easy way to distinguish between people who are good and need a paycheck from people who just need a paycheck. But you sure as hell don’t want the latter in your team.

Therefore detecting this kind of dissonance becomes increasingly harder the larger the company grows. People will be less motivated to report, and contribute to the resolution of, these issues. Those who try will be dismissed with a vague “yeah, we’re working on it” and find ways to work around it in the hope that one day they’ll be in a position where they are taken more seriously.

A concrete example of this is adding logic to work around issues with internal infrastructure or APIs. Imagine you knew the guy who put the Rubik’s cube there and, because he’s not exactly easy to work with, instead of asking him “dude wtf is this?” you realized you can break the cube and reassemble it solved to get your toilet paper. Most people, believe it or not, will do the latter.

I’ve been a witness of this negligence in multiple occasions: a team starts working on a new project that needs support from another team, they start using their service only to realize there’s undocumented behavior or nasty bugs, so they try to report it once, twice, three times and those tickets sit in “Backlog” for weeks, sometimes months.

At some point frustration takes over and the first team decides they can just implement some additional validation, retries etc. to work around all that. That’s a terrible mistake and in the long run will be the cause of cost overruns, unmet deadlines, increased churn and overall bad vibes. And nobody wants bad vibes.

Easy math: imagine this is a service used by N teams and it has a bug that only comes out under specific circumstances so that it goes unnoticed until the first production deployment is completed. Fixing this bug might take 2 man-days. Let’s say some super senior guy who makes $300k/year has to work on this, which prices the bugfix at around $2400 (considering ~250 working days in a year).

Imagine now that it takes 3 days to find such bug, diagnose it and deploy a workaround. And these teams don’t really talk to each other (Big Corp style) so it’s not like they can write once, run anywhere™️ (nor they should, as shown below). Let’s assume the dev on each of these teams, that has to write the workaround, is paid $150k/year. N = 2 is enough to make fixing the bug the most rational, economically sound choice.

Even worse: imagine the same devs realize that they share the same pain and come together in a sort of think tank and start contributing to a single client to access this service. This way the workaround will only have to be implemented once and everyone will benefit from them, right? RIGHT?

Your job as an engineer is to do what’s best for the whole organization, not just for your team. Learning to (blamelessly) hold other teams accountable is an art, more than a science, but it’s what eventually helps the whole organization move forward.

So what can we do to avoid this dissonance in the first place? Easy: question everything and ask the right questions.

There is a fundamental difference between complaining and questioning. A complaint is an expression of frustration that carries no expectation that the source of the frustration will be resolved and no suggestions on how to do it. Questioning is an expression of doubt, accompanied usually by data (to prove the frustration is real and what impact it has) and, when possible, a suggestion on how to resolve it.

And what’s the best question we can ask when trying to determine whether the complexity of a system is justified in comparison with the perceived complexity of the task at hand?

“What’s the worst that can happen?”

And we’re back at criticality!

What’s the worst that can happen if we scp our PHP files straight onto a Linode VPS?

• In a team of 1 with a product that has 10 users? Absolutely nothing. (Ironically we’ve seen this is true with thousands of customers too, at least in the B2C world)
• In a team of 10 with a product that hasn’t launched yet? A “wtf” from a colleague on Slack.
• In a team of 100 with a product that has 1M customers? Absolute madness.

What’s the worst that can happen if we go down for 1 hour?
(Forget that BS scene from The Social Network please)

• For a product that turns your selfies into cat photos? Nothing. Maybe a refund.
• For a product that handles automation for an HR department of a 150 person company (à la Workday)? A few complaints and a post mortem.
• For a beta product that handles customer funds but serves like 20 transactions a day? Some frustration (assuming no loss of funds).

What’s the worst that can happen if there’s data loss?

• Can your data be rebuilt from other data sources over a brief enough period of time (e.g. consolidated accounting from multiple cost centers in a day and the report deadline is a week away)? Nothing.
• Is the data from your customers but ephemeral (think WeTransfer)? Maybe a few bug reports, refund requests.
• Is your data customers’ emails and you can’t recover a backup with an acceptable delta (hours, not days)? You’re screwed. (Video from DHH (👑) that inspired this example)

You get the point.

Disclaimer: it’s one thing to go down 1 hour every 3 months, on average. A completely different thing to go down 1 hour every couple of days. Maybe I’ll add time/SLAs as a 3rd dimension to the confidence/criticality chart.

“Guards!”

Once you’ve assessed what the worst case is the follow up should be “is it worth spending time adding guards to avoid it?”.

In most cases the answer is a resounding “no”.

“Is it worth hardening my code to make sure there is never an uncaught exception, while processing a selfie that turns into a cat, to keep my logs clean?” — No.

“Is it worth adding a catch clause for a specific kind of error that I know how to recover from, to prevent a known issue from blocking a series of sequential steps?” — Hell yeah!

“Is it worth adding retry logic for a function that is being executed every few minutes and that can fail up to 10% of the total times it’s called with no tangible issues?” — Nope. But you should monitor that % and act accordingly.

If the answer is yes and the solution is semi-trivial then go ahead!

If the answer is maybe but you’re unsure about what it would entail you unfortunately need to estimate the opportunity cost of not avoiding it. If it exceeds the estimated cost for implementing the solution (by a fair margin) then green light. Otherwise, why bother?

Keep in mind that part of the cost comes from something that modern development tools prioritize a lot, and rightfully so…

Feedback loop

If your guard increases the feedback loop’s completion time, that’s something you won’t get rid of over time, quite the opposite! These changes tend to quickly compound and getting rid of them in the future becomes exponentially harder the longer they are there. (Ok maybe not “exponentially” but you get what I mean)

If there are 5 devs on your team and they complete, say, 100 feedback loops (e.g. committing to a remote environment) per day, an increase of 10 seconds leads to an increase of almost 1.5 hours in total time spent every day. At $100/hour (which is fair for experienced devs) that puts our increase in costs at about $3000/month. (1.5h/day * $100 * 20 days)

What can possibly cause a 10 second delay on each feedback loop, you ask? Easy: a pre commit hook that runs some integration tests. Or misconfigured caching in the CI pipeline that forces Docker to pull all layers every single time.

And that’s not considering the increase in frustration (and related decrease in shipping confidence) for the team. It might be tiny, but it’s there.

At that point it’s not only suggested, it would be outright wrong not to do it, that you start estimating how effective these changes are in relation to what the consequences might be.

It’s the little things that, put together, make our life miserable.
(Read it somewhere a long time ago, can’t find the source but it’s so true)

Let’s take an example: a service’s SLA forces the company to pay out about $1000/hour for downtime.

An obvious guard would be to have a dev environment that mimics production as much as possible (this isn’t always possible of course). Thankfully there’s a ton of leverage here: the reproducible envs can be automated, together with some sanitization (e.g. you don’t want to have real email addresses in your dev database duh) and tests.

From my experience this is pretty straightforward up to a few hundred thousand users and a DB under a terabyte, assuming reliance on 3rd parties (e.g. payments) is minimal (or mockable).

It becomes pretty much impossible at larger scale. One might be tempted to think “well, if I can’t rely on this when I have 1M users why would I do it at 100k?” and that’s equivalent to asking “well, if I can’t rely on both my legs when I grow so old I can’t use them, why would I do it now?”.

Someone who is overly cautious might think of hiring a QA engineer to run manual tests on some staging environment before shipping. The question becomes: how much are you willing to pay for them and what’s their effectiveness rate?

Assuming that now you go down 1 hour a month and you’d have to pay them $5000/month, the additional resource is clearly anti economic. Things change if the downtime is closer to 3 hours/month and you can outsource the role to someone who’s happy with a $2000/month salary. As long as their effectiveness is ~70% (i.e. they catch 70% of bugs that would lead to a 1h downtime) your monthly savings are $3000 * 70% — $2000 = $100.

Now an overzealous manager might come along and say: “last time we went down the CTO got so angry he almost forced me to switch to a light theme! We cannot go down, no matter what! From now on every deployment will require 2 approvals from selected people”.

Now each dev who wants to add a feature or fix a bug has to:

• Clone the sanitized prod environment. (easy, automated, minimal friction)
• Get the green light of QA on staging. (takes some time but might be worth it)
• Require approval from 2 people, who might not even be familiar with his changes, who might ask him to “jump on a call” to explain what exactly is being deployed and who might eventually tell him they don’t have time to review the changes properly, so better wait until tomorrow. (brutal, brings down morale, dev wonders why 2 people who don’t even know what he’s working on have to review his changes when he’s been writing 80% of the code anyway)

Let’s do some math: let’s say this change is a blocker for such dev. He can’t (or doesn’t want to) work on something else because this is way too important, or because the context switch would be too tough or because there isn’t really anything else worth working on (surprisingly common).

Let’s also assume that it takes 30 mins for these 2 people to review the changes together with the dev. If the changes get reviewed right away, and we pay these people $100/hour, we are wasting $150. That’s 15% of our hourly SLA loss. Seven deployments and we’re back to square one.

In the worst case: time to review (blocking) + requested changes + rollback due to an unexpected issue can exceed, say, 8 hours. That’s $100 * 8 (dev) + $150 * 2 (reviews) = $1100. This policy has just become anti economic with a single deployment.

“What if removing this approval layer leads to downtime increasing?”

Ask yourself a question: do you have any reason to think that your engineers will not do a good job? If the answer is no: why are they still there? If the answer is yes: let them do their damn job. If things go south it’s most likely due to someone being undertrained (easily fixable) or incompetent (get rid of them). Making life harder for everyone due to the shortcomings of one person or team is a non refundable one way ticket to high churn.

Testing on production

Finally the part you were waiting for.

Testing on production is obviously not the right thing to do if your project sits in the top left quadrant of the criticality/confidence chart. But it can be an effective strategy in the top or bottom right one. It’s laughable that some teams, no matter how small or what stage their project is at, will never even consider something like that, out of pure irrationality.

That’s why determining which quadrant your project sits in and acting accordingly can make or break your engineering team. Processes, practices, patterns and workarounds weren’t born because they are the right thing to do. They were born to address specific concerns in specific teams under specific circumstances.

Employing them in a thoughtful manner is the engineering part of, well, software engineering. The absolute worst approach a team can have to “best” practices, patterns and processes is to employ them just because someone wants to or because a team fucked up once and some preventative policy became standard across the whole company.

Developer experience is as important, if not more important, than customer experience. Because if customers aren’t happy, they won’t buy your product.

If developers aren’t happy, that product may never get built.

Notes

Some readers have been kind enough to make some really good points, which I’ll collect below:

• Criticality is multidimensional so we should be careful about not oversimplifying for a whole product or organization
  (NewEntryHN, source)
• “Everybody has a testing environment. Some people are lucky enough enough to have a totally separate environment to run production in”
  (trollied, source) — (not the original author of the quote AFAIK)
• Other executives/middle managers might be extremely scared of something happening, while the core shareholders (often times the CEO) don’t really care. Aligning priorities across teams is fundamental.
  (Scubabear68, source)

For a while I had been looking for the “ideal place to live”, according to a few criteria: English speaking, warm climate all year long (sorry UK!), highly developed, aesthetically pleasing, stable with down-to-earth culture (sorry US!) and a good tech scene. Sydney seems to check all the boxes! In roughly 10 days I was surprisingly unable to find one thing I did not like about the city. Not claiming Sydney is perfect but, to me, it seemed as perfect as one city can get (ignoring the fact that it took me a total of 23 hours of flight to get there). 😍

The venue

Sydney’s ICC is a beautiful glassy building located in the Central Business District (CBD), facing the scenic Darling Harbour. The all glass walls allow sunlight to brighten the common area and make it an ideal space to relax and enjoy the free food and coffee provided by the very well organized catering. Tables with plugs and super fast wifi (a rare finding in a conference) give the attendees a chance to get some work done between talks and, if it gets too noisy, there’s an ample quiet room available just a few meters away. Right next to the ICC there’s the Chinese Garden of Friendship, surrounded by a lively square with green areas and a cool bodyweight fitness track (let me save on a weekly pass for a gym). 🏋🏻‍♀️

The organization was absolutely top notch. Aside from the initial queue for registration (which they overcame by allowing unregistered attendees to attend the early morning talks on the first day) everything else went buttery smooth. The food was, hands down, the best I’ve ever had in a conference. All topped up with lots of free coffee machines available around the common area. ☕️

The talks

Anyone who’s ever attended a tech conference will tell you the talks are usually not the best part. Especially with multiple track conferences (such as this one) you might lose the chance to catch a great talk because the title or description didn’t tickle your interest. On the other hand, it allows for more speakers and more content to be shared. This time was no different and, although all talks are equally important, some talks are more equally important than others 😅, so here are my top 3:

• New Phone, Who Dis?: Human Authentication in the Digital Age 🎫
  A great insight on how primitive personal identification can still be, even after introducing technology in the process. Yaakov shows many examples of how difficult it is to think about our “real” identity in the same terms we think about our digital ones, including finding a vulnerability in NSW new digital driver’s license app.
• Opening Doors with JSON Web Tokens 🚪
  Ben shows us the most secure door in the universe. Well, maybe not the most physically secure but certainly a great improvement over the many commercially available solutions, all flawed one way or another. A comprehensive and clear survey of the flaws in existing domotics products followed by a proposal that makes use of JSON web tokens (which I had only seen mentioned a couple of times, to be honest).
• Building, designing, teaching and training simulation environments for Machine Learning 🚗
  Talk about inspiring presentations! Paris opens the talk with a brief but comprehensive overview of the state of the art on machine learning, to then show how Unity (the game engine) can be used to build visual simulations to train and test ML models. His main example is a self driving kart, built from scratch with Unity assets and a bit of C#, that uses reinforcement learning to learn how to drive itself from historical user’s input. In other words it learns how to drive from the player. Truly encouraging for those who’d like a more fun introduction to ML and AI.

The city

As you might have understood from my opening paragraphs, I was very impressed by Sydney and, although I also had the chance to spend a couple of days in Melbourne, it entered my list of best cities right at the top (if you’re interested, the other 2 are Zurich and Barcelona…So far). I had read about how high the cost of living was but thanks to the favorable exchange rate Sydney was not more expensive than any big European city, rents aside. Real estate is a booming business here in Sydney. During a free walking tour our guide showed us some apartments on the harbour who were selling for $15,000… per square meter! 🤑

I should write about how Sydney has everything: from a NYC-like skyline (after all some scenes in The Matrix were, in fact, shot in Sydney) to huge parks full of interesting plants and animals (watch out for the White Ibis) to the jaw dropping coastline and islands (fun fact: Manly beach is one of the places where surfing was popularized in the 60s with the movie “High On A Cool Wave”), all enjoyable in the middle of winter thanks to the mild temperatures (~15–20 degrees Celsius almost every day I was there) and clear skies. But I’d rather show you some pictures. Yes, yes, the first thing I did was to go see the Opera House. 😒

The very first thing you should do once landed is to get a sim card (I bought one from Yes Optus for 15 AUD with 30 GB of data for the whole month), the second is to get an Opal card (the card is free but the minimum topup is 10 AUD, the train from the airport to the city center is about 16 AUD). Do not forget to tap off! It was not pleasant to see my Opal credit drained after a short bus ride because the system thought I went all the way to the last stop 🤦🏻‍♂️

Moving around and outside the city is easy, thanks to the many trains and ferries. Buses are definitely less reliable but will still work as a backup plan to take you from point A to point B. I do not recommend renting a car (which was my initial plan) since the traffic can really take a toll on your soul, like in most big cities. Sydney can be easily visited exclusively on foot and/or by bike. If I managed to walk from the CBD to Bondi beach, so can you. 🚶🏻‍

With Parth and Vinayak, two great guys I met at the conference, we visited the birthplace of popular surf: Manly. There are 2 kinds of ferry that will take you there: F1 and the Fast Ferry. Use F1 for the view (and Instagram), Fast for the excitement! ⛴

Do not underestimate visiting Taronga zoo! First of all, it’s the only place close to the city where you can see (and occasionally touch) Kangaroos 🦘, Koalas 🐨 and other weird Australian animals, second: it doesn’t even compare to the small zoo in CBD. Taronga looks more like a low budget version of Jurassic Park, the entire place covered by a forest where you can walk around and even get your adrenaline fix by following the Wild Ropes paths around the zoo. Also the view is just…Wow. 🤯

Another activity I highly suggest is to jump out of a plane with a stranger strapped to your back, commonly known as tandem skydiving. It was a very impulsive decision I took on my second day and one I’ll never forget (nor regret since I’m still alive). 🛫

What follows is a selection of some of my favorite photos. This was one of the best trips of my life and I want to thank Hipo for giving me the opportunity to visit what, in my humble opinion, is the best city in the world (so far). G’day!

Keep an eye on posts from Hipo by subscribing to our newsletter and following us on Twitter.

Going Down Under to PyCon Australia 2019 was originally published in Hipo on Medium, where people are continuing the conversation by highlighting and responding to this story.

After finding out that VSCode supports Django/Flask debugging out of the box I was eager to find a way to make it work with our local development setup. Fortunately VSCode also supports remote debugging using ptvsd, a Python debugger developed by Microsoft for the original Visual Studio.

The final result looks great:

The setup

First of all we need to install ptvsd and integrate it into our app as described in the tutorial.

Usually this involves a pip install ptvsd and adding these lines somewhere in your application:

import ptvsd
ptvsd.enable_attach("my_secret", address=('0.0.0.0', 3000))

Note: If you’re on Docker you’ll also need to expose the port in your Dockerfile or docker-compose.yaml.

We then need to configure VSCode to attach to the debugger, we can do this by creating a new debugging configuration:

• Click on the “Debug” tab
• Click on the dropdown next to “DEBUG” and select “Add configuration…”
• Select “Python: Attach” as template
• Customize the configuration as needed. In our case the project root in the Docker container was /fieldguide.

{
    "name": "Fieldguide (Remote Debug)",
    "type": "python",
    "request": "attach",
    "localRoot": "${workspaceRoot}",
    "remoteRoot": "/fieldguide",
    "port": 3000,
    "secret": "my_secret",
    "host":"localhost"
}

The only thing left now is to start the application and start setting breakpoints everywhere!

If this looks too good to be true it’s because, well, it is.

Address in use

I immediately ran into an issue related with the port being used by “another” process:

Traceback (most recent call last):
  File "app.py", line 33, in <module>
    ptvsd.enable_attach("my_secret", address=('0.0.0.0', 3000))
  File "/usr/local/lib/python2.7/site-packages/ptvsd/attach_server.py", line 167, in enable_attach
    server.bind(address)
  File "/usr/local/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 98] Address already in use

As it turns out ptvsd.enable_attach() was being called twice. I wasn't entirely sure why yet and I really wanted to test the debugger on a "real" project so I decided to go with a quick hack: create a temporary file and start the debugger only if that file does not exist.

The application was finally running and the debugger was clearly accepting connections, as tested with netcat:

$ nc localhost 3000
PTVSDBG

Time to debug!

Not so fast…

Debug adapter process has terminated unexpectedly

I tried multiple times but kept getting this nasty error:

After some Googling I found out I had to install version 3.0.0 (the latest stable at the time of writing is 3.2.1). After getting it to work once it suddenly stopped working again and it was either showing the nasty error or just plain hanging.

Note: if VSCode hangs while connecting to the debugger (i.e. the thin blue loading bar keeps looping) you don’t need to restart the editor. Saving any file will terminate the connection attempt.

I dug in the ptvsd source code until I found this snippet:

# ptvsd/attach_server.py
[...]
server_thread = threading.Thread(target = server_thread_func)
server_thread.setDaemon(True)
server_thread.start()

For some reason that thread was hung. Then it dawned on me: the damn reloader!

Pico is based on Werkzeug, a library that abstracts “just enough” of the WSGI specification details (we use uWSGI as the implementation).

Werkzeug allows to start an app with the “autoreload” feature. Whenever there’s a change in the source, the server will pick up the changes and reload itself. When the autoreload feature is enabled Werkzeug always keeps 2 processes alive: the watcher and the actual application. Turns out ptvsd was running as a thread of the watcher, not of the application. I'm not entirely sure why this would be a problem (since the watcher runs the application code as well) but it clearly was.

The solution to all problems

The solution to both the problem above and the double call to ptvsd.enable_attach was to simply run the debugger only within the reloaded instance of Werkzeug, not the watcher. To do so I wrote a simple function that does the following:

• Get the parent process’ PID
• Get the PIDs of all Python processes involved with the app
• If the parent’s PID is within these PIDs it means it’s the Werkzeug watcher and the current process is the reloaded application process, therefore we can start the debugger (the old thread was terminated together with the old application process)

Note that ENABLE_DEBUGGER_PARAM is just some parameter you pass to your application to enable remote debugging, it might as well be the name of the file or what have you. It just makes it easier to pinpoint the processes involved with our app.

def start_debugger():
    parent_pid = os.getppid()
    cmd = "ps aux | grep %s | awk '{print $2}'" % ENABLE_DEBUGGER_PARAM
    ps = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
    pids = ps.communicate()[0].split('\n')
    if str(parent_pid) in pids:
        print('Starting debugger')
        ptvsd.enable_attach("my_secret", address=('0.0.0.0', 3000))

This finally made running and attaching to the debugger consistently work.

Shortcomings

The only shortcoming I could find so far is that, since the debugger server is restarted every time Werkzeug reloads the application, VSCode needs to be re-attached every time. This isn’t a big deal but it can get annoying in the long run.

Update from VSCode Python Tools’ team

Karthik Nadig, one of the engineers behind VSCode’s Python Tools, recently reached out to add a few remarks to this tutorial (this shows how much they care about their product ❤️, thank you Karthik!). Quoting his email:

Setting up the debugger

ptvsd.enable_attach(address=('0.0.0.0', 3000)) is the right way; secret [He’s referring to my_secret in the article] was for the old debugger. Since August 2018, VSCode has been using newer version of the debugger (ptvsd=4.*), that does not support secret.

Launch.json settings to attach to Django

{
    "name": "Attach",
    "type": "python",
    "request": "attach",
    "port": 3000,
    "host": "localhost",
    "django": true, // allows you to set breakpoints in Django template files.
    "pathMappings": [
        {
            "localRoot": "${workspaceFolder}",
            "remoteRoot": "." // automatically maps to the remote cwd where your app is running
        }
    ],
}

To attach to flask:

{
    "name": "Attach",
    "type": "python",
    "request": "attach",
    "port": 3000,
    "host": "localhost",
    "jinja": true, // allows you to set breakpoints in jinja2 tempate files.
    "pathMappings": [
        {
            "localRoot": "${workspaceFolder}",
            "remoteRoot": "." // automatically maps to the remote cwd where your app is running
        }
    ],
}

The highlighted text allows debugging inside template files. You don’t need those if you are not going to debug template files.

This issue: [Errno 98] Address already in use

This occurs if you launch Django without using ( — noreload) or Flask without using ( — no-reload). There are 2 ways to address this.

Option 1: launch Django or flask using corresponding —n o-reload flag. This means auto reload will be turned off.

Option 2: Enable subprocess debugging. This can be done by adding “subProcess”: true, to launch Json. This will support reload

NOTE: with subprocess there is one additional thing you may have to do if the OS is linux/mac. See here for more details (https://github.com/microsoft/ptvsd/blob/master/TROUBLESHOOTING.md#1-multiprocessing-on-linuxmac). This is a temporary thing until we fix the issue in the debugger.

Keep an eye on posts from Hipo by subscribing to our newsletter and following us on Twitter.

Remote debugging with VSCode, Docker and Pico was originally published in Hipo on Medium, where people are continuing the conversation by highlighting and responding to this story.

Docker has been around for a while now. Companies use it in production, people write plenty of articles, record video tutorials, yet most developers don’t “trust” it to handle their development environment. Let alone use it in production.

And that’s usually because understanding Docker is hard. This article tries to be a primer that will save you a few hours of screaming and paranoia without leaving out important details.

Why Docker?

Docker allows you to:

• Wipe differences across environments. That is, your production, staging and development environments will be (should be) identical(ish). The same applies to the development environments of different developers. In other words: no more “works on my machine”.
• Ensure that the exact same environment can be deployed wherever Docker is installed, no matter the hardware or operating system. Good old “build once, run everywhere”.
• Build a reproducible environment with a very small footprint (usually just your source code and a special file called Dockerfile) that you can share with others. No more copying 20+ GB virtual machines to onboard new developers.
• Declare your project’s dependencies as a handy YAML file that can be used to setup everything you need at once (e.g. your app server, database, queues, workers etc.).
• Create a scalable, reliable cluster that can run your entire project across a fleet of heterogeneous physical machines with a few lines of bash (Docker Swarm).

If you are not impressed yet keep reading.

Why not Docker?

“Vagrant is better than Docker”
– someone who has no idea what Docker is

If you have used virtual machines to setup your development environment you probably used Vagrant to handle them. Vagrant is a front-end to some famous virtualization back-ends such as VMWare or Virtualbox.

Comparing Vagrant with Docker is like comparing apples and beer cans. Irrelevant.

Docker is definitely not the ideal option if your intent is to use containers the same way you would use virtual machines. In that case, Docker will just make your life harder.

This is not to say that Docker cannot do that. It can, it’s just not its primary purpose (nor its secondary for that matter).

For example: when using a virtual machine the flow is quite familiar. You turn on the machine, install some packages, configure something at the OS level, maybe write a few files here and there and, once you are done, suspend or shut down the VM just to restart from the exact same point when you have to work again. Now: all your modifications are saved within the VM’s filesystem/memory. With Docker the user is supposed to stop and start different containers every time, therefore losing the changes local to an individual container. The reason to use different containers every time is to ensure that the provisioning process is easily reproducible and they can be deployed and work anywhere, at anytime, without making assumptions on what the underlying state is. Docker is built to deploy applications, not machines. killdoesn't remember the PID of the last process you sent a signal to, does it?

Another huge difference between using a VM and Docker for development is that with Docker you seldom install all services (e.g. database, web server, key-value store) in the same container but rather split those into separate containers that communicate using a virtual network created by Docker itself. This is exactly what happens in the real world: usually all the services needed to run a project (unless it’s a very small one) run on separate nodes.

The Docker Engine

Before diving into more details it’s important to understand that with “Docker” we could be talking about: the company, the project, the tool. Throughout this article we will refer to either the project or the tool, letting the context drive your intuition.

Docker uses a client/server architecture. The server uses the Docker engine to cache images, run and manage containers, handle logs and much more.

Images, containers…What are you talking about?

Right right. The perfect analogy (for programmers at least) to understand how Docker works is the following:

• A Dockerfile is the source code.
• An image is the binary resulting from compiling and building your Dockerfile.
• A container is the running instance of an image, a process by all means.
• Images are cached and containers are run by a Docker host (a machine running the Docker engine).
• Images are stored in a registry, think of it as a repository for package managers such as apt, yum, pip or npm. Docker offers a public registry that anyone can use to store images, as long as these are kept public (one private image is allowed free of charge).

To install Docker on your operating system simply head to https://docker.com and pick the right version.

Let’s get started!

Creating a Dockerfile

Dockerfiles use a simple syntax to express the steps that should be taken to build a specific image. A dead simple Dockerfile is the following:

FROM ubuntu
RUN echo "My first Docker image"

Breaking it down

• FROM ubuntu tells Docker to use the latest ubuntu image as a base. The image will be retrieved from the public registry.
• RUN echo "My first Docker image" tells Docker to run the command echo inside the container.

Building an image

You can now build an image from this Dockerfile with:

docker build .

Breaking it down:

• build tells Docker to build an image.
• . tells Docker to look in the current directory for the Dockerfile and to use the current directory as a "context" so that we can reference files and directories from there.

You will see an output similar to this:

Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu
latest: Pulling from library/ubuntu

b3e1c725a85f: Pull complete 
4daad8bdde31: Pull complete 
63fe8c0068a8: Pull complete 
4a70713c436f: Pull complete 
bd842a2105a8: Pull complete 
Digest: sha256:7a64bc9c8843b0a8c8b8a7e4715b7615e4e1b0d8ca3c7e7a76ec8250899c397a
Status: Downloaded newer image for ubuntu:latest
 ---> 104bec311bcd
Step 2 : RUN echo "My first Docker image"
 ---> Running in f85bd2e0f554
My first Docker image
 ---> 1d4302baa251
Removing intermediate container f85bd2e0f554
Successfully built 1d4302baa251

Breaking it down

• Docker will copy the directory’s content into a temporary directory and use that as context.
• Following STEP 1 Docker will retrieve the latest ubuntu image from the public registry and all of its intermediate images (hold that thought!).
• Following STEP 2 Docker will run the echo command inside the container and route standard output and standard error to our machine so that we can see the result on our terminal.

Now look carefully. Have you noticed something?

My first Docker image
 ---> 1d4302baa251 <--------- THIS
Removing intermediate container f85bd2e0f554
Successfully built 1d4302baa251 <--------- THIS

Every command, in the Dockerfile, that could potentially alter the state of the image (such as RUN, since Docker cannot know if our command has side effects) produces an "intermediate image". That is, every time such a step is encountered an image will be created that holds the state produced by all the previous commands.

For example (assuming the file test.txt exists):

FROM ubuntu
WORKDIR /tmp
COPY test.txt .
RUN cat test.txt

Would produce 4 intermediate images.

Breaking it down

• FROM copies the existing ubuntu image into an intermediate image.
• WORKDIR changes the working directory to the given one.
• COPY copies the given file into the working directory.
• RUN runs an arbitrary command.

Therefore the output of a build will be:

Sending build context to Docker daemon 3.072 kB
Step 1 : FROM ubuntu
 ---> 104bec311bcd
Step 2 : WORKDIR /tmp
 ---> Using cache
 ---> 8b7569f87645
Step 3 : COPY test.txt .
 ---> c515890976fb
Removing intermediate container 7d07b7f6f0fb
Step 4 : RUN cat test.txt
 ---> Running in 9ec4a66f5a05
I'm the content of test.txt
 ---> 27922b2708f1
Removing intermediate container 9ec4a66f5a05
Successfully built 27922b2708f1

Where every line starting with an arrow and ending with a hash (that nice little hex string) represents an intermediate image (e.g. ---> c515890976fb).

Running a container

You will notice something else while we are building our image: Removing intermediate container 9ec4a66f5a05. That's right, to execute RUN commands Docker needs to actually instantiate a container with the intermediate image up to that line of the Dockerfile and run the actual command. It will then "commit" the state of the container as a new intermediate image and continue the building process.

As you can see we’ve already run a container. Now we’ll learn how to do that arbitrarily (and no, you don’t need to build an image every time you need to run a container) and how to create an image from a container’s state instead of a Dockerfile.

Take a note of the hash of the image we have just built (27922b2708f1 in my case) and run a container based on that image with:

docker run 27922b2708f1

You’ll notice that the output is blank. The container is, in fact, doing absolutely nothing. Why is that? Why isn’t our RUNcommand being...Well...Run?

The correct Dockerfile instruction to run a command when a container runs is CMD, not RUN which is, instead, executed only at build time. Let's change the Dockerfile as follows:

FROM ubuntu
WORKDIR /tmp
COPY test.txt .
CMD cat test.txt

And build the image once more. You’ll now notice that the output does not contain the content of test.txt anymore. That's because CMD does not run the command, but simply sets the container up to run that command every time it starts. The hash of the final image will also change, since the last step has changed.

To make things easier we can tag our image so that we don't have to remember an ugly hash but assign a nice mnemonic we choose:

docker build -t test .

This will create an image named test with tag latest. The actual "tag" is the pair <NAME>:<VERSION> which, in this case, will be test:latest. A quick look at the cached images will validate this:

docker images

Output:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
test                latest              f2aba921b459        2 minutes ago       129 MB

If we now run a container from the aforementioned image with:

docker run test

The output will be:

I'm the content of test.txt

(or whatever you put in your test.txt)

Exactly what we were expecting.

How do I interact with a container though?

Fortunately Docker allows us to do that easily:

docker run -it test bash

Breaking it down

• run tells Docker to run a container off the given image.
• -it tells Docker to run the container in interactive mode (as opposed to -d, daemon mode) and to give us a virtual terminal to interact. -i also takes care of wiring the standard input to the container (without which a terminal would be useless).
• test is the image we've just built.
• bash is the command to run once the container starts.

The command we pass is what keeps the container alive. Within the container it will be the process with PID 1 which will act as parent of all subsequent processes. This means that if such process is terminated the container will stop.

You will also notice that the content of test.txt is no longer being printed. That happens because if a different command is passed, while running a container, the one from the CMD instruction will be ignored. We are basically overriding the container's default command. To have a default command run no matter what you will have to use the ENTRYPOINT instruction in your Docker file.

Once inside the container our prompt will look like the following:

root@f1e1064e0958:/tmp#

Two things are important:

• The working directory is /tmp as set with the WORKDIR instruction.
• The default (and only) user is root. That's because containers are supposed to be treated as stateless processes, not fully-functional virtual machines, therefore there is no need to have users with different privileges. If a container is compromised it's enough to shut it down and spawn another one. Some images might use different users but that is definitely optional.

We will immediately see that the ubuntu image is actually a stripped-down version of the Ubuntu server distribution. Many binaries will be missing:

root@f1e1064e0958:/tmp# ping
bash: ping: command not found

We can use the preinstalled package manager to install ping:

apt update
apt install iputils-ping

To have a feeling of Docker’s statelessness let’s exit the container (exit or Ctrl/Cmd+D) and start a new one:

root@f1e1064e0958:/tmp# exit
➜  docker-article docker run -it test bash
root@ef95ac8b41ff:/tmp#

You will see the hostname has changed from f1e1064e0958 (the ID of the old container) to ef95ac8b41ff (Your IDs will be different). And, to our dismay, ping is also gone.

Creating an image from a container

What if we want to “save” our changes to the container in a new image? Let’s install the iputils-ping package again and exit the container once more.

The docker ps command will show currently running containers. Since all of our containers are stopped the list will be empty. To see all cached containers (including those who were stopped/terminated) simply use:

docker ps -a

My output looks like this:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                         PORTS               NAMES
ef95ac8b41ff        test                "bash"                   4 minutes ago       Exited (127) 1 seconds ago                         hopeful_keller

Docker kindly assigns a random mnemonic to each new container (hopeful_keller in my case) so that we don't have to remember the ID. We now want to use the state of this cached (stopped) container as a base for a new image we'll call test2:

docker commit hopeful_keller test2

The output will be the SHA256 hash of the newly created image:

sha256:8b976507170c40526d1e0361631e2e959f150c8cee51fe1b9090f2a56b7e9b35

Our new image is now in the Docker engine’s cache and we can see it by listing all images:

docker images

REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE
test2               latest              8b976507170c        About a minute ago   129 MB
test                latest              f2aba921b459        About an hour ago    129 MB

That’s neat! We have just created a completely new image that we can now reuse to run a container with pingpreinstalled. All in a matter of seconds. Let's run a container off test2 and try pinging something to verify:

docker run -it test2 bash
root@25d285662a5a:/tmp# ping hipolabs.com
PING hipolabs.com (54.83.6.199) 56(84) bytes of data.
64 bytes from ec2-54-83-6-199.compute-1.amazonaws.com (54.83.6.199): icmp_seq=1 ttl=37 time=0.223 ms

As you can see Docker can “kind of” simulate the experience you would have with a virtual machine as long as you remember to commit your changes to an image every time. Alternatively (not suggested) you can keep starting the same container over and over again. From docker ps -a you can get the ID/name of the stopped container and use:

docker start 25d285662a5a
docker exec -it 25d285662a5a bash

To restart the same container with the same state. Docker also provides handy pause and unpause commands to achieve the same result you would have suspending a virtual machine. This goes a bit beyond the scope of this tutorial (and we still have a lot to cover).

The clear disadvantages of creating images this way instead of using a proper Dockerfile are:

• The image is multiple orders of magnitude bigger than a simple text file. Harder to share.
• There is no way to reliably document how you built your image. Equivalent to not sharing the source code.

Summary

In this article we saw:

• How to create a Dockerfile.
• How to build an image from a Dockerfile.
• How to start and interact with a container based on that image.
• How to create a new image from a container’s state.

In the next chapter we will explore two awesome tools: Docker compose and Docker swarm. For questions and comments do not hesitate to get in touch!

Thanks to Fergal Walsh and Semih Basmacı for reading drafts of this article.

Keep an eye on posts from Hipo by subscribing to our newsletter and following us on Twitter.

Understanding Docker without losing your shit was originally published in Hipo on Medium, where people are continuing the conversation by highlighting and responding to this story.

Differences between academia and industry

If you are an engineering student you are probably used to programming courses that focus on solving relatively small problems, often (but not always) in isolation. Perform a binary search, traverse a graph, implement stacks and queues, maybe handle a bit of concurrency, maybe a bit of functional programming. All very interesting and challenging, don’t get me wrong, but not nearly enough for what’s out there.

You see, in academia we rely on a lot of assumptions. A usual group project is based on the following assumptions:

• There will only be one user at a time.
• The code will only be seen by the group members and trashed (be forgotten) after the project is done.
• The project will run only when the instructor has to evaluate it or for testing, probably on the machine of the team member who has a MacBook (my department was GNU/Linux-only).
• Continuous integration means “continuously integrate new code, compile and test until things work”.
• Testing means “make sure it doesn’t throw a segmentation fault when you do that thing that caused a segmentation fault last time”.
• Versioning (the topic of this article) means “remember to start Dropbox when your PC boots or we are screwed”.

Unfortunately this is very true for research as well. I know many graduate students and most of them don’t know what versioning is, how to write tests or even how to deploy an application so that actual people can finally use it. And whenever somebody says stuff like “but I just need to get my results and get my paper published, why would I care about all these things?” I get mad, oh I get very mad.

There’s a contradiction here: the industry is usually known for secretive development, competition and profit-oriented decisions, while academia is usually known for open development, collaboration and decisions based on improving the state of the art, with no regard for the cost. And yet, how many research projects are actually published on a public repository? How many have demos deployed somewhere that can be accessed by interested parties or other researchers who want to improve on that?

Learning how to share your code with the world also teaches you how to write good, maintainable code. It’s a win-win. Learning how to deploy your app, no matter how simple, teaches you how to advertise your accomplishments. If that’s not a good skill to learn I don’t know what is.

But I digress.

Versioning

If you work on a project you’ll likely be part of a team. If you are not, the project is just not big enough yet. Or maybe you are thinking of open sourcing your project. Regardless, learning how to use versioning tools is fundamental if you want to call yourself a “software developer”. Most versioning tools also allow you to trigger events (e.g. using “webhooks”). Imagine updating the code and having your tests (because you do write tests don’t you?!?), deployment scripts, sanity checks and whatnot run automatically. Kinda like This Too Shall Pass by OK Go.

But what is versioning? The name says it all: it’s a way of keeping versions of files stored in a repository. In our case these files happen to be text representing the source code of a program, nonetheless a VCS (Version Control System) can be used for all sorts of files. That is, imagine that a set of files and directory represents a state in time. Without a VCS you end up overwriting those files and, therefore, rewriting the state. With a VCS, you keep the states as a stack. At any point in time you can retrieve an older state or add a new one on top of the stack.

But what if multiple developers work on the same project? Sharing the same stack might yield unexpected results. Imagine Alice and Bob start working on the same state, the one on top of the stack. Now imagine they both change the same set of files but in different ways, for example they change line 48 of utils.py but Alice writes num = 10 while Bob writes num = 5. Now Alice and Bob want to add the new state on top of the stack. There are 3 courses of action:

1. The latest state overwrites the previous one.
2. We merge changes such that both lines are kept but the order is picked according to who submitted the new state first.
3. We warn the developer who submits last that his/her changes cause a conflict with the current state.

The second option is downright silly. Imagine having tens of lines where the same variable is assigned different values. Chaos!

The first option might be viable but, you know, it’s not respectful towards other developers to simply overwrite their changes and, most of the time, it’s the cause of many errors. What happens if Charlie starts working on the state submitted by Bob? Wrong assumptions!

The third option is the one normally picked in modern VCS. This way the developers can decide whether it is Alice assigning the wrong value to num or Bob. Once the conflict is resolved things continue as usual.

But wouldn’t it be nice if every developer could work on a separate stack and, only once a feature is complete or a bug is fixed, “merge” that stack with the main one? This technique is called “branching”. Because of its popularity and relatively ease of use we’ll take a look at the VCS git.

The tree of life

When I had to learn git I started reading lots of articles and asked for help to friends of mine who were more experienced in versioning. They all had the worst approach: they started teaching me how to use the git CLI. There are two groups of developers: those who do git add ., git commit, git push and start praying that their push doesn't get rejected and those who actually know what they are doing. Until a couple of years ago I was in the first group. Now that I'm part of the second, I invite you to join me. We have cookies!

The first, foremost and fundamental lesson about git is the following: a git repository is a tree.

…

Absorb that, internalize it, make it your mantra for the rest of your life or career: a git repository is a tree. This sentence is incredibly simple and, at the same time, incredibly powerful. Following Elon Musk’s beloved first principles every time you have a problem with git you’ll resort to this core concept, find what operations you want to perform on the tree and, only then, find the commands you need.

Now it’s time to define what a repository is and see how to create one. As per the description on their website “Git is a free and open source distributed version control system”. The keyword here is distributed. In practice, there usually exists a central repository (usually hosted on services like GitLab or GitHub) where all changes are “pushed” and from where developers “pull” to keep their own repositories up to date. As you can guess each developer has a subset or a one-to-one copy of the main repository and it’s their responsibility to keep them in sync to avoid inconsistencies.

We will now create a local repository, perform some basic operations, create a remote repository and push our changes there.

Local repository

Depending on your system you might have to install git. Refer to this guide on how to do that. I can wait.

Protip: to master git you can read “Pro Git”, it’s the official book, available for free.

Now that you have git installed create a new directory (I’ll call it git_test) and cd into it. Now let's create a new local repository:

git init

Git will create a hidden directory .git containing all the metadata about the repository. A very important file is .git/config which, unsurprisingly, contains the configuration variables of your repository. We'll visit Mr. config again, let's move on.

Let’s add a file to our repository:

echo "line one" > test.txt

Adding a file does not trigger anything in the git domain. It’s very important to understand one thing: git doesn’t give a damn about your files. It only cares about your commits. As we’ll see now a commit represents a state on the stack we were talking about earlier.

Let’s try to create a commit:

git commit

You will likely get a message along the lines of nothing added to commit but untracked files present. That's because you created the file but you haven't told git to add that file to the new state. In fact try to run:

git status

Protip: if the output of git is not colored (and, trust me, you’ll want colors while using git) you can run git config --global color.ui auto. If the output is still not colored you might want to follow this guide.

You’ll see (similarly to the previous output) Untracked files and a list of the files that git is not tracking. Git will also likely suggest to run git add <file> so let's do that:

git add test.txt

If you run git status again you'll see new file: test.txt (in green if you followed the protip). This means git is now keeping track of that file. But what if we have multiple directories and files and we want to add all of them at once? Git allows us to add directories recursively. Let's try it:

makedir mydir

echo "hello" > mydir/newfile.txt

git add .

Remember, . in *nix represents the current directory so we are adding everything that is in it. If we run git statusagain we'll see that we now have two entries: mydir/newfile.txt and test.txt. We are happy with the state, let's commit!

git commit

This will open a text editor (usually vim or nano if you are on Mac or GNU/Linux). The content will be pre-filled by git with some metadata and an explanation of what to write for your commit message. Commit messages are extremely important, they help keep track of what happened when and, more importantly, why. Writing good commit messages is an art. For now let’s focus on two important factors: you should always describe what you did at a high level and never write a commit message for the sake of it.

Examples of bad commit messages:

• “Wrote some code” (…what?)
• “Fixed something” (fixed what?)
• “Added a new feature” (what feature?)
• “Changed mymodule.py and added a new directory” (not high level)

Examples of good commits:

• “Scaffolding for project XYZ”
• “Wrote module for communicating with API ABC”
• “Refactored module for API ABC”

If you exit the editor without writing a message the commit will be aborted. If you do write a message and exit, the commit will be created in the local repository. Once a commit is created it represents the latest state of that repository. If you go through and run git status you'll see something like:

On branch master
nothing to commit, working directory clean

Branching

To be precise, the last commit represents the latest state of the “branch” you are on. Remember we talked about stacks and branching? The branch is just a stack. You can create multiple branches as copy of existing branches (hence the mantra “a git repository is a tree”) with the main branch, the “root”, called master. We will now create a new branch called new_branch starting from master:

git branch new_branch

This command will create a new branch but we will still be on the master branch. The way git keeps track of what branch we are on is by using a cursor called HEAD. This is the current situation (where A is the first commit you created):

new_branch  -- A

master      -- A <- (HEAD)

To switch to new_branch we run:

git checkout new_branch

The situation is now:

new_branch  -- A <- (HEAD)

master      -- A

As usual we can run git status to check what's going on. In this case you'll see On branch new_branch. If you just want to know what branch you are on git branch is enough. Now let's add a commit to our new branch.

echo "line two" >> test.txt

git add .

git commit

Now let’s check that our commit is on new_branch but not on master:

git log

Should show a total of two commits. If we checkout master and do the same:

git checkout master

git log (shows all the commit history as a list)

We should see only one log. What’s going on here? By committing to new_branch we created the following situation (where B is the new commit):

new_branch  -- A -- B

master      -- A <- (HEAD)

Since we are back on master HEAD is pointing to A and that is why we only see one commit in the history. With git log we explore the history of the current branch. Whenever you run git checkout ${BRANCH} you are simply moving HEAD to the last commit of the specified branch. Guess what? The names of the branches are simply shortcuts for the last commits of those branches! Gitception!

Refer to a commit

This is all very cool but, if you think about it, we still haven’t used git for actual versioning. What if we decide the current state has some problems and we want to roll back to a previous state? As you may have noticed every time we checkout a branch (assuming there aren’t untracked files) the content of our working directory changes to reflect the latest state of that branch (i.e. the commit HEAD is pointing to). But, as we saw earlier, checking out a branch simply means moving HEAD to the last commit of that branch. And if the latest commit of a branch is identified by the branch's name how are the other commits identified? Run git log and take a look at the list. You will see something like commit: d1137cb6824c1893eb6e3c77549d31082c6a3cea. That alphanumerical string is a hexidecimal ID that uniquely identifies that commit across all branches. Therefore the name of the branch is just an alias for the ID of the latest commit, no more no less. If a commit is present in multiple branches (e.g. the initial commit A will be in both master and new_branch) it means we can move HEAD to that commit, without any issue, as long as we are on one of those branches.

Let’s try to rollback to A (identified by another hex string):

git checkout new_branch

git log (copy the ID of the first commit)

git checkout ${ID_OF_FIRST_COMMIT}

You will see something like You are in 'detached HEAD' state. This means you are in a situation like this:

new_branch  -- A -- B
               ^------(HEAD)
master      -- A

HEAD is now pointing to a commit that is not the latest in your branch. In this state you can change and commit as much as you want without touching the original branch (new_branch in our case). But this is not what we want. We want to replace the last commit of the branch with the previous one! There are two ways of doing this:

• git reset --hard ${ID_OF_FIRST_COMMIT}
• git revert ${ID_OF_LAST_COMMIT}

The first method simply erases the last commit from the history of our branch. This is dangerous. Whenever you change the history of a branch (not just the last state) you must always act carefully. The --hard option means that, no matter what's in our working directory, it will be replaced with the state of the commit we are resetting to.

Protip: you can also omit --hard (or explicitly specify --soft) and reset to a commit without touching the files in the working directory. We won't cover this case here but there are numerous use cases for it.

The second method keeps the history intact but adds a new commit that simply reverts the changed applied by the specified commit. With the last commit we added the line “line two” to our test.txt file. Let's try reverting that change:

git revert ${ID_OF_SECOND_COMMIT} (be careful! Do not revert the first commit!)

Again you’ll be given the chance to change the commit messages. I highly suggest you leave the default text and simply add more information on, perhaps, why you reverted that commit.

Now if we run git log we'll see something like this:

commit 9f1fa5084e83efcbe1fe946fde28a31ad60f9b20
[...]
    Revert "Commit 2"

    This reverts commit b5712bd073bd0035326ff92bb23a3206ebe6e5b3.

commit b5712bd073bd0035326ff92bb23a3206ebe6e5b3
[...]
    Commit 2

commit d1137cb6824c1893eb6e3c77549d31082c6a3cea
[...]
    Commit 1

And if you look into test.txt you'll see "line two" is not there anymore. Great! But now new_branch highly differs from master. We want to "merge" the changes into master to keep our branch synchronized.

Merging and rebasing

There are two techniques in git to merge changes into a single branch:

Merging

Merging, as the word says, simply “merges” the commits of the two branches into the new one. To understand what happens let’s imagine we have the following situation:

new_branch  -- A -- B -- C
               
master      -- A -- D -- E <- (HEAD)

If we merge new_branch into master:

git merge new_branch

The situation changes into:

new_branch  -- A -- B -- C
               
master      -- A -- B -- C -- D -- E <- (HEAD)

Every time we merge git will automatically create a commit message along the lines of Merged branch X into branch Y.

Because the correct time order of commits is maintained it’s like shuffling a deck of cards. As you can see we put B and C before D in master since that's their order in time. But most of the time we don't actually care when a commit was created but, rather, when it was merged into a branch. We can either remember to merge master back into new_branchevery time we start to work on something new, to keep the timeline in the correct order (and fill our history of merge commits) or we can use rebasing.

Rebasing

Rebasing allows us to always append commits instead of interleaving them. This is especially useful if we want to keep a linear history without changing the order of commits branch-wise. Let’s start with this situation:

new_branch  -- A -- B -- D
               
master      -- A -- C <- (HEAD)

When merging into master we would, rightfully, expect B to go before C and D after. But before merging we can rebase new_branch on top of master:

git checkout new_branch

git rebase master

Now the situation has changed:

new_branch  -- A -- C -- B -- D
               
master      -- A -- C <- (HEAD)

That’s perfect! If we now merge new_branch into master:

git checkout master

git merge new_branch

The situation will be:

new_branch  -- A -- C -- B -- D
               
master      -- A -- C -- B -- D <- (HEAD)

It makes sense. B was created before C but on another branch! And it was merged into master just now so, branch-wise, B was "created" in master at the time of merging, not before! If all of this is a bit confusing I suggest you draw the states on a piece of paper yourself and follow along. Once it clicks you can come back. I’ll wait.

Conflicts

You might believe it takes at least two people to generate a conflict. You’d be wrong. Let’s create two new branches from master:

git checkout master

git branch conflict1

git branch conflict2

Let’s checkout conflict1 and add a new line to our test.txt file:

git checkout conflict1

echo "conflict1" >> test.txt

And commit. Now do the same for conflict2 but let's change the line:

git checkout conflict2

echo "conflict2" >> test.txt

And commit again. Here’s the current situation:

conflict1   -- A -- B
               
master      -- A

conflict2   -- A -- C

If we merge conflict1 into master (no rebasing necessary since there's only one commit of difference) we get the following:

conflict1   -- A -- B
               
master      -- A -- B

conflict2   -- A -- C

If we merge conflict2 into master we get...A conflict!

conflict1   -- A -- B
               
master      -- A -- B -x- C   <CONFLICT!!!>

conflict2   -- A -- C

What happened is that the commit C has been created on master but not yet attached to B. Git will warn about the conflict with a message like CONFLICT (content): Merge conflict in test.txt. To resolve the conflict we have two options:

• Manually edit the files until all conflicts are resolved.
• Use git mergetool.

The first option is ok for trivial conflicts but you should quickly learn how to use the second one. If we run git mergetoolgit will start iterating over all the files that contain conflicting lines and offer us the chance to apply the due modifications. At some point you'll see something like Hit return to start merge resolution tool (${MERGE_TOOL_NAME}): (on Mac the default tool is opendiff). Because I don't want this article to be OS-specific I will just let you figure out how your merging tool works. It's usually fairly intuitive (unless you get a vim window and you can't use vim, in that case go look for another merging tool).

For each conflict, no matter the tool, we’ll be given the following options:

• Keep the changes in the original branch.
• Overwrite the changes with the new branch.
• Keep both changes (but give more priority to the original branch).
• Keep both changes (but give more priority to the new branch).
• Delete the conflicting lines (almost never what you want).

Once the conflicts are resolved we’ll have the following situation:

conflict1   -- A -- B
               
master      -- A -- B -- C -- M

conflict2   -- A -- C

Where M is the merge commit created by git which will also include which files were conflicting.

Fixing authorship

Before proceeding it is good practice to set the authorship of the commits. By default the commits’ author will be your OS’ user. But if you want to contribute to a project you’ll likely want to set the username and email to values that reflect your actual identity so forget committing as lollipop99.

git config --global user.name "${YOUR_NAME}"

git config --global user.email "${YOUR_EMAIL}"

These two commands will suffice to set your identity straight across all repositories. If you wish to keep separate identities for different repositories simply omit the --global option.

Old commits’ authorship will not change! Figuring out how to do that is left as a google sear…aehm…an exercise for the reader.

Pulling and pushing

If you followed along so far and understood how the mechanics of git work, pushing and pulling are nothing new. Remember: we only created a local repository. Now we want to make this repository available to the public (or our team members). To do so we need to create a remote repository that can be accessed by anywhere in the world. The leading service for creating public git repositories is, hands down, GitHub. Go and create an account. I can wait.

Now create a new repository and fill in the required information (keep it simple, it’s just for testing). I can wait.

Great! Now that your remote repository is ready it’s time to fill it with what we have locally. The first thing we need to do is set a remote repository for the local one. GitHub will likely give you two options once the repository is ready: create a new local repository or push an existing repository. To proceed with the second choice we’ll be given two commands:

git remote add origin ${URL_TO_GITHUB_REPOSITORY}

git push origin master (assuming you are working on master)

A repository can have multiple remote repositories but origin is reserved as the default. The URL to a GitHub repository can follow two schemas: git@github.com:${USER}/${REPOSITORY_NAME} and https://github.com/${USER}/${REPOSITORY_NAME}. The most important difference is that the first one allows to push using SSH and, therefore, authenticate using your key pairs while the second one requires you to type in your password every time. The choice is up to you, if you don't know what "authenticate using your key pairs" means it won't make a big difference so just proceed with the default, whatever it is.

And now the grand finale: push means to submit all of the commits on the specified branch to the remote repository. pullmeans to retrieve all of the commits from the specified branch in the remote repository. That’s it. If you treat a remote branch as “yet another branch” push and pull are just merges! You are either merging from the local branch to the remote one or viceversa. Let’s visualize this:

master        A -- B

origin/master (empty)

When we do push origin master it becomes:

master        A -- B

origin/master A -- B

Now imagine somebody else pushes two more commits to origin/master (C and D):

master        A -- B

origin/master A -- B -- C -- D

To sync your local repository you’ll have to:

git pull origin master

And the result will be:

master        A -- B -- C -- D

origin/master A -- B -- C -- D

All the rules we saw still apply (yes, including conflicts!).

Always remember: git works in terms of commits, not files! Now get out there and start convincing your friends to use git!

Frequently Answered Complaints (F.A.C.)

I hate copy/pasting the commit’s ID every time. Is there a shortcut?

Yes. Remember HEAD is pointing to the latest commit of the current branch? Well, HEAD~1 will point to the previous commit, HEAD~2 to the one above and so on. Using ~ will always select the first parent in case of multiple commits at the same level. You can use ^ to explicitly select a parent. More info are in this awesome Stack Overflow thread.

I want to reset to a previous commit without touching the files.

git reset --soft ${COMMIT_ID} . --soft is the default so it can be omitted.

I made a mess, haven’t committed my changes yet and want to rollback to the last working state.

git reset --hard . The --hard flag makes sure to restore the tracked files to their last commit's state. Untracked files will not be touched. So if you run git status you will still see them untracked and what you do with them only depends on your purpose.

I want to hard reset to a specific commit.

git reset --hard ${COMMIT_ID}. The default for COMMIT_ID is HEAD, the current commit (if you are not in detached mode), hence the tip above.

I want to clean all the untracked files.

git clean -n (-n is for a dry-run) will tell you which untracked files would be removed. git clean -f will actually remove them.

I want to reset a single file to its previous state.

git checkout -- ${PATH_TO_FILE}. It's like git reset --hard but on a single file. DO NOT try git reset --hard ${PATH_TO_FILE}. In the best case you'll get an error along the lines of fatal: Cannot do hard reset with paths., in the worst you'll throw away any uncommitted change.

I want to retrieve a specific file from a specific branch.

Similarly to the above: git checkout ${BRANCH} -- ${PATH_TO_FILE}.

Conclusion

I hope you now appreciate the importance of using a VCS, regardless of the size of your project or the number of people in the team. Dropping that DropBox premium subscription and moving all of your projects to a hosted VCS is your first step towards a happier life (and a sustainable career as a software developer).

(Special thanks goes to XKCD for being an awesome webcomic.)

Keep an eye on posts from Hipo by subscribing to our newsletter and following us on Twitter.

How to work in a team: Version control and git was originally published in Hipo on Medium, where people are continuing the conversation by highlighting and responding to this story.

Except…It will keep your CPU at 100%, fans will likely start spinning and your battery life will suffer. Do we really need advanced machine learning to achieve something as simple as reducing background noise? There’s a better way (if you’re willing to use an external microphone).

Keep the input volume fixed to a minimum so that your voice can be heard but the noise cannot.

Depending on where you are you might want to set it to 5, 10, 20 etc. Personally I found a value of 10 to be good for most of my calls. macOS offers the osascripttool to set system properties such as the input volume. A simple script to periodically reset it to the desired value looks like this:

while [ 1 ]; do osascript -e “set volume input volume {YOUR_VALUE}”; sleep 10; done

This will prevent macOS from automatically resetting the input volume level depending on the loudness of your voice or the noise around you. You can test everything by using an app such as Linein.

The same can be done on Linux (using PulseAudio):

while sleep 1; do pacmd set-source-volume alsa_input.???-?????.analog-stereo {YOUR_VALUE}; done

(No idea what’s available on Windows)

Now we can go back to employing ML for more serious tasks, such as recognizing whether something is a hotdog.

If you ever found yourself having to integrate push notifications into your project you’ll likely share my views on the subject:

Push notifications are a pain in the butt.🤦🏻‍♂️

Testing push notifications can still be pretty damn difficult. I don’t own an iOS device and using the Xcode simulator is not an option. Android makes things a bit easier but you still need to download the entire Android Developer Studio (and SDK) just to test a couple of push notifications.

Then Firebase Cloud Messaging came and changed the way push notifications can be tested. The fact that a single platform integrates with Web, Android and iOS and it’s supported by AWS SNS means we can easily test push notifications on web and be 100% sure they will work the same way on all other platforms. How awesome is that? 🤯

Firebase setup

First of all let’s create a Firebase project (create a Firebase account if you don’t have one).

Once the project is created retrieve the Server key from the settings (replace {PROJECT_NAME} with your project’s name) by following this URL: https://console.firebase.google.com/u/0/project/{PROJECT_NAME}/settings/cloudmessaging/

In “Cloud Messaging -> Web Push certificates” create a key pair and store the public key somewhere.

Gitpod setup

Gitpod allows you to easily create a fully featured development environment in the cloud. Their browser extension will enable a nice little green button that will immediately create an environment and clone the repository where you clicked it. It currently supports GitHub, GitLab and BitBucket.

Go to https://github.com/firebase/quickstart-js/tree/master/messaging and smash that little green button. Creating the environment will take a while.

Once the environment is ready you’ll be given a fully functional VSCode instance, together with a terminal, running in a Docker container. The next steps assume you’re using Gitpod but they should work on a local environment too.

Development environment setup

Install the Firebase CLI on your system:

1. npm install -g firebase-tools
2. firebase login
3. firebase use — add (select the project created above)

Open index.html .

Pass the public key created above to usePublicVapidKey (replace <YOUR_PUBLIC_VAPID_KEY_HERE>).

From the messaging folder run firebase serve --port 8081 to serve the project.

Gitpod will ask you to expose the chosen port (which, by the way, is completely arbitrary) and provide you with an option to open a preview or a new tab showing the running project. Regardless of the option you choose you’ll be presented with a big “Request permissions” button. Click it to allow your browser to receive push notifications.

Note: the button won’t work if the app isn’t served via HTTPS, which Gitpod provides by default.

Once permissions are granted you’ll be shown an Instance ID (IID) which represents your device (your browser, in this case). You can use it to send a test notification via cURL:

curl --request POST \
  --url http://fcm.googleapis.com/fcm/send \
  --header 'authorization: key=AAAA4TVzF60:APA91bHxmpq7Va8JtpGple9nkfki7ez91E7gjPENNYwevEOQVH-xsxSrYVIG4B2xZC-AvBRZW7_w5wnrMLonHJScxRVvVnRWswIFs3ROK9m1BwmWRSiXjtHnWchSLJmNjIyR3m1QO64N' \
  --header 'content-type: application/json' \
  --data '{
  "notification": {
    "title": "Push notification test",
    "body": "It works! 🎉",
    "icon": "firebase-logo.png",
    "click_action": "http://localhost:8081"
  },
  "to": "<IID>"
}'

The response will show whether the request was successful (i.e. "success": 1 ) or not. If the tab with the running app is not in focus (even if your browser is in the background) the notification will show up as a native app of your operating system.

The future: automated testing

Thanks to Firebase API and tools such as Node and Google’s Puppeteer (which will hopefully support the push API soon) it will finally be possible to automate end-to-end testing when it comes to push notifications without using a device or setting up the SDK of the related platform (assuming you trust Firebase with the delivery).

This will greatly increase confidence when implementing push notifications without having to resort to expensive services or (physical) device farms.

If you have more tips and tricks on how to implement and test multi-platform push notifications, without exiting the comfort of your own laptop, please do reach out! 🤝

UPDATE: a previous version of this article recommended setting up the environment locally and using Localtunnel to serve the project via HTTPS. Turns out Localtunnel is very unstable and using Gitpod makes everything a lot easier.

Testing push notifications locally with Firebase was originally published in Hipo on Medium, where people are continuing the conversation by highlighting and responding to this story.

A while ago I found myself wanting to test a flow, in a Django (+ Rest Framework) based API, that included sending an email to a client containing a so called magic link, similar to the ones used by Slack.

We normally rely on services such as AWS SES and fire an async task using Celery. Obviously relying on a 3rd party service and, more importantly, on an async task for testing is very, very bad.

Let’s talk about settings

So the first step consists of overriding the default settings for your Django application to do 2 things: 1) tell Celery to execute tasks in a synchronous manner (i.e. they will act exactly as a normal function call, even when using delay ), 2) use a local backend to send/receive emails that allows easy and fast access.

The answer to the former problem is the two settings below (here’s a mapping between Django settings and Celery settings):

CELERY_TASK_ALWAYS_EAGER = True
CELERY_TASK_EAGER_PROPAGATES = True

The first tells Celery to treat any task invocation as a “normal” function call. In reality the execution is an emulation of the Celery API, i.e. the task will not be put in the queue but the returned value will be a EagerResult object, which emulates the AsyncResult object normally returned by a task invocation.

The docs clearly warn against using task_always_eager, if you don’t know what you’re doing.

The second tells Celery to propagate exceptions to the caller. Obviously this will only work for eagerly executed tasks.

The answer to the latter problem is to change email backend to the following:

EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

This will store any sent/received email in django.core.mail.outbox or inbox and print the text on the standard output.

Fishing for links

Now that we have an easy way to access emails sent by the system we need a way to parse the link and send the relevant token(s) to the API. This assumes that links are constructed so that native clients (e.g. iOS/Android) can intercept the link without having to open a web browser (known as Universal Link).

As usual regexes are our friends. In particular we’re going to use a feature called named groups. Imagine we send the following email:

Hi there! Here's your magic link:

https://example.com/magic/MzIzNjQ2MTE2_MTY5NzQ0NDg1NQ/

Have a nice day!

We can access the body of the email, find and parse the link in just a few lines (the HOSTNAME_REGEX is just too long to include here):

from django.core import mail
...

email_body = mail.outbox[0].body

invitation_link_re = r'(?P<host>{})/(?P<path>magic)/(?P<token>\d+_[\w\-]+)/'.format(HOSTNAME_REGEX)

link_match = re.search(invitation_link_re, email_body)

host = link_match.group('host')    # example.com
path = link_match.group('path')    # magic
token = link_match.group('token')  # MzIzNjQ2MTE2_MTY5NzQ0NDg1NQ

Now we can use the parsed components of the URL as we please. The power of regex!

Obvious solution: change:

id = models.BigIntegerField(primary_key=True, default=utils.generate_unique_id, editable=False)

To:

id = models.CharField(primary_key=True, default=utils.generate_unique_id, editable=False, max_length=100)

Build and run migrations.

django.db.utils.ProgrammingError: foreign key constraint "tasks_task_project_id_a2815f0c_fk_projects_project_id" cannot be implemented

DETAIL:  Key columns "project_id" and "id" are of incompatible types: bigint and character varying.

Damn it…

Another obvious solution: destroy and rebuild current schema (a simple PostgreSQL backup/restore will not work).

Yeah…No. So how about we use…

Fixtures!

./manage.py dumpdata --natural-foreign --natural-primary -e contenttypes --indent=4 > dump.json

--natural-foreign and --natural-primary tell Django to skip the primary/foreign keys and simply regenerate them when storing the data. This will avoid problems such as:

django.db.utils.IntegrityError: Problem installing fixture '/app/dump.json': Could not load contenttypes.ContentType(pk=23): duplicate key value violates unique constraint "django_content_type_app_label_model_76bd3d3b_uniq"

DETAIL:  Key (app_label, model)=(tasks, historicaltask) already exists.

Now we can wipe out the DB using a PostgreSQL client (e.g. psql ):

DROP SCHEMA public CASCADE;
CREATE SCHEMA public;
GRANT ALL ON SCHEMA public TO public;
GRANT ALL ON SCHEMA public TO postgres;

Wipe out the old migrations and re-create new ones (make sure you don’t delete any overridden/data migration!):

find . -maxdepth 6 -path "./*/migrations/*.py" -not -name "__init__.py" -delete

Rebuild and run new migrations:

./manage.py makemigrations && ./manage.py migrate

And finally load the fixtures back in:

./manage.py loaddata dump.json

(Praying should not be necessary but it’s highly suggested)

A few notes:

• Any reference to a previously stored primary key will be lost (e.g. a client’s cache).
• Make sure the data can be loaded into the DB before running the migrations.
• This is obviously not suggested for mature projects.

You don’t even need a todo app (e.g. Wunderlist, Todoist), an app to scan documents (e.g. Camscanner) or even an app to take notes. Everything you need has been under your nose for a long long time…

UPDATE: Google recently added some neat URLs to be used as shortcuts for creating new documents, slides, spreadsheets etc. Check them out!

Google Docs on Twitter

Introducing a ✨ .new ✨ time-saving trick for users. Type any of these .new domains to instantly create Docs, Sheets, Slides, Sites or Forms ↓ https://t.co/erMTHOsdyH

The requirements

First of all we all use apps in different ways so it shouldn’t come as a surprise that requirements for what a good todo or note taking app is differ from person to person. These are my personal requirements:

A todo app should:

• Support web and mobile (mobile app should be fast).
• Support one time and periodic reminders.
• Support tagging/colors or allow for lists/folders.
• Support setting and sorting/filtering by due date.
• Star/pin a task (similar to moving a task to “Doing” on a Kanban board).
• Allow to export items in an open format (JSON, HTML or similar)
• Reliable sync.
• (Optional) Calendar integration.

A note app should:

• Support web and mobile (mobile app should allow taking a quick note easily).
• Support tagging/colors or allow for lists/folders.
• Allow to attach any type of file, preferably with very high or no size limits.
• Support formatting (preferably Markdown).
• Allow to attach any type of file, preferably with very high or no size limits.
• Allow to export items in an open format (JSON, HTML or similar)
• Reliable sync.

As you can see there is a big overlap in terms of requirements between the two apps, that’s why when I discovered Notion I thought I had found the Holy Grail of productivity apps. Unfortunately their mobile app is still not as optimized as I would like it to be, which is the primary reason why I was wary of moving all my tasks and notes over to them.

I started experimenting with Google Keep to take notes but the lack of formatting was a huge turn off. Also I couldn’t understand why a note taking app would have reminders…

Using Keep as a todo app

So I thought: why not use Keep for tasks instead of notes? It clearly has a lot more features than Google Tasks itself! So from now on notes in Keep will be called “tasks”.

• It supports periodic and location based reminders.
• It allows attaching almost any kind of file (a big plus).
• It allows to dictate the task (and the voice recording will be automatically attached to the task).
• Time based reminders can be used to mark the due date of a task.
• By clicking on “Reminders” it allows to sort tasks by due date.
• It allows to pin tasks to the top.
• It automatically integrates with Google Calendar, showing the reminders as events.
• It allows to add notes to the task itself (duh).
• It allows to draw, which is pretty cool!

I just couldn’t find a reason why Keep couldn’t replace Wunderlist for my todos. So a week ago I did the switch and it’s been great! I don’t miss Wunderlist at all. The only shortcoming was the lack of a feature to import from Wunderlist. In my case I patiently copied the few periodic tasks I had setup in Wunderlist and the few remaining tasks left to do.

Now I had to find an app to take notes. Initially I tried separating tasks and notes in Keep by tagging and/or color coding but 🤮 too messy.

So I started looking into various alternatives (other than Notion): Trello (good support for Markdown and attachments but app a bit slow), Simplenote (cool publish feature but no attachments), Bear, Inkdrop, Evernote…

And then it dawned on me. I’ve always had the solution under my nose.

Using Drive (Docs) for taking notes

This is the biggest revelation. Google Drive (specifically Docs) is an excellent note taking app. There, I said it.

• The mobile and desktop/web apps are blazing fast.
• Everything stays in your Google Drive so notes are just other documents.
• You can take notes quickly and just leave them in the root to organize later.
• It supports folders.
• It has great formatting on both desktop and mobile (and you can import/export Markdown with other add-ons).
• Super fast to scan documents, with a great UX.
• Virtually no limits to what you can store (files of any kind up to 15 GB for free or $1.99 for 100 GB)
• Great integration with other apps (through add-ons or Zapier/IFTT) and can create your own add-ons with Javascript 🤤.
• You can search in PDF and images.
• You can convert a scanned document into an editable document (web only).
• Android widgets that make it super easy to access Drive, create a new document or scan.
• Easy sharing with anyone with or without a Google account (via sharing links).
• Decent integration with Medium (thanks to the fact that everything in Docs is HTML, where do you think I wrote this article?).

And best of all both Keep and Drive are free.

I won’t bring up privacy issues but I would definitely discourage anyone from storing sensitive data on any medium unless they are absolutely sure that nobody has/had access to the encryption keys ever (at the time of writing you are probably more likely to win the lottery).

And, at the end of the day, if you use Gmail why wouldn’t you use Drive?

As you can see the combination Keep + Drive works wonderfully and I consider my search for an “all in one” productivity app concluded.

If you have any other requirement not covered here please do share it in the comments.