It’s hard to be able to condense a full year of hard work in a few paragraphs but there’s a reason why people say that a picture is worth a thousand words:

This is a video of what happened in the Unsplash API codebase during the last year.

It’s naive to talk about the future without taking a step back and looking at what we’ve accomplished the past 365 days of building the Unsplash API.

The API received ~19.5 billion requests during 2019. That means we had almost two times more traffic than in 2018.

The team merged 731 PRs in just the API repository, excluding all of the satellite apps, like the frontend, data pipeline, CDNs, and helper services.

All of this was supported by a team of three people and a Rails application that day by day becomes less Rails and more pure Ruby.

A lot of people make Unsplash work everyday but this is a small recap of what a team of three can do.

Monitor every heartbeat

There’s a lot happening within Unsplash, a lot of services, databases and different teams. Our current weapon of choice for monitoring is DataDog. We not only have agents in most of our runtimes but we also have specific metrics we use through our monitors.

Monitors exist within DataDog as a way to identify outliers, anomalies or just things passing a threshold. Mostly we use them to check response times across our architecture but they can serve other purposes like checking if some fail safes are in place or if our experiments are running smoothly.

Experiments did you say? Yes. We run performance experiments when we want to try refactors while ensuring that performance is not going to be affected. All of that data goes to DataDog.

Recaptcha

We love bots, we have some doing tasks for us and a special one we use for chatops. But if we learned something from Terminator it’s that eventually they go rogue.

2019 saw a rise in spam accounts and finally we took the decision to force ReCaptcha on our Join page

This was not an easy decision but as bots become smarter and smarter, we decided to add that filter to keep the content quality high.

There are also some internal tools that detect bots on the platform but that’s a post for another day.

Penumbra

Search is a big and complicated problem to solve and for the past year we’ve been migrating to solutions more close to the metal than before.

To allow better interactions and more responsibility in our code we built Penumbra, a Ruby library to interact with our Elastic cluster, handle mappings, data migrations and search algorithms.

We feel that taking ownership of the core features of our app allows us to take it to the next level. This is obviously a long path that involves a lot of learning but it’s the only one that allow us to squeeze our servers to the fullest and take control of every part of the experience.

Eventually Penumbra will be released as open source.

Autobahn

Popularity comes to the cost of DOS, crawlers and abusive api applications so certain things need to be put in place to ensure that the uptime of the public API is met.

Autobahn is a small middleware that allows us to ban and block IPs based on their behaviour. Since speed is key the backend of it is Redis.

Dynamically stopping abusive IPs was was key to keeping our response times low in 2019.

Goodbye Cervantes, hello Lisboa

Taking ownership of our search algorithms allowed us to start experimenting with different iterations of them. Everything happened behind the curtains and via different experiment flags.

The initial set of algorithms after migrating to Penumbra was aptly called Cervantes. Its main goal was to keep all of the old features without affecting conversion rates, so strictly speaking it did nothing new but did everything the old version did.

We tweak the algorithms quite often but from time to time when the time is right and the changes are big enough we “bump” the name.

Lisboa was the next version of search from which involved a lot of work of one of our teammates: Roberta. Due to that extreme commitment to the craft we named it because of Adriana Lisboa which shares, among other things, the same nationality as her.

This version significantly improved multi-keyword searches versus the prior versions multi keyword searches from the prior version.

Color filtering

As part of our ongoing integrations with the Data team we collaborated to add color filtering to search.

An AI will “look” at the photos and check for color coverage to return color information. All of this was later on added to our search cluster so now we can use the new specific color information to allow filtering among the photos.

Due to the vast amount of fantastic black and white photography and the limitations of AI, the black and white filtering was more challenging to implement and get right.

This colorful version of search is an example of the many experiments we run on regular basis to test new features.

Specialized workers

As one would expect a lot of the work that happens on our servers happens asynchronously and we have a lot of it.

Due to that and with the spirit of DRY we are migrating some of our workers to enqueue jobs as we do normally but process them in bulk.

This MapReduce approach allows us to reduce our execution time for things that need to happen in the background but can wait so the load in our databases/services is not that big.

To homogenize our asynchronous work we started migrating our current cron jobs to be part of the background jobs queue so we get better retry capabilities and cleaner code overall.

Safer together

I don’t want to end this year in perspective without saying thanks to the security researchers that use their time to check our security practices, send emails and communicate with us.

Their love to the craft and help is invaluable to make Unsplash the website it is. Some of them are found here https://unsplash.com/security and we are truly thankful for their support.

Future

There’s so much stuff already for the 2020 edition. I’d love to tell you all it but you’ll have to wait.

As a sneak peek we are shipping a new statistics cluster which is being fed by our first Go service that handles the ingestion and transformation of data.

A lot is happening on API right now and we’ll have a lot to share this year while we push the greatest photo library to the next level.

A year in perspective was originally published in Unsplash Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

As you may know (and expect) Unsplash is built on the shoulders of giants by leveraging open source and proven projects. There’s a bird’s eye view by Luke here.

I’m not here to tell the same ol’ story about how open source is the best and using it is the way to go.

Since I’m the product of an open source community I take it very seriously, not only by doing open source and talks myself but also pushing the cultural part of it.

Lately I’ve been working on tools that try to solve specific problems we’ve encountered but with a twist: treating it like open source, coding like everyone is watching.

Penumbra

The 24 hour bookstore

The name of this library comes from a book that I really enjoyed some years ago called Mr. Penumbra’s 24 hour bookstore that tells the story of a web designer working in a library, Ruby and the importance of books.

It was the perfect name for our internal Elastic Search library ^_^.

Penumbra was born with a specific purpose: being the nuts and bolts of our new search autocomplete.

However, there were some specifics to be resolved since it was supposed to be stupidly fast and accurate with the recommendations.

I think that the best tools are the ones that solve a specific problem without adding layers of complexity while being honest and clear about what’s going on.

Any fancy stuff happening behind the scenes? Nah, just a `proc`.

Since we were building something for us — with the idea of making it open source — I started with this paradigm:

People want a tool to make the job easier, not to be completely oblivious to what’s happening so Penumbra is a tool. It does not force you to do anything, it will just try to make things easier.

For example: we have a lot of photos so reindexing takes time. That’s a problem. As a result, Penumbra’s sole focus is to make that fast and safe so that you spend less time being worried, and instead spend more time on what actually matters: building a better algorithm.

MRQ

Mr Q — Map/Reduce Queue

The name comes from the beloved MrQ from the James Bond series. That sweet man in charge of all the gizmos and gadgets… and also happens to describe what it actually does: a map/reduce queue for Ruby.

We have a lot of sources of information here at Unsplash, different systems that extract and process all we can get from a photo.

But one thing is important: most of these systems are not dependent on each other, they just extract data so they can be run in parallel.

For that we use concurrent-ruby and push the tool to a feature complete state-meaning it will do one thing and one thing only: map then reduce.

Every map runs in a thread fetching information and the reduce will give you everything once all the sources are ready.

Present Proofing

I like to feel like I’m able to predict the future. That I’ll be able to know where the codebase should be a week, a month.

I like to feel that I do, that I can reach the code utopia I feel we deserve. That I’m no different from Wayne Gretzky saying this:

I skate to where the puck is going to be, not where it has been.

The truth is that I don’t. I have no clue where we are going to be.
Don’t get me wrong, I really know our codebase and I have an idea of where it’s headed-but code is only a tiny percentage of our product.

Businesses will change, performance will fluctuate, team mates might be different, an AI might overtake the world.

However, the thing I’m certain of is the now. So instead of future-proofing for a future that most likely will not be as I imagine it, I prefer to present-proof the code.

And that’s by solving the actual problem that we’re having in the most simple, elegant and real way.

If something is painful for the team, if it takes too much time, if it’s not reliable or if we don’t have the control we need, we must change it.

Ruby will change, JavaScript will definitely change, we will change so instead of imagining the best and fanciest palace that will be nice to look at, but completely useless, let’s just build the stairs we need; one step at a time.

Dance like no one is watching. Code like everyone is. was originally published in Unsplash Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

I was born in Montevideo, the capital of Uruguay, a small country in the south side of the world on a rainy Tuesday. Anxiety was always my thing I guess since I was born 2 months early.

Uruguay was exiting a dictatorship that decade so things (and people) were trying to find a place in this new republic.

People want to belong somewhere and I wanted to belong as well. As the son of a single mom we have our struggles, things were not easy, and hard decisions had to be made.

I made the decision that I wanted to work. I wanted to stop seeing my mother sleep only a few hours a day, starve herself and walk miles to her job just to allow me the opportunities that she never had.

How can you change your reality if you don’t have the money to go through a career?

You start from the bottom.

My first job was when I was 15 and when I was 17 I started working at a cyber café on the night shift on a regular basis. It was a quiet and calm time to read and learn. Eventually, I started fixing the computers and understanding how the underlying network worked. Having unlimited internet access, during a time that it was unthinkable to have that in your own house, gave me an endless world of information and I took every bit I could.

PHP was the thing I learned thanks to the Uruguay Linux Group and a person that took time of his life to give me pointers and teach me that in the internet no one cares about where are you from or how much money you have.

I got a job as a web designer. HTML, ajax, and all that early 2000s tech. Only two people on that company coded, me and an old school C developer. He shared what he knew with me because I took the time to learn.

Then in my next job, something changed. The internet changed. Communities changed. GitHub was there, letting everyone share their code, millions of lines of code to read, of projects, of ideas.

If you’re not from the world of open source software, you need to understand how alien the idea of sharing all the things you know, all the information it might take you years to learn, all the shortcuts. Free? Why? Why would someone do that? What’s the catch?

I learned, I coded, I kept trying and (more often than not) failing, but I had a million teachers, so I learned Javascript, then Ruby, worked on Ruby on Rails.
I gave a talk for the Python community about distributed systems.

My code was reviewed by a lot of people, most of them I didn’t know their faces, nor did they know mine.

I gave talks in Argentina, Netherlands, Scotland, etc. All opportunities from which I gained exposure, pushing my limits every time.

A career built on top of sharing what I knew led me to spend some months in NYC working on a project with ICANN, and then working for several startups in San Fransisco. Only because I shared my code and I learned from people I had never met, but I cherish deeply.

I’ve been working for Unsplash for almost two years now, and I love the product and the people behind it, but frankly: I know better than anyone the effect of sharing, the effect of just sharing because you have something to give, to learn, and to try.

It’s been a long trip with ups and downs but let me tell you something, I chuckle when someone tells me that exposure is useless.
Because it is useless, you can’t take 10 exposures and change it for food. But it’s NOT worthless.

Exposure and its side effects not only gave me a career, it gave me the tools to grow.

My life changed thanks to exposure, thanks to a community of people that wanted the profession to grow beyond what anyone would expect.

They didn’t care that I was born in a third world country, nor that I learned English thanks to HBO.

They cared about what I’m capable of doing and for one I’m proud of working with people which mission is to accomplish the same in the photography community.

One day I hope I meet people that changed their lives, their reality, thanks to Unsplash.

There are people out there that want to learn, that want to change their life. They have hope that one day, the career they love so passionately will give them a chance.

I started this post talking about part of my story but, as the knowledge that I have, this story is not only mine. There’s a lot of people out there like me, like you.

A Liberian man looking for an opportunity with a limited set of tools labeled as a scammer who proved his valor by literally taking photos.

It took someone doubting him, thinking it was a trap, trying to make him lose his time, to change the life of one person, of one village, of a lot of kids.

There are people like Michele:

A woman that wanted to change her life by becoming a programmer got a free ticket and became a speaker herself.

For my story and the things in between, you might now see the world how I see it. Or you could still think that exposure is useless and that’s okay. But for me, my friend… exposure is priceless.

Exposure is useless was originally published in Unsplash Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.