The backstory

Yammer was initially built with REST APIs but slowly as our microservices and APIs became more complex, we moved to a backend for frontend [1] pattern. But our backend from frontend APIs ended up being tightly coupled with our other services, making it extremely hard to ship changes. Three years ago, we made a leap to GraphQL and have not looked back since.

Microsoft’s Yammer codebase is powered by 120+ microservices. Our tech stack consists of Ruby on Rails for our GraphQL server. Yammer’s GraphQL service specifically uses the open source Ruby implementation of the GraphQL specification. The only thing that makes it “GraphQL” is adherence to the spec[3], so lots of different implementations exist — Typescript/Node, Go, Java/Kotlin, you name it.

Our clients use everything from typescript+React to Kotlin alongside Apollo. We use Jaeger Tracing for distributed tracing. This helps us monitor our performance and optimize our queries.

Refresher: What is GraphQL

Facebook created GraphQL for internal use in its native mobile apps in 2012 to handle sloppy networks and low internet speeds

In REST, you have many endpoints which return predefined static structures of data. However, in GraphQL, you have ONE endpoint where you send all your queries and let GraphQL handle the fetching of data to build a highly customized response, just for you.

This is called declarative data fetching, it means what you ask is exactly what you get.

Suppose, I want to build a Simpsons website to fetch the names of characters and the actors who voiced them and these are exposed via two different APIs by the backend. In REST, the clients would need to make two API calls whereas in GraphQL, we would only need to make one.

Lifecycle of a GraphQL query

The client will use the query language from the spec to speak to the server implementation. The query language is declarative — you specify what data you want and the shape it should be returned in.

The GraphQL server has a schema which defines what you can query, and also houses the engine which knows how to parse queries and execute them. During execution, it will make calls to various backend services, we call these as ‘downstream service’. A single client query can fetch data from multiple resources. This is what’s happening over the lifetime of a query:

1. The client sends a GraphQL query to the service.

2. The service parses the query into an abstract syntax tree.

3. The AST is used to validate against schema and make sure that the types and fields being requested exist and are in the right places. If there are mistakes in the query language, they are caught at this stage.

a. There are steps after this that simplify the AST representation and optionally analyze the query.

4. After that, the query is ready to be executed. It will begin contacting downstream services to resolve the requested resources. We’ll be covering later how that happens.

5. Lastly, it sends the request back to the client. If something failed while retrieving resources, you might have an errors array here detailing that something failed, alongside whatever partial data we successfully got. You can also have validation errors or query parsing errors that will show up here.

Resource Limiting

Our service has some limits to protect against excessive or abusive calls to Yammer’s servers. So we imposed limits enforced on a query inside the GraphQL service:

· We can have no more than 50 levels of depth in the query.

· A query can make no more than 30 downstream calls at one depth level.

· And the max execution time of a query is at most 1 minute.

Designing GraphQL friendly REST APIs

Most GraphQL queries make calls to multiple downstream services. It’s important to make sure that all downstream endpoints adhere to some common guidelines in order to meet end-to-end latency and availability SLAs. Here we will cover 3 case studies:

· High Span Query: GraphQL resolves queries in a breadth-first way. This causes some problems. Namely, something mysteriously called the N+1 problem. The n+1 problem arises because GraphQL executes a separate resolver function for every field, whereas REST has one resolver per endpoint. These additional resolvers mean that GraphQL runs the risk of making additional round trips to the database than are necessary for requests.

We use GraphQL Batch, the Shopify-developed Ruby implementation of DataLoader[2], which is the node implementation. GraphQL Batch uses promises during query resolution to defer calling downstream services. After everything at that depth is finished resolving. It groups API requests by downstream endpoints. One thing to note is that we never batch calls at different depths. We can do some interesting things at this point, like deduping API requests for the same user, for instance. Finally, we make whatever downstream calls we need and resolve the promises with the results

In this case, one of the downstream apis is failing with 429s and this would cause the entire query to be marked as failed. Recommendations for solving high span query problems including providing batched REST APIs for every resource. If downstream REST APIs support batching, then the N+1 problem can be entirely avoided. Batched APIs should support partial results. They should not return an error when only a subset of ids are found. Whereas, single resource endpoint would return a 404 when even when one id is not found.

· Slow Query: In the case above, after batching — one downstream API call has high latency which is causing the entire query to slow down. Recommendations for slow queries include Client side optimization to Split queries such that slowest fields are lazy loaded and optimized for debuggability and performance. On the server side, when developing APIs that return sets of objects support paging & filtering wherever possible

· Lookahead in Queries: GraphQL resolvers queries in breadth first fashion, so parent fields are always resolved before children, causing batch loaders to not be able to optimise child queries.

Lookahead is a GraphQL concept that allows you to peek at sibling and child fields and trigger resolution outside of the BFS pattern to save latency and make your APIs more performant.

Parting thoughts

· In REST, we have a bunch of many endpoints which return predefined static structures of data. However, in GraphQL, we have ONE endpoint where you send all your queries and let GraphQL handle the fetching of data to build a highly customized response, just for you. Main catch here is declarative data fetching, it means what you ask is exactly what you get.

· GraphQL might not be for everyone, it is important to understand the use cases and best practices before you decide to commit to it

· Designed properly, graphql can help clients to handle sloppy networks and low internet speeds, which is extremely invaluable for mobile clients

Building GraphQL at scale was originally published in Code Like A Girl on Medium, where people are continuing the conversation by highlighting and responding to this story.

Last week, I alongside my RGSoC teammate, Ipshita Chatterjee spoke at Codemotion Berlin about our project during the summer, coala which is a static code analysis tool and about the program itself. While it wasn’t the first technical conference we were attending, it was the first time we were going to be giving a technical talk and hence, we were quite nervous. We also had a very strong imposter syndrome as many speakers came from stalwarts such as Google, Microsoft, IBM, Salesforce and we were just college students. Needless to say, our worries were totally unfounded and we ended up having a blast at the conference! Here is what we learned :)

Know your audience

As our first step towards preparing for the talk, we researched about our target audience and the conference itself. Codemotion guidelines stated that the attendees usually came from strong technical backgrounds and most of them were already pursuing careers in technology so talks were expected to be of intermediate/advanced level than beginner. This gave us an indicator that we could comfortably use technical terms in our talk and would not necessarily have to simplify the content.

Tell your story

One of the earliest advice that we had received from the people who had given technical talks was to structure it in the most engaging way possible. We began by talking about how we had a very strong impostor syndrome before RGSoC and how we felt empowered enough to talk about it now, we talked about our process of choosing our project and interspersed the dialogue with anecdotes of why it is was awesome. Most importantly, we told our “RGSoC and coala’ story.

Practice, practice, practice (and engage)

One of the best things we learnt while preparing for this was that we were supposed to give a ‘talk’ not a speech. We were supposed to keep the material engaging, ask questions rather than just provide answers. And it helped us immensely. We didn’t rehearse the talk much, except being thorough with what came next and that helped us in having a really really successful first talk!

Berlin and Codemotion

It was our first time travelling to Berlin, so we had a lot of fun in the city. We met up with other teams from the current and past editions (we also met a few people who applied and didn’t get in — so we encouraged them to apply again). Codemotion, in itself, was a very satisfying and enriching experience. Some of the talks, especially the keynotes, were very relevant to us as minorities in tech, and as new graduates looking to join the industry. We met with a lot of interesting people, networked, spoke to recruiters and in general, had a blast!

Codemotion and the entire experience of speaking at our first technical conference is something we will never forget! A huge, huge shout out to the RGSoC organising team for giving us this opportunity!

Giving your first technical talk was originally published in Code Like A Girl on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’ve been holding a few workshops on getting high school students in North India interested in software development and open source. Here is what I learned.

Busting Myths

There is a misconception among educators and parents alike that at school level, the only thing that interests students is getting admission in a good college. However, I have found that several of them are in fact looking for ways to fill the gaps in between their studies and would love to do something constructive.

During one of the sessions, we called for a show of hands among students for who wanted to pursue computer science or engineering as a career. I was very surprised that none of the girls sitting in the audience raised their hands. However, when I spoke with them one on one, I found out this was NOT actually the case. They were very interested in math and science. Several of them were very smart and knew their way around computers as well. They were simply too afraid to ask for help and speak up. This made me wonder if it was indicator of an underlying problem about diversity in tech?

Explaining Open Source to a Newbie

I began by talking about some of the products that they have come across in their lives — Android, Mozilla Firefox, Visual Studio and proceeded to link all of them by the common thread of open source software.

I also talked about communities and projects (which can easily get too scary and big, so I had to make sure to keep telling them it wasn’t too tough) Most of all, I introduced them to Google Code In, an open source program for 13 to 17 year olds. I stepped them through some of the examples from the past projects.

For fielding questions about their ability to participate in such a program, I quickly chose to divide my audience in groups:

• Beginners: These students who had never written any code in their lives except for the basic Logo programming and using HTML to build websites. (mainly 13/14 year olds) I gave them easier tasks such as building and installing an application, writing technical blog posts, or suggesting changes to a website.
• Intermediate: They had some knowledge of programming constructs, had built some programs in C++ and Java. For them, we introduced the slightly tougher tasks such as fixing small bugs, adding features and getting engaged with the open source community, in general.
• Advanced: Unfortunately I did not have the good fortune of interacting with any students who were already contributing but ideally, I’d let them pick and choose whatever they wish to work on with some mentoring assistance available instead of walking them through.

Student Take Aways

Mostly they took away a very enthusiastic impression of me harping about how they should make the most of this opportunity. I was happy to field some queries about GCI later in the day and GSoC from those who were turning 18 soon. I also had questions about how to get started to create a GitHub account and advice around learning GIT.

We announced a 15 day competition (Ala Hacktoberfest) wherein the students were to submit their contributions and then send them to us. We would then award the top three participants! We are yet to get the entries in from this so I can’t comment on the success of the competition but hopefully it will be something promising :)

How do you replicate this in your school?

If you have a working knowledge of how the community works here, you can access my slide deck. Please feel free to reach out to me about any inconsistencies, mistakes or areas of improvement.

If you are an educator and looking for help or any other learnings from my experience conducting these sessions, you can tweet to me @prachim1210.

I do this for outreach and awareness and mostly because I have a lot of fun interacting with kids :)

Catch’em young was originally published in Code Like A Girl on Medium, where people are continuing the conversation by highlighting and responding to this story.

From the code that sent Apollo 11 to the moon to the operating system that powers most of our cellphones, open source is booming and you would have to be living under a rock to not have heard of it, yet. As of 2016, GitHub was home to projects written in 316 programming languages (source: Octoverse) and to be honest, I didn’t even know that many existed. But contributing to open source projects has helped me write cleaner code, ship less bugs and improve community interactions considerably.

So what is Open Source really?

Typically speaking, open source refers to software that users can run, distribute, study or modify for any purpose. It is simply collaborative software development that relies on peer reviews and community feedback to get better quality code.

The Open Source Business Model

Why would companies want to give away their software for free?

The business incentives behind releasing software as open-source are many. Take for example, Google’s decision to open-source Android that helped with fast adoption and its growth to become the most prominent operating system on mobile platforms. Major tech companies such as Facebook, Microsoft have also started open-sourcing their software now.

Summers of (Awesome) Code

Today, several organisations fund programs to push students to open source notably among them Google Code In, Google Summer of Code, Outreachy, Rails Girls Summer of Code and the list goes on. Most of these provide a stipend to cover living expenses of the student for a period of ~3 months while working remotely on the project of their choice.

What is in it for you?

How can Open Source be helpful?

• Write cleaner, less buggy code: Open Source is fundamentally built on the practice of peer reviews and community feedback. So, you usually receive comments from a lot of developers who have worked on the project in the past about programming idioms, good community practices and writing maintainable code.
• Contribute to great projects: A lot of great projects today are open source, notably among them DuckDuckGo, VSCode, React, Swift, Go, Kotlin and many, many more.

• Build a network of great developers: Since community interactions is a fundamental part of open source, one gets to interact with developers from different backgrounds in different timezones which eventually evolves into a support network with a lot of talented coders.
• Bag internships and jobs: Your GitHub profile is your online portfolio and a good way to showcase the quality of your code to recruiters. I, have myself, received internship offers by recruiters who had a look at my projects on GitHub.
• Swag, swag, swag: Once you have made some valuable contributions to a project, they will ship you some cool goodies and stickers as a thank you!

Okay, I love Open Source now but how do I start?

Ask and ye shall be answered

Set it up: Find a project that you really, really like (and in a language you are familiar with) and clone it on your desktop. Set it up, follow the installation guide and try running it. If you face any issues, go ahead and create an issue for the project maintainers to look at (Telling maintainers what problem you faced in the project is also a valuable contribution for the community).

Introduce yourself to the community: Say hi to the other people working on the project on Gitter, Slack or whatever chat room they use. Community interactions are as important in FOSS as writing code is.

Find a suitable issue and work, work, work: Most communities have labelled some issues as /easypick or /newcomer and do not require much knowledge of the code base. Start with those first. Ask intelligent questions (Now “What should I do” is not one). Poke around. Send a Pull Request once you think you have fixed it. More often than not, you will receive a lot of comments on it. Address those, fix your code, iterate over till it passes all the tests and voila, you have made your first contribution!

Most importantly, don’t get discouraged or stop. It may happen that you feel overwhelmed by the huge code base most of which you do not understand. That is okay. Do not give up, the community is there to help. Ask for it. :)

Git Out and Contribute was originally published in Code Like A Girl on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Summer So Far

This is the third blog in our fortnightly blog series for coala as a part of Rails Girls Summer of Code. We have previously written about writing bears for our project coala and community outreach by the Rails orga team. This blog post shall focus on writing test suites and our experiences with Test Driven Development (TDD)

What is code testing and why is it important?

Unit testing is a software development process in which the smallest testable parts of an application, called units, are individually and independently scrutinized for proper operation. Unit testing can be done manually but is often automated. In simpler words, unit tests help you find bugs in software and make sure the same bugs don’t cause the application to break again.

Testing often seems like a pain to developers and a time consuming job with no actual output except a validation of the development code itself but it is important to realize that testing is as essential for a developer as writing production code.

For one, tests will help you avoid delivering sub standard code. A comprehensive test suite is extremely useful when adding a new feature to ensure that any modifications that you made doesn’t break the existing code base. Further, when refactoring or removing redundant code, tests would be there to ensure that no corner cases are being violated by your changes. Keeping code is clean is incredibly important, especially for vast open source projects.

Secondly, tests help increase code stability. If each commit is tested before deploying, it would ensure that the release version is always a stable build and no users are being impacted by buggy developmental code.

Not just this, tests also supplement your development code by helping new contributors get familiar with the possible tests cases they should keep in mind while making any changes.

coala uses CircleCI, Appveyor and Travis CI for their testing suites and while at times, the broken builds do make us pull out our hair in frustration, we realize the importance of having a robust CI pipeline

When should tests be written?

Test driven development (TDD) guidelines mandate that developers must write test functions before the actual code but most developers prefer writing testing suites retroactively. However in our opinion, it doesn’t actually matter when you write the tests as long as you actually write the tests.

How do you check the quality of your tests?

A common metric for testing the quality of your testing suite is called code coverage. Code coverage basically is a number conveying what percentage of your development code is being covered by your test function. It helps you catch the corner cases you might have missed testing.

coala uses codecov as a part of their development pipeline for checking code coverage and no commit is allowed to be deployed unless it achieves the required code coverage (a minimum of 90% across most projects). We can’t adequately stress the importance of this as it has helped us catch several cases that we had missed while writing test functions.

Is code coverage by itself sufficient?

While code coverage is important and absolutely essential, it is not an adequate metric for checking the quality of test suite. It simple checks for the number of lines of development code that are being tested.

For example: If you are testing a function for finding maximum in an array, you maybe testing the functionality of the function and get 100% code coverage but not test cases such as when the array is out of bounds. Hence, manual testing are also important.

Key Takeaways

• The best practice is to write tests as you go. The developer who writes the development code is the best person for writing the testing suite.
• Ensure maximum code coverage. Keep in mind the 80–20 rule of code coverage (20% of time will be spent in achieving 80% code coverage and 80% of time will be spent in achieving the remaining 20%)
• Manual testing of code is important after automated testing is complete. There is no substitute for peer review and feedback.

If you would like to follow our progress over the summer, you can read our daily logs here or give us a shout on twitter! We’d love to hear from you :)

What is Rails Girls Summer of Code?

Earlier this year, I heard about this ‘Summer of code’ program called RGSoC sponsored by the Rails Foundation. What was different and interesting about it was that they sponsored ‘teams’ instead of individual students. The motive of the program was to help promote women in tech by creating a network of supportive learning through your code-pair, coaches, mentors and team buddy (or, supervisor). Whoa, that is a lot of people! Needless to say, I was intrigued and ended up applying with a friend and fellow developer, Ipshita.

The Selection Process

Each year during the application period, the RGSoC organizing team releases the list of open source projects that they have received applications from and they think will be suitable for students to work on during the summer. Students are then supposed to interact with the project mentors, community and contribute to them while filling out the application proposal. The proposal had a very diverse set of questions ranging from community outreach to code samples in the language of your project.

After looking at different projects, we decided that we wanted to apply for coala. Unlike Google Summer of Code, RGSoC only allows you to apply for only two projects and since we had our exams going on, we weren’t able to interact with any other project. We planned a timeline of tasks for the summer with our mentor and sent it with our application.

Around 6 weeks after the applications closed, we received a call for interview with the RGSoC organizing team. More than a formal interview, it was a discussion about our goals and aspirations from the program (and discussions about the stipend :D ). We were also asked about our choice of project and what is it about coala that interested us.

The community bonding period

Our organization, coala requires any contributor to complete a set of tasks before they can be upgraded to developer. Amid those are getting merged two Pull Requests of ‘difficulty/newcomer’ and ‘difficulty/low’ type and reviewing two Pull Requests. Doing this not only helped us know the people who were working on different projects (corobo, coala-quickstart, coala-bears) but also helped us understand the guidelines that coala suggests that you follow when contributing (Regarding commits, code styling et al)

Fast Forward to July

In our first week, we met with our supervisor (or team buddy), Mayar who was herself an RGSoC fellow last year.

Fun fact: We are from India, our supervisor is from Egypt, and two of our coaches will be based in London and Amsterdam by September. Talk about bringing international flavor (and the pain of co-ordinating across timezones, Just kidding it is fun) to the summer.

We also started working on our timeline in all earnest. In our first week, we began by solving some issues to get a hang of not just writing quality code but also write corresponding test suites. It also helped us get familiar with the code base (I, for one, had to refer a lot to the existing codebase to get it right). Oh, and the feeling of seeing green builds was pure bliss!

About coala

Our project, coala is a code analysis tool. So, developers at coala write bears to help you check for bad coding practices, styles etc. in different languages.
Bear writing can be basically of types:
 — Native Bears
 — Linter Bears

In our second week, we started writing our first bears which have been linter bears. A lot of programming languages already have linters implemented, so if a project uses a language that does not already have a linter bear we do not need to implement it on our own. Linter bears are python wrappers (since coala is in python) around existing code linting tools available in other languages.

So far we have sent PRs for two linter bears:
1. RubyFlogBear which gives as output “most tortured code in an easy to read pain report. The higher the score, the more pain the code is in.”
2. HAMLLintBear which wraps around a tool haml-lint that helps keep HAML files clean and readable. In addition to HAML-specific style and lint checks, it integrates with RuboCop to bring its powerful static analysis tools to your HAML documents.

One of the hurdles that we faced was finalizing the implementation details, we weren’t sure how many of the flags should be included and in what way. (Code linting tools have a LOT of options). Also, coala has a very robust CI pipeline, so even if one of the builds (Appveyor, CircleCI or Travis) were red, we’d become disappointed. And then we’d figure out hours fixing absolutely nothing before realizing that coala-bears was broken and the builds weren’t failing for any fault of ours. Haha.

Oh, and something that we have experienced for the first time at an open source org — at coala, the more code reviews you do — the faster your code gets reviewed by the community (they have a lot of developers and so a lot of PRs!) So, we have been steadily performing code reviews too :D

What’s next?
So far, our experience during RGSoC has been filled with a lot of learning. We plan to work on a few more linter bears for now, and then progress onto more challenging issues.

The organizing community will also be providing us tickets to some conferences that we can attend (and give lightning talks at) so super excited for that!

Keep up with us
If you want to follow our progress, we write daily logs and I’d also be writing a fortnightly blog about our summer (hopefully). And of course, you can always tweet to us. We promise we’d get back to you.

This winter I had the opportunity to intern at Kayako.

I’d heard about Kayako when a friend had applied there an year ago. So when I received a call from their HR, Ruhi Mahajan regarding a possible internship opportunity, I jumped on it. During summer 2016, I wrote a bot script that trended on GitHub and that was how I caught their attention. It goes without saying that I was excited (and a teeny bit, apprehensive) to be joining their product engineering team, but whoa! what an experience it was!!

Onboarding

Alright, so before I deep-dive into what I did, let me get all the razzmatazz out of the way. I was based out of their Gurgaon office where I worked for 6 weeks. A few weeks before joining, I received a book and a welcome note from Ruhi. More was to follow. On my first day at Kayako, I was hooked up with a MacBook Pro, headphones and some swag. ❤

The Project

My initial days at Kayako were spent familiarizing myself with their tech stack, in house framework and the project that I was supposed to work on. Being a customer support platform, Kayako takes end-user satisfaction very seriously and so, improving the performance of the product was given top billing. Some of the key actions in product were excruciatingly slow and hampered the user experience, resulting in the leads being lost to competitors.

The Idea

Identify → Patch → Monitor

My first week was spent analysing the performance graphs profiled by Blackfire to identify bottlenecks in critical API endpoints. We chose Blackfire because it provides a detailed breakdown of the slow functions and helps you to identify the bottlenecks by giving an average response time aggregated over multiple requests for any end point with minimum overhead. Also since blackfire is triggered only with specific headers in requests, it meant zero overhead on regular requests.

The insights gained from this exercise were remarkable. For example, a seemingly innocuous method that should have been O(1) theoretically was found to be O(n) because of the way it was being called. Once we identified the bottlenecks, we needed to send patches and fix those bottlenecks. After the patch was rolled out, we validated the performance gains by re-running the blackfire profile and confirming if the patch fixes the issue.

While we noticed the performance gains immediately, it was the outpouring of appreciation from the customers and SME team regarding the faster response time that really felt like an achievement.

The Engineering Culture

• Code Reviews: Since my project specifically concerned product performance, getting code reviewed by multiple members in the team not only helped in fixing bugs before they got shipped off to production but also helped me in gaining perspective if the fixes I made might cause something not overtly obvious to break!
• Distributed Release Management: Engineers at Kayako deploy 10+ times per week to kayako.com, so instead of having a dedicated person who takes out product releases, each developer at Kayako, takes out their own tickets in their releases.
• Continuous Integration: One of the most important Agile practices is a robust deployment production pipeline, and although failing a quality test over a lower-cased commit message did make me pull my hair out in the beginning, but I cannot adequately stress on the number of times it helped me catch bugs that were not immediately visible.

This winter has, by far, been the most productive vacation of my college life and working at Kayako alongside the most talented people (shout out to Shivam Dixit for being the coolest mentor ever) is something I’ll certainly miss. Every day there threw new challenges at me and left me with a thirst to learn more and to strive to become a better programmer (not coder :P)