Unleashing the Power of Open Source in DL: Does Global Collaboration Accelerate Code Evaluation?
Four Master’s students from the University of Groningen explore the impact of timezones on code review speed in Deep Learning non-industry projects.
TL;DR
Does Global Collaboration Accelerate Code Evaluation?
nooooooope :)
Overview
Have you ever wondered how the extent of globalization of a project affects the development time? Guess what — you are not the only one! In this blog post, we are sharing our findings on the impact that global collaboration has on code review speed in Deep Learning non-industry repositories.
Why research Deep Learning repositories?
A major chunk of research conducted in the realm of code review speed only concerns itself with code bases that show the highest level of activity in terms of commits made, popularity, relevance in the industry, and so on. However, in our case, we aim to narrow our investigation to repositories belonging to a specific domain of interest. Our primary focus lies in delving into repositories of Deep Learning and comprehending the potential influence of global collaboration on the speed of code review in such repositories.
By investigating this relationship, we aim to gain valuable insights into how collaborative efforts in non-industry Deep Learning projects are manifested across different regions, and thereby the impact on the efficiency of code review processes. It can also motivate cross-border work, fostering a global community. Additionally, identifying areas for improvement can enhance code review processes, increasing productivity and driving innovation in Deep Learning.
What are we investigating?
Research Question 1 (RQ1): What is the impact of the number of different time zones from where developers collaborate on the speed of code review?
Research Question 2 (RQ2): How does the variation in time zones from where developers collaborate influence the speed of code review?
The key metric for assessing code review speed is the “time-to-merge,” defined as the duration from the opening of a pull request by its author to its eventual merge (and closure). The metric we use to quantify global collaboration is based on the count of distinct time zones and the difference in hours between them.
Get Software Analytics (RUG) Group 7’s stories in your inbox
Join Medium for free to get updates from this writer.
The analysis for each research question will be conducted through two approaches:
- Repository-Level Analysis: Calculate an aggregated “time-to-merge” metric for all pull requests within a given repository. This figure is then analyzed in relation to the diverse time zones from which the repository’s collaborators operate.
- Pull Request-Level Analysis: Determine the “time-to-merge” for each individual pull request, and examine this in correlation with the time zones of the participating contributors.
How was the data collected?
- Retrieving repositories. The GitHub API was used to retrieve Deep Learning repositories. To strike a balance between focusing on widely used and established libraries, while staying within the GitHub API’s 1000-repository limit, we narrowed down our findings to three libraries. After evaluating multiple combinations, we found that PyTorch, Keras, and TensorFlow yielded the highest number of results. In this way, we collected 800 repositories.
- Filtering repositories. We filter down the repositories by Open Source GitHub licenses. We also enforce the selected repositories to have Python as the main programming language. Afterwards, we filter out the repositories with insufficient commit count, lower number of collaborators or unavailable locations.
- This leaves us with 51 repositories. On this gathered data, we perform statistical analysis to answer our research questions.
Insights from the data
- Our dataset encompasses 16 distinct time zones. The most common ones include UTC-7, which covers the West Coast of the USA, such as Silicon Valley; UTC-4, spanning the East Coast of the USA, parts of Canada including Ontario and Quebec, as well as Central South America; UTC+1, representing Central European Time and covering Western and Central Europe; and UTC+8, which includes China, Singapore, and Western Australia.
2. No significant difference has been found in time-to-merge between repositories with a low number of distinct timezones and a high number of distinct timezones.
3. With respect to the distribution of collaborators’ timezones, repositories with a more homogenous spread of timezones exhibit faster time-to-merge values when compared to repositories where the distribution/frequency of timezones is skewed.
4. The lower the number of time zones in a Pull Request, the faster the code review speed.
5. Repositories with a smaller average difference in time zones tend to have a quicker merge time compared to those with a larger average timezone difference.
6. We found that Pull Requests with participants from the same timezone display a faster time-to-merge than participants with a timezone difference greater than zero.
Brace yourselves… Here are the limitations!
During the data collection phase, we encountered repositories that are owned by companies but also incorporated contributions from the open-source community. To ensure an adequate amount of data, we decided to include these repositories, as removing them would have reduced our available repositories to only one-fifth of the original count.
Not enough users had their location on, so the locations we collected may not be the true representative of the list of time zones from where the collaborators participate. Even if the location is specified by a user in their GitHub profile, it may not always accurately reflect their current location, and in extension, the time zone they reside in.
It’s important to acknowledge the significant diversity among our repositories. We have large ones boasting hundreds of collaborators and thousands of pull requests and commits, which tend to overshadow the smaller ones. These smaller repositories typically have fewer than ten collaborators and only a few hundred commits and pull requests.
What comes next?
Many other domains of Software Engineering could also be explored. How does global collaboration affect the code review speed in data processing, full-stack development etc.?
In addition, a more thorough analysis of the pull requests, time to merge, code reviews and code comments can be done in a study related to technical debt in software engineering. This research can investigate how technical debt is affected by the number of timezones in a repository or timezone difference.
Lastly, as our study examined non-industry repositories, it would be interesting to see whether there is a significant impact of global collaboration on industry Deep Learning repositories.
Interested to see some technical details?
If yes, then check out our public GitHub repository.
