Commit!
Long-term consistency in commitment to open-source
Open-source projects are not just about the latest exciting updates and features, but also the principles and people that drive these projects to success. In this blog post, I reflect on the dynamics of open-source communities, their challenges, and our approach when it comes to navigating them through a few examples. I detail some core principles that have guided us since QuantStack’s inception in 2016, including the subject of long-term consistency and commitment in open-source projects, and how it can be a double-edged sword for maintainers and contributors.
The people behind the projects
It is well-known that the success of large open-source projects often hinges on the dedication of a small distributed group of individuals. At any given time, a small team of maintainers are working in a sustained fashion on the project, while a larger group of regular or occasional contributors lend their expertise. Additionally, there’s a long tail of contributions from those who have encountered issues while using or extending the project and took it upon themselves to improve it. The same can be said for contributors who work on the organizational aspects of the project, either professionally or as volunteers, in the long run, or for a single event.
Projects like Jupyter, with millions of end-users worldwide, are a testament to the power of this model. It’s hard to imagine a proprietary software application, used at such an extensive scale, thriving with such limited resources. In open-source projects, this is often compensated for by the overinvestment of maintainers who remain engaged in the long run.
Working on impactful projects is a privilege. It brings a unique sense of purpose that can be dreadfully missing when working on projects that only impact a few colleagues within an organization. This sense of purpose from working on large open-source projects is a strong driver behind the engagement of contributors.
Unfortunately, it also places a significant burden on the shoulders of a few individuals who feel responsible for the large community of end-users that rely on open-source software. They may feel trapped, unable to move on like they could with a traditional job. The demands on issue trackers and critiques on social media from both users and other community members can be crushing as well.
On the other end of the spectrum, and sometimes as a reaction to maintainer burnout, some open-source developers are taking the opposite approach by distancing themselves from the idea of responsibility to those who depend on their work. While it’s important to be able to walk away from a difficult situation, my philosophy so far has been to acknowledge responsibility for the work that we put out into the world. Besides, as a business owner, I also have a commitment to the open-source maintainers who have joined us in this endeavor.
Long-term consistency
The Jupyter project has a large scope. While a single dedicated developer can bring about significant progress over time, certain challenges require the concerted efforts of several maintainers for a few months. Teams like that of QuantStack have been well-positioned to tackle this kind of endeavor.
One noteworthy project that required such a team effort was the creation of the JupyterLab visual debugger. This task involved modifications to the kernel protocol, the implementation of the modified protocol in a Jupyter kernel, tweaks to the Jupyter server, and the development of the debugger’s frontend in JupyterLab. The project’s scope spanned multiple repositories and areas of expertise, and crucially, it required the community’s consensus on the proposed core protocol changes. To ensure that we could deliver a functional debugger within the timeframe expected by the organization funding us, it was essential to make this project a known quantity in terms of software engineering and social undertaking.
To achieve this, we decided to build the frontend as a standalone JupyterLab extension. Simultaneously, we embarked on the development of a new Python Jupyter kernel from scratch, named xeus-python, which would be capable of supporting debugging. This approach allowed the community to engage in discussions and reach a consensus on debugging at a suitable pace while we delivered on the first iteration. We then submitted a Jupyter Enhancement Proposal (JEP) backed by our implementation, which was ultimately accepted. Consequently, the debugger frontend was incorporated as a core component of JupyterLab, and we proceeded to enable debugging support in the main Jupyter kernel, ipykernel.
The entire process, from the initial prototypes, the first release of the debugger extension a few months later, and then the JEP process, to the incorporation of the debugger in the core codebases took about two years. However, this was only the beginning of a second phase in which team members regularly came back to the project to consolidate these developments and improve user experience. What the story also fails to mention is the considerable amount of skepticism, and sometimes suspicion and resistance from community members, including some accusations that we were attempting to deceive them.
When you embark on the next phase of your project or seek backing for an upcoming milestone, it’s essential to understand that public validation or support may not be forthcoming. Progress comes from continuous and long-term commitment, and the reward is the fulfillment of seeing end users benefitting from your efforts. Cohesion, mutual support, and trust in your team are also key to making the journey worthwhile in the long run.
Notebook v7
The path towards the release of “Notebook v7” is a prime example of long-term consistent commitment. For those who may not be aware, Version 7 of the Jupyter Notebook, which was released in 2023, is a complete overhaul of the notebook project. The new notebook reuses components of the JupyterLab project to create a new version of the classic notebook UI. Notebook v7 and JupyterLab nearly operate from the same codebase.
This release introduced many improvements to the notebook UI, including:
- support for the JupyterLab visual debugger in the notebook,
- theming and dark mode,
- internationalization through installable language packs,
- collaborative editing,
- accessibility improvements, most notably through the migration to CodeMirror 6.
Releases of the notebook package now closely follow those of JupyterLab, immediately benefiting from improvements made to the JupyterLab codebase. JupyterLab extensions, such as mime type renderers, widgets, or more sophisticated frontend extensions, can also work in the notebook or be adapted to do so with minimal effort. The notebook maintenance burden has been significantly reduced as the Jupyter frontend codebase has been consolidated into a single project.
The development of Notebook v7 started with a prototype of a JupyterLab-based notebook UI by Jeremy Tuloup, then called “retrolab”. At the time, the classic notebook codebase was lacking maintenance and was built upon unmaintained technologies. This situation sparked challenging discussions about the future direction of the repository. A proposal was eventually made for the next major release of the notebook (Version 7) to be based on the retrolab codebase and the JupyterLab 4.0 release. This proposal took the form of a Jupyter Enhancement Proposal and included a pledge to transition some of the well-liked notebook extensions that were not yet compatible with JupyterLab to Notebook v7 and JupyterLab.
Once the proposal was accepted by the projects’ governance, we took it upon ourselves to make Notebook v7 a reality. Despite having no funding for this endeavor, we put significant effort and time towards this goal in both 2022 and 2023. In 2023 alone, QuantStack spent over 1200 work-days on unfunded Jupyter work. Roughly 40% of that time was focused on the Notebook v7 plan, including the port of nbgrader to JupyterLab and accessibility improvements following up on the upgrade to CodeMirror 6. (Other key areas towards which unfunded work was done on the Jupyter project are: JupyterLite development, JEP reviews and implementations, UX research and user tests, and volunteering on the organization of JupyterCon 2023.)
After over two years of sustained development, the update from Notebook 6 to Notebook 7 was seamless for most users, who found the same familiar UI in the new version. This smooth transition was a testament to the power of consistent long-term commitment. QuantStack, which is home to many Jupyter maintainers, was acknowledged as an official sponsor of the project on the Jupyter website.
Candyce, Capytale, and JupyterLite
Jupyter has been widely adopted in higher education since the early days of the project. The project is credited for being a successful implementation of the concept of “literate computing”. The ability to integrate a narrative into computational contexts has proved especially useful to teachers and students.
Get Sylvain Corlay’s stories in your inbox
Join Medium for free to get updates from this writer.
In the French education system, the Paris school district (Académie de Paris) pioneered the use of Jupyter notebooks for high school students. Initially, the project required launching a container on the server side for each user session, which is a common practice in Jupyter deployments. However, this approach posed significant scalability challenges, as it demanded substantial cloud resources to support the user sessions of tens of thousands of students and teachers.
Recognizing the need for a more efficient solution, the project, aptly named “Capytale”, pivoted to adopt a different execution model. This innovative approach was championed by Romain Casati in his visionary “Basthon Notebook.” By combining the classic Jupyter Notebook with Pyodide (a WebAssembly Python distribution), Basthon enables client-side execution, thereby liberating Capytale from the requirement of providing a container for each user session. The remarkable scalability and reliability of this execution model allowed Capytale to expand its reach by two to three orders of magnitude.
Around the same time, in late 2020, I was contacted by the leadership of Inria, the French “Institute for Research in Computer Science and Automation”, to help put forward a proposal for a national Jupyter-based teaching and research platform. With substantial funding (€12m) from a “Plan d’Investissement d’Avenir (PIA)”, one of the post-covid investment initiatives, this ambitious project aimed to create a product-level platform that would cater to both high school and college education. The Capytale, QuantStack, and Inria teams would join forces for a unified platform, named “Candyce”. We were soon joined by Université Paris-Saclay and France Université Numérique (FUN) in the project.
Our team dedicated a significant amount of time to crafting the proposal, which spanned hundreds of pages. It included detailed mockups of proposed user interfaces and integrations with learning management systems, as well as references to several technical prototypes that we had developed to support the proposal. One of these prototypes, spearheaded by Jeremy Tuloup and later joined by Martin Renou, Thorsten Beier, and Nick Bollweg, was JupyterLite. The goal of JupyterLite was to bring the same client-side execution model as Basthon to the JupyterLab and Retrolab user interfaces. We put forward a very ambitious plan, including a major focus on accessibility, LMS integration, learning analytics, security, and continuous improvement based on user testing.
As our energy began to wane, Nicolas Thiéry (Université Paris-Saclay), who had earlier been the Principal Investigator on the OpenDreamkit EU grant, managed to bring a renewed sense of enthusiasm and momentum to the project. He spent a large amount of time reworking and refining the proposal based on the feedback from the administration, bringing us over the finish line. The project appeared to have succeeded when it was announced in a press release by the French prime minister that the project had been funded.
However, our triumph was short-lived. Inria soon informed us that the project’s funding vessel did not allow for private companies to be involved as partners. All references to QuantStack had to be removed from the proposal. Although there was a chance that a public tender could be opened, our involvement in putting together the project would make it challenging for us to apply. Eighteen months passed, and it became generally understood that the project would not come to be, despite never receiving a formal notification.
In the meantime, the Capytale project was adopted more broadly, and we shifted our focus to assist the Capytale team, including Basthon’s author, Romain Casati, and the main Capytale project manager, Nicolas Poulain, in transitioning to JupyterLite. The primary advantage of JupyterLite over the classic notebook upon which Basthon is currently based is that it is built on the JupyterLab codebase, benefiting from all the accessibility improvements in the notebook UI. With half a million registered high school students, Capytale’s scale of adoption makes accessibility a requirement. The ongoing (unfunded) collaboration between the Capytale and QuantStack teams will eventually make it possible for many students who have disabilities to use the Capytale project. This is an area in which we need to collectively acknowledge our responsibility towards the people using our work.
The fact that Jupyter is the primary tool for coding education in France, my home country, is a great source of pride. Despite the disappointment with the Candyce project, we continue to apply our principle of long-term consistency to our work. Funding open-source work can be challenging, but I am convinced that the involved government agencies will understand the need to have an ambitious plan around these technologies. We will collectively continue to explore opportunities.
Bringing people together
I am firmly convinced that face-to-face, in-person communication is essential for fostering a sense of community and establishing trust beyond immediate work-related relationships. After all, open-source projects and communities are composed of people, and to move forward, we must be able to connect on a personal level.
Since its inception in 2016, QuantStack has been dedicated to bringing people together through events and workshops. I took the initiative to establish the PyData Paris meetup group in 2017, and have since organized or co-organized twenty meetups, attracting over 4,500 members. We are always seeking new venues and speakers. We invite you to join our community if you are interested in such events in the Paris area. In the realm of Jupyter, we have been actively involved in organizing community workshops, starting with the inaugural workshop on Jupyter widgets. Since then, we have organized workshops on various topics, including the notebook format, Jupyter kernels, the use of Jupyter in education, JupyterLite, and dashboarding with Jupyter. After the community workshop program was discontinued, we carried on with Jupyter Studio Days and workshops.
Building a critical mass takes time, and our principle of long-term consistency in this commitment is key to the gradual process of building a sense of community.
In terms of large-scale conferences, we have also played a significant role in organizing JupyterCon in both 2020 and 2023 (serving as vice chair and general chair, respectively). The 2020 edition was abruptly jeopardized by the global pandemic and an initial financial setback due to the cancellation of the physical venue we had booked in Berlin. Under the leadership of Lorena Barba, we shifted to an online event, with QuantStack investing resources to provide attendees with readily accessible JupyterHub deployments, both with and without GPU computing for tutorials. Despite these challenges, the 2020 edition was ultimately profitable. The 2023 edition was highly regarded, featuring top-quality keynotes (Paul Romer, from NYU, Economics Nobel Laureate, and Alyssa Goodman, from Harvard University), and high-quality presentations. While sponsorship numbers were very good, attendance was lower than expected, and the event incurred a sizable financial loss.
This year, building from the experience of organizing in-person events in Paris, we are organizing PyData Paris 2024 in late September, which will be held at the same venue as JupyterCon 2023. We partnered with NumFOCUS to use the PyData trademark in the context of the conference in 2024, making QuantStack the legal organization backing the event and taking responsibility for all associated costs. We are making a donation to NumFOCUS to support their continued work. We hope that PyData Paris 2024 will mark the beginning of a long series of events, each one celebrating and strengthening the scientific open-source community.
NumFOCUS
An organization that has been instrumental in promoting and supporting scientific open-source projects in the long run is NumFOCUS, a non-profit organization based in the United States that is dedicated to promoting and supporting the development of open-source software for scientific applications.
I have had the privilege of serving on the board of directors of NumFOCUS since late 2018, and my tenure will conclude with the upcoming election of new board members. During my time on the board, I have been struck by the remarkable reliability of my fellow board members, consistently achieving quorum and engaging in numerous additional calls and working sessions beyond the scheduled meetings. This high level of participation is particularly remarkable, considering the challenges that many volunteer-based organizations face in maintaining long-term engagement.
In my opinion, not only the board, but also the leadership of the organization have embodied this principle of long-term consistency in their commitment, even in the face of challenging personal circumstances and public criticism on social media. Progress and growth are steady and the work pays off.
Without delving into recent debates, I believe that this resolute dedication and reliability are the key characteristics that we must seek in an organization serving as a pillar of our open-source community. I am utterly convinced that they will still be at work a decade from now.
Ongoing long-standing projects
This post has already covered a great deal of ground, and yet we have only scratched the surface of our decade-long commitment to open-source. There are many more ongoing initiatives in the projects that we have been steadfastly supporting over the years.
One of the most significant endeavors in terms of size is the push for collaborative editing in Jupyter. This project involves several full-time developers and is a prime example of an undertaking that requires long-term focus. Although much of our work on collaborative editing is currently funded through client contracts, a challenge is to deliver value to users early on while also ensuring that we make the right decisions for the future.
Collaborative editing made us collectively more productive when it comes to word processing. By pushing collaborative editing features in JupyterLab, we start exploring new possibilities towards enabling such enhanced productivity in other areas, such as CAD, GIS, and many more. I am convinced that collaborative editing workflows will have an even bigger impact for building much more complex things than documents. Building a football stadium or a jet engine are collective and social endeavors, and enabling collaborative workflows for these kinds of projects will help us take advantage of such features at a new level.
Beyond the Jupyter ecosystem, we remain deeply committed to several other projects, such as mamba and conda-forge. With Mamba 2.0 around the corner, we will see another major milestone in the area of package management. Mamba 2.0 will help conda-forge to not be completely dependent on a single service for package hosting by enabling support for mirrors and multiple storage backends.
Smaller projects spearheaded by QuantStack, such as Xsimd, have met significant adoption in the past two years, with Firefox and Apache Arrow using it for SIMD acceleration. We just signed a three-year partnership with “Fondation HPC” to push on the Xtensor and XSimd software stack for high-performance computing applications. Stay tuned to hear more about this!
Final remarks
The tech community often appears to be swayed by hype, which can create the illusion that progress happens in sudden leaps. I firmly believe that the reality is quite different. Progress is more often the result of steady, persistent efforts to push technologies beyond their tipping points, which is when our collective attention focuses on the latest exciting development. Great things happen when we continue focusing on our goals, without shying away from hard problems.
There is a strong multiplicative effect in open-source communities, especially when people care about the quality of the technology that they help create. Regardless of your level of engagement as a contributor, whether you are doing it professionally or as a volunteer, if there is this small thing that you think is not quite right, that you can get back to and fix, please do it! It could make the difference in unlocking someone else building upon the project. This collective attention to continuous improvement is what allows projects like ours to reach new heights.








