Building Blocks for AI in Jupyter: A Modular Approach

[Zsailer]

I'd like to develop and coordinate a strategy for introducing artificial intelligence into the Jupyter Notebook authoring experience. Our goal should be to create modular, reusable building blocks that can be combined to create rich AI experiences while keeping the ecosystem lean, fast-moving, and on the cutting edge.

Philosophy

Rather than building monolithic AI features, we want to create discrete, well-tested components that can be:

• Released and versioned independently
• Combined in different ways by different projects
• Contributed to in parallel by different teams
• Easily swapped out or extended

Think of this as the "Unix philosophy" for AI in Jupyter: small, focused tools that do one thing well and can be composed together.

Proposed Building Blocks

Browser Client Components (JupyterLab/Notebook)

Chat Interface & UI Components:

• Core chat interface widget
• Tool call approval/rejection UI
• AI execution stop/interrupt controls
• Available tools display/browser
• Tool execution status and arguments viewer
• AI-generated todo/task list UI
• Cell diff viewer (git-like) for AI changes
• AI action trigger buttons/shortcuts
  • Example: https://github.com/jupyter-ai-contrib/jupyterlab-magic-wand
• Conversation history management
• Multi-agent conversation UI

What other client-side components do we need?

Jupyter Server Components

AI Integration Layer:

• Model/agent integration framework
• AI workflow orchestration
• MCP (Model Context Protocol) server interface
• Extension tool registration system
• Server control tools (kernel management, monitoring)
• Authentication/authorization for AI services
• AI session management
• Tool execution sandboxing

What other server-side abstractions are missing?

Kernel Integration

Runtime Introspection:

• Variable inspection
• Namespace introspection tools
• Code execution context sharing
• Kernel-to-server communication protocols
• Runtime state snapshots
• Interactive debugging interfaces

What other kernel-level capabilities do we need?

Cross-Cutting Concerns

• Security and sandboxing frameworks
• Telemetry and observability
• Configuration management
• Plugin/extension APIs
• Testing frameworks for AI components

Implementation Strategy

1. Separate Repositories: Each building block should live in its own repository under the jupyter-ai-contrib organization
2. Independent Releases: Components should be versioned and released independently
3. Lean Implementation: Each component should do one thing well
4. Reference Implementation: jupyter-ai becomes an opinionated distribution that combines these components
5. Community Contributions: Clear interfaces allow parallel development

Getting Started

We're looking for contributors to help identify, design, and build these components. If you're interested in working on any of these building blocks:

1. Comment below with which component(s) interest you
2. We'll create dedicated repositories in jupyter-ai-contrib
3. Start with a clear specification and API design
4. Build, test, and iterate

Questions for the Community

• What building blocks are we missing from this list?
• Which components should we prioritize first?
• What are the key interfaces between these components?
• How do we ensure these components work well together while remaining independent?
• What security and safety considerations should be built into each layer?

Let's divide up the work and build the future of AI in interactive computing together! 🚀

---

This discussion aims to coordinate the development of modular AI building blocks for Jupyter. Please share your thoughts, volunteer for specific components, or suggest additional building blocks we should consider.

[ellisonbg]

Zach and I talked about this earlier today and I am fully on board. Thanks for writing it down Zach!

[Zsailer]

Thanks, Brian!

[3coins]

Thanks for putting this together. Should we first work on decomposing the jupyter-ai components into one of these repos?

[Zsailer]

Hey y'all! Yeah, I think so. I think we should develop a plan for how we can decompose jupyter-ai into multiple repos/packages for tighter iteration on multiple components. We can do this in a separate discussion thread. Let's keep this thread focused on defining what the "blocks" are.

My hope is that this decomposition with actually help jupyter-ai distribute some of the pressure and load to release.

[KirstieJane]

Thanks for sharing this @Zsailer!!

I'm following along because at the moment I think the only person on UC Berkeley campus who is thinking about this work is @fperez and that feels like a risky contribution / communication pathway 😉

I think @cboettig might be interested in following along too...

Really looking forward to seeing how this design work unfolds!

[fperez]

Thanks so much for writing this down, @Zsailer, and agreed with @KirstieJane that I should NOT be the pathway for this :) In addition to Carl, from the Berkeley team I'd also flag @ryanlovett @minrk @balajialg.

On the technical side, one thing that may need additional granularity is the problem of sandboxing. I don't think we should be looking at sandboxing only as a tool execution question, as prompt injection attacks continue to show (just one recent example among many how subtle and pernicious they can be.

I think a deeper sense of isolation for all "project" activities is going to be necessary as we start having agents with API access to a UI as rich and powerful as Jupyter's. I'm not sure at all what that will look like, but I get a sense that simply trying to isolate tool execution won't be sufficient.

[krassowski]

What other client-side components do we need?

• inline completion (we already have it)
• a way to hook into input boxes like for generating git commit messages
• inline chat in editor
• quick edit suggestions
• whatever else is there in LSP specification really as this shows solidified features that work for human-automation interface and are used by other editors to integrate AI

[krassowski]

@mbektas what extension points would have made it easier for you to develop https://github.com/notebook-intelligence?

[jtpio]

Jupyter Server Components

All frontend components should be designed in a way so they work without Jupyter Server, or with a different Jupyter Server implementation.

For example by isolating the parts that makes requests to the server in a different plugin (which is the case for https://github.com/jupyter-ai-contrib/jupyterlab-magic-wand).

This will help frontend driven use cases such as https://github.com/jupyterlite/ai.

[ryanlovett]

Thanks for opening this discussion. In case you're looking for use cases, jupyter-a11y-checker (@Chanbinski, @balajialg) uses AI to help authors add alt text to their notebooks. It'd be useful if it could tap into modular AI machinery built by the community rather than using custom code in the extension.

[dlqqq]

@Zsailer Thanks for posting this new exciting direction for Jupyter AI. A few high-level questions:

1. The jupyterlab/jupyter-ai repo has a monorepo configuration. We can fix our release flow to allow separately-versioned packages. Given this, does it still make sense to use separate GitHub repos for every sub-package?

2. Who will be directing the work across all of the different packages? We will need to align on what PyPI & NPM packages we want and what the expected behavior of each package is.

3. How do we plan for this to affect the official release timeline? Our JupyterCon 2025 talk has already been accepted, and I'm wondering if we should plan to cancel that talk & instead focus on development.

[Zsailer]

1. ...
   I think separate Github repos is still important. This allows us to manage merge + release rights for different parts. For instance, we might want to give someone merge rights to a core AI provider interface package, but not to packages that handle sensitive user data.

The monorepo creates scenarios where we might want to give someone merge rights to a part of the project, but not all parts. In that case, we often err on not giving those rights.

Separate repos also enable independent CI/CD pipelines and clearer issue tracking per component.

2...

I think we should operate like a Jupyter subproject, even though we aren't an official one (yet).

We should bootstrap a council of people who can lead the effort and adopt a team-compass approach. I propose we identify 3-5 initial council members from current active contributors and borrow a lightweight governance document from e.g. jupyter-server.

This gives us a trial run at governance before potentially becoming an official subproject. Then, if we decide to become a subproject, it's an easy transfer.

3...

Maybe we can repurpose your talk to be about how we're creating a place for people to collaborate on developing the building blocks for AI in Jupyter. jupyter-ai will be repurposed to ship an experience from these blocks.

This could be perfect timing to announce the new architecture and invite the broader Jupyter community to contribute to specific building blocks that match their expertise. We can demonstrate how this modular approach enables faster innovation.

[nsingl00]

Hey @Zsailer Thanks for writing this down. This is super exciting. Would like to work together on this.
Is it worth to mention *AI Agent Memory and state management within Jupyter Server* as well?

[jasongrout]

For client-side, I would suggest a component that is providing API-style access for extensions to programmatically query an agent. The components listed above focus on the UI around a particular set of user workflows (for example, centering on chat and agent work), but I think we should also have programmatic access to the server apis so any other extension out there can query an agent. This may be implicit in how you design this functionality in layers, but I think providing a programmatic api for other extensions should be an explicit design goal.

[SylvainCorlay]

Many many thanks @Zsailer for posting this. I strongly support the idea of building modular components, released independently, and assembled in different ways. This is a fast-moving area and we should avoid having a bottlenecks on a single repository.

[tonyfast]

• What building blocks are we missing from this list?

😢 accessibility is missing from the list! there have been so many opportunities to shift accessibility to the left and they've been passed on. maybe this time? adding more access to AI stuff is not as inclusive as it sounds when the jupyter interface is fundamentally broken for AT users.

• Which components should we prioritize first?

accessible cells and notebooks MUST exist. there has been little movement on making the notebook itself accessible. there have bits and pieces of the interface made accessible, but overall the accessibility of the notebook interface in lab and stand alone is broken. until this fundamental component is accessible any other feature added will be a privilege for those who access needs are already met.

• What are the key interfaces between these components?

a sizable minority of your potential users may use assistive technologies, and focusing on left shifting accessibility could improve the fidelity of the interface between jupyter and its assistive technology users.

• How do we ensure these components work well together while remaining independent?

from an accessibility perspective, there need to be automated tests, which means a test engineer, and manual testing by affected users who use assistive technology in their daily life is required. Fable is a service that could provide this support.

• What security and safety considerations should be built into each layer?

continually neglecting accessibility effects the personal security of assistive technology users. it is inequitable for them to experience second/third tier access to the jupyter while others are benefit without extra labor. inaccessibility of jupyter means many AT users can't take classes using jupyter, can't have jobs using jupyter, and can't benefit from the same literacy privileges current users have. it would be a real shame if they can't use jupyter's AI tools too because that would exclude a lot of people from modern tech employment. if AI is to be so much of jupyter's future it needs to include disabled folx.

[Zsailer]

@tonyfast thank you for calling this out. I'm going to ask around my organization to see if we can devote some attention to this.

I wonder if we can leverage the funding opportunity here to chip away at making accessible notebooks and cells: https://jupyterfoundation.org/community-funding-proposals/

[tonyfast]

thanks for noticing the work, it takes energy. it would be really great if some larger organizations could devote effort to the accessibility. it will take resources.

i'm pretty sure that the notebook accessibility work is in scope for that funding. there are several external accessibility efforts like notebooks for all that have worked with and designed with disabled users. we've learned a lot and have solutions for an accessible, however integrating these ideas into jupyter will take work and resources to organize. i don't have time before the deadline to submit an issue for this funding, but i'd be down to collaborate with folx on one if it got started.

[Zsailer]

Thank you to everyone who has added to this discussion. I will try to synthesize this feedback and add it to the list above (unless someone else beats me to it 😃 ).

[choldgraf]

I just want to give my support to @Zsailer 's framing here. It feels like this is an opportunity to align Jupyter's efforts around AI with the broader design philosophy that has made it impactful over the years: composability, modularity, customizability.

I also suggest that we use interactive and human-centric computing workflows as a guiding function to prioritize what to do work on, because it's the unique space where Jupyter has built its expertise and brand over the years (rather than trying to re-invent every wheel in the AI space just because we didn't invent it).

Let's create the building blocks that allow hundreds of individuals and organizations to experiment and innovate with AI and interactive computing.

[KirstieJane]

@tonyfast @Zsailer - I've opened up jupyter-governance/funding-proposals#10 (Accessible AI: Articulating a roadmap to ensure JupyterAI enables data scientists who use screen readers)

(Sorry - I clicked send too soon 🤦‍♀️ - but maybe that's good - I'll go add some thoughts over in the funding proposal! Please come and join me if you have any time to just dump some thoughts! Or find me on Zulip!)

[KirstieJane]

Ok - more details in jupyter-governance/funding-proposals#10 (comment)

I'm going to be bitty online today - but I will check in with Liz and see what she thinks - this feels important enough to get in front of the EC even if we don't get the best proposal pulled together by Sunday night!

[kevin-bates]

Hey @Zsailer - thank you for putting this together - this is really fantastic! I'd love to be involved with the server-side aspects of this, although I'm still unsure regarding my level of contribution (new gig).