Skills are hard. (Part 2)

A robot interviews a human while being observed by other robots.

I’ve interviewed a lot of engineers.

In this blog, I’ve written mostly about the intersection between software and security, but I’ve spent more of my career in some software engineering leadership position than as a security professional. Thankfully, in those situations I’ve been given the opportunity to hire, and along the way I’ve seen some great interview techniques and some really awful ones used by those helping me.

Something which I frequently employ in my interviews is depiction of a familiar scenario that I expect someone in this role to encounter. Think of it as inverse-STAR: I define the situation (with some information hidden on purpose), I give the task, I roleplay responses based on the actions voiced by the interviewee, and I review the end result with the interviewee to confirm if they see something they missed. In turn, I see what they know based on a realistic situation, using events that trigger their tacit knowledge, and considering what things they appeared not to know despite my probing.

When it comes to the problem of constructing a context harness that can be used for an organization’s software engineers to write good code with AI, I’ve used my experience in the above to shape a structure for how it could be captured and provided. Here, I’ll define “good code” as code that reflects the principles and conventions of the software engineers in a company, as well as reflecting language used by the company when it talks about its operations. Code should be easier to maintain when written this way, and thus should help your company make progress towards that 100x goal I talked about in my last post. In this post, I want to share some lessons learned from this experience, some folks in the space I’m tracking related to this, and where I think we have a ways to go.

First, I want to give a shoutout to @rohit4verse for this excellent post titled “The Harness Is Everything: What Cursor, Claude Code, and Perplexity Actually Built”. Written in March 2026, it summarizes how LLMs process information, then goes into detail about what purpose the “harness” serves that wraps your interactions with the LLM underneath. Near the end of the post is this quote:

The harness is everything. The model is the reasoning engine. The harness is the context, the constraints, the feedback loops, the memory, the tools, and the scaffolding that determines what the reasoning engine can actually accomplish. Getting the harness right is not a prompt engineering problem. It is a systems engineering problem. And it is the most important engineering problem in applied AI right now.

I would agree, and I’ve been learning about how best to do this since the time he posted this. Around the same time, I attended the [un]prompted conference in San Francisco, and my greatest takeaway was that if I couldn’t help my current company build this harness well, then our software engineers would never be able to keep up with the pace of vulnerabilities being identified and exploited. As a result, we’d be breached by a 0-day before the end of the year.

I knew we could assemble a pipeline that used AI to process findings into useful vulnerability reports, but I didn’t have confidence in our ability to fix them with automation in a maintainable way. A slop PR (as in one that isn’t steered at all by developers in a company) could probably fix most simple vulnerabilities. But a dozen PRs for a dozen individual holes is always going to be harder to manage and subsequently maintain than one PR for a thoughtful refactor that follows your principles and conventions and fixes everything with a design change. A dozen slop PRs won’t get reviewed (there’s an attention limit) and kicks the maintainability can down the road; a single thoughtful PR has compounding value.

So how do we capture these principles and conventions to make maintainable code? One way is through something I’ve called “The Brain Extraction Ritual”. This operation is performed on subject matter experts in my organization in order to transcribe their existing expertise into an AI doppelganger of themselves. These in turn are summoned to support other engineers by applying this expertise when an engineer’s agent takes a prompt and starts developing a solution. Maybe another time I’ll cover the full architecture I’ve defined for this. For now, I will focus on the salient bits of the design and the extraction ritual itself.

The Brain Extraction Ritual is part of a larger system built to capture the collective knowledge of a company. When asked to design how a “panel of experts” could help engineers write good, maintainable code with their agents, I figured that it’s hard to do this with only the perspective of specific software engineers. These engineers are themselves shaped by those they report to and the projects assigned to them. Those in turn are influenced by still other forces inside and outside the company and the operations being performed there. These things change over time and at a different rate than best-practices for software engineering in general. Thus, I believe it’s important to step back and look at the whole problem from the perspective of domain-driven design and Conway’s Law while considering the organizational structure that actually exists. As a manager, I’ve frequently played with these to inform the structure of the software I wanted to see built. The systems modeled this way always had staying power.

For example, take a look at this illustration of various organizational structures at major tech companies:

From https://bonkersworld.net/organizational-charts

Looking at Microsoft’s organizational structure, if they all wrote software, Conway’s Law would expect those three major business units to produce three disparate systems that mostly don’t talk to each other. When they do, it’s probably due to some operation that Microsoft itself has forced upon them all equally, and they’re each more likely to perform the same operation independently than trust each other to handle some component of it before handing it off to another for completion.

I know that organizational structures vary from company to company and will change over time. I know that the ubiquitous language used in a company to describe what it does will vary over time. In-between, companies have leaders responsible for domains of operation that provide value and set norms for what they do and how they communicate about it. So, if I want to capture how a company currently communicates and conducts its operations, I need to construct a model of these leaders in an organization using a group of AI “expert” agents. I want to capture the vision, goals, principles, conventions, and references used by these leaders into prompts for these agents. I want to listen for who the experts reference collaborating with when approaching specific problems. I also want them to have connections that reflect actual organizational structure, so that their feedback remains aligned with the leadership over them.¹ The result should be a graph of interlinked experts, with connections that reflect how the company actually organizes around work.²

Separately, a graph of specific knowledge sources may be extracted and maintained, used when an expert mentions a specific source (node) of knowledge. This may also be queried directly. Examples of this range from external reference documentation, an internal architectural design doc, or a specific code reference seen as a good example of a pattern. If done well inside a place where a company does work, the ubiquitous language here will match that of interviewed experts, helping you stitch references together.

When a prompt comes in that uses this system, the planning phase for building a solution extracts key terms (ubiquitous language) from the prompt, starts with the expert at the heart of the problem, and walks the expert graph from there to derive high-level concepts about how to approach the problem. References to what those experts would consider along the way are included, pointing at the knowledge graph for when it’s time to go into detail. Later, if anything unexpected comes up, the same experts can be queried, and this system does its best again to respond with what is already known. Novel solutions will still be left to the developer driving the project, but this system hopefully frees up their capacity to focus on the novel while the known is accounted for in the work of their agent.

This system relies heavily upon capturing knowledge (expertise) and keeping it up to date. As I’ve discussed in my last blog post, however, this is potentially impossible to do completely. To illustrate why this is with another example, let’s walk through the first iteration of the Brain Extraction Ritual I had in mind.

The Brain Extraction Ritual looks like a multi-phase interview. It expects two to three individuals (two interviewers and a subject matter expert) to meet over at least two hours. During the first phase, the interviewers record the subject matter expert (SME)’s responsibility, goals, and principles. This defines the domain they work in and some values they think are important to have when working within it. Then, it tests, develops, and refines these by looking at different scenarios in subsequent phases: a greenfield development project, a recent code review, a recent bug fix, and a recent incident. Invariably, the subsequent phases will identify gaps in what was initially mentioned, so long as the interviewers are paying attention and asking clarifying questions. Breaks are given between each phase, because there’s a lot of ground to cover and everyone needs to stay focused for the captured knowledge to be accurate and, in turn, valuable.

This process quickly showed some problems. First, getting two to three people together for this long was difficult. Second, the skill of the interviewers was important because it helped them ask good clarifying questions. Third, the output of the interview needed to be consistently structured so it could be easily incorporated into the corpus of experts and knowledge, but this itself is an endeavor. Fourth, even two hours is inadequate to capture much, thanks to the sheer volume of expertise that our human experts can actually share through varying scenarios.

The next iteration of this tackled points 1–3 through creation of a /brain-extraction skill that was designed to be run by Anthropic’s Fable model. Fable’s good at identifying vagaries and defining boundaries, and with its large context window, it’s able to stay coherent over a long session. It was good at making up greenfield scenarios that were relevant. With a sub-skill, it can also consistently structure the knowledge captured (though it sometimes misclassifies things). It was also pretty good at reviewing its own notes and asking clarifying questions when inconsistencies were detected. Lastly, because the skill was written for use in both Claude Code and in Chat/Cowork (with some differences for each mode of communication), the human expert could conduct the interview whenever they had time with their preferred interface.³

This allowed more people to contribute in parallel, but we still have a comprehensive coverage problem. This goes back to the issues of tacit knowledge discussed in my last post: we know more than we can tell. But hey, this is 2026, and loops are all the rage again, so we figured we could eventually solve this problem with a loop.

The loop: while developers are writing code and using this system, capture the agent session’s reasoning outputs and tool calls. In places where an agent has to churn through a lot of tool calls and reasoning steps to determine what should be a valid solution, we can infer a gap in the corpus of expertise. So, feed these sessions into a data lake, run a process on the data that extracts these gaps, and package the result into a consistent structure (situation, original task, actions taken, resulting code). Then, tag these by the domain of expertise they’re closest to, and forward them to the human experts to review. The real experts can choose to ratify the finding as good, thereby incorporating it into the corpus. Or, they can disagree and provide their own take.⁴

Thus, we remove the need for experts to be aware of their tacit knowledge; we observe where gaps are, then challenge them to explain what they can. It’ll also be self-improving, as when issues occur, they’ll be detected (again through multiple inference cycles by the coding agent since it’s been led astray), forwarded for human review, and rectified over time. Through this loop, we could theoretically reach a point where 100% of coding problems being solved by AI agents are aligned with the principles, conventions, vision and goals of our domain experts and the company at large.

The Brain Extraction Ritual is an answer to how to effectively capture relational knowledge that our company’s experts have, and hopefully along the way it’s capturing some of the collective knowledge that is localized to the organization they work in. Theoretically, 100% of what knowledge can be captured will be captured, and as a result the AI-generated code will be better than ever because it’s always enriched by the knowledge of the organization’s experts. Novel problems can safely be the focus of every engineer, and known pitfalls will be avoided because of the support of codified expertise.

Get Christopher Blanco’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

I still think it’s impossible, at least without another dimension of observation and some way to internalize the knowledge. For starters, there will be constant conflict with all the viewpoints of all experts involved in responding to a prompt; “too many cooks in the kitchen” is a real problem. It’s exacerbated if you emphasize the input of higher-level leaders’ agent-experts, as they could suck away attention from relevant nuance or speak in intangibles that confuse the coding agent. Even if you try to artificially limit the number of experts from your graph to consult, you could have an overwhelming number of details to consider.

This enters into the next problem, which comes from the sheer volume of context necessary to reason through at arguably every step of the development process. If you’ve captured all the expertise you can capture, then theoretically you’d benefit from the experts advising every time the coding agent needs to stop and infer what to do when it hits a roadblock. This could provide far more than what’s possible to inject from nested AGENTS.md files, because the experts could recognize the situation and share relevant knowledge to guide the agent out of a pit. This should be better than the flat, generic knowledge in your .md files or deterministic quality checks. But, this could be hugely expensive in terms of time and tokens, even if your experts are Haiku-class models. Later, during code review downstream, what quality bar will be good enough to pass through each expert? The potential churn here as each expert gives an opinion on their areas of concern could likewise blow up cost.⁵

Mapping things back to Rohit’s post about models and the harness, all of what we’ve built in this system is a very big, very thorough harness. It’s trying its best to cover everything the company does so the reasoning engine (the underlying LLM) always writes software like a company expert. Unfortunately, time, money, and hardware limits the depth of this harness’ coverage and the frequency of its application. What I think is necessary are some intermediate layers between the LLM and its explicit knowledge harness, and observation capacity provided by ambient AI presence in where work is communicated. Not all of this should be the responsibility of each company to implement. Otherwise we’ll keep hitting these walls.

Let’s go back to tacit knowledge and how an apprentice learns skills from a master. They don’t work with a master breathing down their neck every step of the way, forever. Over time, they observe the master’s behavior and internalize lessons from their own experiences. This grows their tacit knowledge. Later, through application of their skills in different contexts, perhaps under different masters, in different companies or indeed even different societies, their tacit knowledge grows. At some point, they’re considered proficient enough to move from apprentice to journeyman status, and eventually they may be trusted to train others as a master (or even become an influencer of other masters).

This process takes years. We shouldn’t each try to do it for everything that we count ourselves experts in. Furthermore, as described in my last post, you probably can’t write it all down and convey it adequately. Companies shouldn’t need us to. Companies need agents that are able to reliably execute specific skills that serve their operations, like coding; this is why interviewing happens, not investment in company colleges.

More important for companies is code generation that reflects their context: the ubiquitous language it uses, how they organize around solving problems, the principles and conventions used when they do, and changing these over time to reflect the current reality of the company’s circumstances. However, do they really need to spend time writing down what they already communicate constantly, if not directly?

To build a better harness, I think what we actually need looks like the following:

1. The same reasoning module that we have in today’s frontier models
2. An additional module that’s pre-trained with relational tacit knowledge from leading experts in the profession (similarly to how Anthropic has added to Claude’s system prompt over time to make it better at coding, but ideally internalized in some way so it doesn’t consume the context window)⁶
3. A company-specific module that is constantly updated by observing how the company communicates its work
4. The AI skills we write today for specific processes we routinely follow

1 is thankfully already done for us and is generally improving with every model release.

2 is a professional expertise layer, the creation of which is beyond what most of us have time for when writing skills or when extracting knowledge from experts. Additionally, both of these only provide the explicit knowledge we can share. This is where I’d love to see a purpose-built model that’s incorporated tacit knowledge from experience and observation. AfterQuery seems to be working on this problem from the same perspective I have, here.

3 is where I’ve seen talk of a company’s “business brain” emerge: a collective knowledge layer. Folks working alongside me at ClickUp are working on a solution for part of this, populating that knowledge graph I described above from observation of what’s in the workspace. Technologies like Google’s OKF and SurrealDB provide components as well for your own solution. I believe the panel of experts architecture helps the other side of the solution (the reasoning that determines applicability).

4 is the land of explicit skills and the other things we can do with our harnesses to intelligently invoke them: AGENTS.md files, workflows for specific operations, and hooks that force certain checks. We each follow processes that take what we know and apply them in specific ways for repeatable success; this layer still has value.

I think that if we can’t get 2 and 3 added, we’re adding a tremendous burden to every individual writing an agent skill today. They can get something going that supports them in day-to-day work where an agent works as a partner, and they can steer its output when things go wrong. They still share their work for review by other humans to fill expertise gaps, removing the impossible responsibility of covering everything at once. I believe that to increase the rate of software delivery (or any operation), you require more codified knowledge. Establishing teams of agents sustainably conducting business operations requires both industry-leading expertise (to avoid mistakes) and local expertise in what matters to the company and how it’s communicated (to produce relevant and recognizable results). Otherwise, you’ll just be producing slop at 10x or 100x the rate you could have on your local machine.

Without both that professional expertise layer and collective knowledge layer, we’re forced to compensate with systems that provide varying levels⁷ of human involvement. This is doable! This is where many companies are today! This is not 100x. What I’ve described above presents a path towards comprehensive coverage in these operations. This path can take you from skill apprenticeship to skill mastery, and it has components that are necessary to maintain masterful application of these skills in a company over time. Real constraints exist today in fully capturing and deploying that knowledge, but things are changing, and hopefully more of this can be captured without your direct effort. You also can find a compromise in how you inject it that suits your needs. In the meantime, you might not get to 100x by writing some AI skills and getting AI to scrape knowledge from existing docs and code, but with awareness of the fundamental gaps it has and the tacit dimension of knowledge, you can work around those gaps and get closer.

Footnotes

1. Companies have a structure that exists on paper, but they also have one that exists in practice. The actual structure typically diverges because of personal relationships and trust that people build with each other over time. The process here naturally targets the latter, but curators of the knowledge system can still incorporate official leaders from the org structure to keep perspectives balanced. What you do with the disparity between who is mentioned and who should be is an interesting management problem for a different day.
2. The first instance of this actually started with a flat set of .md files curated by a single team of architects. This worked, but it creates a nasty problem over time where their decisions dictate the way the whole organization works. When there’s conflict, engineers are no longer disagreeing with a single domain expert, but that of a panel that has decided unilaterally what is right. In the real graph of experts, there can be disagreements with rationale for the developer to think through. In this first instance, the developer is arguing with an abstract, monolithic council. It is stifling in the same way that architecture review boards are. This is a much bigger topic for a different day, especially considering that software architecture is one of the last places where we still have leverage over what code is written.
3. I did not start with supporting Chat, but I found that using this mode with voice enabled allowed the mind enough time to think about what Claude was saying and for myself to begin formulating a response. Artifact references could also be included in-line, which were particularly helpful for the greenfield scenario. In pure text inside a terminal window, provided at the rate Claude typically generates it, I found that you quickly become pressured to respond as quickly to the prompts, which is impossible and overwhelming, resulting in shallower responses that did not reflect your full knowledge. Capturing specific code references in Chat mode was borderline impossible, though.
4. A variant of this process can occur as well for existing knowledge that led to similar churn. Sometimes our experts make mistakes, but also their expertise can be misconstrued. Both allow room for refinement.
5. For reference, we tried this system at ClickUp. Even with just three agents doing a PR review for another coding agent, each with distinct responsibilities and a single set of universal principles, the PR churned for multiple hours until every agent was satisfied. The problem addressed in the PR wasn’t abnormally large, nor was the surrounding codebase. Artificial limits on the number of iterations allowed needed to be inserted.
6. Here you might ask why isn’t the code that LLMs are trained on already good enough. I’d say that open source code, the only code most LLMs can reach for training data, is not the best code out there. It’s a bunch of side projects, polished to a point, functional, but not what most developers would consider to be their best. That’s reserved for their day jobs.
7. Varying by the maturity of the operations being automated. Even operations automated with software have humans around its use.

Acknowledgements

I need to thank my collaboration with Levi Thomason at ClickUp for inspiring my work on the Brain Extraction Ritual and the surrounding architecture. Though we’re in different places now, I’m grateful for the discussions and lessons learned!