AI Batch Requesting For Lawyers
Introduction
Batch AI processing is being used in commercial legal products, but can now be used by resource constrained lawyers in a variety of ways that could impact the criminal justice system. Functioning similar to a mail merge, batch processing allows a user to submit a large number of prompt requests to a model and receive responses within a relatively short amount of time (typically 24 hours). Lawyers can use this technology to review large amounts of documents searching for particular material or to draft many similar documents such as motions to dismiss, but that are particularized to the facts of different cases.
In this post, I expand on ideas for applications of AI in the criminal justice system by outlining a few specific ways batch processing can be used in the resource constrained, criminal law context. I will start by describing how I’m using AI batch processing to further research into possible AI bias and conclude with a brief discussion of the externalities that such use can impose on the system.
How I am using batch processing
I believe many lawyers, including prosecutors, are using AI chatbots like ChatGPT to assist with their work. There is a lot of evidence that this is happening. My concern is that it is not clear whether and how the chatbots might be biased. Chatbots could be explicitly discriminatory, though most AI companies are continually working to mitigate this. But chatbots could hold more subtle, implicit biases that are harder to recognize and that still impact human decision-making. For example, a chatbot asked to perform a risk assessment might consistently rate prior drug convictions as a high-risk attribute without recognizing historical racial patterns in drug arrests such that Black people are then consistently rated as higher risk.
These types of issues can be uncovered by running large scale tests to gather many AI responses to similar prompts and assessing whether there are underlying patterns that would manifest into biased results. To generate the data though, requires thousands of prompts and responses, which, when run through a chatbot’s normal interface, is error-prone and can take a long time. To deal with this, I am now creating AI batches to submit entire experiments’ worth of requests in a single batch. This saves a massive amount of time and money.
How batch processing can be used by criminal lawyers
There are likely many applications for batch processing within the criminal justice system, but in this blog, I am going to outline two that I think could be readily implemented and would have real impact. Both ideas require a batch processing system with web search and possibly web fetch capabilities, neither of which ChatGPT currently supports, though Claude does. These ideas also require an organization to carefully structure how case data is stored, so that there’s essentially a single link per case, which is itself a folder containing all case materials. Because of the size of some case files, these approaches (1) may not currently work, but as token limits increase (essentially, how much data the model can process) may become possible, and (2) token limits may require breaking up the proposed processes into multiple staged batches. Finally, implementation likely requires some network security expertise to ensure that any sensitive case materials are not exposed.
Get Justice Innovation Lab’s stories in your inbox
Join Medium for free to get updates from this writer.
Prosecutor conviction review units. I know of at least one fellow researcher who had been proposing to funders a project to use AI in conviction review. This has also been discussed in criminal justice-related podcasts, so it is not a novel idea. The general thrust of those proposals, though, was focused on using AI to evaluate a single case. Because of the volume of case material, a single case review takes a massive effort that AI could shorten. Batch processing these reviews multiplies this workflow many times over. Here’s how I can imagine an office doing this:
- Create a data set with links to case folders. For a lot of offices, this is not trivial, depending on how the office has structured evidence/data management and the various technologies the office uses.
- Because of the volume of documents in each case, each folder should likely include a table of contents that lists and categorizes each file. Write a prompt for the AI to review materials, categorize them, and return a table of contents. Then put the table of contents into each folder.
- Since there are different reasons for reviewing a conviction, write prompts for each of these reasons. In the prompts, include directions for how to use the table of contents and what documents to focus on. For instance, reviewing for witness inconsistencies would require a prompt that directs the AI to focus on reviewing witness statements. Have the return from these prompts be a short summary and a prioritization ranking for reviewing cases.
From this, an office would have datasets ranking cases by the likelihood of various conviction review patterns, which could then help prioritize human review. Though there is significant investment needed to set up the process, these costs are likely small compared to the cost of having lawyers review case materials, especially since the review process itself — searching for things like inconsistencies across many documents — may better suit a model than a human as well.
Writing motions. Offices could use batch processing to generate first drafts of various motions on a daily or weekly basis. For example, a prosecutor’s office could gather from staff all motions that need to be written on a daily basis, create template prompts for each type of motion, again use links to case folders, and then nightly run a batch job to write drafts. I think this is unlikely to happen, in part because writing motions is what many lawyers, at least nominally, enjoy doing. But many convicted persons would benefit from a legal review, and with the introduction of processes like those in California’s Racial Justice Act, many people could submit legitimate motions but lack the expertise and support to do so.
The Racial Justice Act creates an avenue for a convicted person to submit a motion to reconsider a conviction based upon either discrimination by the state in that person’s case or based on evidence of systematic discrimination by the state — see challenges based on statistical disparities. The latter claim is potentially broadly applicable for many convicted people in California, and a process to mass produce well-written motions that can be particularized just enough to a defendant’s case means that hundreds to thousands of such motions could be drafted in a few weeks. Using the same structure outlined above, a legal services provider could create a system for organizing case materials and a set of basic prompts to draft the motions. In fact, an actor could create a platform whereby convicted persons would create their own evidence folders, submit them to the service, which then could organize the folders and run daily batches to generate the motions. Though there is not much evidence that these motions have succeeded often, the ability to file hundreds of motions, possibly modeled after successful motions, could significantly change the law’s impact.
What could happen
Scaling labor-intensive activities like conviction review can help to address long-standing equity issues, like conviction review being mostly reserved for those able and willing to navigate the process. In the past, it was defendants with the financial resources, advocacy support, and ability who were most likely to push a conviction review forward. Now that the process can be extended to nearly all defendants at an extremely low cost. Similarly, processes like motion writing can be scaled such that all persons within the criminal justice system receive a higher floor of service. This can be managed by those already working within the system, who can also reduce and prioritize workloads.
Unfortunately, batching can also be used to break systems like courts that have a narrow funnel. Individuals are already finding AI useful in legal proceedings and are able to produce high-enough quality legal filings to harass others and bog down courts and lawyers in process at a reduced cost. So just as batch processing can scale the benefits of AI in legal processes, it can also be used to scale potentially harmful practices that could grind courts to a halt. In turn, courts might limit the use of AI and do so in a way that prevents us from taking advantage of it to better enact our principles for due process and justice.
Rather than ignoring these realities, criminal justice agencies should begin to take a more active role in incorporating AI into practice. Without doing so and absent state or federal regulations, individual actors will control how AI impacts the system, which may or may not result in a net public benefit.
By: Rory Pulvino, Justice Innovation Lab Chief Implementation Officer. Admin for a Prosecutor Analytics discussion group.
For more information about Justice Innovation Lab, visit www.JusticeInnovationLab.org.
