Intro

A few years ago, I had what you could describe as a steady and unremarkable tech career. Master’s degree, a few MLE and data science jobs at midsize companies. Then one day I realized that my youth was vanishing in the rear-view mirror while I was not getting anywhere.

Friend: “Sorry I missed your call. I was giving my keynote at the Amalfi Coast Machine Learning Conference and then had a grant-funded dinner at a Michelin three-star restaurant with three Nobel laureates.”

Me: “Sorry I missed your text. I was failing the same CAPTCHA seven times while trying to log in to a mandatory compliance training on Python indentation.

Today, things look different. My first-author paper just got accepted to the prestigious ICSE conference — my third accepted paper this year. I guess it’s fair to say I’ve managed to break into research from what felt like an impossible position: a mid-career industry engineer, a decade out of grad school.

This post is about how that happened, and what I learned from it.

Why Research?

I was always excited about bridging the worlds of research and product. But I started my career on the product side of the house and mostly stayed there. I did on-calls, built data pipelines, engineered features, trained and deployed simple models to production. More recently, like everyone else, I spent a lot of time building on top of LLM APIs.

It was all good, honest work — but I missed the thrill of venturing into the unknown: trying out bold ideas, collaborating with people across the world, having public work artifacts. Instead — like many engineers — I often found myself in a loop of scrum sprints with well-intentioned project managers trying to “derisk” their teams’ work.

I wanted to spend more time doing research, but wanting to do research and knowing how to do it were very different things

The Challenge

Getting a paper accepted to a peer-reviewed venue is a high bar — especially with no PhD or prior research experience beyond my master’s thesis, which, to be honest, was already hard for me to get through. Unlike back then, I had no advisor, no lab, no community, and no free time. It wasn’t clear how to navigate any of it: idea development, scoping, submissions, rebuttals — all of it was uncharted territory.

First Footholds

I started small: I did a bit of work for the LAION OpenAssistant project, which led me to join the Ontocord community. It was a freewheeling affair, but it gave me my first real research foothold. We eventually published an LLM paper at COLING’s industry track. It wasn’t groundbreaking, but it proved that I could contribute meaningfully to a research collaboration and gave me motivation to pursue additional research projects.

Things Started Moving

For a time, my team was going through some changes, and I had more room for side projects. I signed up to contribute to a paper led by a collaborator I’d worked with at Ontocord. I also had an idea for a paper based on a project I did at work. Masochistically, I decided to do both. It turned out to be an incredibly hard stretch, but I powered through. Both papers were accepted to top-tier conferences.

Then another Ontocord project came along — again somewhat chaotic, but with interesting results that seemed worth writing up. I spent a decent amount of time polishing it and submitted that too. By the end of 2025, I had three accepted papers and one more still in review.

Thinking in Terms of a Research Portfolio

Now I have a growing backlog of ideas and some opportunities to join projects led by others. My next challenge is learning to think more strategically about shaping my research portfolio — how to balance risk and reward, match projects to the right venues, and build a broader network of collaborators to increase throughput.

To grow in these areas, I’m looking for mentorship from more experienced researchers. I’m also beginning to explore whether I can formalize this work within my organization — perhaps in a research engineer or a research scientist role.

Learnings

Looking back, one thing I kept as a priority was to focus my efforts where my comparative advantages actually were and look for help elsewhere, instead of trying to become good at everything. I’m not a great math person, but I can implement things quickly, coordinate people, and make projects move.

One thing that I did feel I have to dial in is the ability to think creatively and let my mind wander off the beaten path. A part of what helped me make progress is priming my mind to be open to ideas when they show up. Another was to avoid consuming too much content, as I found that it saturates my thinking and doesn’t leave space for exploration.

Leveraging the OSS community to latch on to more experienced researchers was a great idea. I made some long-term relationships that were and continue to be very fruitful. An experienced collaborator can help on prioritization, scoping and give valuable early feedback — these are very helpful for a beginner.

John Schulman’s Opinionated Guide to ML Research also left a mark on me, especially his distinction between idea-driven and goal-driven research. For me, goal-driven research turned out to be far more effective for generating ideas.

What surprised me most was how powerful it is to just keep showing up. Some of the people I’ve met who ended up having an impressive publication record weren’t the most technically brilliant — they were just willing to put in work, consistent and easy to collaborate with.

Epilogue

In retrospect, I realize that for far too long I’d been underestimating my own agency — my ability to steer my life in the direction I want, despite the circumstances — and especially the power of setting a goal and committing to it.

The goal I set for myself was to make it as a researcher, and eventually to turn it into a core part of my day job. I’ve already taken big steps toward that, despite all the difficulties, and I’m excited to see how far I can take it in 2026.

If you’re trying to start your own research journey from industry, I’d love to hear what’s worked for you.

There are a lot of resources on how to optimize LLM inference for latency with a batch size of 1. This is important for the use-case of an end-user running a model locally for chat. However, I couldn’t find much on the subject of batch inference. Batch inference is important, for example, for evaluating models over large benchmarks.

In this blog, I’ll show how to use Hugging Face Accelerate to do batch inference. I’ll focus on a Multi-GPU setup, but a Multi-node setup should be pretty similar.

Implementing the Inference Script

The main flow of the batch inference script should look like roughly as follows:

accelerator = Accelerator()

with accelerator.main_process_first():
  tokenizer = get_tokenizer(...)
  model = get_model(...)
  dataloader = get_dataloader(...)

model, dataloader = accelerator.prepare(model, dataloader)
output_sequences = run_generation(dataloader, tokenizer, model, accelerator)

accelerator.wait_for_everyone()

if accelerator.is_local_main_process:
  save_output(output_sequences, output_path)

So what’s happening here? The Accelerator is the main class of the Accelerate library, so we need to create it. When we launch a script with Accelerate, it spawns a process for each GPU. The Accelerator class provides utilities to handle this multi-process workflow.

Now, let’s focus on each of the calls where the accelerator is referenced — everything else is standard HF Transformers stuff.

The purpose of the with accelerator.main_process_first() context manager is to ensure that the code inside runs, well, on the main process first. Why is this important? For downloading datasets and models, obviously there is no point in downloading them separately on each worker process. For multi-worker CPU bound work, likewise the documentation recommends that will be done on the main process. Therefore, we load the model and the data and also do the tokenization within this context manager.

The accelerator.prepare() method prepares data and models for distributed processing by moving them to the GPU’s where they are needed. Therefore, there is no need to make calls such as model.to(‘cuda’).

The accelerator.wait_for_everyone() call is done in order to ensure that all workers finished their part of the inference. When that is done, we can collect the output from all workers and create the output file. Conveniently, the output file will have the same order as the input dataset.

Now, let us take a look into the run_generation() method, which has a few more implementation details.

def run_generation(dataloader, tokenizer, model, accelerator):
    model, dataloader = accelerator.prepare(model, dataloader)

    output_sequences = []

    for batch in tqdm(dataloader):
       with torch.inference_mode():
            generated_tokens = model.generate(**batch)

            generated_tokens = accelerator.pad_across_processes(
                generated_tokens, dim=1, pad_index=tokenizer.pad_token_id)

            generated_tokens = accelerator.gather_for_metrics(generated_tokens).cpu().tolist()

        outputs = [tokenizer.decode(ids, skip_special_tokens=True) for ids in generated_tokens]
        output_sequences.extend(outputs)

    return output_sequences

Inside the inference loop, we make two calls to the accelerator: pad_across_processes and gather_for_metrics. The latter’s purpose is to gather the outputs from the workers into the main process. The former is needed so that the batch sizes will be the same across all workers — the gather_for_metrics method requires that.

Executing the Inference Script

To start multi-GPU inference using Accelerate, you should be using the accelerate launch CLI. The command should look approximately as follows:

accelerate launch \
--num_processes=2 \
 generation.py \
 --model_name [MODEL_NAME]\
 --adapter_checkpoint_dir [ADAPTER_CHECKPOINT_DIR]\
 --data_path [DATA_PATH]\
 --output_dir [OUTPUT_DIR]

Accelerate does have quite a bit of configuration options, so it’s a good idea to run accelerate config before starting an inference run.

Conclusion

We looked at how the Accelerate package from Hugging Face can be leveraged to implement a simple solution for multi-GPU inference. For convenience, I am providing a full working example script for Llama2.

Background

Recently, I tried to fine-tune the Falcon-40b model on a question answering dataset. To that end, I adapted some example Jupyter notebooks that are available on the web, e.g. here and here.

To my surprise, the fine-tuned model couldn’t quite finish its answers — it usually kept generating tokens until it hit the max_tokens limit. The issue turned out to be specific to Falcon models — it didn’t happen for example when I swapped the base model to OpenLlama.

In theory, this is not supposed to happen because training examples have a prompt format such as “{question}<separator>{answer}<eos token>”. The model should learn that the end-of-sequence token has a high probability when an answer ends. However, this wasn’t happening in my experiments.

Diagnosing The Problem

Googling the issue has led to a couple of relevant threads (1, 2). From these, it became clear that the problem is with the padding token, which the Falcon actually doesn’t have. This is a problem because we need to pad input examples to create batches for training. The way that the example notebooks deal with this is by setting:

tokenizer.pad_token = tokenizer.eos_token

In theory, this should be fine because the pad tokens should get masked out, while the real end-of-sentence token should not get masked out.

The problem is that when fine-tuning the model according to the recipe suggested by the tutorials, we collate the input with DataCollatorForLanguageModeling. This collator applies the following logic:

if self.tokenizer.pad_token_id is not None:
 labels[labels == self.tokenizer.pad_token_id] = -100

As we know from the PyTorch doc, labels that equal -100 are excluded from the loss calculation. Our pad token being same as the end-of-sequence token, that means that the model never learns to finish generating responses.

Solutions

One way to mitigate this issue is, of course, to set the pad token to something else. For example, create a new special token:

tokenizer.add_special_tokens({'pad_token': '[PAD]'})
with torch.no_grad():
  model.resize_token_embeddings(len(tokenizer))
model.config.pad_token_id = tokenizer.pad_token_id

In the snippet above, the last line is needed for the model.generate() method to use the correct padding token — otherwise it defaults to use the end-of-sentence token. One extra comment: it’s probably a good idea to set the new token’s weight to the mean of all existing token embedding, as discussed here.

Another solution is to use a different collator that doesn’t handle the padding token in the same way. Since I was working on question answering, the DataCollatorForSeq2Seq collator was a natural candidate. With this collator, there are two options:

1. Treat the question as the input and the labels as the output. Then, the loss is calculated only over the labels.
2. Treat the whole prompt as both the input and the output.

I didn’t experiment with which of the two options leads to better results, as that was out of scope at the time. I did implement the 2nd option, roughly as in the snippet below:

def tokenize(prompt, max_length=1024):
    result = tokenizer(
        prompt,
        truncation=True,
        max_length=cutoff_len,
        padding=False,
        return_tensors=None,
    )
    result["labels"] = result["input_ids"].copy()
    return result

tokenized_dataset = dataset.map(tokenize, remove_columns=list(dataset.features))

data_collator = transformers.DataCollatorForSeq2Seq(
                  tokenizer,                  
                  return_tensors="pt",
                  padding=True)
      
trainer = transformers.Trainer(
    model=model,
    train_dataset=tokenized_dataset,
    data_collator=data_collator
)

Conclusion

In this blog post, we addressed the issue of the Falcon model indefinitely generating tokens until it hits the max token limit. We diagnosed that the issue stemmed from the model’s lack of a unique padding token which resulted in the model’s inability to learn when to end responses. We proposed two potential solutions: Firstly, creating a new special token to act as the padding token, and secondly, using a different collator that does not handle padding tokens in the same way as DataCollatorForLanguageModeling.

A common use-case for LLM-based applications is an API server that makes a call to an LLM API, does some processing on the response and returns it to the caller. Since LLM API calls can be very slow, a common method to improve the user experience is to use a streaming API. In this post, I will show how to build a scalable API service that serves streaming and potentially post-processed LLM API responses.

Background

FastAPI is a modern and reliable framework for building REST API’s. However, due to the Python GIL, a single process of a FastAPI or similar Python service can handle only one request at a time, making the service difficult to scale.

However, in our use-case the bottleneck for a Python REST API service is the I/O time of calling an OpenAI model. This allows us to get around the GIL issue via concurrency, as multiple requests can all wait for their output to arrive simultaneously. This can be achieved in a few ways, such as good old thread-pools or the more modern concept of co-routines, which is implemented in Python by the asyncio package. The latter is better suited for scaling the I/O-bound use-case as co-routines are lighter than system threads.

API Server Implementation

Given that there is any number of other resources on how to create a boilerplate FastAPI application, let us focus on the parts that are specific to the task at hand.

First, we will define the input schema for our API endpoint using Pydantic. More details on this can be found for example here.

from pydantic import BaseModel

class InferenceRequest(BaseModel):
    model_name: str
    input_text: str 
    api_key: Optional[str]
    org_id: Optional[str] = None
    generation_cfg: Optional[dict] = None

Now, let us define the handler of the API endpoint.

from fastapi import FastAPI, HTTPException
from starlette.responses import StreamingResponse
import openai

app = FastAPI()

@app.post(f"/openai_streaming")
async def openai_streaming(request: InferenceRequest) -> StreamingResponse: 
  try:    
      subscription = await openai.ChatCompletion.acreate(
          model=request.model_name,
          api_key=request.api_key,
          organization=request.org_id,
          messages=text,
          stream=True,
          **request.generation_cfg
      )
  
      return StreamingResponse(stream_generator(subscription),
                               media_type='text/event-stream')
  except openai.OpenAIError:
    raise HTTPException(status_code=500, detail='OpenAI call failed')

A couple of things to note here:

1. We are using the async version of the OpenAI ChatCompletion.
2. In order to return a stream of Server-Sent Events , our handler method returns an object of type StreamingResponse.

Now, we will create the stream_generator method.

import async_timeout
import asyncio

GENERATION_TIMEOUT_SEC = 60

async def stream_generator(subscription):
    async with async_timeout.timeout(GENERATION_TIMEOUT_SEC):
        try:
            async for chunk in subscription:
                yield post_processing(chunk)
        except asyncio.TimeoutError:
            raise HTTPException(status_code=504, detail="Stream timed out")

Here, the post_processing method contains whatever business logic that needs to be applied to each data chunk. Of course, this code could be modified to buffer multiple chunks and post-process them jointly and so forth.

Its is important to have the async_timeout.timeout in the generator to ensure that the method yields. Otherwise, it can get stuck indefinitely and leave the client hanging.

Client Implementation

For completeness, here’s an example client implementation using the httpx package.

import httpx

from inference_request import InferenceRequest

ENDPOINT_URL = 'http://0.0.0.0:80/openai_streaming'

payload = InferenceRequest(key='MY_API_KEY', 
                           input_text='What is the capital of Italy?',
                           model_name='gpt-4').dict()

with httpx.stream('POST', ENDPOINT_URL, json=payload) as r:
     for chunk in r.iter_raw():        
         print(chunk)

Benchmarking

In my experiments, I clocked about 10k requests per minute when hosting a single worker of the service (using uvicorn) on a machine that has a single CPU and 2gb RAM.

Conclusion

FastAPI, when combined with asyncio, can provide a robust solution for building high-performance streaming applications leveraging OpenAI models or other LLM API’s.