Testing LLM applications is inherently challenging. Unlike traditional software, where deterministic outputs are expected for given inputs, LLM responses can vary significantly, making it hard to validate their correctness.

LLM evaluation frameworks like Deepeval/Ragas/Deepcheck measure the quality of responses using automated scoring or human feedback which make them expensive, slow and a bit inconsistent. They are crucial as integration tests and better mechanisms exist for testing application code changes faster.

An approach which provides robust unit tests involves regression testing your code against responses from the LLM. This is where record/replay testing frameworks like pytest-recording/VCR.py help tremendously. By capturing LLM responses and replaying them during test runs, developers can ensure that the test fails only if the input passed to the LLM changes.

How do record/replay tests help?

VCR.py provides a record/replay mechanism that captures API responses during a test run and reuses them in future executions. If you’ve been using pytest, this is available as a decorator via the pytest-recording library. This technique works well for LLM-based applications because:

1. The tests run once with live LLM API calls and and subsequent test runs replay the recorded responses instead of making live calls.
2. If the API request payload (request url, body, headers) changes in any way, the tests fail. This provides an opportunity to identify if this is a regression in application behavior and accordingly update the recordings.

Practical Example

Consider the following function that interacts with an LLM to generate responses:

import openai

def generate_response(prompt: str) -> str:
    response = openai.ChatCompletion.create(
        model="o3-mini",
        messages=[{"role": "user", "content": prompt}]
    )
    return response["choices"][0]["message"]["content"]

To test this function using pytest-recording, install the plugin and write a test:

import pytest
from myapp import generate_response

@pytest.mark.vcr()
def test_generate_response():
    response = generate_response("What is pytest?")
    assert "testing framework" in response.lower()

This generates a cassette in yaml format which records the entire interaction and is used for subsequent test runs automatically. These should be pushed to version control so that all team members have the exact same behavior.

Note: Cassettes need to be explicitly¹ created using pytest --record-mode=once or with--record-mode=rewrite

interactions:
- request:
    body: '{"messages":[{"role":"user","content":"What is pytest?"}],"model":"o3-mini"}'
    headers:
      content-type:
      - application/json
      user-agent:
      - OpenAI/Python 1.63.2
    method: POST
    uri: https://api.openai.com/v1/chat/completions
  response:
    body:
      string: "{\n  \"id\": \"chatcmpl-B3ljAwsvwZ8zV4UbkASdW1P1TzPRA\",\n  \"object\":
        \"chat.completion\",\n  \"created\": 1740238148,\n  \"model\": \"o3-mini-2025-01-31\",\n
        \ \"choices\": [\n    {\n      \"index\": 0,\n      \"message\": {\n        \"role\":
        \"assistant\",\n        \"content\": \"pytest is a testing framework for Python
        that makes it easy to write and run tests for your code. \",\n        \"refusal\": null\n      },\n      \"finish_reason\":
        \"stop\"\n    }\n  ],\n  \"usage\": {\n    \"prompt_tokens\": 10,\n    \"completion_tokens\":
        419,\n    \"total_tokens\": 429,\n    \"prompt_tokens_details\": {\n      \"cached_tokens\":
        0,\n      \"audio_tokens\": 0\n    },\n    \"completion_tokens_details\":
        {\n      \"reasoning_tokens\": 192,\n      \"audio_tokens\": 0,\n      \"accepted_prediction_tokens\":
        0,\n      \"rejected_prediction_tokens\": 0\n    }\n  },\n  \"service_tier\":
        \"default\",\n  \"system_fingerprint\": \"fp_42bfad963b\"\n}\n"
    headers:
      Content-Type:
      - application/json
    status:
      code: 200
      message: OK
version: 1

⚠️ The default config will log everything to the YAML as is (inclusive of credentials). You must override the VCR configuration to redact authentication headers, decompress gzipped response amongst other things. See the sample repository in the References section for a working example.

Bonus: Using Record/Replay to Explore LLM Capabilities

Beyond testing, record/replay can help developers better understand LLM behavior:

1. Function Calling: While OpenAI’s documentation explains function calling quite clearly, If you’ve been curious how it actually passes this through, record/replay tests are a quick way to validate the same.
2. Comparing Model Versions: Detect subtle changes when switching between LLM versions.

Conclusion

Given record/replay tests capture and reuse LLM responses, they are ideal for detecting unintended prompt regressions and isolating non-LLM-related issues (e.g., code changes, input handling). In contrast, LLM evaluation tests assess response quality dynamically, focusing on overall system performance rather than isolating LLM-specific regressions from external factors.

Resources:

1. Sample repository
2. VCR.py
3. pytest-recording

In this article, we’ll explore what data versioning entails, the challenges it addresses, and how it can be easily integrated into your ML workflow.

How is data versioning different?

The aspects of the right side of the diagram look very similar to packaging a service:

1. CI pipeline: Individual stages of the pipeline e.g. test/build/promote have not been displayed in the image above.
2. Experimentation: You can experiment with different variants by using a feature toggle.
3. Code changes: If you’d run into any issues, you’d rollback your code so that it continues to function as before.

Things that are different:

1. Training pipeline: The artifact in an ML project is heavily influenced by the code, configuration and data. Changes to any of them affects the model weights and hence prediction accuracy significantly. In certain cases, the model may no longer be relevant if the data changes significantly.
2. Experimentation: Depending on the complexity and desired accuracy of the problem, the experimentation cycle can even go on for several months. Through this window, it is essential to be able to identify the best model. Managing this effectively in a wiki/document is not sustainable since different types of changes are involved:
   a. Code changes: Apply different pre-processing steps
   b. Configuration changes: Experiments done with combinations of hyper parameters
   c. Data: The distribution of the data, its schema and more can all change over time.
3. Metric tracking: Data quality metrics and any evaluation metrics can be tracked via data versioning so that there is full traceability.
4. Collaboration: The complexities associated with collaboration over code have mostly been solved by git and other version control tools. Data and configuration add additional layers to the problem since the traceability of an artifact, its accuracy and the corresponding data/config becomes a challenge.

Data versioning solves these problems by providing capabilities similar to git:

1. Dataset Versioning: Track changes to raw, processed and intermediate datasets.
2. Leverage metadata for context: The input datasets, the code and the config parameters and the output need to be linked together for each run of an experiment. This provides full traceability and reproducibility of the results.
3. Storage strategies: The code repository only has a reference to a commit/ref of the data version repository. The actual files are centrally managed on object stores like S3/GCS.

Lifecycle with data versioning

Tool considerations

Few popular tools in this space include DVC, GitLFS, doIt, LakeFS. These tools provide an excellent interface over hand-rolling your own tooling over object stores like S3. Choosing an appropriate tool requires one to think about a few aspects:

1. Dataset size: With larger datasets, depending on the underlying storage representation, the data storage requirements may grow multi-fold and some tools do not handle this effectively. Appropriate data retention policies should be set up so that unreferenced data can be safely removed.
2. Team collaboration: Multiple individuals may work on the same problem statement requiring close collaboration.
3. Access control: For sensitive datasets like financial datasets, access control and auditing capabilities are required.
4. Tool Integration: Integration with other systems like Experimentation (MLFlow / Kubeflow etc), Visualisation of run metrics, model packaging etc.
5. Infrastructure: Some data versioning tools require additional data stores to manage state which adds to the infrastructure footprint.

Detailed comparisons have been done by Neptune, Dagshub, Deepchecks, and LakeFS. Please be aware that this is a rapidly evolving space and the examples and references given here are only meant to give you an idea.

Conclusion

Data versioning plays a key role in building reliable, reproducible, and scalable machine learning systems. By tracking changes to datasets and associating them with experiments and model versions, teams can streamline their development cycles and bring in auditability and traceability. Validating the capabilities of data versioning on small, critical datasets can help teams validate and benefit from its capabilities.

Thanks to Ajinkya, Jaju, Rhushikesh and Swapnil for reviewing early drafts and providing detailed feedback. Ajinkya, Harshad, PV, Preethi and Sanket for building internal ML accelerators and sharing their experiences with these tools.

Reproducibility in ML: The Role of Data Versioning was originally published in inspiringbrilliance on Medium, where people are continuing the conversation by highlighting and responding to this story.

Time-tested principles for building robust Spark applications

Spark is a distributed data processing engine that is widely used in batch processing and stream processing platforms. Building such platforms comes with a fair share of challenges beyond those required for continuous delivery in the microservices world such as data drift, bad data, and data security.

Similar to the Twelve-Factor app that is an outstanding methodology/set of principles for building web apps, we at Sahaj have realized over time that a set of concepts/patterns can be applied to building data processing applications using frameworks like Spark. Through this article, we attempt to document these observations that we believe can serve as ideal practices for defining a basic structure to build production-grade Spark applications.

I. Codebase

One codebase tracked in revision control, many deploys.

Ensure that the code is version controlled and changes are promoted from a lower environment (dev/qa) to a higher environment. This ensures that you can test changes before rolling them out and in case of any issues, you can easily rollback. It further provides a traceable artifact for investigating issues.

II. Dependencies

Explicitly declare and isolate dependencies.

There are several choices for declaring and locking dependencies. To pin dependencies (inclusive of transitive dependencies) to specific versions and have a reproducible build, tools such as Poetry/Pipenv are available in Python, and in Java/Scala, sbt, Gradle, and Maven do the same.

In the Python ecosystem, several libraries rely on binary dependencies directly installed in the system. In such scenarios, use containerized images to ensure that the system dependencies are packaged into a tested image and are promoted across environments.

III. Config

Store config in the environment.

Spark lets you override several parameters at runtime via environment variables or command line parameters to tweak application parameters and handle changes in data volume or infrastructure without changing code.

For other configurations, parameterise using named CLI args. Every language has libraries for processing command-line arguments that can validate the types as well as supported values and ensure that the application doesn’t run with junk values.

Externalise secrets and access them at runtime. Use tools like Vault, AWS Secrets Manager and service roles to ensure that keys can be easily rotated.

IV. Backing services

Treat backing services as attached resources.

Spark provides abstractions for dealing with various data sources such as JDBC, HDFS, S3 etc. By treating these as resources and externalising configuration, you can avail simplified testing, pluggable resources, and reduced vendor lock-in. Use custom connectors, readers instead of directly interacting with external sources like http/ftp using client libraries so that the spark job focuses primarily on business logic.

Build/Use abstractions that make it easier to run jobs locally with minimal external dependencies. For example, Instead of relying on a shared s3/dev database location, treat them as attached resources so that locally it can be swapped with the filesystem/stub apis.

For infra services such as performance metrics, Spark supports configurable sinks so that key metrics are pushed to systems such as Prometheus, StatsD, and Ganglia.

V. Build, release, run

Strictly separate build and run stages.

For Spark applications, during the build stage, you should create a packaged artifact inclusive of all dependencies. For Python, this involves creating a wheel for your target system. For Scala/Java, it involves creating an uber jar using tools like shadow. To reiterate, using a containerized image isolates any system dependencies and helps with reproducible builds that can be promoted from one environment to another.

Invest heavily in writing tests and clean code practices to provide safety nets as well as to avoid data bugs. Depending on the data, the cost of data bugs can be astronomical and fixing it would involve recomputing the data along with any derived insights. Run tests automatically in a pipeline before building the artifact.

If the configuration is fully externalised, the release stage is mostly a no-op and a workflow orchestrator can ensure that the right configuration is applied and used in each environment.

VI. Data Schema

Validate data schema and semantics

Prefer data formats such as Parquet/Avro/ORC so that schema is implicitly a part of the data and schema does not need to be explicitly managed/specified across separate data jobs. For CSV/JSON, explicitly specify the schema so that unintentional changes are caught early.

Create separate schema versions whenever the semantics of existing columns have changed or columns have been introduced/deleted. The intent is to avoid unintentional impact on downstream consumers.

Fail fast! Validate schema and data semantics as part of your job for all inputs so that unintentional changes are caught early. Frameworks such as Deequ, GreatExpectations let you integrate checks into your data pipeline.

VII. Immutability

Prefer atomic, repeatable jobs with partitioned data

Immutability in data is similar to immutability in programming. Since tasks can fail for various reasons, it is important to ensure that the re-computation doesn’t corrupt the output state. Partition your data so that older/irrelevant data can be phased out easily. For batch processing, fail on presence of existing data and write to a separate output location for each run.

For a deterministic model/job, this also helps in rolling out newer versions of the model since they can be easily compared against a previous model for a given input.

Write each application as an atomic job that does one thing well. This ensures that a rerun on failure doesn’t affect anything else. Compose multiple jobs using a workflow orchestrator.

VIII. Scalability

Scale out horizontally with balanced tasks

For Spark applications, leverage parallelisation and balanced tasks to ensure that the tasks can scale with additional infrastructure. Use CPU/memory/disk optimally and monitor systems to ensure infrastructure usage scales on the basis of data.

Don’t collect data on the driver node which might exceed the resources available to a single node. Instead, identify mechanisms of distributing the processing across all nodes. For example, Spark lets you define user defined functions (UDF) which can do complex model/algorithmic processing on your data across nodes/tasks. If you need to collate and create a single artifact (e.g. a geojson file) , this can be done in a separate stage which unifies smaller parts with different infrastructure considerations.

IX. Disposability

Idempotent tasks with no side-effects on re-run

Given the distributed nature of Spark, be wary of issues that arise due to intermittent availability of backing services and the implications of rerunning tasks and side-effects such as updating data stores/HTTP endpoints.

Cloud infrastructure makes it easy to spawn on-demand infrastructure with spot pricing which further helps reduce cost and manage immutable infrastructure. However, this also increases the odds of infrastructure instances being terminated on short notice.

X. Dev/prod parity

Keep development, staging, and production as similar as possible.

As far as possible, keep the environment similar as it helps ensure that cluster sizing, debugging, model validation etc., can all be done in the dev environment.

However, concerns such as cost, privacy, time etc., might lead to choosing alternatives such as sampling and anonymisation of data for test environments. Scale down infrastructure/configuration proportionately if you choose to sample. This offers a good middle ground where data on dev still lets you know of data bottlenecks and other issues before they hit the production environment.

XI. Observability

Provide insights into runtime application behaviour

Use structured logging to provide visibility into the application behaviour. The event logs and spark history server provide detailed insight into the processed tasks and can help in identifying unbalanced tasks, their duration, DAG visualisation, shuffle issues etc.

Publish infrastructure and spark metrics to monitoring systems so that the infrastructure can be effectively utilised and issues due to incorrectly configured system resources can be investigated.

XII. Data Quality

Define data quality checks to safeguard against data drift

In addition to testing, running data quality checks in your data pipeline helps ensure that any degradation due to Data Drift is caught early before downstream jobs consume the data.

Define a measure of accuracy for any models and ensure the accuracy score meets a static/dynamic threshold.

Choose actionable data quality metrics which can be acted upon to avoid alert fatigue and noise.

12 Factor Spark Applications was originally published in inspiringbrilliance on Medium, where people are continuing the conversation by highlighting and responding to this story.

OOH agencies have been traditionally managing and running campaigns manually. From campaign planning to media buying, pricing, placement and tracking, every step of the process is manual. The lack of industry standards add another layer of complexity by creating information silos that in turn make measurement and accountability complicated. A wholesome reimagining of the medium has long been overdue.

Today, disruptive brands that adopt OOH require data-driven campaigns to adapt to dynamic aspects of audience behaviour. OOH needs to leverage digitization and data-led insights to transform itself into a near real-time medium to deliver responsive and geographically tailored campaigns.

Talon, a specialist OOH agency with more than 30% market share, partnered with Sahaj to build a holistic platform that could make the entire cycle of planning, buying, implementation and ROI attribution of campaigns scalable, intuitive, and automated. By doing so, they would effectively lay down the foundation for a path-breaking, data-led disruptive framework that would yield significantly better results for brands.

This led to the creation of two innovative platforms - Plato and Ada.

Plato, an automated trading platform, aggregates all OOH inventory into a single, intuitive campaign planning and delivery system. It allows users to explore, plan, check availability, reserve, and trade paper and digital inventory across multiple markets through direct integrations with media owners and industry systems.

Plato has direct integrations with multiple media owners and industry systems. It uses more than a dozen web services, a document store running heavy geospatial queries with dense geo-location data and fast messaging queues to distribute the load with elastic infrastructure to rapidly ingest, process, and store data to the tune of a billion data points for optimized brand/business radius searches.

Plato increased the efficiency of Talon’s planning and buying teams by at least 50%. Users can now plan and book a campaign in just 5 minutes!

Ada, unlike Plato that aggregates the universe of inventory of OOH media in the UK, uses multiple data science models to analyse billions of device-level audience data points, real-world locations and advertising inventory to enable granular segmentation of audience, intelligent decision-making, campaign management and track ROI.

It leverages state-of-the-art data science models and technologies like Airflow, Apache Spark, Terraform, PostgreSQL, and Micronaut/Spring to recognise, classify and relay behaviors near-real time for a cohort of people.

Navigating the OOH landscape with Talon’s Ada has been hugely successful for businesses such as McDonald’s and Starbucks who have used it for their brand building campaigns.

To launch Starbucks’ new Blonde Espresso Roast, Ada built a custom behavioural audience profile of frequent visitors to high street coffee houses over a 6-month-period. The best performing large format digital inventory was then selected for OOH media buy. Finally, the campaign was measured against brand metrics to determine its effectiveness.

Starbucks’ campaign impact

To drive sustained awareness of the Saver Menu for McDonald’s, Ada created insights about how people travel and how to effectively reach and engage them on the go using billions of data points alongside information about real world places. The campaign was then executed using a blend of 6 sheets and kiosks to drive cost effective reach nationally and bursts were aligned with payday every month when sales spike.

McDonald’s campaign impact

Our View

As audience planning, delivery, and measurement become increasingly data-driven, advertising strategies derived from behaviours will assume significance for the agility they offer. Operational costs of running a campaign will reduce considerably.

Technology will play a pivotal role in enabling us to navigate the ever-evolving landscape of OOH media. Game-changers such as Ada will enable media owners to react faster to audiences and the changing dynamics with their surroundings — the pandemic being a classic example — and thus focus on impactful delivery. While the idea of delivering targeted content in OOH is exciting enough by itself, there will be a push to provide qualitative attribution by cross-pollinating data from target businesses, OOH environments, and other data sources.

Data science-led, game-changing solutions to disrupt OOH media was originally published in inspiringbrilliance on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are innovative use cases that are currently driven using Data Science, and given the cheap cost of storage and cloud infrastructure, it is tempting for enterprises both big and small to start this journey. While the rest of engineering has reached a phase where Agile/Agility has become an abused term, Data Science, still needs to catch up on a lot of practices.

Here are three ways to accelerate your Data Science journey:

Iterate on the models

For any new model, start with simple, explainable models with a measure of accuracy and roll it out to your users. Improve the model in subsequent cycles based on any new observations and stop once the cost exceeds the benefit. If required, build capabilities to A/B test your models.

Timebox exploration for your hypotheses so that you don’t get lost in data. You will discover new patterns/outliers in our data, note them, and see if it makes sense to analyse these further for future business use cases.

Rather than spending more time choosing the best model, pick one that provides acceptable accuracy and then improve over time. Delaying a feature for minor improvements prevents you from discovering other aspects in production.

Work with Stakeholders and business users

Involve business users early and often for data analysis and validation. Provide access to users via interactive widgets/dashboards so that they can explore and come back with specific data points for any issues.

Ensure that your stakeholders are aware of the difference in approach for rolling out a model. Given that a lot of things can be exploratory, data analysis discovery/steps play an important role in addition to the outcome. Agree on what is an acceptable accuracy threshold. For some models, you can start with enabling people in their roles and later evolve to provide direct recommendations.

Faster delivery

Do not delay confidence-building measures like testing and validation to get to a deliverable faster. Instead, leverage them to evolve the model frequently. For systems dealing with big data, fixing a broken model costs more in time, money and reputation.

Leverage on-demand immutable cloud infrastructure to reduce upfront infrastructure costs and to run larger workloads faster. While the cost for running it with more infrastructure is higher, you save on the overall time to ship the model.

Avoid silos and have Data Scientists and Data Engineers working closely on all aspects. All of them possess specific skills and can benefit hugely from cross-pollination that in turn would improve the rate as well as the quality of what is delivered. Reduce cycle time for your models by having more people work on it simultaneously rather than running one stream per person.

Ways to Accelerate Your Data Science Journey was originally published in inspiringbrilliance on Medium, where people are continuing the conversation by highlighting and responding to this story.

X-ray lets us filter traces by latency, status codes, failure/error and a lot more.

Trace Inbound HTTP Calls

We can easily configure the SDK middleware to intercept inbound requests. This adds basic information to the trace which includes the HTTP method, timing and a few more HTTP headers. The segment name can be configured to respect the Host header. The alternative is to specify a static name for the node.

We can also force sampling by passing a custom header with the request X-Amzn-Trace-Id with the value as Sampled=1 . This feature isn’t documented but the SDK libraries typically look for this header in the incoming request.

Override Sampling Rules

By default, the first request every second and 5% of any additional requests are sampled. However if we have certain routes which are triggered less often but are important to monitor, we can tweak the sampling rules based on url pattern, service_name and http method. We use this to increase the sampling to 100% for important low traffic integration endpoints. See https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-nodejs-configuration.html#xray-sdk-nodejs-configuration-sampling for an example in NodeJS.

With 2.0.0 version of the X-Ray SDKs, the sampling rules can now be managed from the web console itself allowing us to tweak the rules on runtime without redeploying the application.

A maximum of 1000 traces can be seen at a time in the list view. The data is retained for one month and traces can be seen for a time window between 5 mins to 6 hours.

Trace Outbound HTTP Calls

Outbound HTTP calls from the application can be traced as a sub-segment. This allows us to easily trace requests which exceed a specified duration or have been failing with a particular status code. This usually involves replacing the http client with an equivalent instrumented client. See https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-nodejs-httpclients.html

Capture Custom Metadata Per Trace

Using the AWS X-ray SDK, we can add custom metadata and annotations.

The default X-ray HTTP instrumentation doesn’t log the query parameters. This can be easily incorporated in the application code by using some middleware/interceptor to add additional metadata attributes.

We use a custom requestId annotation which lets us map requests from X-ray to those in our logs. This helps us investigate the issue further.

Annotations are more powerful since they are searchable via filter expressions and also allow us to group the traces by an annotation. Metadata however are only displayed with each trace.

Capture Custom Sub-Segments

We use custom sub-segments to capture any background tasks executed in a separate thread. This makes it easy to visualise the time taken for background processing.

Trace calls to AWS Services

The X-ray SDK automatically captures calls to other AWS services. The attributes reported can be customised using a whitelist file. In our case, knowing the path to the S3 file is useful. This can be tweaked using the whitelist. E.g. https://github.com/aws/aws-xray-sdk-node/blob/master/packages/core/lib/resources/aws_whitelist.json which can be set from a TracingHandler. With X-ray 2.x versions of the Java SDK, the S3 file path is tracked automatically using the same mechanism.

SQL Queries

SQL queries associated with an application can be traced by using an instrumented database client. We don’t use this currently but this can be quite useful.

Colorise the X-Ray Service map

Lastly, the default X-ray service map only highlights nodes for throttled/failed requests. We use a lot of AWS components and this makes it difficult to identify hot spots. I wanted a color scale depending upon requests/sec per node and so I ended up writing a Chrome extension for the same.

The service map currently cannot be combined with filter queries. This would have been quite useful to filter nodes only invoked in a particular flow/application.

Overall, AWS X-ray provides us with insights into our services without the hassles of maintaining the corresponding infrastructure.