Most teams spend weeks wrestling with RAG infrastructure before they can answer their first question. I built this open-source AWS RAG application to cut that timeline from weeks to hours — giving you a solid foundation that demonstrates enterprise-grade patterns and scales from proof-of-concept to production deployment.

The RAG Infrastructure Problem

Here’s what I see repeatedly: engineering teams get excited about Retrieval-Augmented Generation, spin up a quick prototype with OpenAI embeddings and a local vector database, then hit a wall when they need to ship something that handles real traffic, integrates with existing AWS infrastructure, and meets enterprise security requirements.

The gap between “RAG demo” and “RAG in production” is massive. You need:

• Managed embedding models that don’t require ML infrastructure
• Enterprise vector databases with proper access controls and monitoring
• Scalable ingestion pipelines that handle thousands of documents
• Production APIs with error handling, logging, and health checks
• Cost-efficient architecture that doesn’t break the budget during experimentation

That’s exactly what this AWS RAG application delivers.

How This Open-Source RAG Pipeline Works

I designed this implementation around two core principles: use managed services wherever possible and optimize for developer velocity. The result is a RAG pipeline that leverages AWS Bedrock and Pinecone to eliminate infrastructure complexity while demonstrating enterprise-grade patterns.

The Technical Stack

AWS Bedrock Integration:

• Titan Text Embeddings V2 for 1024-dimensional vectors with superior retrieval performance
• Claude Sonnet 4 for response generation with built-in safety guardrails
• Native AWS IAM integration for enterprise security and compliance
• Pay-per-use pricing that scales from $10/month POCs to enterprise workloads

Pinecone Vector Database:

• Serverless vector search with sub-second query latency
• Metadata filtering for source attribution and access control
• Automatic scaling that handles traffic spikes without configuration
• Free starter tier perfect for development and testing

FastAPI Application with Enterprise Patterns:

@app.post("/query", response_model=QueryResponse)
async def query_documents(request: QueryRequest, service: RAGService = Depends(get_rag_service)):
    """Process RAG queries with comprehensive error handling and monitoring."""
    start_time = time.time()
    
    # Generate query embedding using Bedrock Titan
    query_embedding = await service.generate_embedding(request.query)
    
    # Search similar chunks in Pinecone
    matches = await service.search_similar_chunks(
        query_embedding, 
        request.max_chunks or settings.top_k,
        request.similarity_threshold or settings.similarity_threshold
    )
    
    # Generate response using Claude Sonnet 4
    response = await service.generate_response(request.query, matches)
    
    processing_time = (time.time() - start_time) * 1000
    return QueryResponse(
        answer=response,
        query=request.query,
        sources=matches,
        processing_time_ms=processing_time
    )

What You Get Out of the Box

Complete Document Ingestion Pipeline:

• Support for local files and S3 buckets
• Intelligent text chunking with LangChain text splitters
• Batch processing for efficient embedding generation
• Comprehensive error handling and retry logic

FastAPI Server with Best Practices:

• Async request handling for high concurrency
• Pydantic validation for type safety
• Structured logging with contextual information
• Health checks for all external dependencies
• CORS middleware for web application integration

Developer Experience Tools:

• Interactive setup wizard that validates your environment
• Comprehensive test suite for API validation
• Docker containerization for consistent deployments
• Sample documents for immediate experimentation

Enterprise-Grade Patterns:

• IAM-based access control through AWS Bedrock
• Secrets management best practices
• Cost monitoring and optimization guidance
• Scalable deployment options (Lambda, ECS, Kubernetes)

Performance Characteristics

I optimized this implementation for real-world usage patterns:

• Query latency: Sub-2-second response times for most queries
• Concurrent users: Handles 50+ simultaneous requests
• Document processing: 1000+ documents per hour ingestion rate
• Cost efficiency: Linear scaling with predictable pricing

The vector database uses cosine similarity with 1024-dimensional Titan embeddings, providing superior retrieval accuracy compared to smaller embedding models. Claude Sonnet 4 generates responses with built-in citation capabilities, ensuring transparency in source attribution.

Getting Started: From Zero to RAG in 30 Minutes

The fastest path to a working RAG pipeline:

# Clone and setup
git clone https://github.com/ColeMurray/aws-rag-application.git
cd aws-rag-application
pip install -r requirements.txt
 
# Interactive configuration
python scripts/quickstart.py
 
# Ingest sample documents
python src/ingest.py --source-type local --path ./data
 
# Start the API server
python src/app.py\

The quickstart script handles environment validation, AWS credential verification, and service connectivity testing. Within minutes, you’ll have a working RAG API that can answer questions about your documents.

For production deployment, the included Docker configuration supports both local development and cloud deployment:

# Local development with hot reload
docker-compose up --build
 
# Production deployment
docker build -t rag-pipeline .
docker run -p 8000:8000 rag-pipeline

Why This Matters for Your Team

This isn’t just another RAG tutorial;it’s a demonstration of how we approach RAG implementations in our consulting practice. The application showcases modern best practices that we apply when building production systems for clients:

• Type-safe configuration with Pydantic prevents runtime errors
• Structured logging provides observability for production debugging
• Comprehensive error handling ensures graceful degradation
• Security-first design follows AWS IAM best practices
• Cost optimization leverages managed services to minimize operational overhead

Whether you’re building internal knowledge bases, customer support automation, or document analysis tools, this RAG pipeline demonstrates the architectural patterns and best practices that scale with your requirements.

The complete implementation, documentation, and deployment guides are available on GitHub. Start with the sample documents, then point the ingestion pipeline at your own data sources — S3 buckets, document repositories, or any text-based content.

Ready to build your RAG pipeline?

The code demonstrates proven patterns, the documentation is comprehensive, and the architecture showcases how to scale from prototype to enterprise. Clone the repository and start experimenting with your first intelligent document system this week.

Generating synthetic data with LLMs can remove the need to manually collect, annotate and review large datasets and be used to fine-tune smaller language models for machine local inference.

Imagine staring down an 80TB compressed dataset. That’s exactly where I found myself when I decided to explore the Common Crawl dataset. My first task? To classify the intent of each webpage. Simple enough in theory, but when you’re dealing with a dataset that makes most hard drives weep, the challenges stack up fast.

The Common Crawl Conundrum

For those unfamiliar, Common Crawl is a nonprofit that creates and maintains an open repository of web crawl data that can be accessed and analyzed by anyone. It’s a goldmine of information, but it’s also a behemoth that demands respect — and a solid game plan.

Calling OpenAI or Anthropic was not viable as it is slow and costly.

The challenges:
- 80TB+ of compressed data to process
- Significant latency for each API call
- Costs that would make even the most generous budget blush
- Rate limits

I needed a solution that was fast, cost-effective, and could scale to handle the enormity of Common Crawl. What if I could create a model that had the smarts of GPT-4 but could run locally at lightning speed?

The Lightbulb Moment: BERT Meets GPT-4

I decided to combine the power of GPT-4o for generating high-quality synthetic data with the efficiency of a fine-tuned BERT model.

The plan was simple:
1. Use GPT-4 to generate a diverse, high-quality dataset of synthetic web pages
2. Fine-tune a BERT model on this synthetic data
3. Deploy the fine-tuned model to classify the Common Crawl dataset locally

This approach offered several advantages:
- No need for API calls during classification, drastically reducing latency
- A one-time cost for data generation and training, rather than per-page API fees
- The ability to process the entire dataset on local hardware, solving scalability issues

In this tutorial, I’m going to walk you through this process step by step. Whether you’re grappling with your own massive dataset or just curious about pushing the boundaries of what’s possible with AI and big data, this guide is for you.

Let’s get started!

The Solution: GPT-4 Meets BERT

By leveraging the latest advancements in GPT-4, we can generate synthetic HTML data that’s not only realistic but also diverse. This approach allows us to create a custom model for classifying web pages without the need for a large, manually labeled dataset. Let me walk you through the process I’ve developed.

Step 1: Generating Synthetic HTML Data with GPT-4o

First things first, let’s set up our environment. You’ll need to install the OpenAI library:

pip install openai

We’re going to use GPT-4 to generate our synthetic HTML data.

import random
import json
import os
import logging
from openai import OpenAI
from datetime import datetime

# Set up logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# Add your key to env vars
client = OpenAI(api_key=os.environ['OPENAI_API_KEY'])

def generate_synthetic_html(page_type):
    prompt = f"""
    Generate an HTML page for a {page_type} website. Make it realistic and diverse.
    Include appropriate tags, headings, and content based on the page type.
    For an informational page, include a main question and its answer.
    For a navigational page, include a navigation menu and some basic content.
    For a commercial page, include product information and a call-to-action.
    For a transactional page, include a form or a clear action for the user to take.
    Return the result as a JSON object with two keys: 'html' for the HTML content, and 'summary' for a brief description of the page.
    """
    logging.info(f"Generating synthetic HTML for {page_type} page")
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are a web developer creating diverse and realistic HTML pages."},
            {"role": "user", "content": prompt}
        ],
        max_tokens=3000,
        n=1,
        response_format={ "type": "json_object" },
        stop=None,
        temperature=0.8
    )
    content = response.choices[0].message.content
    logging.info(content)
    return json.loads(response.choices[0].message.content)

def generate_dataset(num_samples):
    page_types = ["informational", "navigational", "commercial", "transactional"]
    dataset = []
    for i in range(num_samples):
        page_type = random.choice(page_types)
        logging.info(f"Generating sample {i+1}/{num_samples}")
        result = generate_synthetic_html(page_type)
        dataset.append((result['html'], page_type, result['summary']))
    return dataset

def write_dataset_to_file(dataset, filename):
    logging.info(f"Writing dataset to file: {filename}")
    with open(filename, 'w', encoding='utf-8') as f:
        json.dump(dataset, f, ensure_ascii=False, indent=2)
    logging.info(f"Dataset successfully written to {filename}")

def main():
    num_samples = 10  # Replace with your desired count
    logging.info(f"Starting synthetic data generation for {num_samples} samples")
    
    dataset = generate_dataset(num_samples)
    
    # Create a timestamp for the filename
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    filename = f"synthetic_dataset_{timestamp}.json"
    
    write_dataset_to_file(dataset, filename)
    
    logging.info("Synthetic data generation complete")

if __name__ == "__main__":
    main()

This script uses GPT-4 to create HTML content based on different page types, giving us a rich, diverse dataset to work with.

Step 2: Preparing the Data for BERT

Now that we have our synthetic data, it’s time to prepare it for BERT. Here’s where the magic happens:

pip install transformers[torch] torch datasets

import torch
from transformers import BertTokenizer, BertForSequenceClassification, Trainer, TrainingArguments
from datasets import Dataset
import json
import glob
import os

# Load the most recent dataset
def load_most_recent_dataset():
    list_of_files = glob.glob('synthetic_dataset_*.json')
    if not list_of_files:
        raise FileNotFoundError("No synthetic dataset files found.")
    latest_file = max(list_of_files, key=os.path.getctime)
    with open(latest_file, 'r', encoding='utf-8') as f:
        return json.load(f)

# Load the dataset
dataset = load_most_recent_dataset()

# Unpack our dataset
html_texts, labels, summaries = zip(*dataset)
label_map = {"informational": 0, "navigational": 1, "commercial": 2, "transactional": 3}
encoded_labels = [label_map[label] for label in labels]

# Create a Hugging Face Dataset
hf_dataset = Dataset.from_dict({
    "text": html_texts,
    "summary": summaries,
    "label": encoded_labels
})

# Now, let's tokenize our data
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

def tokenize_function(examples):
    combined_text = [f"{html}\n\nSummary: {summary}" for html, summary in zip(examples["text"], examples["summary"])]
    tokenized = tokenizer(combined_text, padding="max_length", truncation=True, max_length=1024)
    tokenized["labels"] = examples["label"]
    return tokenized

tokenized_dataset = hf_dataset.map(tokenize_function, batched=True, remove_columns=hf_dataset.column_names)

# Split into training and validation sets
train_val_dataset = tokenized_dataset.train_test_split(test_size=0.2)
train_dataset = train_val_dataset["train"]
val_dataset = train_val_dataset["test"]

Step 3: Fine-tuning BERT

Here’s where it all comes together. We’re going to fine-tune BERT on our synthetic dataset:

# Load pre-trained BERT model
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=4)

# Set up training arguments
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir="./logs",
    logging_steps=10,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    remove_unused_columns=False,
)

from transformers import DataCollatorWithPadding

data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

# Create Trainer instance
trainer = Trainer(
 model=model,
 args=training_args,
 train_dataset=train_dataset,
 eval_dataset=val_dataset,
 data_collator=data_collator
)

# Fine-tune the model
trainer.train()

# Save our fine-tuned model
model.save_pretrained("./fine_tuned_bert_page_intent")
tokenizer.save_pretrained("./fine_tuned_bert_page_intent")

The Result: A BERT Model That Understands Web Pages

After running this process, we end up with a BERT model fine-tuned on high-quality, diverse synthetic data. This model has the potential to classify web pages with between informational, navigational, commercial, and transactional content.

If you liked the content, be sure to follow for me. Also, follow and Twitter and feel free to reach out

• Cole Murray - Personal Website
• x.com

Introduction

In this article, you’ll learn how to build a code generation dataset using Large Language Models (LLMs). This tutorial will guide you through scraping code data, cleaning it from various artifacts, and refining it into a structured dataset. We’ll navigate the challenges and demonstrate the power of LLMs in transforming raw code into a valuable resource for code generation.

The Journey of Scraping Code

Scraping code involves extracting code snippets from various online sources. However, this raw code often contains unwanted elements like line numbers, HTML tags, or commentary that aren’t part of the actual code. Let’s start by scraping some code using Python’s requests library.

import requests
 
# URL of the webpage where the code is located
url = 'https://example.com/code-snippet'
 
# Sending a GET request to the URL
response = requests.get(url)
 
# Checking if the request was successful
if response.status_code == 200:
    # Extracting the text content from the response
    raw_html = response.text
    print("html fetched successfully:")
    print(raw_html)
else:
    print(f"Failed to retrieve page. Status code: {response.status_code}")

Extracting Code from HTML

After fetching the raw HTML content using Python’s requests library, the next challenge is to convert this HTML into a more readable, text-only format. This is where the html2text library becomes useful. It converts HTML into Markdown, which is much closer to the plain text we want. Here’s how you can use html2text to achieve this:

import html2text
 
# Initialize html2text converter
converter = html2text.HTML2Text()
converter.ignore_links = True  # Optionally, ignore converting links
converter.mark_code = True  # wrap code in [code]...[/code] tags
 
# Converting HTML to Markdown
markdown_text = converter.handle(raw_html)
 
print("Converted Markdown text:")
print(markdown_text)
 
code_blocks = re.findall(r'\[code\](.*?)\[/code\]', markdown_text, re.DOTALL)
print("Code blocks extracted:")
print(code_blocks)

Introducing LLMs in the Cleaning Process

The Initial Challenge

Consider the scraped code snippet in our example. It includes line numbers and annotations irrelevant to the actual code. Cleaning this manually is tedious and error-prone.

[code]
copy
    1
    2
    3
    $ curl https://api.openai.com/v1/models \
      -H "Authorization: Bearer $OPENAI_API_KEY" \
      -H "OpenAI-Organization: YOUR_ORG_ID"
[/code]

Using LLMs for Initial Cleaning

We can use LLMs to remove these artifacts. Here’s a simplified version of how we might use an LLM like GPT-3.5-turbo:

import openai
 
# The code snippet with artifacts
example = """[code] 1 2 3 4...[/code]"""
 
# Basic prompt for the LLM
prompt = f"Please clean the following code snippet by removing all non-code elements like line numbers and annotations:\n{example}"
 
# Send the request to the OpenAI API
response = openai.Completion.create(
    model="gpt-3.5-turbo-1106",
    prompt=prompt,
    max_tokens=1000
)
 
cleaned_code = response.choices[0].text.strip()
print("Cleaned Code:")
print(cleaned_code)

This script sends our code snippet to the LLM, which returns a cleaner version.

[code]
    curl https://api.openai.com/v1/models \
      -H "Authorization: Bearer $OPENAI_API_KEY" \
      -H "OpenAI-Organization: YOUR_ORG_ID"
[/code]

Cost Considerations

Using GPT-3.5-turbo for tasks like code cleaning or parsing documentation webpages, the expected cost per generation with this model ranges from $0.30 to $0.75 per 1000 generations depending on the code snippet length. While this might seem minimal at a glance, the expense can accumulate significantly in large-scale projects.

If a project entails parsing 100,000 snippets, at the higher end ($0.75 per 1000 generations), this cost escalates to $75. While this isn’t wildly expensive, it’s important to assess whether the complexity of the task justifies the use of such a high-end model.

Conclusion

In this post, we’ve seen how LLMs can be used to clean code scraped from the web. We’ve also seen how the cost of using such models can add up quickly. In my next post, I’ll show you how to use a cheaper model to achieve the same results.

Introduction

Recently, I undertook the task of rebuilding my personal website using Next.js. My goal was to build a basic static blog using next.js and host it on AWS Cloudfront and S3. Despite finding several tutorials, none seemed to fit my specific requirements, leading me to experiment through trial and error. In this blog post, I aim to share the effective solution I discovered, hoping to streamline this process for others.

Prerequisites

Before diving into the details, here are the prerequisites:

• A Next.js application ready for deployment.
• An active AWS account with necessary permissions.

Preparing Your Next.js Application

The first step involves preparing your Next.js application for deployment. Ensure your application is production-ready by running:

npm run build

and you have next.config.js configured in export mode

const nextConfig = {    
  output: 'export',    
  // other configs
}

This command generates a out folder containing optimized production builds of your application.

Issues with Next.js and AWS Cloudfront

While setting up my Next.js application on AWS Cloudfront using a modified dev-kit static build template, I encountered a significant challenge with path routing. This issue arises from the mismatch between the URL path on the website (like site.com/contact) and the corresponding file in the S3 bucket (contact.html).

Path Routing Problem

In a typical Next.js setup, navigating to site.com/contact expects a contact.html at the root. Most web servers can handle this by serving the contact.html file when the request URI is /contact. However, s3 is not a web server, and only understands key paths.

Using Next.js’s trailingSlash option creates nested files like contact/index.html, but this still doesn't solve the problem due to the .html extension mismatch.

Solution: Cloudfront Edge Function

To overcome this, I implemented a Cloudfront edge function to rewrite incoming request URIs. The function modifies the incoming URI to point to the correct HTML file in the S3 bucket. Below is the TypeScript code for the edge function:

exports.handler = async (event) => {
    const request = event.Records[0].cf.request;
    const uri = request.uri;

    if (uri === '/') {
        // turns "/" to "/index.html"
        request.uri += 'index.html';
    } else if (uri.endsWith('/')) {
        // turns "/foo/" to "/foo.html"
        request.uri = uri.slice(0, -1) + '.html';
    } else if (!uri.includes('.')) {
        // turns "/foo" to "/foo.html"
        request.uri += '.html';
    }

    return request;
};

Integration with AWS CDK

For those using the AWS Cloud Development Kit (CDK), integrating this solution is straightforward. Below is a CDK stack example that includes the edge function:

import {Certificate} from 'aws-cdk-lib/aws-certificatemanager';
import {
    CloudFrontWebDistribution,
    CloudFrontWebDistributionProps,
    OriginAccessIdentity,
    ViewerCertificate,
} from 'aws-cdk-lib/aws-cloudfront';
import {BlockPublicAccess, Bucket} from 'aws-cdk-lib/aws-s3';
import {BucketDeployment, ISource} from 'aws-cdk-lib/aws-s3-deployment';
import {Construct} from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import {Runtime, Code} from 'aws-cdk-lib/aws-lambda';
import {LambdaEdgeEventType} from 'aws-cdk-lib/aws-cloudfront';


export interface StaticSiteProps {
    readonly certificateArn?: string;
    readonly domainNames?: string[];
    readonly sourceAsset: ISource;
}

export class StaticSiteWithRewrite extends Construct {
    constructor(scope: Construct, id: string, props: StaticSiteProps) {
        super(scope, id);

        if ((props.certificateArn && !props.domainNames) || (!props.certificateArn && props.domainNames)) {
            throw new Error('Both certificateArn and domainNames must be provided if one is provided.');
        }

        // Lambda@Edge Function
        const edgeLambda = new lambda.Function(this, `${id}EdgeLambda`, {
            runtime: Runtime.NODEJS_LATEST,
            handler: 'index.handler',
            code: Code.fromInline(`
exports.handler = async (event) => {
    const request = event.Records[0].cf.request;
    const uri = request.uri;


    if (uri === '/') {
        // turns "/" to "/index.html"
        request.uri += 'index.html'
    } else if (uri.endsWith('/')) {
        // turns "/foo/" to "/foo.html"
        request.uri = uri.slice(0, -1) + '.html'
    } else if (!uri.includes('.')) {
        // turns "/foo" to "/foo.html"
        request.uri += '.html'
    }

    return request;
};
      `),
        });

        const bucket = new Bucket(this, `${id}WebsiteBucket`, {
            websiteIndexDocument: 'index.html',
            blockPublicAccess: BlockPublicAccess.BLOCK_ALL,
        });

        const originAccessIdentity = new OriginAccessIdentity(this, 'OAI');
        bucket.grantRead(originAccessIdentity);

        new BucketDeployment(this, `${id}DeployWebsite`, {
            sources: [props.sourceAsset],
            destinationBucket: bucket,
        });

        let distributionProps: CloudFrontWebDistributionProps = {
            originConfigs: [
                {
                    s3OriginSource: {
                        s3BucketSource: bucket,
                        originAccessIdentity: originAccessIdentity,
                    },
                    behaviors: [
                        {
                            isDefaultBehavior: true,
                            lambdaFunctionAssociations: [{
                                eventType: LambdaEdgeEventType.ORIGIN_REQUEST,
                                lambdaFunction: edgeLambda.currentVersion,
                            }],
                        }
                    ],
                },
            ],
        };

        if (props.certificateArn && props.domainNames) {
            distributionProps = {
                ...distributionProps,
                viewerCertificate: ViewerCertificate.fromAcmCertificate(
                    Certificate.fromCertificateArn(this, `${id}certificate`, props.certificateArn),
                    {aliases: props.domainNames},
                ),
            };
        }

        new CloudFrontWebDistribution(this, `${id}WebsiteDistribution`, distributionProps);
    }
}

In this CDK stack, we define a Lambda@Edge function that rewrites the request URI. This ensures that the path routing issue is resolved, making the Next.js application function correctly when hosted on AWS Cloudfront.

By implementing these solutions, I was able to successfully deploy my Next.js application on AWS Cloudfront and overcome the routing challenges. This approach can be beneficial for anyone facing similar issues with Next.js and AWS Cloudfront integration.

This article was first published on my personal site, click through for more content.

Retrieval-Augmented Generation (RAG) has been a game-changer in the field of natural language processing, especially for tasks that blend information retrieval with generative models. As part of my contributions to the field, I have recently implemented support for the FlagEmbeddingReranker in the llama_index library. This tool performs reranking, bringing more accuracy and relevance to the retrieval phase. In this article, I'll provide a comprehensive tutorial on the usage of FlagEmbeddingReranker, diving into its functionalities, the problems it solves, and its importance in improving RAG systems.

Introduction to FlagEmbeddingReranker

The FlagEmbeddingReranker is a class in the llama_index library designed for re-ranking nodes (or documents) based on their relevance to a given query. It leverages a pre-trained model to compute relevance scores and select the top N nodes. This re-ranking mechanism is particularly valuable in RAG, where the quality of retrieved documents directly impacts the generation quality.

The re-ranker can be used with any of the BAAI/bge-reranker models, such as BAAI/bge-reranker-base , BAAI/bge-reranker-large , BAAI/bge-reranker-large-en-v1.5

You can view the available models here

Why Use FlagEmbedding Reranking in RAG?

The latest BAAI/bge-reranker models, integral to the FlagEmbeddingReranker, are designed to:

• Directly Compute Relevance: They take a question and document as inputs and directly output a similarity score, providing a more nuanced measure of relevance compared to vector-based embedding models.
• Optimize Scoring Range: Optimized using cross-entropy loss, these models offer flexibility in scoring, not limited to a specific range.
• Adapt to Specific Use Cases: Their fine-tuning capabilities make them ideal for re-ranking the top-k documents returned by embedding models, ensuring high relevance and accuracy.

Setting Up

Before diving into the implementation, ensure you have the llama_index and FlagEmbedding packages installed. If not, install them using:

pip install llama-index FlagEmbedding

Implementing FlagEmbeddingReranker

Here’s how to incorporate FlagEmbeddingReranker into your RAG setup:

1. Import and Initialize

First, import necessary classes and initialize FlagEmbeddingReranker.

from llama_index.postprocessor.flag_embedding_reranker import FlagEmbeddingReranker

reranker = FlagEmbeddingReranker(
    top_n=3,
    model="BAAI/bge-reranker-large",
    use_fp16=False
)

2. Prepare Nodes and Query

Assume you have a list of NodeWithScore objects, each representing a document retrieved by your RAG's initial query phase.

from llama_index.schema import NodeWithScore, QueryBundle, TextNode

documents = [
    "Retrieval-Augmented Generation (RAG) combines retrieval and generation for NLP tasks.",
    "Generative Pre-trained Transformer (GPT) is a language generation model.",
    "RAG uses a retriever to fetch relevant documents and a generator to produce answers.",
    "BERT is a model designed for understanding the context of a word in a sentence."
]

nodes = [NodeWithScore(node=TextNode(text=doc)) for doc in documents]
query = "What is RAG in NLP?"

3. Re-Rank Nodes

Invoke FlagEmbeddingReranker with the nodes and query.

query_bundle = QueryBundle(query_str=query)
ranked_nodes = reranker._postprocess_nodes(nodes, query_bundle)

4. Analyze Results

The returned ranked_nodes will be a sorted list based on relevance to the query.

for node in ranked_nodes:
    print(node.node.get_content(), "-> Score:", node.score)

Expected Output

The output should reflect the relevance of each document to the query “What is RAG in NLP?”. Ideally, the document explicitly mentioning RAG and its role in NLP should rank higher.

1. “Retrieval-Augmented Generation (RAG) combines retrieval and generation for NLP tasks.” -> Higher score
2. “RAG uses a retriever to fetch relevant documents and a generator to produce answers.” -> Next highest score

Documents about GPT and BERT, while relevant to NLP, are less pertinent to the specific query about RAG and thus should score lower.

Conclusion

The FlagEmbeddingReranker equipped with the new BAAI/bge-reranker models is a powerful addition to any RAG pipeline. It ensures that your retrieval phase is not just fetching a wide array of documents but is accurately focusing on the most relevant ones. Whether you are developing sophisticated NLP applications or exploring the frontiers of language models, integrating this advanced reranking tool is pivotal in achieving high-quality, relevant outputs.

A guide on deploying Next.js on AWS amplify with AWS CDK

Table of Contents

1. Introduction
2. An Overview of AWS Amplify and Hosting Next.js
3. Prerequisites
4. Setting Up the Next.js Starter Kit
5. Introducing AWS CDK and Its Key Concepts
6. Configuring the AWS CDK Stack for Deployment
7. Deploying with AWS Amplify
8. Tips for Effective Maintenance
9. Conclusion

Introduction

While Vercel stands out as a popular platform for deploying Next.js applications, there are alternative paths that cater to specific requirements and offer greater flexibility. Enter AWS Amplify with AWS CDK. With this guide, we’ll explore the step-by-step process of deploying your Next.js app on AWS Amplify using AWS CDK. Ready to embark on this voyage? Let’s dive right in!

An Overview of AWS Amplify and Hosting Next.js

AWS Amplify

AWS Amplify is a development platform from Amazon Web Services (AWS) that makes it easy for developers to build and deploy applications. Amplify abstracts the process of configuring cloud services and provides a straightforward way to interact with them.

Key Features of AWS Amplify:

• Backend Functions: Simplifies setting up cloud services like authentication, APIs, storage, and more.
• Hosting & CI/CD: Automated deployment, hosting, and continuous integration/continuous deployment options are available out of the box.
• Scalability: Built on top of AWS, it offers immense scalability options for your applications.
• Integrated Development: Amplify’s libraries and UI components help in seamlessly integrating cloud services with your app.
• Monitoring with CloudWatch: AWS Amplify integrates seamlessly with CloudWatch, offering real-time monitoring of your Next.js application. This allows you to track and diagnose issues, set alarms, and gain insights into application performance and usage patterns.

Hosting Next.js on AWS Amplify

Next.js, a popular React framework with server-side rendering capabilities, can be efficiently hosted on AWS Amplify. The primary advantages include:

• SSR (Server-Side Rendering): AWS Amplify fully supports Next.js’s SSR, allowing for faster page loads and enhanced SEO capabilities.
• Environment Variables: Easy configuration and usage of environment variables to manage secrets and application settings.
• Continuous Deployment: Integrated CI/CD from your repository, ensuring that your Next.js app is always up to date on the cloud.
• Branching: Features like branch-based deployments make it simpler to manage different stages of your application.

In essence, AWS Amplify offers a comprehensive hosting solution for Next.js, merging the flexibility of AWS infrastructure with the simplicity of frontend-focused deployment.

— -

Prerequisites

Before we proceed, ensure:

- An AWS account is set up
- Node.js and npm are installed
- AWS CLI is configured with the right permissions
- You’re familiar with Next.js, AWS CDK, and pnpm (a fast, disk-space efficient package manager)

— -

Setting Up the Next.js Starter Kit

Kickstart your Next.js app:

1. Initialize a new project: npx create-next-app nextjs-cdk-app
2. Enter the project directory: cd nextjs-cdk-app

Introducing AWS CDK and Its Key Concepts

The AWS Cloud Development Kit (CDK) lets us define cloud resources using familiar programming languages. In this guide, we’re using TypeScript. Here are some elements you should be familiar with:

- App: The main unit where cloud components are defined.
- Stack: A collection of AWS resources.
- BuildSpec: A collection of build commands and related settings.

Configuring the AWS CDK Stack for Deployment

Here’s the meat of our deployment: the CDK Stack. Below is an outline of the steps our provided code will execute:

1. Define Role: The stack starts by defining a role for Amplify. This role has permissions to manage resources for our application. Note: This role is the role for Amplify’s build, and not the runtime role!
2. Source Code Provider: We’re linking our application to a GitHub repository. The GitHub token is stored securely in AWS Secret Manager. Each code change will trigger a rebuild and automatically update.
3. Amplify App Creation: Here, we configure and create our Amplify app.
4. Main Branch Attachment: We attach our main branch and define its settings.

Deploying with AWS Amplify

With our stack defined, here’s the step to deploy:

First, create a configuration file in config.ts

export const configuration = {
  accessTokenName: "__ACCESS_TOKEN__",
  repoOwner: "__REPO_OWNER__",
  repoName: "__REPO_NAME__",
}

import * as cdk from 'aws-cdk-lib';

import { AmplifyStack } from './stack';
import {configuration} from "./config";

const app = new cdk.App();

new AmplifyStack(app, `NextjsStack-${configuration.repoName}`, {
  description: 'Cloudformation stack containing the Amplify configuration',
});

Deploy the CDK stack: cdk deploy

AWS Amplify will then pick up the changes and initiate the build and deployment process.

Tips for Effective Maintenance

• Regularly Update: Ensure Next.js, AWS packages, and pnpm are up to date.
  - Monitoring: Use AWS CloudWatch to keep an eye on your application.
  - Backups: Regularly backup your app and database.
  - AWS Best Practices: Stay updated. AWS frequently revises its best practices.

Conclusion

Kudos! You’ve now deployed a Next.js app on AWS Amplify using AWS CDK. With this robust foundation, the opportunities are endless. You can find the complete code here

GitHub - devkit-io/nextjs-amplify-starter-kit

Simplified Deployment

If you’re feeling a bit lost with the CDK deployment steps, check out dev-kit.io/managed-nextjs. Dev-kit will automate the entire starter kit setup and deployment, giving a one-click template deploy into your AWS account, completely free.

Get started today!

Host Next.js on AWS: Effortless Deployment Your Own Account

In Plain English

Thank you for being a part of our community! Before you go:

• Be sure to clap and follow the writer! 👏
• You can find even more content at PlainEnglish.io 🚀
• Sign up for our free weekly newsletter. 🗞️
• Follow us on Twitter(X), LinkedIn, YouTube, and Discord.

Alternative to Vercel: Deploying a Next.js Application on AWS Amplify using AWS CDK was originally published in JavaScript in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ever wondered how the top tech companies deploy their applications so seamlessly? It all starts with an efficient CI/CD pipeline, the assembly line of the software world. In today’s fast-paced software development environment, mastering this process is crucial for delivering high-quality applications. But there’s another piece to the puzzle: ‘infrastructure as code.’ Imagine being able to write, test, and deploy your entire infrastructure using code! Sounds intriguing?

In this blog post, we will explore how to build a CI/CD pipeline for a serverless express.js backend application using AWS CDK (Cloud Development Kit). By using serverless architecture, you’ll not only streamline your development but also significantly reduce operating costs. By the end of this post, you’ll have a

• full CI/CD pipeline
• staging and production environment
• serverless-express lambda function
• error alarm with pipeline rollback functionality

Table of Contents

1. Prerequisites
2. High Level Architecture
3. Serverless Express
4. Backend Pipeline Stack CDK
5. Lambda Construct
6. Service API Stack
7. Deploying the project

Prerequisites:

• Understanding of Express.js
• Basic understanding of AWS CDK
• Docker installed

Complete code can be found here:

GitHub - ColeMurray/serverless-express-lambda-cdk: An express serverless lambda project deployed with AWS CDK

Glossary

• CI/CD Pipeline: Continuous Integration and Continuous Deployment/Delivery. It’s an automated process that allows code changes to be automatically built, tested, and deployed to production environments. This practice enhances software development speed, quality, and reliability.
• AWS CDK (Amazon Web Services Cloud Development Kit): A software development framework used to define cloud infrastructure as code (IAC) and provision it through AWS CloudFormation. The CDK allows developers to define resources using familiar programming languages such as Python, TypeScript, and Java, enabling more reusable and modular cloud service deployments.
• Express.js: A fast, unopinionated, minimalist web framework for Node.js, designed to simplify the process of building web applications and APIs. It provides a robust set of features for web and mobile applications, including routing, middleware support, and template engines, allowing for rapid development and deployment.

High-Level Architecture

Deployment Pipeline

Application Architecture

Serverless Express

Serverless Express is a powerful solution that allows developers to run REST APIs and other web applications using popular Node.js application frameworks such as Express, Koa, Hapi, Sails, and more. The beauty of Serverless Express lies in its ability to host these applications on scalable and cost-effective platforms like AWS Lambda and Amazon API Gateway or Azure Function while being able to use express.js for local development.

In this code block, we are setting up a serverless Express.js application using the @vendia/serverless-express package, which enables you to run your Express app on AWS Lambda and API Gateway.

Here, we define our express middleware, as well as our api routes.

Class Definition: BackendPipelineStack

The BackendPipelineStack class extends the CDK's Stack class, enabling the orchestration of various AWS resources.

Properties

The class accepts properties defined in the PipelineStackProps interface:

• Repo Name: The repository name for your backend code.
• Branch: The branch to pull the code from.
• Connection ARN: Amazon Resource Name for the connection.
• Pre-Prod and Prod Accounts: Account and region details for pre-production and production environments.
• Cross Environment Deployment: Optional flag for enabling cross-account keys.

Source Configuration

First, a CodePipelineSource is created by specifying the repository name, branch, and connection ARN. This source configures the origin of the code that will be built and deployed.

You can create a connection within code pipeline here:

https://us-west-2.console.aws.amazon.com/codesuite/settings/connections/create?origin=settings&region=us-west-2

Synthesis Step

The synthesis (synth) step is responsible for building the application. In this code, it involves the following commands:

• Navigating to the deployment directory.
• Installing dependencies using npm ci.
• Building the project with npm run build.
• Synthesizing the CDK resources using npx cdk synth.

Pipeline Creation

A CodePipeline is created to orchestrate the build and deployment:

• Cross Account Keys: Allows or restricts cross-account access.
• Synthesis Step: The aforementioned synthesis step.
• Self Mutation: If set to true, the pipeline will automatically update itself when changes are made to its definition.
• CodeBuild Defaults: Specifies the environment and policies for the build process.

Staging

The pipeline is divided into stages, including pre-production (preprod) and production (prod). These stages are created using the BackendServiceStage class, which encapsulates the logic for deploying the backend services.

• Pre-Production Stage: This stage is added to the pipeline without any prerequisites.
• Production Stage: Before deploying to production, a manual approval step is included. This ensures that changes are reviewed before being promoted to the production environment.

Lambda Construct

AWS CDK provides a powerful way to encapsulate complex constructs, allowing developers to build reusable modules. This construct represents an AWS Lambda function that’s ready for deployment, along with some error monitoring and deployment strategies.

Here’s a deep dive into what this construct does:

Properties

The ApprovedNodeLambda construct accepts properties defined in the ApprovedNodeLambdaProps interface, such as:

• Alarm Threshold: Defines a threshold of errors for triggering an alarm.
• Code Directory: The directory containing the source code.
• Build Commands: Optional build commands and environment for the Lambda function.
• Runtime Settings: These include the memory size, runtime duration, and environment variables.

Code Asset Definition

First, the construct defines an asset code by reading from the provided code directory and bundling it using a Docker image for Node.js 18.x. The code asset includes commands for installing dependencies, running a build, and copying the necessary files.

Lambda Function Definition

A Lambda function is created with the following properties:

• Runtime: It uses Node.js 18.x.
• Timeout: A timeout period for execution, with a default value of 1 minute.
• Memory Size: The amount of memory available for the function, defaulting to 2048 MB.
• Environment Variables: Optional environment variables to pass to the runtime.

Error Metrics and Alarm

The construct sets up a CloudWatch metric to monitor the ‘Errors’ metric for the Lambda function. An alarm is also created based on this metric, with a configurable threshold and evaluation period. If the error count surpasses the threshold, the alarm is triggered.

Lambda Deployment Group

Finally, an alias named ‘Current’ is created to represent the current version of the Lambda function. A LambdaDeploymentGroup is defined with this alias, setting up a linear deployment strategy that deploys 10% of the changes every minute. The alarm previously created is also associated with this deployment group.

Service Stack: Constructing the API Gateway

The DeploymentStack class is responsible for deploying the backend server, Lambda function, and defining the API Gateway's characteristics. Here's a walkthrough:

Lambda Function Definition

First, an instance of ApprovedNodeLambda is created to define the backend server Lambda function. This is where we specify the code directory, handler, and runtime environment.

const serverFunction = new ApprovedNodeLambda(this, 'backend-server', {
  codeDir: '../source/',
  description: 'backend server lambda function',
  handler: 'src/lambdaServer.handler',
  runtimeEnvironment: props.envVars ?? {}
});

API Gateway Definition

Next, we create an API Gateway, giving it a name and description. The binaryMediaTypes property is defined to accept all media types.

const api = new RestApi(this, 'api', {
  restApiName: 'BackendApi',
  description: 'Api gateway for backend api',
  binaryMediaTypes: ['*/*']
});

CORS Configuration and Integration

Finally, we configure the CORS policy and set the Lambda integration with the API root. This defines how the backend should interact with requests from different origins.

Backend Service Stage: Organizing Deployment Stages

The BackendServiceStage class represents a specific stage of deployment. It is used to define various deployment environments, such as development, testing, or production as well as other stacks that should deployed as part of this stage.

Deploying the Project

Update the configuration file

export const config = {
  codeRepository: "<YourCode/repo>,
  codeBranch: <BRANCH>",
  connectionArn: "arn:aws:codestar-connections:us-west-2:<ACCOUNT_ID>:connection/<CONNECTION_ID>",
  preprodAccount: {
    account: "<ACCOUNT_ID>",
    region: "<REGION_NAME>"
  },
  prodAccount: {
    account: "<ACCOUNT_ID>",
    region: "<REGION_NAME>"
  },
  crossEnvDeployqment: false
}

To deploy the project, run the aws-cdk command

npm install
npx cdk deploy

After the deployment completes, you can view your pipeline. You may need to update the below url specific to your AWS region

https://us-west-2.console.aws.amazon.com/codesuite/codepipeline/pipelines?region=us-west-2

Complete code can be found here

serverless-express-lambda-cdk/deployment at main · ColeMurray/serverless-express-lambda-cdk

Conclusion

By leveraging AWS CDK, we have successfully built a CI/CD pipeline for our backend application. This pipeline automatically triggers whenever changes are made to the repository, builds the application, and deploys it to both pre-production and production environments. With the ability to define infrastructure as code, we can easily manage and scale our backend application with confidence.

AWS CDK provides a powerful and flexible way to define and manage cloud resources, making it an excellent choice for building CI/CD pipelines. By automating the deployment process, we can reduce manual errors, increase efficiency, and deliver high-quality applications to our users.

Get an email whenever Cole Murray publishes.

Building a CI/CD Pipeline for a Serverless Express Application with AWS CDK was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my last tutorial, you learned about how to combine a convolutional neural network and Long short-term memory (LTSM) to create captions given an image. In this tutorial, you’ll learn how to build and train a multi-task machine learning model to predict the age and gender of a subject in an image.

Overview

• Introduction to age and gender model
• Building a Multi-task Tensorflow Estimator
• Training

Prerequisites

• basic understanding of convolutional neural networks (CNN)
• basic understanding of TensorFlow
• GPU (optional)

Introduction to Age and Gender Model

In 2015, researchers from Computer Vision Lab, D-ITET, published a paper DEX and made public their IMDB-WIKI consisting of 500K+ face images with age and gender labels.

DEX outlines an neural network architecture involving a pretrained imagenet vgg16 model that estimates the apparent age in face images. DEX placed first in ChaLearn LAP 2015 — a competition that deals with recognizing people in an image — outperforming human reference.

Age as a classification problem

A conventional way of tackling an age estimation problem with an image as input would be using a regression-based model with mean-squared error as the loss function. DEX models this problem as a classification task, using a softmax classifier with each age represented as a unique class ranging from 1 to 101 and cross-entropy as the loss function.

Multi-task learning

Multi-task learning is a technique of training on multiple tasks through a shared architecture. Layers at the beginning of the network will learn a joint generalized representation, preventing overfitting to a specific task that may contain noise.

By training with a multi-task network, the network can be trained in parallel on both tasks. This reduces the infrastructure complexity to only one training pipeline. Additionally, the computation required for training is reduced as both tasks are trained simultaneously.

Building a multi-task network in TensorFlow

Below you’ll use TensorFlow’s estimator abstraction to create the model. The model will be trained from raw image input to predict the age and gender of the face image.

Project Structure

.
├── Dockerfile
├── age_gender_estimation_tutorial
│   ├── cnn_estimator.py
│   ├── cnn_model.py
│   └── dataset.py
├── bin
│   ├── download-imdb.sh
│   ├── predict.py
│   ├── preprocess_imdb.py
│   └── train.py
├── requirements.txt

Environment

For the environment, you’ll use Docker to install dependencies. A GPU version is also provided for convenience.

docker build -t colemurray/age-gender-estimation-tutorial -f Dockerfile .

Data

To train this model, you’ll use the IMDB-WIKI dataset, consisting of 500K+ images. For simplicity, you’ll download the pre-cropped imdb images (7GB). Run the script below to download the data.

chmod +x bin/download-imdb-crop.sh

./bin/download-imdb-crop.sh

Preprocessing

You’ll now process the dataset to clean out low-quality images and crop the input to a fixed image size. Additionally, you’ll format the data as a CSV to simplify reading into TensorFlow.

docker run -v $PWD:/opt/app \
-e PYTHONPATH=$PYTHONPATH:/opt/app \
-it colemurray/age-gender-estimation-tutorial \
python3 /opt/app/bin/preprocess_imdb.py \
--db-path /opt/app/data/imdb_crop/imdb.mat \
--photo-dir /opt/app/data/imdb_crop \
--output-dir /opt/app/var \
--min-score 1.0 \
--img-size 224

After approximately 20 minutes, you’ll have a processed dataset.

Next, you’ll use TensorFlow’s data pipeline module tf.data to provide data to the estimator. Tf.data is an abstraction to read and manipulate a dataset in parallel, utilizing C++ threads for performance.

Here, you’ll utilize TensorFlow’s CSV Reader to parse the data, preprocess the images, create batches, and shuffle.

Model

Below, you’ll create a basic CNN model. The model consists of three convolutions and two fully connected layers, with a softmax classifier head for each task.

Joint loss function

For the training operation, you’ll use the Adam Optimizer. For a loss function, you’ll average the cross-entropy error of each head, creating a shared loss function between the heads.

TensorFlow estimator

TensorFlow estimators provide a simple abstraction for graph creation and runtime processing. TensorFlow has specified an interface model_fn, that can be used to create custom estimators.

Below, you’ll take the network created above and create training, eval, and predict. These specifications will be used by TensorFlow’s estimator class to alter the behavior of the graph.

Train

Now that you’ve preprocessed the data and created the model architecture and data pipeline, you’ll begin training the model.

docker run -v $PWD:/opt/app \
-e PYTHONPATH=$PYTHONPATH:/opt/app \
-it colemurray/age-gender-estimation-tutorial:gpu \
python3 /opt/app/bin/train.py \
--img-dir /opt/app/var/crop \
--train-csv /opt/app/var/train.csv \
--val-csv /opt/app/var/val.csv \
--model-dir /opt/app/var/cnn-model \
--img-size 224 \
--num-steps 200000

Predict

Below, you’ll load your age and gender TensorFlow model. The model will be loaded from disk and predict on the provided image.

# Update the model path below with your model
docker run -v $PWD:/opt/app \
-e PYTHONPATH=$PYTHONPATH:/opt/app \
-it colemurray/age-gender-estimation-tutorial \
python3 /opt/app/bin/predict.py \
--image-path /opt/app/var/crop/25/nm0000325_rm2755562752_1956-1-7_2002.jpg \
--model-dir /opt/app/var/cnn-model-3/serving/<TIMESTAMP>

Conclusion

In this tutorial, you learned how to build and train a multi-task network for predicting a subject’s age and image. By using a shared architecture, both targets can be trained and predicted simultaneously.

Next Steps:

• Evaluate on Your Own Dataset
• Try a different network architecture
• Experiment with Different Hyperparameters

Questions/issues? Open an issue here on GitHub

Complete code here.

Call to Action

If you enjoyed this tutorial, follow and recommend!

Interested in learning more about Deep Learning / Machine Learning? Check out my other tutorials:

- Building an image caption generator with Deep Learning in Tensorflow

- Building a Facial Recognition Pipeline with Deep Learning in Tensorflow

- Deep Learning CNN’s in Tensorflow with GPUs

- Deep Learning with Keras on Google Compute Engine

- Recommendation Systems with Apache Spark on Google Compute Engine

Other places you can find me:

Cole Murray (@_ColeMurray) | Twitter

How to build an age and gender multi-task predictor with deep learning in TensorFlow was originally published in We’ve moved to freeCodeCamp.org/news on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my last tutorial, you learned how to create a facial recognition pipeline in Tensorflow with convolutional neural networks. In this tutorial, you’ll learn how a convolutional neural network (CNN) and Long Short Term Memory (LSTM) can be combined to create an image caption generator and generate captions for your own images.

Overview

• Introduction to Image Captioning Model Architecture
• Captions as a Search Problem
• Creating Captions in Tensorflow

Prerequisites

• Basic understanding of Convolutional Neural Networks
• Basic understanding of LSTM
• Basic understanding of Tensorflow

Introduction to image captioning model architecture

Combining a CNN and LSTM

In 2014, researchers from Google released a paper, Show And Tell: A Neural Image Caption Generator. At the time, this architecture was state-of-the-art on the MSCOCO dataset. It utilized a CNN + LSTM to take an image as input and output a caption.

Using a CNN for image embedding

A convolutional neural network can be used to create a dense feature vector. This dense vector, also called an embedding, can be used as feature input into other algorithms or networks.

For an image caption model, this embedding becomes a dense representation of the image and will be used as the initial state of the LSTM.

LSTM

An LSTM is a recurrent neural network architecture that is commonly used in problems with temporal dependences. It succeeds in being able to capture information about previous states to better inform the current prediction through its memory cell state.

An LSTM consists of three main components: a forget gate, input gate, and output gate. Each of these gates is responsible for altering updates to the cell’s memory state.

For a deeper understanding of LSTM’s, visit Chris Olah’s post.

Prediction with image as initial state

In a sentence language model, an LSTM is predicting the next word in a sentence. Similarly, in a character language model, an LSTM is trying to predict the next character, given the context of previously seen characters.

In an image caption model, you will create an embedding of the image. This embedding will then be fed as initial state into an LSTM. This becomes the first previous state to the language model, influencing the next predicted words.

At each time-step, the LSTM considers the previous cell state and outputs a prediction for the most probable next value in the sequence. This process is repeated until the end token is sampled, signaling the end of the caption.

Captions as a search problem

Generating a caption can be viewed as a graph search problem. Here, the nodes are words. The edges are the probability of moving from one node to another. Finding the optimal path involves maximizing the total probability of a sentence.

Sampling and choosing the most probable next value is a greedy approach to generating a caption. It is computationally efficient, but can lead to a sub-optimal result.

Given all possible words, it would not be computationally/space efficient to calculate all possible sentences and determine the optimal sentence. This rules out using a search algorithm such as Depth First Search or Breadth First Search to find the optimal path.

Beam Search

Beam search is a breadth-first search algorithm that explores the most promising nodes. It generates all possible next paths, keeping only the top N best candidates at each iteration.

As the number of nodes to expand from is fixed, this algorithm is space-efficient and allows more potential candidates than a best-first search.

Review

Up to this point, you’ve learned about creating a model architecture to generate a sentence, given an image. This is done by utilizing a CNN to create a dense embedding and feeding this as initial state to an LSTM. Additionally, you’ve learned how to generate better sentences with beam search.

In the next section, you’ll learn to generate captions from a pre-trained model in Tensorflow.

Creating captions in Tensorflow

# Project Structure

├── Dockerfile
├── bin
│ └── download_model.py
├── etc
│ ├── show-and-tell-2M.zip
│ ├── show-and-tell.pb
│ └── word_counts.txt
├── imgs
│ └── trading_floor.jpg
├── medium_show_and_tell_caption_generator
│ ├── __init__.py
│ ├── caption_generator.py
│ ├── inference.py
│ ├── model.py
│ └── vocabulary.py
└── requirements.txt

Environment setup

Here, you’ll use Docker to install Tensorflow.

Docker is a container platform that simplifies deployment. It solves the problem of installing software dependencies onto different server environments. If you are new to Docker, you can read more here. To install Docker, run:

curl https://get.docker.com | sh

After installing Docker, you’ll create two files. A requirements.txt for the Python dependencies and a Dockerfile to create your Docker environment.

To build this image, run:

$ docker build -t colemurray/medium-show-and-tell-caption-generator -f Dockerfile .

# On MBP, ~ 3mins
# Image can be pulled from dockerhub below

If you would like to avoid building from source, the image can be pulled from dockerhub using:

docker pull colemurray/medium-show-and-tell-caption-generator # Recommended

Download the model

Below, you’ll download the model graph and pre-trained weights. These weights are from a training session on the MSCOCO dataset for 2MM iterations.

To download, run:

docker run -e PYTHONPATH=$PYTHONPATH:/opt/app -v $PWD:/opt/app \
-it colemurray/medium-show-and-tell-caption-generator \
python3 /opt/app/bin/download_model.py \
--model-dir /opt/app/etc

Next, create a model class. This class is responsible for loading the graph, creating image embeddings, and running an inference step on the model.

Download the vocabulary

When training an LSTM, it is standard practice to tokenize the input. For a sentence model, this means mapping each unique word to a unique numeric id. This allows the model to utilize a softmax classifier for prediction.

Below, you’ll download the vocabulary used for the pre-trained model and create a class to load it into memory. Here, the line number represents the numeric id of the token.

# File structure
# token num_of_occurrances

# on 213612
# of 202290
# the 196219
# in 182598

curl -o etc/word_counts.txt https://raw.githubusercontent.com/ColeMurray/medium-show-and-tell-caption-generator/master/etc/word_counts.txt

To store this vocabulary in memory, you’ll create a class responsible for mapping words to ids.

Creating a caption generator

To generate captions, first you’ll create a caption generator. This caption generator utilizes beam search to improve the quality of sentences generated.

At each iteration, the generator passes the previous state of the LSTM (initial state is the image embedding) and previous sequence to generate the next softmax vector.

The top N most probable candidates are kept and utilized in the next inference step. This process continues until either the max sentence length is reached or all sentences have generated the end-of-sentence token.

Next, you’ll load the show and tell model and use it with the above caption generator to create candidate sentences. These sentences will be printed along with their log probability.

Results

To generate captions, you’ll need to pass in one or more images to the script.

docker run -v $PWD:/opt/app \
-e PYTHONPATH=$PYTHONPATH:/opt/app \
-it colemurray/medium-show-and-tell-caption-generator  \
python3 /opt/app/medium_show_and_tell_caption_generator/inference.py \
--model_path /opt/app/etc/show-and-tell.pb \
--input_files /opt/app/imgs/trading_floor.jpg \
--vocab_file /opt/app/etc/word_counts.txt

You should see output:

Captions for image trading_floor.jpg:
 0) a group of people sitting at tables in a room . (p=0.000306)
 1) a group of people sitting around a table with laptops . (p=0.000140)
 2) a group of people sitting at a table with laptops . (p=0.000069)

Conclusion

In this tutorial, you learned:

• how a convolutional neural network and LSTM can be combined to generate captions to an image
• how to utilize the beam search algorithm to consider multiple captions and select the most probable sentence.

Complete code here.

Next Steps:

• Try with your own images
• Read the Show and Tell paper
• Create an API to serve captions

Call to Action:

If you enjoyed this tutorial, follow and recommend!

Interested in learning more about Deep Learning / Machine Learning? Check out my other tutorials:

- Building a Facial Recognition Pipeline with Deep Learning in Tensorflow

- Deep Learning CNN’s in Tensorflow with GPUs

- Deep Learning with Keras on Google Compute Engine

- Recommendation Systems with Apache Spark on Google Compute Engine

Other places you can find me:

- Twitter: https://twitter.com/_ColeMurray

Building an image caption generator with Deep Learning in Tensorflow was originally published in We’ve moved to freeCodeCamp.org/news on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my last tutorial , you learned about convolutional neural networks and the theory behind them. In this tutorial, you’ll learn how to use a convolutional neural network to perform facial recognition using Tensorflow, Dlib, and Docker.

Overview

• Introduction to Facial Recognition
• Preprocessing Images using Facial Detection and Alignment
• Generating Facial Embeddings in Tensorflow
• Training an SVM Classifier

Prerequisites

• Basic understanding of Linear Algebra
• Basic understanding of Convolutional Neural Networks
• Basic Tensorflow understanding

Introduction to Facial Recognition Systems

Facial recognition is a biometric solution that measures unique characteristics about one’s face. Applications available today include flight checkin, tagging friends and family members in photos, and “tailored” advertising.

To perform facial recognition, you’ll need a way to uniquely represent a face. In 1960, Woodrow Bledsoe used a technique involving marking the coordinates of prominent features of a face. Among these features were the location of hairline, eyes and nose.

In 2015, researchers from Google released a paper, FaceNet, which uses a convolutional neural network relying on the image pixels as the features, rather than extracting them manually. It achieved a new record accuracy of 99.63% on the LFW dataset.

FaceNet: In the FaceNet paper, a convolutional neural network architecture is proposed. For a loss function, FaceNet uses “triplet loss”. Triplet loss relies on minimizing the distance from positive examples, while maximizing the distance from negative examples.

Conceptually, this makes sense. Faces of the same identity should appear closer to each other than faces of another identity.

Vector Embeddings: For this tutorial, the important take away from the paper is the idea of representing a face as a 128-dimensional embedding. An embedding is the collective name for mapping input features to vectors. In a facial recognition system, these inputs are images containing a subject’s face, mapped to a numerical vector representation.

Since these vector embeddings are represented in shared vector space, vector distance can be used to calculate the similarity between two vectors. In a facial recognition context, this can vector distance be applied to calculate how similar two faces are. Additionally, these embeddings can be used as feature inputs into a classification, clustering, or regression task.

Preprocessing Data using Dlib and Docker

# Project Structure

├── Dockerfile
├── etc
│ ├── 20170511–185253
│ │ ├── 20170511–185253.pb
├── data
├── medium_facenet_tutorial
│ ├── align_dlib.py
│ ├── download_and_extract_model.py
│ ├── __init__.py
│ ├── lfw_input.py
│ ├── preprocess.py
│ ├── shape_predictor_68_face_landmarks.dat
│ └── train_classifier.py
├── requirements.txt

Preparing the Data

You’ll use the LFW (Labeled Faces in the Wild) dataset as training data. The directory is structured as seen below. You can replace this with your dataset by following the same structure.

# Download lfw dataset
$ curl -O http://vis-www.cs.umass.edu/lfw/lfw.tgz # 177MB
$ tar -xzvf lfw.tgz

# Directory Structure
# ├── Tyra_Banks
# │ ├── Tyra_Banks_0001.jpg
# │ └── Tyra_Banks_0002.jpg
# ├── Tyron_Garner
# │ ├── Tyron_Garner_0001.jpg
# │ └── Tyron_Garner_0002.jpg

Preprocessing

Below, you’ll pre-process the images before passing them into the FaceNet model. Image pre-processing in a facial recognition context typically solves a few problems. These problems range from lighting differences, occlusion, alignment, segmentation. Below, you’ll address segmentation and alignment.

First, you’ll solve the segmentation problem by finding the largest face in an image. This is useful as our training data does not have to be cropped for a face ahead of time.

Second, you’ll solve alignment. In photographs, it is common for a face to not be perfectly center aligned with the image. To standardize input, you’ll apply a transform to center all images based on the location of eyes and bottom lip.

Environment Setup

Here, you’ll use docker to install tensorflow, opencv, and Dlib. Dlib provides a library that can be used for facial detection and alignment. These libraries can be a bit difficult to install, so you’ll use Docker for the install.

Docker is a container platform that simplifies deployment. It solves the problem of installing software dependencies onto different server environments. If you are new to docker, you can read more here. To install docker, run

curl https://get.docker.com | sh

After installing docker, you’ll create two files. A requirements.txt for the python dependencies and a Dockerfile to create your docker environment.

To build this image, run:

$ docker build -t colemurray/medium-facenet-tutorial -f Dockerfile .

# This can take several minutes depending on your hardware
# On MBP, ~ 25mins
# Image can be pulled from dockerhub below

If you would like to avoid building from source, the image can be pulled from dockerhub using:

docker pull colemurray/medium-facenet-tutorial # Recommended

# A GPU supported environment can be found here
# nvidia-docker pull colemurray/medium-facenet-tutorial:latest-gpu

Detect, Crop & Align with Dlib

After creating your environment, you can begin preprocessing.

First download dlib’s face landmark predictor.

$ curl -O http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
$ bzip2 -d shape_predictor_68_face_landmarks.dat.bz2

You’ll use this face landmark predictor to find the location of the inner eyes and bottom lips of a face in an image. These coordinates will be used to center align the image.

This file, sourced from CMU, provides methods for detecting a face in an image, finding facial landmarks, and alignment given these landmarks.

Next, you’ll create a preprocessor for your dataset. This file will read each image into memory, attempt to find the largest face, center align, and write the file to output. If a face cannot be found in the image, logging will be displayed to console with the filename.

As each image can be processed independently, python’s multiprocessing is used to process an image on each available cpu core.

Getting Results

Now that you’ve created a pipeline, time to get results. As the script supports parallelism, you will see increased performance by running with multiple cores. You’ll need to run the preprocessor in the docker environment to have access to the installed libraries.

Below, you’ll mount your project directory as a volume inside the docker container and run the preprocessing script on your input data. The results will be written to a directory specified with command line arguments.

$ docker run -v $PWD:/medium-facenet-tutorial \
-e PYTHONPATH=$PYTHONPATH:/medium-facenet-tutorial \
-it colemurray/medium-facenet-tutorial python3 /medium-facenet-tutorial/medium_facenet_tutorial/preprocess.py \
--input-dir /medium-facenet-tutorial/data \
--output-dir /medium-facenet-tutorial/output/intermediate \
--crop-dim 180

Code up to this point can be found here

Review

Using Dlib, you detected the largest face in an image and aligned the center of the face by the inner eyes and bottom lip. This alignment is a method for standardizing each image for use as feature input.

Creating Embeddings in Tensorflow

Now that you’ve preprocessed the data, you’ll generate vector embeddings of each identity. These embeddings can then be used as input to a classification, regression or clustering task.

Download Weights

You’ll use the Inception Resnet V1 as your convolutional neural network. First, create a file to download the weights to the model.

By using pre-trained weights, you are able to apply transfer learning to a new dataset, in this tutorial the LFW dataset:

$ docker run -v $PWD:/medium-facenet-tutorial \
-e PYTHONPATH=$PYTHONPATH:/medium-facenet-tutorial \
-it colemurray/medium-facenet-tutorial python3 /medium-facenet-tutorial/medium_facenet_tutorial/download_and_extract_model.py \
--model-dir /medium-facenet-tutorial/etc

Load Embeddings

Below, you’ll utilize Tensorflow’s queue api to load the preprocessed images in parallel. By using queues, images can be loaded in parallel using multi-threading. When using a GPU, this allows image preprocessing to be performed on CPU, while matrix multiplication is performed on GPU.

Train a Classifier

With the input queue squared away, you’ll move on to creating the embeddings.

First, you’ll load the images from the queue you created. While training, you’ll apply preprocessing to the image. This preprocessing will add random transformations to the image, creating more images to train on.

These images will be fed in a batch size of 128 into the model. This model will return a 128 dimensional embedding for each image, returning a 128 x 128 matrix for each batch.

After these embeddings are created, you’ll use them as feature inputs into a scikit-learn’s SVM classifier to train on each identity. Identities with less than 10 images will be dropped. This parameter is tunable from command-line.

$ docker run -v $PWD:/medium-facenet-tutorial \
-e PYTHONPATH=$PYTHONPATH:/medium-facenet-tutorial \
-it colemurray/medium-facenet-tutorial \
python3 /medium-facenet-tutorial/medium_facenet_tutorial/train_classifier.py \
--input-dir /medium-facenet-tutorial/output/intermediate \
--model-path /medium-facenet-tutorial/etc/20170511-185253/20170511-185253.pb \
--classifier-path /medium-facenet-tutorial/output/classifier.pkl \
--num-threads 16 \
--num-epochs 25 \
--min-num-images-per-class 10 \
--is-train 

# ~16 mins to complete on MBP

Evaluating the Results

Now that you’ve trained the classifier, you’ll feed it new images it has not trained on. You’ll remove the is_train flag from the previous command to evaluate your results.

docker run -v $PWD:/$(basename $PWD) \
-e PYTHONPATH=$PYTHONPATH:/medium-facenet-tutorial \
-it colemurray/medium-facenet-tutorial \
python3 /medium-facenet-tutorial/medium_facenet_tutorial/train_classifier.py \
--input-dir /medium-facenet-tutorial/output/intermediate \
--model-path /medium-facenet-tutorial/etc/20170511-185253/20170511-185253.pb \
--classifier-path /medium-facenet-tutorial/output/classifier.pkl \
--num-threads 16 \
--num-epochs 5 \
--min-num-images-per-class 10

After inference is on each image is complete, you’ll see results printed to console. At 5 epochs, you’ll see ~85.0% accuracy. Training @ 25 epochs gave results:

Conclusion

In this tutorial, you learned about the history of machine learning and how to implement a state of the art pipeline. You utilized docker to manage your library dependencies, offering a consistent environment that is platform agnostic. You used Dlib for preprocessing and Tensorflow + Scikit-learn for training a classifier capable of predicting an identity based on an image.

Complete Code Here:

ColeMurray/medium-facenet-tutorial

Next Steps:

• Test on your own dataset!
• Experiment with different hyper parameters
• Train on other labels such as gender or age
• Implement a clustering algorithm to group similar faces

Call to Action:

If you enjoyed this tutorial, follow and recommend!

Interested in learning more about Deep Learning / Machine Learning? Check out my other tutorials:

- Deep Learning CNN’s in Tensorflow with GPUs

- Deep Learning with Keras on Google Compute Engine

- Recommendation Systems with Apache Spark on Google Compute Engine

Other places you can find me:

- Twitter: https://twitter.com/_ColeMurray

Building a Facial Recognition Pipeline with Deep Learning in Tensorflow was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.