In 2013, Target’s massive data breach exposed 40 million credit card details and cost the company $292 million in settlements. Attackers infiltrated the network through a third-party HVAC vendor — a risk overlooked during manual threat assessments

Why Threat Modeling Matters (And Why It’s Broken)

The Equifax and Target breaches weren’t anomalies — they were preventable. Both catastrophes stemmed from failures to systematically identify risks during development. Threat modeling, using frameworks like STRIDE, could have flagged these oversights before attackers struck. For example:

• Equifax’s unpatched Apache Struts vulnerability represented a classic Information Disclosure risk.
• Target’s third-party HVAC vendor access was an Elevation of Privilege loophole.

Yet, most organizations still rely on manual processes, overburdened experts, and outdated checklists — a recipe for disaster. Here’s why traditional methods fail:

Human Bottlenecks: A global shortage of 3.4 million cybersecurity professionals (ISC² 2022) leaves teams drowning in alerts. Equifax’s engineers knew about the Apache Struts flaw but lacked bandwidth to prioritize it amid competing tasks.

Inconsistency: Manual reviews miss critical patterns. Target’s threat model likely focused on internal systems, ignoring third-party attack surfaces like HVAC vendors — a fatal oversight.

Speed: Equifax left the vulnerability unpatched for 78 days — plenty of time for attackers to exploit it. Manual threat modeling cycles can’t keep pace with DevOps pipelines deploying code every 11 seconds (GitLab 2023).

AI-driven automation fixes this. By applying STRIDE systematically, AI scans PRDs for risks like:

• Tampering: Unpatched dependencies (e.g., Equifax’s Apache Struts).
• Spoofing/Elevation of Privilege: Weak third-party access controls (e.g., Target’s HVAC vendor).

It generates actionable reports in minutes, not months.

The AI-Driven Solution: A Technical Deep Dive

A practical implementation of AI-powered threat modeling. Let’s dissect its workflow:

1. Inputs: PRD and Threat Modeling Template

• sample_prd: A Product Requirements Document (PRD) outlining system functionality. It can be a list of documents, flow diagrams etc.
• threat_modeling_template [prompt]: A markdown template/prompt defining the structure for the threat report (e.g., risk categories, mitigation steps).

2. Crafting the AI Prompt

You can create a get_prompt() function. The get_prompt function constructs a detailed instruction set for GPT-4, combining:

• Role Definition: “You are a cybersecurity expert with 10+ years of experience.”
• Task: Analyze the PRD using the STRIDE model and populate the template.
• Formatting Rules: Use ** bold ** key terms and adhere strictly to Markdown.

3. Querying AI models and Generating the Final Report

Your Tool sends the prompt to GPT-4 via OpenAI’s API or Claude Sonnet 4, DeepSeek (R1), or any other AI tool, with a low temperature of 0.2 to prioritize accuracy over creativity. The model returns a structured markdown report based on your template.

Note: A better prompt will give you a better response.

Sample PRD — — — — — — — — — — — -

For a full PRD document, please refer here.

Sample Threat Modeling Report — — — — —

For a Full report, please refer to this link.

Possible Implementation: A User will create a JIRA ticket and will attach the PRD; the automation will pick this PRD and will do the threat modelling and attach it back to the JIRA ticket. It can provide the checklist as well to the developers so they can check what they have implemented and what’s left.

Thank you for reading.

If you liked the blog, follow me on Twitter/LinkedIn.

We’ve all been there.
The team has spent countless hours documenting everything—onboarding guides, deployment steps, architectural decisions, and troubleshooting steps. It’s all in Confluence, searchable, and yet… the same questions keep landing in Slack.

“Hey, where’s the link to that dashboard again?”
“What’s the process for production deployments?”
“Who owns the XYZ microservice?”

At some point, it stops being a documentation problem. The information exists, but no one is reading it.

This got us thinking:
What if we could make documentation more accessible? Not just as static pages, but as something you could talk to.

The Challenge

Most teams aren’t short on documentation. The real problem is retrieval and friction.

• Searching Confluence manually isn’t always intuitive.
• Context-switching to find the right page disrupts flow.
• And often, it’s just easier to ask a teammate.

The result? High-context, repetitive questions that interrupt everyone’s time.

We wanted to explore whether we could build a lightweight assistant that understands your internal documentation and answers questions based on it.

The Idea

What we’re exploring is this:

Can we build an internal chatbot that reads Confluence and answers your team’s questions — instantly, privately, and accurately?

The idea is simple:

• Ingest content from Confluence pages.
• Generate embeddings to represent meaning, not just keywords.
• Store these using a vector database.
• Query this store with natural language.
• Respond using an LLM based on the most relevant documents.

Imagine asking:

“How do I request an AWS role?”
“What’s the incident response process?”

And getting a clear, context-aware answer — within seconds — based on your docs.

💡 Here Comes the ConfluenceGPT

We built a proof-of-concept project: ConfluenceGPT — a simple, local-first chatbot that can read your Confluence pages and answer your team’s questions using LLMs.

Let’s break down how it works and what it includes.

🔧 How It Works — Architecture Overview

Document Ingestion: We use the Confluence API to extract the raw text content from specified Confluence pages and sections. This is handled by the script: ingest_confluence_to_chroma.py

• Pulls data from Confluence using basic auth or token.
• Cleans and chunks the documents using LangChain’s TextSplitter
• Converts text into vector embeddings using a Hugging Face model.
• Stores these vectors locally in a Chroma DB.

Model Caching (Offline Ready): Since Hugging Face models are used, we’ve included: cacheHuggingFaceModel.py

• Downloads and caches the BAAI/bge-base-en-v1.5 model locally.
• Ensures the system doesn’t rely on internet access to use the model later.

Question Answering with LLM + Embeddings

This is the heart of the project — querying your Confluence documents using natural language: query_chroma_with_llm.py

• Takes a user’s question as input.
• Uses BAAI/bge-base-en-v1.5 to generate embedding for the question.
• Searches Chroma DB for the most relevant chunks.
• Sends the context to an LLM (like GPT, Mistral, or OpenChat) via LangChain. (Refer to Note)(*You can host an in-house model as well, to use it.)
• Returns an answer based on both the question and the most relevant documentation chunk, along with the source URL so that hallucinations can be avoided.

Note: We are utilizing the GPT-4 model, as per OpenAI’s official documentation:

• OpenAI does not use data submitted via the API to train or improve its models.
• All data is retained for a limited period and is automatically deleted after 30 days.

Future Plans & Improvements

While ConfluenceGPT is currently a lightweight, local-first assistant, there’s a clear roadmap to evolve it into a more robust, production-grade internal knowledge bot.

• If you think downloading and managing the model is too much, you can use the online model as well.
• You can integrate it with Slack slash command and use it directly from Slack. You can refer to my Slack + AWS: Secure, One-Click Role Access (With Zero Standing Privileges) blog, where I have talked about implementing automations using Slack slash commands.
• You can consider using scalable vector stores like: Pinecone, Weaviate, Qdrant, Milvus, etc.
• Improved Embeddings and LLMs like mxbai-embed-large, OpenAI’s ada-002.
• Web dashboard for people who are not using Slack.
• It can be extended not just to Confluence but to anything or everything as far as they exist in the doc.

Github Repo: https://github.com/ankitsaini2609/confluence-gpt

Thank you for reading.

If you liked it, you can follow me on Twitter/LinkedIn for more.

Imagine this: Developers maintain permanent AWS role access “just in case,” while DevOps drown in frantic Slack requests for ProdDB access, and credentials are casually shared in public channels. When auditors ask, “Who approved this role?” the answer is met with a shrug.

This isn’t fiction — it’s the reality that led to catastrophic breaches. In 2019, Capital One’s misconfigured IAM roles allowed an attacker to access over 100 million customer records. In 2016, Uber’s exposed AWS credentials contributed to a major data compromise affecting millions of users. And in 2018, misconfigured Kubernetes clusters in a high-profile case involving Tesla enabled unauthorized AWS access.

This isn’t hypothetical — it’s the reality for teams using manual IAM role management. The risks?

• Permanent roles amplify breaches by serving as an irresistible treat for attackers.
• Human Error: When users have excessive privileges and policies are misconfigured, it paves the way for security breaches.
• Compliance Gaps: The lack of an audit trail for role assumptions hinders accountability.

To eliminate permanent access risks, I architected a solution that ties AWS roles to Slack approvals and time-bound credentials.

The solution: Slack-driven, approval-based role access by sharing temporary credentials

I automated the entire workflow into a Slack app that:

• Lists Assumable Roles: Users type /aws-assign-role list-roles to see what they’re allowed to request.
• Requests Approval: /aws-assign-role <role name> triggers an API Gateway → Lambda workflow.
• Securely Delivers Credentials: Approved? Temporary credentials land in the user’s Slack DMs — ephemeral, and never in public channels.

How It Works

Let us break down the whole setup step by step:

Part- 1: The manifest: your Slack app’s security blueprint

display_information:
  name: AWS_JIT_ACCESS_MANAGEMENT
features:
  bot_user:
    display_name: AWS_JIT_ACCESS_MANAGEMENT
    always_online: false
  slash_commands:
    - command: /aws_assign_role
      url: <AWS_API_GATEWAY_URL>/roleassignment
      description: This function will assign IAM role to the user.
      should_escape: false
oauth_config:
  scopes:
    user:
      - channels:write
      - groups:write
      - im:write
      - mpim:write
      - chat:write
    bot:
      - chat:write
      - commands
      - groups:write
      - im:write
      - mpim:write
settings:
  interactivity:
    is_enabled: true
    request_url: <AWS_API_GATEWAY_URL>/approvalCheck
  org_deploy_enabled: false
  socket_mode_enabled: false
  token_rotation_enabled: false

1- App Identity & Core Features

display_information:
  name: AWS_JIT_ACCESS_MANAGEMENT  # Name visible to users
features:
  bot_user:
    display_name: AWS_JIT_ACCESS_MANAGEMENT  # Bot’s display name
    always_online: false  # No need for 24/7 uptime (security-first!)  

Why It Matters: Clear naming avoids confusion (e.g., “AWS_JIT” signals its purpose).

always_online: false reduces attack surface when idle.

2- Slash Command — The User’s Gateway

slash_commands:
  - command: /aws_assign_role
    url: <API_GATEWAY_URL>/roleassignment  # Endpoint for role assignment logic
    description: This function will assign IAM role to the user.
    should_escape: false  # Allows markdown formatting in responses  

/aws_assign_role: The command triggers role assignment — ensure it is only accessible to authorized users.

url: Points to your secured API Gateway endpoint (validate Slack signatures here).

should_escape: false: Lets you format error messages (e.g., code blocks for clarity).

3- OAuth Scopes — Least Privilege in Action

oauth_config:
  scopes:
    user:
      - channels:write  # Post to public channels (for audit logs)
      - groups:write    # Post to private channels
      - im:write        # Send DMs to users
      - mpim:write      # Post to multiparty DMs
      - chat:write      # General message posting  
    bot:
      - chat:write      # Post messages as the bot
      - commands        # Enable slash commands
      - groups:write    # Join private channels
      - im:write        # Initiate DMs  
      - mpim:write      # Post to multiparty DMs

4- Interactivity — Secure Approval Workflows

settings:
  interactivity:
    is_enabled: true
    request_url: <API_GATEWAY_URL>/approvalCheck  # Endpoint for button clicks (approve/deny)  

Validate Requests: Use Slack’s signing secret to verify that interactions (e.g., approval button clicks) come from Slack — not attackers.

Ephemeral Messages: Deliver credentials via DMs, not public channels.

Part-2: Secure AWS JIT Role Access: From Slack Command to Tamper-Proof Approvals

In Part 1, we configured the Slack app. Now, let’s dissect the secure flow from command invocation to approval — designed to eliminate standing privileges and prevent tampering.

1- Slack Slash Command → API Gateway

• When a user runs /aws_assign_role, Slack sends a POST request to your API Gateway endpoint (/roleassignment) with application/x-www-form-urlencoded data.
• Problem: Lambda expects JSON, but Slack sends URL-encoded payloads.
  Solution: Use API Gateway’s mapping template(Integration Request) to transform the payload:

{
  "body" : $input.json("$"),
  "headers": {
    #foreach($header in $input.params().header.keySet())
      "$header": "$util.escapeJavaScript($input.params().header.get($header))" #if($foreach.hasNext),#end
    #end
  }
}

2- Request Verification — Guarding Against Spoofing

The JIT_Role_assignment Lambda first verifies the request is from Slack using the X-Slack-Signature header:

• Fetch Slack Signing Secret: Retrieve it from AWS Secrets Manager (never hardcode).
• Signature Verification: Whether the request came from the authorized Slack app or not.
• Reject Old Requests: If timestamp is >5 minutes old, discard it (anti-replay).

3- Role Authorization Check — Least Privilege Enforcement

The Lambda fetches a role assignment file from an S3 bucket (encrypted at rest):

[{
    "slack_userID": "U12345",  // Slack user ID
    "aws_username": "dev-user-1",  // Corresponding AWS IAM user
    "can_assume_role": ["read-only-role", "s3-admin"],  // Allowed roles
    "slack_approver_userid": "U67890"  // Approver’s Slack ID
}]  

• Check if the requesting slack_userID is allowed to assume the role.

• Validate the role exists in AWS IAM.
• Fetch the slack_approver_userid to send the approve request in the slack.

4- Approval Workflow — Tamper-Proof Design

To prevent malicious approvals, the Lambda:

• Create a signature using HMAC-SHA256(timestamp + userID + role) and then encode it with a secret key.
• Posts an ephemeral Slack message to the approver with ✅/❌ buttons and Embeds the signature and timestamp in the callback URL :

For example:
https://<APIGatewayURL>/approvalCheck?user_id=<userid>&action=approve&role_name_to_assume=LambdaS3PolicySecuringRole&timestamp=1740679860&signature=f1e3d234672e2cdb88fe6c413844564e9c3da4eac8e1096c46f6c44294372649

• Expires in 5 Minutes: Reject approvals if the timestamp is stale.

Why it matters:

No Standing Access: Roles are inert until approved.

Tamper-Proof Links: Signature ensures approval requests can’t be forged.

Least Privilege: Users only see roles they’re explicitly allowed to request.

Audit Trail: S3 access logs + CloudWatch track who requested/approved roles.

Part-3: The Approval Workflow — From Button Click to Secure Credentials

In Part 2, we validated requests and triggered approvals. Now, let’s dissect the final steps: tamper-proof approval handling, STS credential generation, and automated cleanup.

1- API Gateway Setup — Handling Approvals

• When an approver clicks ✅/❌, Slack sends a GET request to /approvalCheck with query parameters:

/approvalCheck?user_id=U123&action=approve&timestamp=1700000000&signature=abc123&role_name_to_assume=read-only  

• API Gateway Configuration for GET Request:

This feeds clean JSON to Lambda.

2- Tamper-Proof Validation

Like the First lambda function, this lambda function will also do the following validations

• Recompute HMAC-SHA256(timestamp + user_id + role_name_to_assume) using a secret key, Match against the signature param. Mismatch? Reject instantly.

• Timestamp Check: If (current_time — timestamp) > 5 minutes, reject. Blocks replay attacks.

• Whether a user is authorised to assume the role or not.
• Anti-Duplication via S3: Fetch database.json from S3, if signature exists: it will return “Request already processed” to Slack. If not, Add signature + timestamp to database.json, then proceed.

3- Credential Generation & Delivery

• If approved: Lambda will assume role using STS and will share the credentials with the user.

Notes:

• If you make a HEAD request in the /approvalCheck endpoint, it will do the s3 cleanup, meaning removing all the signatures that are not valid anymore.
• Ensure the lambda role should have proper access to the S3 bucket and the trust policy is correctly set up in both the lambda role and the role lambda is going to assume.
• The Lambda role can only assume specific roles — nothing else.
• API Gateway throttles to block brute-force attacks.

Why This Stops Most of Cloud Security Incidents

• No Standing Privileges: Roles are inactive until approved.
• Slack = Audit Trail: Every request/approval is logged in Slack + CloudWatch.
• Least Privilege, By Default: Users only get what they need, when they need it.

Future Plans

• This can be used to attach and remove the policies on a role or a user.
• You can use the privatebin to share the credentials with the user rather than sending them in plaintext.
• You can add the user to the group or remove it as well, after x hours, in case you don’t want to give temporary credentials using assume role by lambda.
• You can play around with permission sets as well in IAM identity center.

Thank you for reading.

If you liked the blog, you can follow me on Twitter/LinkedIn.

Slack + AWS: Secure, One-Click Role Access (With Zero Standing Privileges) was originally published in DevSecOps & AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Amazon GuardDuty, a Security Monitoring Service, analyzes AWS data sources and uses threat intelligence and machine learning to identify malicious activity. It covers issues like unauthorized access, malware detection, unusual login patterns, etc.

Most organization uses Slack/Microsoft Teams for their day-to-day communications.

In this blog, we will try to get the Guard duty alerts in Microsoft Teams not only it will enhance security incident response but also fulfill one of the ISO27001 requirements. It can be achieved by SIEM tools as well. You can configure these alerts and take action on a real-time basis.

Prerequisite: Guard Duty should be enabled.

Architecture Diagram

As you can see, GuardDuty alerts will be triggering the EventBridge rule which will run the lambda function and will send the alerts in MS Teams.

Creating a Lambda function

• Login to your AWS account, Go to Lambda and create one function with basic permissions.

• Fill in the function name and select runtime as python 3.7 then click on create function.

• Paste the code given in the link.
• Set up the Environment variable by navigating to Lambda -> Configuration -> Environment Variables -> Edit and Set the TEAM_WEBHOOK_URL.

• To create a Microsoft Teams incoming webhook refer to this article.

Setting Up EventBridge Rule

• Search for EventBridge in the AWS console, open it, and click on Create Rule on the right side.

• Fill in the name and click on next.

• In the next screen make the following changes:

• Select the Target as a lambda function and choose its name from the drop-down.

• Add the relevant tag then click on review and create then click on create.

That’s it.

Output

The same solution can be extended to Slack, Discord, etc.

Future Plans

You can make it automatically actionable based on the alert type using the same lambda function.

Thank you for reading.

You can follow me on Twitter/LinkedIn.

The Core Problem?

The sensitive data shared with human beings, ideally should not be accessible to human users also instead of storing them in a file, you should use password managers, for example, Bitwarden, Lastpass, etc & be available for systems from a creds store.

Sometimes we use a third party to share the credentials instead what if we can use in-house tools to share it? Like deploying in-house PrivateBin behind the VPN and using it to share credentials.

Why PrivateBin?

• It gives you the flexibility to adjust the validity of the link as per the user’s need (Even a one-time link)
• Links can be password protected
• It’s free and open source
• It’s easy to use

Why should PrivateBin be behind the VPN?

It is because only people having access to the VPN can access the link. If it is not behind a VPN anyone having the slack token can get the link and access it over the internet.

Automating the entire solution

User opening the PrivateBin and generating the link will be a lot of work for the user, What if the user sends the message and it can get replaced by the PrivateBin link 😍.

Let’s create a slack app and with the help of AWS Lambda(Scalability) and AWS API Gateway, we will achieve our goal.

Architecture Diagram

Implementation Steps — Configuring Slack app:

• Open the slack apps link.
• Click on Create New App in the top right corner.

• Click on From Scratch.

• Give the Slack App name and choose the workspace and click on Create App.

• Click on Oauth & Permissions and scroll down to the scopes and add the below-given scopes in Bot Token Scopes and User Token Scopes:

channels:read
files:read
groups:read
im:read
links:read
mpim:read
remote_files:read
search:read
team:read
users:read
users:read.email
chat:write

• Click on Events subscriptions on the left pane and click on the toggle button to enable it.

• You can see the Request URL, It is the URL where slack is going to send all the events. So Let’s create a URL with the help of AWS API Gateway and AWS Lambda function.

Configuring AWS API Gateway and AWS Lambda function

• We are going to configure AWS API Gateway and going to associate a lambda function which will get invoked every time that API endpoint gets a hit. Login to your AWS account, Go to Lambda, and create one function with basic permissions.

• Fill in the function name and select runtime as python 3.8 then click on create function.

• Copy the code of the lambda_function.py and paste it into the lambda function and upload the remaining two zip(pbincli.zip and slack-sdk.zip) files as layers.
• Remember to set environment variables in your lambda. These variables are Slack signing secret, server(URL of the private bin, for example, https://privatebin.net/), and user slack token( to post the link in the chat). You can get the Signing secret by going to the basic information of your slack app and App credentials, you can get the signing secret.

• Now click on the install app, on the left pane of the slack app. After installing you will get a user OAuth token. The environment variable in your lambda will look like this:

server = <Your Server URL>
SLACK_USER_TOKEN = <User Oauth Token>
SLACK_SIGNING_SECRET = <Slack Signing Secret>

• Do not forget to increase the timeout of the lambda to 5 min.
• We have completed the setup of lambda, Let's create an API gateway and associate it to the lambda function. Go to the API Gateway and click on build.

• Choose the integration as lambda, give it a name and click on review and create and then click on create.

• Now copy the Invoke URL and append the API name which you have given in the previous stage. It will look something like this:

Final Configuration in Slack Bot

• Copy above URL and open the slack app, go to the Event subscription on the left pane, Click on the toggle and paste the URL in the request URL and wait until it does not show verified.

• Click on Subscribe to events on behalf of the users and Subscribe to bot events and add the following OAuth permissions.

message.groups
message.channels

• Click on save and reinstall the app by going to the Install app on the left pane and clicking on reinstall to the workspace.
• All Set !!!

Verifying the Solution

• Open your slack, Add the bot to a channel and try to post a secret in it.

Future Plans

• The same Solution can be expanded for Microsoft Teams, Confluence, etc.

Thank you for reading.

Thanks to Abhinav, Priyal, and Jeyanthi for your ideas and suggestions.

You can follow me on Twitter/LinkedIn.

In this series of containers, we have learned what containers are and how to run them securely. What if, the base image you are using in a container is not secure in itself, then no matter how securely you are running the container, it will have some side effects. This blog will talk about the container registry (The place where we store container images) and the security measures around container images.

While using docker, container images play a very crucial role, so it is important to establish trust that the container image you are using is secure. There are a few methods using which it can be achieved:

• Private registry
• Docker official container images
• Content trust in Docker

Private Registry

The Registry is a stateless, highly scalable server-side application that stores and lets you distribute Docker images, some examples of registries are Harbor, Dockerhub, Amazon ECR, etc. You can either use a private or public registry, but they have their pros and cons. In a public registry, anyone can upload a malicious image and it is up to the user to decide whether he wants to use it or not, that’s why it is very hard to establish trust in the case of a public registry. Meanwhile, private registries provide the flexibility to control who can push images and pull images. In addition, Role-based access control can be enforced to have granular control over users by specifying which department or environment a specific user has access to. The private registry provides various other features like:

• Tag Immutability: By default, if you are uploading an image, it will be tagged as the latest. It might be possible that the developer forgets to put the tag in the image and uploads it so that it will override the image having the latest tag. To prevent these situations there is a feature called Tag Immutability where you cannot alter the tag by pushing, retagging, or replication.
  It also prevents the image from accidental deletion.
• Vulnerability Scanning: There is a static vulnerability scanner available in private registries for example Trivy, Clair, etc. On the basis of the results, you can even block all the deployments which are using the vulnerable image. You can scan the images at the time of upload or you can schedule the scanner to scan them at a particular time.

Docker official container images

You can use the images stored in the docker registry as well but before using it make sure to check whether it is an official image or not. If the below-given tags are present it means, they are trusted:

• DOCKER OFFICIAL IMAGE
• VERIFIED PUBLISHER
• OPEN SOURCE PROGRAM

To check the tag refer to the below-given image:

Content trust in Docker

When transferring data among networked systems, trust is a central concern. In particular, when communicating over an untrusted medium such as the internet, it is critical to ensure the integrity and the publisher of all the data a system operates on. You use the Docker Engine to push and pull images (data) to a public or private registry. Content trust allows you to verify both the integrity and the publisher of all the data received from a registry over any channel.

The content trust provides the ability to use digital signatures to verify the integrity and authenticity of the image. It is possible that an image having one tag is signed and the same image having another tag is not signed. Each image has a particular set of keys using which publishers sign an image tag.

In order to use content trust user must have notary installed with the registry.
Notary is a tool for publishing and managing trusted collections of content. Publishers can digitally sign collections and consumers can verify integrity and origin of content.

Let’s refer to the below image to understand it better:

Using an offline key, we can generate the tagging keys which can be used to sign the images. Similarly, a timestamp key is generated on the server-side which is associated with the image repository. To know more you can read this.

This is about container registry, feel free to reach out to me if you have any doubts/queries.

Thank you for reading.
If you liked it follow me on Twitter: d3afh3av3n

If you are new to the containers, you can read my previous blog to get a glimpse of the container’s basics.

Organizations are shifting from a monolithic architecture to microservices to containers. While deploying a microservice, it is very important to keep in mind that the architecture of the machine in production should be the same as where you are developing your application because of dependencies and architecture issues. To avoid these issues containers come into the picture. Hence many organizations are using containers to deploy their production application which are dealing with databases, AWS S3 buckets, etc. It is very important to keep containers secure, It can be achieved by using various techniques which are:

• Control Groups (CGroups)
• Docker Rootless
• Docker Namespace
• Kernel Capabilities
• AppArmor
• Seccomp profiles

Let's get started …

Control Groups(CGroups)

By default, a container has no resource constraints and can use as much of a given resource. CGroups implement resource accounting and limiting. They provide various helpful metrics and ensure that each container will get its fair share of the resources(CPU, Memory, I/O, etc) so that, a single container cannot bring down the whole system by exhausting all the resources.

They are the Linux capabilities so it is important for your kernel to support these capabilities. You can confirm whether the kernel supports it or not by running the following command:

docker info

If you see the output given below in the image it means it is disabled.

You can use -m (or --memory) and --cpus to restrict the memory and number of cpus. To know more, read this.

Docker Rootless

Running container as a non-root user

You can create a group and a user and include a username in the dockerfile. It is important because even if an attacker gets hold of a container he won’t be able to run commands as root. Remember you are running container as non-root but there are chances that docker daemon might be running as root.

RUN groupadd -g 1000 someUser && useradd -u 1000 -g 1000\
-d /usr/share/someUser -M someUser
USER someUser

Running docker daemon as a non-root user (Rootless mode)

Rootless mode allows you to run docker daemon and container as a non-root user. It uses user namespace to run both docker daemon and docker container. The rootless mode does not use binaries with SETUID bits or file capabilities, except newuidmap and newgidmap, which are needed to allow multiple UIDs/GIDs to be used in the user namespace. To know more, read this.

Docker Namespace

The concept of a namespace is the same here as it is used in other contexts. So here you can achieve more isolation since containers in one namespace won’t be able to access the containers in other namespaces. In case of a breach in one namespace, if configured properly attacker won’t be able to access the container in other namespaces. In the Docker namespace, the docker daemon is running as root and containers as non-root. The process running inside the containers assumes that is running as root within the container and it happens because of user remapping within a namespace.

The remapping itself is handled by two files: /etc/subuid and /etc/subgid. Each file works the same, but one is concerned with the user ID range, and the other with the group ID range. Consider the following entry in /etc/subuid:

testuser:231072:65536

This means that testuser is assigned a subordinate user ID range of 231072 and the next 65536 integers in sequence. UID 231072 is mapped within the namespace (within the container, in this case) as UID 0 (root). UID 231073 is mapped as UID 1, and so forth. If a process attempts to escalate privilege outside of the namespace, the process is running as an unprivileged high-number UID on the host, which does not even map to a real user. This means the process has no privileges on the host system at all. To know more, read this.

Kernel Capabilities

By default, docker restricts a lot of kernel capabilities within a container. For example, processes(like web servers) that just need to bind on a port below 1024 do not need to run as root: they can just be granted the net_bind_service capability instead. And there are many other capabilities, for almost all the specific areas where root privileges are usually needed. We can deny these capabilities so even if an intruder manages to escalate to root within a container, it is much harder to do serious damage or to escalate to the host.

AppArmor

AppArmor is a Mandatory Access Control(MAC) system that confines a program to a limited set of resources. We can create a security profile that can be loaded in the kernel and AppArmor can be set to either enforce the profile or complain when profile rules are violated.

Difference between DAC (Discretionary access control)(LINUX, UNIX etc) and MAC(Mandatory Access Control)(AppArmor, SELinux etc)

With DAC, files and processes have owners. You can have the user own a file, a group own a file, or other, which can be anyone else. Users have the ability to change permissions on their own files. The root user has full access control with a DAC system. If you have root access, then you can access any other user’s files or do whatever you want on the system.

But on MAC systems like SELinux, there is administratively set policy around access. Even if the DAC settings on your home directory are changed, an SELinux policy in place to prevent another user or process from accessing the directory will keep the system safe.

AppArmor profiles are simple text files. AppArmor profiles are default-deny by default, which means if you have not defined something in the policy file it means it is denied you cannot execute it inside the container. I have attached a sample profile for Nginx. You can run the Nginx container and load this profile and when you will try commands like ping, touch, sh, dash, etc it will not work as deny is set in the profile. To know more, read this.

Seccomp profiles

Secure computing mode (seccomp) is a Linux kernel feature. Like AppArmor, it is also used to restrict the actions available within the container. We create seccomp profiles and pass them to the container using --security-opt option with docker run command. Seccomp is used to block syscall. This feature is available only if Docker has been built seccomp and the kernel is configured with CONFIG_SECCOMP enabled. To check if your kernel supports seccomp run the below-given command:

Like AppArmor seccomp profile is an allowlist that denies access to system calls by default, then allowlists specific system calls. To know more, read this.

Few extra key points …

• Do not expose /var/run/docker.sock file to the container. This socket file is used for container management like collecting the logs, creating containers, stopping them, running commands on it, etc.
• Volume mounts as read-only.
• Ensure the host network interface is not shared with the containers.
• Do not mount system volumes like /etc, /dev etc.

Thank you for reading and Keep following for the next blog in the Container series.
You can always reach out to me if you have any questions/queries.
If you liked it follow me on Twitter: @d3afh3av3n.

This blog is going to tell you about the fundamentals of the container. It is a pre-requisite for my upcoming blog on container security. You all must have heard of docker. If not, Docker is one of the means of building and running containers.

What are containers and why do we need them?

Before the Container era, developers used to write the code and try to deploy it in a different environment and because of dependencies and architecture issues sometimes it runs and sometimes it’s not. You all have heard about “It worked on my machine”. So basically containers resolve these issues and many others. In containers, you pack your application and deploy it in any environment you want.

Difference between VMs and Containers

As you can see in the above diagram in VMs we share the hardware, we delicately assign a portion of the memory and CPU to the VMs while containers are just a process that shares the operating system. Containers are limited to what resources can be accessed and exit when the process running inside them stops.
When you start a VM, it takes a significant amount of time in booting up because they are having its own OS, also they are big in size. On the other side, containers have a relatively lower size and can quickly spin up and start running.

Important Component of the Docker

• Dockerfile: It is a simple file where you write instructions that are being executed by the docker client.
• Docker Image: It is a collection of app binaries and dependencies. It is neither a complete OS nor a Kernel. It can be as small as golang binary and as big as ubuntu distro. You build dockerfile(executing the instructions written in the dockerfile) and it creates a docker image(read-only template containing instructions for building a container). Each instruction written in the dockerfile is called a layer.
• Docker Container: When you run a docker image we call it a docker container.
• Container Registry: It is a place where you store container images.
• Docker Host: Host machine where we have installed docker.
• Docker Daemon: It is a process that runs on the host machine and facilitates docker client commands.

How to use docker client

You can install the docker engine by following this link.

Let’s get started …

docker --help
Most useful command, whenever you are stuck with any command just append --help in front of it, It will be a great savior.

Example:

docker container --help: You can see it will show you all the commands which you can run on the container.

Few key points …

• What is the difference between docker container run, docker container start, and docker container exec?
  the run will always start a new container.
  the start will start the existing stopped container.
  exec will execute the command inside the existing running container.
• What is the difference between docker container top, docker container stats, and docker container inspect?
  the top will list the process in a container.
  stats will give us the live performance stats of the container(s).
  inspect will give us details of one container config. (metadata like config, volumes, networking, etc)
• By default docker container ls will list only the running container so in order to get all the containers(running + stopped) use the -a option with it.
• A container is just a single read/write layer on top of the image.

docker image --help: It will show you all the commands you can run on an image.

Few key points …

• docker image history will show layers of changes made in an image.
• Each layer will be stored only once in the host machine and can be uniquely identified. This saves the space on the host machine and transfers time on push/pull.
• Image tagging is used to manage different versions of an image. You can create a custom tag, and re-tag an existing image.

There are several other commands which you can explore like docker network, docker volume, docker system, docker secret, etc.

Thank you for reading.
You can always reach out to me if you have any questions/queries.
If you liked it follow me on Twitter: @d3afh3av3n.

Master the Container Security — Overview was originally published in System Weakness on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Blinkit, we follow the “You build it, you run it” philosophy. This means that developers deploy applications and provision services independently. This philosophy has enabled us to move fast but it comes with its own set of challenges. Case in point — Kubernetes LoadBalancers. When Kubernetes LoadBalancer are provisioned, they are public by default. As a developer, you can choose to override this configuration and declare your LoadBalancers as private by adding the ”service.beta.kubernetes.io/aws-load-balancer-internal” annotation. However, this creates an inherent risk. The security of our setup hinges on whether a developer decides to add this annotation every time they create a new service. Should a developer forget to add this annotation, they would end up exposing internal applications to the public network. We don’t want that.

We needed better safety nets, better security controls in our service provisioning process in Kubernetes.

Policy Enforcement and Governance using Kyverno

Kubernetes has an interesting concept of admission controllers that can be used to intercept requests to Kuberenetes’ API server and decide whether they should be allowed. This kind of control pattern can be leveraged to enforce policies and govern your Kubernetes cluster. We started looking for frameworks that had been built for this purpose and arrived at Kyverno.

What is Kyverno and how is it useful?

Kyverno is a policy engine designed for Kubernetes. It can validate, mutate, and generate configurations using admission controls and background scans. Kyverno policies are Kubernetes resources and do not require learning a new language. You can write policies declaratively. The policies themselves are objects in Kubernetes. Another, more popular, alternative to Kyverno that deserves a mention is Open Policy Agent (which we use as well) but it requires you to learn Rego — an OPA specific language — to implement policy logic.

How do we use Kyverno for secure provisioning of Services?

Let us refer back to our problem statement. Kyverno allows us to write a policy that makes sure that developers are not able to provision any public facing LoadBalancers. The policy instructs Kyverno to inspect all the incoming requests for the creation of a new Service object. These requests are then rejected if they don’t contain the annotation required to declare them as private. Ofcourse, there are cases when we genuinely want to expose certain services to the public internet. For this purpose, we maintain a whitelist of service in a configmap that our Kyverno policy can refer to while considering if the request should be allowed.

Note: To whitelist your load balancer, add an entry like namespace/service_name.

Let’s look at our policy specification line-by-line:

• Line 2: You can see that it is a cluster policy which means it is going to be applied cluster-wide irrespective of the namespace.
• Line 6–7: These lines represent that this policy is going to be of an “enforced” nature which means that it will block any incoming resources which will violate this policy.
• Line 10–14: These lines make the policy fetch the values from a configmap named allowed-elbs from the security namespace and become aware of whitelisted services.
• Line 15–18: These lines make sure that the policy only acts upon incoming requests of type Service.
• Line 19–31: These lines consist of the core logic of blocking any incoming service if it matches all of the following conditions:
  - The service is not annotated with the annotation service.beta.kubernetes.io/aws-load-balancer-internal.
  - The service is of the type load balancer.
  - The namespace/service_name is not present in the configmap of whitelisted services.

As Kubernetes is deployed in organizations to enable more and more developers to deploy applications, governance, and security of Kubernetes clusters are becoming an area of concern. Writing policies for security and governance with Kyverno is easy and the barrier to entry is extremely low. We are looking at adopting Kyverno for more governance-related use-cases.

Ankit Saini used to work as a Security Engineer at Blinkit. Follow him on Twitter.

We are hiring across various roles! If you are interested in exploring working at Blinkit, we’d love to hear from you.

Secure provisioning of LoadBalancer Services on Kubernetes using Kyverno was originally published in Lambda by Blinkit on Medium, where people are continuing the conversation by highlighting and responding to this story.

BUG! BUG! BUG!
We have seen many bugs related to AWS S3, for example, Instagram Million dollar bug, Facebook data breach, Leak of GoDaddy trade secrets, Voters database Leak. As people started using AWS, Simple Storage Service(S3) is one of the essential services used to store data, and most of the time some credentials get leaked or because of some misconfiguration in the bucket policy results to a data breach. so here are some preventive measures which can be implemented to avoid such kind of scenarios up to some extent.

AWS Simple Storage Service(S3) is used to store and retrieve any amount of data, it is a highly scalable, reliable, fast, inexpensive data storage service and because of all these features, it is highly used. Each item that we store inside the bucket is called an object. We can apply policies around the object, bucket, and account where account level policy will always have high priority.

A few days back I wrote a script, which revokes the AWS access keys after certain days of inactivity. The assets lying around that are not used and maintained are often the biggest sources of incidents as nobody is watching over them also it enforces the key rotation.

Versioning: It is like storing multiple versions of the object in the same bucket. You can use versioning to retrieve, preserve, and restore the object. Using versioning you can recover the object even if it was deleted intentionally or unintentionally. You can enable versioning by going to the properties of the bucket.

Multi-Factor Authentication (MFA): It will help you to prevent the accidental deletion of the bucket. For now, It can be only enabled from CLI.
Only the root account can enable MFA. Run the below command to enable it:
aws s3api put-bucket-versioning --bucket <yourbucket> --versioning-configuration Status=Enabled,MFADelete=Enabled --mfa "arn:aws:iam::<account number>:mfa/root-account-mfa-device <mfa code>".

Allow only your infrastructure IPs: A Simple bucket policy that can be applied to allow your IPs to access a particular s3 bucket in your infrastructure.

{
  "Version": "2012-10-17",
  "Id": "S3PolicyId1",
  "Statement": [
    {
      "Sid": "IPAllow",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::somebucketname/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "X.X.X.X/X"
          ]
        }
      }
    }
  ]
}

By default, all S3 bucket objects are private.

You can always set the IAM Policies for a user who can access the bucket, or control the permissions on bucket level, or set the permission for a specific object as well. Enabling encryption is just another option.

Lets set some monitoring…

AWS Config: It is used to assess, audit, and evaluate the configuration of AWS resources, you can setup config rule against your bucket to check for the public access. It will help you to detect any public bucket as soon as possible.

Trusted Advisor: It is a tool provided by AWS for real-time guidance to provision your resources. There is one section related to AWS S3 bucket permissions which will tell you about the open-access permissions.

There are several other ways, using which you can quickly track if your bucket gets public and once it is detected. You can set lambda function to again change its visibility to private.

Thanks for reading!!
You can follow me on twitter: @d3afh3av3n