At NewDay, we deeply care about software testing and, while we are big fans of TDD, there is no denying that being able to debug software effectively is a crucial aspect of software engineering.

Favouring a debugging session instead of writing a new unit test is a big topic on its own, so here we will only focus on a specific aspect: being able to step into an internal library from a consuming application or service in .NET.

.NET has a long history of providing a fully featured debugger via Visual Studio. When working with Microsoft libraries or many open-source ones, we get the ability to step into the code, how is that possible? And how we can achieve the same behaviour for our company internal libraries?

Leveraging debug symbols

In general, to debug a compiled language, on top of having access to the source code, you need the debug symbols, which map the binary to the original source code. In .NET, this role is fulfilled by the Portable Database (or PDB).

For Microsoft libraries or many open-source ones, the PDBs are automatically retrieved by Visual Studio from the Microsoft public symbols server or from the NuGet.org symbol packages (.snupkg) feed, respectively.

When working in an enterprise scenario, we don’t usually have either of those mechanism available for our internal libraries: symbols servers have never been popular and are tricky to setup. On the other hand, while snupkg are the standard for modern .NET libraries, they are not supported by the most popular software used to host internal package feeds, like JFrog Artifactory or GitHub Packages registry. For this reason, the best approach is to include the PDBs as part of our actual library NuGet package.

Including PDBs in a NuGet package

The common solution to provide debug symbols for private packages used to be to include the PDBs in the NuGet packages alongside the dll.
For example, in early versions of .NET Core you could add a, admittedly quite hard to remember, line to your csproj to achieve this.

Unfortunately, with changes in .NET Core 3.0 to the project system, it is no longer possible to consume the PDB files included in NuGet packages.

Luckly, we can rely on another mechanism that, while deeply different in nature, allow us to achieve the same result: embedding the PDB inside the DLL. It has a few drawbacks, like having larger packages and not being able to easily strip out PDBs from deployables, however these problems are rarely relevant for internal libraries, especially when used in internal microservices and APIs.

In current .NET versions, this is easily achievable by adding the <DebugType>embedded</DebugType> property to your library csproj, which is much easier to remember than the previous technique and one day it could become the default behaviour.

There is a final aspect that we need to mention when talking about including a PDB: making sure it is deterministic. While .NET builds have become deterministic by default in .NET, ensuring the same DLL is produced no matter where it is built, to have PDBs with normalized paths to our source file, we also need to add the <ContinuousIntegrationBuild>true</ContinuousIntegrationBuild> property, usually with a conditional to only apply the normalization when building in CI, but we will look into an easy way to achieve all these later on.

Getting the source code

Now that our library contains the required metadata to map the DLL to the source code, we need to be able to easily retrieve it. Nowadays we all use version control systems, so the easier way is to retrieve it directly from it. The .NET platform provides Source Link, a nifty tool integrated into Visual Studio, which provides the ability to automatically include metadata into NuGet packages that point to the source code hosted in our version control hosting platform of choice from which Visual Studio can then automatically download it when stepping into the code. Thanks to the support to multiple authentication mechanism as well as on-premises solutions like GitHub Enterprise, this works well in the enterprise scenario, allowing us to leverage the same tool as the opensource community without extra configuration, as easy as adding a package reference and a couple of project properties. But can it be even easier to integrate it?

Putting it all together

As we had seen, by putting a few pieces together and using the right configuration, we can provide a brilliant debugging experience for our internal libraries which has nothing to envy to opensource or Microsoft libraries. Of course, we would not want to have to replicate a bunch of settings across every project and, while creating shared props or targets is an option, there is no need for it.

In fact, there is a .NET Foundation’s package called DotNet.ReproducibleBuilds that takes care of all these steps necessary to get Source Link fully working and CI builds deterministic, including logic to detect your correct CI server. All you need to do is to add this package either to your csproj, or more convenient if you have multiple projects published as library in the same solution, to your Directory.Build.props` file.

Investigating Source Link issues

If you are having issues in navigating to the source code, you should check the logs in Navigate to External Sources in the Visual Studio’s Output View.
Also make sure you have Enable Source Link support enabled and consider enabling the integration with Git Credential Manager if you experience authentication issues.

Furthermore, don’t forget that to be able to step into libraries, apart from thrown exceptions, you need to disable Just My Code and set the Symbols loading rules appropriately.

Extra goodies

In recent versions of Visual Studio 2022 there is now the ability to use “Go To Definition” on references of our internal libraries and magically go to the source code relying on the same mechanism previously described used during debugging. While the feature is still not perfect, it works in most scenarios, and it keeps getting refined with each feature update!

So easy!

Thanks to the ability of the IDE to automatically retrieve debug symbols and the source code, allowing both to “step in” during debug as well as using “Go To Definition” when navigating the codebase, working with private libraries has never been so easy!

Easily making internal libraries debuggable in .NET was originally published in NewDay Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to Part 4 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

• Add container scanning to our GitHub Action workflow
• Resolve security vulnerabilities

TL;DR

We add the Azure Container Scan GitHub Action to our actions’ workflow file, before pushing the container image to our registry. Add a group and user to our Dockerfile and use the USER command to specify the user for the process to run; ensuring our copied files can be accessed by this user. The security attack surface is reduced by running as non-root, and this is raised in CIS; the linting provided by Dockle. We resolved some slightly older CVE’s as examples; these can be raised by Trivy and can be allowed through or not based on configuration. Both Trivy and Dockle are wrapped by the Azure Container Scan GitHub Action.

peteking/Samples.WeatherForecast-Part-4

Introduction

In Part 3 we were able to get into GitHub Actions and build and push our Docker image to the GitHub Container registry. This is a great start, we are producing a Docker image in CI and it’s a consistent image, however, what about vulnerabilities?

Requirements

We will be picking-up where we left off in Part 3, which means you’ll need the end-result from GitHub Repo — Part 3 to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn’t necessary if previous posts are knowledge to you already.

Vulnerabilities

A software vulnerability is a glitch, flaw, or weakness that is present in software or in an Operating System. CVE (Common Vulnerabilities and Exposures), contains a list of records each containing an identification number, a description, and at least one public reference — for publicly known cybersecurity vulnerabilities.

CVE Records are used in numerous cybersecurity products and services from around the world, including the U.S. National Vulnerability Database (NVD).

There are many open source and commercial tools specifically designed for CI purposes; to be used before pushing a Docker image to a container registry. This can ensure any CVE’s of certain thresholds can be caught a build time.

Commercial tools can also scan containers at runtime and apply custom mediating actions.

CIS (Center for Internet Security)

CIS is a community of cybersecurity experts, who have developed CIS Benchmarks: more than 100 configuration guidelines across 25+ vendor product families to safeguard systems against today’s evolving cyber threats.

CIS also produced a list of hardened images — https://www.cisecurity.org/cis-hardened-image-list/

Why is this important?

It’s important because the software we design, build, and ship needs to be secure, we want to ensure safety of our end users’ data and be responsible. We must take security seriously and both CVE and CIS provide a trusted source of information.

The open source community has done an amazing job of utilising the information and provided fantastic tooling we can all take advantage of.

To continue reading this post, please visit our blog here

API’s from Dev to Production — Part 4 was originally published in NewDay Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to Part 3 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

• GitHub Actions:
• Build the docker image
• Publish the docker image to the GitHub Container Registry

TL;DR

Instead of building locally like we did previously, we utilised GitHub Actions and the GitHub Container Registry to store our Docker image. GitHub Actions workflow is basic YAML, and we used a few actions to achieve what we needed. We checked-out the repository, docker build, logged into our GitHub Container Registry and executed a docker push command. We used a Personal Access token and stored it in GitHub secrets to use in our workflow, and finally, we pulled down our public image from the GitHub Container Registry and tested locally using Postman.

peteking/Samples.WeatherForecast-Part-3

Requirements

We will be picking-up where we left off in Part 2, which means you’ll need the end-result from GitHub Repo from — Part 2 to start with.

What are GitHub Actions?

GitHub Actions makes it easy to automate all kinds of software workflows, right inside GitHub. You can build, test, and deploy your code and more. Workflows can be triggered off many GitHub events like push, issue creation and more. There is support for Linux, macOS, Windows, ARM and of course containers Even matrix builds — so you can build the same software across multiple OS’s. There are live logs, secret store and more, plus the UX is rather nice too! All of this is driven by YAML files, nice and neat, and stored under source control management.

There is also GitHub Packages which has various package management providers like npm, nuget and more. For containers, there is a new GitHub Container Registry, so images no longer need to be stored in GitHub Packages but there is a real container registry — We will use this in this blog post :D

To continue reading this post, please visit our blog here

API’s from Dev to Production — Part 3 was originally published in NewDay Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to Part 2 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, , GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

• Optimising the Docker image size

TL;DR

We managed to optimise the resulting Docker image from 210MB down to 109MB; this 109MB could be even smaller if you were happy not to use AOT (ahead of time compiling) with .net — you can get this down to ~69MB. We did this by using Alpine Linux — a small, lightweight Linux distribution, using the .net 5 SDK to restore, build, publish, and then we used the minimal .net runtime dependencies image (which is only 9MB!). We took advantage of self-contained, PublishReadyToRun and PublishTrimmed.

--self-contained=true \ -p:PublishReadyToRun=true \ -p:PublishTrimmed=true

peteking/Samples.WeatherForecast-Part-2

Requirements

We will be picking-up where we left off in Part 1, which means you’ll need the end-result from Part 1 to start with.

We have the same requirements as last time, but to save you some time, I’ve put them below anyway :)

• Git — https://git-scm.com/downloads
• Visual Studio Code — https://code.visualstudio.com/download
• Docker — https://www.docker.com/products/docker-desktop
• .net 5 — https://dotnet.microsoft.com/download/dotnet/5.0
• WSL 2 (Windows 10 only) — https://docs.microsoft.com/en-us/windows/wsl/install-win10

I’m using Windows 10, for other OS’s like MacOS and Linux, there will be small differences that are not covered here.

VS code extensions

• C# — https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp
• .gitignore Generator — https://marketplace.visualstudio.com/items?itemName=piotrpalarz.vscode-gitignore-generator
• Docker — https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker

Dockerfile optimisation

Dockerfiles works in distinct layers and each of the following commands creates a new layer.

In relation to a previous file which is below.

FROM creates a layer from the mcr.microsoft.com/dotnet/aspnet:5.0 Docker image.

COPY adds files from your source to destination directory.

RUN in our case will restore, build and publish using dotnet.

FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
WORKDIR /app
EXPOSE 80

FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /app
COPY . .

WORKDIR /app/src/Samples.WeatherForecast.Api

RUN dotnet restore "Samples.WeatherForecast.Api.csproj"

RUN dotnet build "Samples.WeatherForecast.Api.csproj" -c Release -o /app/build --no-restore

FROM build AS publish
RUN dotnet publish "Samples.WeatherForecast.Api.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Samples.WeatherForecast.Api.dll"]

Docker build cache

When building an image, Docker steps through the instructions in your Dockerfile, executing each in the order specified. As each instruction is examined, Docker looks for an existing image in its cache that it can reuse, rather than creating a new (duplicate) image.

If you do not want to use the cache at all, you can use the --no-cache=true option on the docker build command. However, if you do let Docker use its cache, it is important to understand when it can, and cannot, find a matching image. The basic rules that Docker follows are outlined below:

• Starting with a parent image that is already in the cache, the next instruction is compared against all child images derived from that base image to see if one of them was built using the exact same instruction. If not, the cache is invalidated.
• In most cases, simply comparing the instruction in the Dockerfile with one of the child images is sufficient. However, certain instructions require more examination and explanation.
• For the ADD and COPY instructions, the contents of the file(s) in the image are examined and a checksum is calculated for each file. The last-modified and last-accessed times of the file(s) are not considered in these checksums. During the cache lookup, the checksum is compared against the checksum in the existing images. If anything has changed in the file(s), such as the contents and metadata, then the cache is invalidated.
• Aside from the ADD and COPY commands, cache checking does not look at the files in the container to determine a cache match. For example, when processing a RUN apt-get -y update command the files updated in the container are not examined to determine if a cache hit exists. In that case just the command string itself is used to find a match.

Once the cache is invalidated, all subsequent Dockerfile commands generate new images and the cache is not used.

It’s preferable to use COPY instead of ADD.

This is simply because the term COPY is more transparent in terms of what it is doing, whereas ADD supports features like local-only tar extraction and remote URL support that are not immediately obvious.

See references [1]

How can we optimise?

For starters, we can use a different base image, a different OS (Operating System), one of the smallest is Alpine Linux. Alpine is great, it’s small, is not bloated with anything extra, even curl isn't installed!

With .net, the question becomes — How do I know what’s available?

Well, DockerHub is your best friend, it has everything you need to find out about the image that companies and individuals have published.

Let’s take a look at .net 5 from Microsoft.

You can see there are over 1 billion downloads overall!

The build

What do we actually need to build? When you think about it, you need the SDK (Software Development Kit) in order to build your projects.

If you navigate to ‘dotnet/sdk’ you’ll see a page with all the relevant information you need about the SDK.

From there you can see the various versions (tags) are available, we want Alpine. There are various tags there, but they all equate to the same Dockerfile; therefore, the same image. For us, using the tag, ‘5.0-alpine’ is enough.

Always specify a version

It’s best practice to pin a version, it is then a known thing — You don’t want to be caught out by using ‘latest’ tag. i.e. if you build later on, and the version with the ‘latest’ tag has changed, but you haven’t tested it, you may find some issues.

Rewrite the Dockerfile

Let’s start by rewriting our Dockerfile, don't worry, it will still only be about 20 lines of code.

ARG VERSION=5.0-alpine

FROM mcr.microsoft.com/dotnet/sdk:${VERSION} AS build
WORKDIR /app

Here you can see a couple of things, we are using ARG to hold 5.0-alpine and we use that variable in the FROM statement with the dollar sign and curly braces ${ }

If you’re wondering what this FROM is and why we have more than one (even in our generated Dockerfile from Part 1), this is known as a multi-stage build - We essentially take advantage of different base images to do certain things, and use another final image to ensure it is the most optimal it can be. For more information about multi-stage builds, please see, Docker Docs - Multistage Build

DotNet restore

Next up is dotnet restore we want to restore because if that fails, there is certainly no point in going any further with the build. We will also optimise for the runtime too, because we are targeting x64 Linux, we can specify the -r or -runtime for more information about the options available for dotnet restore please see, Microsoft Docss - dotnet restore

Our Dockerfile now looks like:

# Copy and restore as distinct layers
COPY . .
WORKDIR /app/src/Samples.WeatherForecast.Api
RUN dotnet restore Samples.WeatherForecast.Api.csproj -r linux-musl-x64

Here you can see a few more things, one the COPY command and two, the WORKDIR command. We copy everything from the source to our destination, hence the period . for the source and the period . for the destination. Next, the WORKDIR changes the current directory.

DotNet duild (publish)

Now we move on to the build, or rather publish, we do this because publish will also build as well, but what we don’t want it to do is restore, so we use the option of --no-restore. I feel it is down to personal preference, you may want to do a dotnet build before a dotnet publish and then in addition specify --no-build, but I'll leave it up to you to decide. Here, I'm going to go straight to publish.

FROM build AS publish
RUN dotnet publish \
    -c Release \
    -o /out \
    -r linux-musl-x64 \
    --self-contained=true \
    -- no-restore \
    -p:PublishReadyToRun=true \
    -p:PublishTrimmed=true

There are a few more dotnet publish options happening here.

-c defines the build configuration, in our case Release.

-o specifies the path for the output directory.

-r or -runtime specifies the given runtime, just like in dotnet restore

Please see the RID catalogue for more information. https://docs.microsoft.com/en-us/dotnet/core/rid-catalog

--self-contained publishes the .net runtime with the application so the runtime doesn't need to be installed in the image.

--no-restore doesn't execute an implicit restore when running the command.

-p:PublishReadyToRun=true compiles the apps assemblies as ReadyToRun (R2R) format. R2R is a form of ahead-of-time (AOT) compilation.

This will make our start-up time a great deal faster, but it comes at the cost of size.

-p:PublishTrimmed=true trims unused libraries to reduce the deployment size of an app when publishing a self-contained executable.

Take great care in using this option
For more information, please see, Microsoft Docs — dotnet trim-self-contained

What is linux-musl-x64 ?

Lightweight distributions using musl like Alpine Linux

See references [2]

The final stage

With the new commands we have recently used, we can use the `FROM` to indicate a different base image. This is the most important part to take note around Docker multi-stage builds.

We don’t want the SDK in the final runtime image, for one reason, it’s too large, but most importantly we don’t actually even need it!

We have compiled AOT (Ahead-of-time) and including the .net runtime as part of our app.

We go to Docker Hub again and we can find the runtime dependency images. It’s name is mcr.microsoft.com/dotnet/runtime-deps and it has a bunch of tags like we've seen previously. So we'll attach ${VERSION} to it.

For more information, please see, Microsoft Docs — dotnet Runtime Dependencies

# Final stage/image
FROM mcr.microsoft.com/dotnet/runtime-deps:${VERSION}
WORKDIR /app
COPY --from=publish /out .

We set the WORKDIR again and COPY the output from publish, and if you noted the directory with the -o option, this is where we specified the output from the dotnet publish command.

Finally, we can expose the port and specify an entry point.

EXPOSE 8080
ENTRYPOINT ["./Samples.WeatherForecast.Api"]

However, you may have clocked it, but if you haven’t, our ENTRYPOINT looks a little different compared to what it was in Part 1. This is because we are taking advantage of - p:PublishReadyToRun=true.

Full Dockerfile

ARG VERSION=5.0-alpine

FROM mcr.microsoft.com/dotnet/sdk:${VERSION} AS build
WORKDIR /app

# Copy and restore as distinct layers
COPY . .
WORKDIR /app/src/Samples.WeatherForecast.Api
RUN dotnet restore Samples.WeatherForecast.Api.csproj -r linux-musl-x64

FROM build AS publish
RUN dotnet publish \
    -c Release \
    -o /out \
    -r linux-musl-x64 \
    --self-contained=true \
    -- no-restore \
    -p:PublishReadyToRun=true \
    -p:PublishTrimmed=true

# Final stage/image
FROM mcr.microsoft.com/dotnet/runtime-deps:${VERSION}
WORKDIR /app
COPY --from=publish /out .

EXPOSE 8080
ENTRYPOINT ["./Samples.WeatherForecast.Api"]

Let’s build it

Navigate to your repo root directory and execute the docker build command.

docker build -t samples.weatherforecast:v2 .

We have changed the tag so we can compare from Part 1.

Now, execute a docker image ls to see your images.

Notice the size different, we’ve been able to optimise the size from 210MB down to 109MB!; and this includes .

With AOT, you do pay a size price, you need to determine if this size is worth paying for in terms of execution speed, or you’d rather an even smaller image size and the JIT compile.

If you prefer the image size is more important, simply remove the option and adjust your ENTRYPOINT.

You end image size will be about 69MB!

Image size is **not* everything and does not equate to an increase in security hardening — However, it does help, the less bloat an base image has, i.e. less tooling etc. including in an image, the slightly less likely a hacker could take advantage of any vulnerabilities of those.*

You’ll also notice the TAG, and hopefully seeing this will really hit home about what it really is, and the concept of using latest can be misleading. You can see latest is 25 hours old, whereas v2 is about 1 hour, the tag can be anything, just because I’ve used the term latest, it doesn’t actually mean anything. In our case, the latest version is really the image with the v2 tag.

It comes down to good practice around the image tag — Using latest should point to the latest production image, however, using latest is no longer good practice.

Let’s test it

We should test the image to make sure it serves requests as we expect.

docker run -it --rm -p 8080:8080 samples-weatherforecast:v2

Don’t forget the tag change to v2.

Let’s use Postman again… It works!

What have we learned?

We have learned about multi-stage builds in Docker, how the build cache works, and how to really optimise your final image. There are many different base images supporting different OS’s, Alpine, Buster, Ubuntu and more. This give engineers the ultimate flexibility to chooser what is right for their solution. For this example, we’ve opted for something very minimal and lightweight.

If you’re new to Docker, I recommend that you spend further time to understand more docker commands that are available in the Dockerfile.

Up next

Part 3 in this series will be about:

• GitHub Actions — Build the docker image
• GitHub Actions — Publish the docker image to the GitHub Container Registry

More information

• https://www.azure.com
• https://dotnet.microsoft.com/
• https://www.github.com
• https://www.docker.com
• https://www.pulumi.com

References

[1] — https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

[2] — https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-publish

API’s from Dev to Production — Part 2 was originally published in NewDay Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Interview techniques, it’s a subject that we don’t like to really talk about, but in reality we should, it is a skill many of us need to learn; not everyone is born with it.

We will go through the what, and how of the usual, famous last stage interview, the behavioural / competency-based interview.

Behavioural, competency-based interview

What is it?

Competency-based interviews are based on the premise that past behaviour is the best predictor of future performance. During an interview, the interviewer will ask questions where you will be required to answer based on your previous experience.

Do not talk about what you would do, it’s more geared towards what you have done, what you have experienced.

These style of questions could include challenges you have faced and how you have handled them, opportunities you spotted and taken advantage of, or successes and failures and what you have learned from them.

It’s important to be honest, even in failure, showing humility, recognising those failures, and how you’d react to such a situation in the future is something you can lean on; turning a negative into a positive.

How?

First of all, acknowledge the fact you need to prepare, prepare, prepare!

Remember, the competency-based interviewer are seeking answers that are based on your past experience in certain situations, and you need to have a detailed answer.

One method you should consider utilising is known as, STAR.

• Situation — Describe the context within which you performed a job or faced a challenge at work.
• Task — Describe your responsibility in that situation.
• Action — Describe how you completed the task or deal with the challenge.
• Result — Explain the outcome/result. It may be helpful to emphasize what you accomplished, or what you learned.

It may feel quite repetitive as you answer the ongoing questions from the interviewer, but it’s a good method to keep focused, on-track, and to the point; being concise in this style of interview is a good thing.

When a question is asked, you can structure your response something like the following ( by no means this is the only way of course):

“The Situation I found myself in was X, and because of this situation, I was responsible for X; given my objective of O. The Actions I took for these where, A, B, and C. After taking these decisive actions, the end Result was Y (and I learned Z).

You can of course mix it up and use different words than I to keep it slightly less repetitive, more your own style and a lot less rigid, but please ensure you use the STAR method in sequential order.

The example phrases I give is small, but remember you are telling a story, make your interviewer feel that way, express the actions you took, and ensure there is a connection to the situation you’ve described in just enough detail.

If you can attribute any metric in your story, make sure you highlight it in your situation and in the end result, again so there is a connection between the two.

Your goal is to help your interviewer understand the situational context all the way through to the end result.

Once you understand this method, it’s time to put in a real effort to prepare yourself.

Open up your favourite word processor etc. and find a bunch of sample questions, and go through and answer each one using the STAR method.

Below is a small list, but there are many you can Google for, and over time, you can build your back catalogue:

• Tell me about an occasion when you provided exceptional service to a client or stakeholder.
• Tell me about a time when you failed to deliver.
• Give me an example of a time when you successfully persuaded others to do something.
• Describe a situation where you have identified the need for change in a business.

Why do this?

• This will help reinforce the STAR method
• Provide yourself a good sum of situations you’ve been in in the past. It’s important to have a back catalogue if you will, of ready situations you can draw upon.

You of course cannot prepare for everything, which is why putting in this effort now will give you the tools at your disposal to be more agile on day.

Be prepared for follow-up questions, some, but not all questions, the interviewer may dive further into a situation you have described, this is just a warning, and another thing to prepare for. It’s hard, because you may not remember further information about that particular situation. Be honest with the interviewer if that is the case; otherwise, describe another thin layer of detail that you believe will satisfy the follow-up question.

Final thoughts

I cannot stress enough that preparation is quite necessary, if however, you are sitting many interviews, you may not need to prepare as much each time. This is where your previous preparation becomes a valuable asset.

Before each competency interview, scan through your prep-material the night before and/or just before the interview.

Utilise the STAR method, yes, it’s repetitive (I keep saying that :P), I know, but interviews for a lot of people can be stressful, by utilising a method that you can follow each time can really relieve a little bit of stress.

Your back catalogue will prove useful for years to come, you can refer back to your content and it will help you recall situations that occurred in previous job roles you’ve held.

Now, go on, prepare yourself, you can do it!

Vacancies at NewDay

For open vacancies, please visit, Working at NewDay

Interview Techniques was originally published in NewDay Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to a blog post series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, , GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

It will be a long multi-part series covering pretty much everything… I hope!

Our goal is to have an API that follows modern practices, including everything as code with the shift-left philosophy, inclusive of testing, security and infrastructure; in order to facilitate the ability to have an increased level of autonomy as an engineering team.

We will host our services in Microsoft Azure using PaaS services, now you may think how? Surely we need Kubernetes, well the answer is no. There are other options that provide a good middle-ground before going into full-blown Kubernetes. The AppService in Azure supports containers (and there are other products too; such as ACI). This is a fantastic product, those of you who are experienced with the “standard” AppService will know the benefits it provides such as SSL, scaling and much, much more. If you couple this with the benefits of containers, it becomes a good stepping stone into containerised workloads; before taking the leap into Kubernetes such as AKS (Azure Kubernetes Service).

This is a practical hands-on journey from Dev to Production.

TL;DR

Visual Studio Code is great to use as an editor and take advantage of various extensions to make working with Docker and more much easier. We use dotnet new to create our C# project, generate our Dockerfile, modify it just a little bit, build, run and test to ensure it works. We learn some basic commands of the dotnet CLI and Docker CLI.

peteking/Samples.WeatherForecast-Part-1

Why Docker?

Consistent and Isolated Environment

Using containers developers can create predictable environments that are isolated from other apps. Regardless of where the app is deployed, everything remains consistent and this leads to massive productivity: less time debugging, and more time launching fresh features and functionality for users.

Cost-effectiveness with Fast Deployment

Docker-powered containers are known for decreasing deployment time to seconds. That’s an impressive feat by any standard. Traditionally, things like provisioning, getting the hardware up and running would take days or more. In addition, you faced massive overheads and extra work. When each process is put into a container, it can be shared with new apps. The process of deployment becomes swift and you are essentially ready to go.

Mobility — Ability to Run Anywhere

Docker images are free of environmental limitations, and that makes any deployment consistent, movable (portable), and scalable. Containers have the added benefit of running anywhere, providing it is targeted at the OS (Win, Mac OS, Linux, VMs, On-prem, in Public Cloud), which is a huge advantage for both development and deployment. The widespread popularity of the Docker image format for containers further helps. It has been adopted by leading cloud providers, including Amazon Web Services (AWS), Google Compute Platform (GCP), and Microsoft Azure. In addition, you have powerful orchestration systems such as Kubernetes, and products such as AWS ECS or Azure Container Instances are mighty useful in terms of mobility.

Repeatability and Automation

You are building code with repeatable infrastructure and config. This speeds up the development process tremendously. It must be pointed out that Docker images are often small. Consequently, you get fast delivery and, again, shorter deployment for new application containers. Another advantage is straightforward maintenance. When an application is containerized, it’s isolated from other apps running in the same system. In other words, apps don’t intermix and application maintenance significantly easier. It lends itself to being automated; the faster you repeat, the fewer mistakes you make, and the more you can focus on core value for a business or application.

Test, Roll Back and Deploy

As we said, Environments remain more consistent in Docker, from start to finish. Docker images are easily versioned, which makes them easy to roll back if you need to do so. If there is a problem with the current iteration of the image, just roll back to the older version. The whole process means you are creating the perfect environment for continuous integration and continuous deployment (CI/CD). Docker containers are set to retain all configs and dependencies internally. Now, you have a fast and easy way of checking for discrepancies.

Flexibility

If you need to perform an upgrade during a product’s release cycle, you can easily make the necessary changes to Docker containers, test them, and roll out new containers. This sort of flexibility is another key advantage of using Docker. Docker really allows you to build, test, and release images that can be deployed across multiple servers. Even if a new security patch is available, the process remains the same. You can apply the patch, test it, and release it to production. Additionally, Docker allows you to start and stop services, or apps rapidly, which is especially useful within the cloud environment.

Collaboration, Modularity and Scaling

The Docker method of containerization allows you to segment an application so you can refresh, clean up, repair without even taking down the entire app. Furthermore, with Docker you can build an architecture for applications comprising of small processes that communicate with each other via APIs. From there, developers share and collaborate, solving any potential issues quickly. At this stage, the development cycle is completed and all issues are resolved, with no massive overhaul needed — this is extremely cost-effective and time-saving.

See references [1]

Requirements

• Git — https://git-scm.com/downloads
• Visual Studio Code — https://code.visualstudio.com/download
• Docker — https://www.docker.com/products/docker-desktop
• .net 5 — https://dotnet.microsoft.com/download/dotnet/5.0
• WSL 2 (Windows 10 only) — https://docs.microsoft.com/en-us/windows/wsl/install-win10

I’m using Windows 10, for other OS’s like MacOS and Linux, there will be small differences that are not covered here.

VS Code Extensions

1. Create Repository in Git → Samples.WeatherForecast
2. Create ‘src’ directory
3. Within ‘src’ execute:

dotnet new webapi -n Samples.WeatherForecast.Api

At this point you should have a nice new .net 5 webapi.

A nifty trick to launch VS Code in your terminal, simple type code .

. refers to the the file path. In this case: current directory. You can specify alternative location, OR even simply write code at the terminal, which will open code with last workspace.

If you are prompted to install the C# extension, be sure to select, ‘yes’ if prompted; this will create a .vscode folder.

GitIgnore

I assume you probably haven’t created a .gitignore file yet, there is a really useful extension in VS Code called, ".gitignore Generator". I recommend you install it.

In VS Code — Command Palette — Windows (Ctrl+Shift+P) → .gitignore Generator.

For this project, it’s best we select 4 items form the list:

• visualstudiocode
• visualstudio
• Dotnetcore
• csharp

Once you have your .gitignore file, it should look something like the below:

Dockerfile

The good thing about the Docker extension is that it can do loads of things for you, it is a fantastic timesaver; a real Docker companion.

As a first step, we can generate our Dockerfile, and it can generate a docker-compose file too!

VS Code — Command Palette — Windows (Ctrl+Shift+P) → Docker: Add Docker Files to Workspace

Select .NET: ASP.NET Core.

Select Linux.

Specify port 80, lets deal with SSL later.

Select ‘yes’ to include docker-compose files, we’ll need them later.

At the end of this, you should have the following files all generated for you:

1. Dockerfile
2. docker-compose.debug.yml
3. docker-compose.yml
4. .dockerignore

Now, if you carried this operation out in the src/Samples.WeatherForecast.API directory (like I did), these files would have been created there. However, we should move them...

It is best practice to have your Dockerfile etc. in the root directory of your repository.

Let’s move them up into the root directory, if you open VS Code in the root directory, it should appear like below, and we should also move the ignore files; both .gitignore, and .dockerignore too!

Now, we have to modify the Dockerfile, why you ask? Well, given we moved it, plus it’s a little clunky. So let’s do a very quick change to make it just slightly better and then build the image.

You can see below the paths will no longer work for us.

We can simplify the file a little bit by using WORKDIR and make use of COPY . . to copy everything from source to destination.

In addition, if you look carefully, you can see it is doing a dotnet restore then a dotnet build, which means it's going to check to see if it needs to restore again. As part of this initial optimisation (and there will be more to come), we can make use of the --no-restore option.

Your final Dockerfile should appear as the below — Don’t worry if you’ve already spotted further optimisations at this point, there are many we will do in future posts in this series.

FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
WORKDIR /app
EXPOSE 80

FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /app
COPY . .

WORKDIR /app/src/Samples.WeatherForecast.Api

RUN dotnet restore "Samples.WeatherForecast.Api.csproj"

RUN dotnet build "Samples.WeatherForecast.Api.csproj" -c Release -o /app/build --no-restore

FROM build AS publish
RUN dotnet publish "Samples.WeatherForecast.Api.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Samples.WeatherForecast.Api.dll"]

At this point, I think we are good to go!

If you haven’t already done so, this will be a really good time to commit your code to your git repository. For this you can use git directly from your command line, use a VS Code extension, or even use GitHub Desktop; this choice is yours.

Let’s build the image

The command you need to build a docker image is:

docker build -t samples-weatherforecast:latest .

Note, you’ll need to run this in the root directory where the Dockerfile is located, if not, it won’t be able to find it — there is a -f option available to specify the location of the Dockerfile if you ever need to.

-t is the name of the image and tag in name:tag format.

If you’ve followed along to this point, everything should build just fine, if not, please carefully go over the steps, it doesn’t take long.

Docker image built — Now what?

We have built the image, so what do we do next, let’s list the images in our local repository and see what we have there.

Type the follow command in your terminal:

docker image ls

You should see something like the below.

There is a single image called samples-weatherforecast with a Tag of latest, it has an Image ID, and we can even see the Size; 210MB - Don't worry about this size, we will optimise this further later.

Let’s run it!

We can see our image there, it all looks good, so let’s run it and make sure it works. We can do this a couple of way, my preferred is the terminal with the below command, however, Docker Desktop has improved so much recently, it has become much more user friendly, in particular for first time users, or just for those people who either prefer a UI, or just can’t be bothered with more commands.

docker run -it --rm -p 8080:80 samples-weatherforecast:latest

You can see we are using a few options on the command, and we can break these down so you can understand why we have used them.

-i is for interactive mode; so our terminal will wait.

-t is to allocate a pseudo-TTY

--rm automatically removes the container image once the command is terminated.

-p specifies the port configuration in the format of host:container

You can always type --help to find out what commands are available.

If all is well with your container and your run command, you should see a screen like mine below:

We now have a working container running, and there are loads of things we could do to inspect the container etc. we can get into that in another blog post.

Don’t forget the Docker Desktop has that new UI, please check it out, it’s fantastic.

Send a request

We have the container running, just waiting there to serve requests, it would be a shame not to send one to make sure it actually works.

For me, my best friend is Postman, if you haven’t heard of it, I ask you where have you been for the last 10 years! Anyway, if you send a request like the following, your container should reply and come back with the weather forecast.

We’re Finished!

Unfortunately not, there is still lots we can do to optimise the size of the container, and there are also loads of Docker commands you should explore to help with your everyday engineering lives too.

What have we learned?

We have learned the basics of creating a brand new .net 5 webapi, ensuring you have a valid .gitignore file, and the very basics of Docker. We built and ran the image, and tested it to make sure it worked as we expected it to.

Furthermore

Docker cannot be understated, since 2016 there has been a lot of changes in the container ecosystem, we’ve seen Docker Swarm pretty much come and go through the container wars (Kubernetes won if you didn’t know ). Although through Docker Enterprise sale to Mirantis, Docker Swarm is still alive and kicking inside their Enterprise Container Cloud solution, in addition to a secure, zero-downtime managed Kubernetes product.

Not to mention there are other orchestrators, to mention just one, there is Nomad from Hashicorp, and there are even other containers similar to Docker like Podman too.

The benefits of containers are there and paramount no matter what you are really building, even if you’re mixing IaaS and PaaS.

Up next

Part 2 in this series will be about:

• Optimising the Docker image size

More information

• https://www.azure.com
• https://dotnet.microsoft.com/
• https://www.github.com
• https://www.docker.com
• https://www.docker.com/why-docker
• https://www.pulumi.com

References

[1] — https://hentsu.com/docker-containers-top-7-benefits/

API’s from Dev to Production — Part 1 was originally published in NewDay Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Shift-left is a practice intended to shorten the feedback loop in the lifecycle in order to take decisive action.

The idea is to improve overall quality and security and more by moving tasks to the left as early in the lifecycle as possible.

What are the main benefits?

Automation

• Much fewer human errors
• Increased test confidence
• Fewer production issues

Increased delivery speed

• The time between releases can reduce significantly
• The quality of software improves

Increased satisfaction

• Faster delivery of software with less defects is a major benefit for all; from product owners, stakeholders and even customers!

Where are some of the places we can shift-left?

Testing

When building API’s for example, the team owns the API product, its design, its code, its quality, in order to ensure a high-quality product is produced for its customers (consumers), the engineering team is responsible for testing its own product. From unit-testing to performance testing and all those in-between, the list above is not an exhaustive one, and there are many other testing practices you can put in place.

I do hope many of the items listed above are being carried out by an engineering team who owns the product already, for those who are not, the next step is how? How do we, “shift-left”? Well, we need to automate, if you can automate these items, and let’s be honest, who hasn’t automated some of these before in a CI (Continuous Integration) pipeline? — but not everyone has. If you haven’t, these should be an essential items in your shift-left shopping basket!

Security

Security is paramount these days, and a lot of the time, InfoSec (Information Security) or security teams are on the right. Quite often, these teams are not large teams, and adopt a decentralised model, which puts the responsibility on engineering teams. This means the pressure is already on the engineering team to shift-left and bake security practices into the lifecycle, early and often.

Through automation, we can achieve fantastic results, as part of CI, it is easy to achieve static / dynamic code analysis, security scanning of internal and open source libraries, and even container scanning. This all means as an engineer, you’ll know very early on if something isn’t quite right, instead of waiting later, and possibly relying on another team.

As the world of tech continues its love affair with containerisation, security is always at the forefront of security engineers, and with shift-left, it now should also be at the forefront of every software engineer. With containers specifically, container scanning tools are abundant, and some container registries also scan images that are stored. Scanning before pushing a container and when it lands in the registry, is a typical practice to follow.

With great power comes great responsibility, security should never be an afterthought.

Infrastructure

Infrastructure is something we simply can’t avoid, even with serverless; serverless may have integration on vnets (virtual networks). Usually there will be a centralised Operations or Platform team who owns, designs, builds and operates all infrastructure. This can become a big blocker for engineering teams, teams who cannot deploy their own API’s for example, are reliant on another team, and this will in-turn break team autonomy.

So what do we do, we shift-left? Now, each group of teams will be different, each infrastructure will be different, it is likely you may not shift-left 100%, but you can easily shift-left quite a lot in most setups.

A centralised Operations / Platform team can design, build, and operate baseline, foundational, bedrock infrastructure. Thereby, product teams can build on top of this, and come together when needed, for instance, virtual networks, firewalls, a SEIM as examples.

As a product team, what can you now do? Well, now the engineering team can shift-left; they can build on top of the foundational infrastructure. This improves team autonomy, a sense of ownership, and responsibility, furthermore, you breakdown that barrier to release frequently — Deployment frequency is a key DevOps metric.

One way to achieve this is through infrastructure as code (IaC), and we treat it like any other code, it lives in git. We create PR’s (Pull Requests) and go through CI and CD (Continuous Deployment), it can all be automated, checks can be executed, linters, most tools such as Terraform and Pulumi can preview the infrastructure changes before they are applied.

If you’ve never done IaC before, don’t be afraid… It’s just like learning a new language, and as an engineer, it’s a healthy and a nice, challenging thing to do. Terraform is HCL — Hashicorp Language; although they are building cloud development kits (CDK’s) in other languages now. Then there is Pulumi, a fantastic platform where the barrier to entry is incredibly low, if you’re engineer building API’s and use C#, Go, Python or even JavaScript with NodeJS, you’ll feel at home with Pulumi. Finally, if you’re more front-end, Pulumi supports JavaScript as mentioned, and also TypeScript too! These are just two mainstream examples, cloud vendors also support IaC with their solutions, Azure with ARM Templates (Bicep is a new slimmer version that is in very early alpha), AWS with Cloud Formation etc. The point is to embrace IaC into product teams so an engineering team is more in control of their own destiny. It doesn’t mean never speak to these centralised Operations / Platform teams, if means an engineering team is less reliant and can go at their pace — which is hopefully at a good fast. One key element to remember is…

DevOps is not a role, it’s a culture!

Operations

Once an API, or a server-side rendered web application, or even a statically generated web application is in production, it needs to be monitored; I wonder who should do that….? A centralised team…. or the team that designed it, built it and deployed it….? It’s of course the engineering team that designed it, built it etc.

In many organisations just like infrastructure above, the Operations or Platform teams operate the web applications and API’s etc., this is rather right-heavy so-to-speak. As with infrastructure, the same problems occur with operating services, they define monitoring and alerting, are on-call and more. This can work great for smaller organisations, however, if you are in an organisation that is growing and growing as rapidly as NewDay for instance, this model starts to become a hinderance. Now, what do we do, that’s right, we shift left!

As part of our normal engineering lifecycle, we define performance metrics, degradation metrics, any kind of metric we need for our apps and API’s, we go further and define alerts, anything that is perhaps close to breaching our metrics where we have clear and concise SLA’s (service level agreements) or SLO’s (service level objectives). As part of IaC, these can be baked in as code, we can test early and often in our engineering lifecycle. Once we get to production, it’s the job of the engineering team to monitor there apps and API’s, as they are the ones who designed it, built it and deployed it, the team will be the one to operate it. The engineering team will most likely be on-call if your organisation requires 24/7 operations, because if something goes wrong, the best people to fix it is the team that built it!

What’s next?

If you’ve reached this far, thank you, I’m sure you’re now wondering what’s next? Well, we will be showcasing practical examples as part of a series soon.

NewDay → Shift-Left

Here at NewDay, we are on our journey to shift-left as we scale, it’s not a short journey, but it will be an enjoyable one, one where we come together not just as individual teams, but also as a community of engineers to help drive forward-thinking, innovative change.

The shift-left philosophy will bring unprecedented value to engineers and our customers lives.

NewDay embraces a culture of team autonomy, ownership, responsibility and collaboration. We always aim to do the right thing and pull together, taking on the challenge to aspire to the extraordinary and adapting as we build for the future.

Shift-Left Engineering was originally published in NewDay Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.