<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by GarryPas on Medium]]></title>
        <description><![CDATA[Stories by GarryPas on Medium]]></description>
        <link>https://medium.com/@garry.passarella?source=rss-826782ece220------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Gxy9wRH4PKWrU47QYB4YFQ.png</url>
            <title>Stories by GarryPas on Medium</title>
            <link>https://medium.com/@garry.passarella?source=rss-826782ece220------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:39:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@garry.passarella/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Automating a business process with OpenAI]]></title>
            <link>https://medium.com/@garry.passarella/automating-a-business-process-with-openai-48654686c611?source=rss-826782ece220------2</link>
            <guid isPermaLink="false">https://medium.com/p/48654686c611</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[GarryPas]]></dc:creator>
            <pubDate>Thu, 15 Feb 2024 21:58:07 GMT</pubDate>
            <atom:updated>2024-02-15T21:58:07.521Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*ScqIJKrb3eL6CjsRqNuUZg.jpeg" /></figure><h3>Introduction</h3><p>As industries evolve, the integration of OpenAI into business operations represents a significant milestone. In this article, I’ll explain how I used OpenAI to automate a business process, unraveling challenges, gaining insights, and realising tangible benefits. From ideation to implementation, this article offers a roadmap for readers eager to embrace AI, providing insights into the potential pitfalls and triumphs that come with incorporating technology into everyday business practices (yes, ChatGPT did write this paragraph).</p><h3>Background</h3><p>I am bootstrapping a startup in my spare time; one of the user stories involves automotive specialists (mechanics, body repairers etc) signing up to a platform and creating a profile that is made visible to the public. One of the things that concerned me was the possibility that spammers might start posting on the platform, so I added a step that allowed me to manually approve applications before they become visible.</p><p>Luckily (or not) the site isn’t getting a lot of traffic at the moment, most likely because I’ve done no marketing. However, if/when it does, someone (probably me, since I haven’t got the finances to employ anyone yet) would need to approve new repairers. Unsurprisingly I can tell you that this chore does not appeal to me. Furthermore repairers will endure a wait while someone (me) gets the time to do this. This process seemed like a very good candidate for leveraging some of the developments in natural language processing that have caused so much hype in the last couple of years.</p><p>So was born the idea of automating this process by integrating the systems I’ve built with OpenAI.</p><h3>Architecture</h3><p>The architecture of this application will be pretty familiar to software engineers. An existing API posts a message onto an SQS queue; this triggers a lambda, which gets the profile and then calls OpenAI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iwU6Hr_sNTFIe0xFj8-M7w.png" /><figcaption>Sequence diagram illustrating the components of the repairer approval system</figcaption></figure><p>There’s nothing too alien here. All of the AI cleverness is hidden behind OpenAI’s API. The code used in the lambda (modified for brevity) is given below (using the openai NPM library).</p><pre>  const response = await openai.chat.completions.create({<br>    model: &#39;gpt-3.5-turbo&#39;,<br>    temperature: 0.1,<br>    max_tokens: 100,<br>    messages: [<br>      {<br>        role: &#39;system&#39;,<br>        content: `You are an automated agent working for a site that connect automotive specialists with customers.<br>          Auto specialists sign-up and request to be approved to become repairers. You are responsible for <br>          approving or rejecting these requests.<br>          Look at the profile and decide if the profile looks like a legitimate repairer.<br>          You will be given their profile in JSON format.<br>          Reject things that are obviously spam, such as viagra, porn etc.<br>          If the profile looks legitimate, approve it. If it looks suspicious, reject it.<br>          If you are unsure then indicate this too.<br>          You should respond with one of the following 3 responses:<br>          - &quot;approve&quot;<br>          - &quot;reject&quot;<br>          - &quot;unsure&quot;<br>          Only respond like this. Don&#39;t add any other text.<br>          `,<br>      },<br>      {<br>        role: &#39;user&#39;,<br>        content: JSON.stringify({<br>          name: profile.name,<br>          aboutYou: profile.aboutYou,<br>          // ....<br>        }),<br>      },<br>    ],<br>  });<br><br>  let result = &#39;pending&#39;;<br>  const responseContent = response.choices[0].message.content;<br>  if (responseContent === &#39;approve&#39;) {<br>    result = &#39;basic&#39;;<br>  } else if (responseContent === &#39;reject&#39;) {<br>    result = &#39;rejected&#39;;<br>  }</pre><p>After this an API call is made to set the status to approved or rejected; if OpenAI can’t reach a decision the profile remains in a pending state for a human to review.</p><h3>Does it work?</h3><p>So I completed the following (obviously spam) profile and submitted my application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l-at5vbOfM-VlkL9JaW-4Q.png" /><figcaption>Spam profile that was submitted for approval</figcaption></figure><p>OpenAI takes several seconds to process the request (worth noting when setting Lambda timeouts), but after a few seconds the status was set to rejected. Testing so far has been positive and there have been no cases where an obviously wrong decision was made. For a few cents in charges each month, and a couple of hours of coding, this felt like a great investment of time.</p><h3>Summary</h3><p>This article delves into the integration of OpenAI into the business operations of a startup focused on automotive specialists. The potential challenges of manually approving profiles due to the risk of spammers leads to a solution that leverages OpenAI’s natural language processing capabilities to automatically approve or reject profiles within seconds.</p><p>The architecture of the application involves an existing API posting messages to an SQS queue, triggering a lambda function. This lambda function retrieves the profile and then calls OpenAI’s API to make decisions on whether to approve or reject the application.</p><p>To me it appears there are likely a huge number of applications of this type of approach to a wide variety of processes that are currently performed manually. It is likely organisations will seek to take advantage of OpenAI to automate these in the very near future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48654686c611" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a ChatGPT chatbot with OpenAI]]></title>
            <link>https://medium.com/@garry.passarella/building-a-chatgpt-chatbot-with-openai-4eab7b4c1041?source=rss-826782ece220------2</link>
            <guid isPermaLink="false">https://medium.com/p/4eab7b4c1041</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[typescript]]></category>
            <dc:creator><![CDATA[GarryPas]]></dc:creator>
            <pubDate>Fri, 09 Feb 2024 22:08:43 GMT</pubDate>
            <atom:updated>2024-02-09T22:08:43.831Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Header image showing a laptop with an AI bot illustration" src="https://cdn-images-1.medium.com/max/1024/1*d-1wISnAQ8BUaYNq6rMCdA.jpeg" /></figure><p>In the past 18 months the tech job market has cratered, but amidst this upheaval, one sector is booming: artificial intelligence (AI). As the driving force behind many cutting-edge AI initiatives, ChatGPT has not only weathered the shifting tides of the tech job market but has also emerged as a key player shaping the future of artificial intelligence.</p><p>Moreover, it’s worth noting that OpenAI, the organisation behind ChatGPT, continues to be at the forefront of AI research and development. OpenAI’s commitment to pushing the boundaries of what’s possible in artificial intelligence has contributed significantly to the sustained growth and profitability of this sector, solidifying its position as a trailblazer in the ever-evolving tech landscape. In fact, ChatGPT wrote most of what you’ve read so far! 😀</p><p>It also so happens that I’ve been attempting to bootstrap a startup for the last 9 months, and it seemed like the opportunity to add a chatbot would increase interest in my product, and give me a chance to try out OpenAI’s API (always fun to learn about inovative technology).</p><p>I promise I wrote the rest of this article myself…</p><h3>Getting up and running</h3><p>The first thing you need to get up and running is an account with Open AI, which is <a href="https://platform.openai.com/docs/overview">over here</a> at the time of writing. Getting up and running is pretty simple, and they have lots of code samples in several popular languages, including Javascript and (of course) Python.</p><p>To make requests to OpenAI’s API you need an API key. This is pretty straight forward to generate. I also had to put a small amount of money on my account as it turned out I was out of free credits.</p><h3>The backend code</h3><p>Once this is setup you can get up and running with NestJS (a Typescript framework for building APIs) you can do something simple like this. Nest has the concept of providers for bootstrapping depencies…</p><pre>export const chatProviders = [<br>// ...<br>{ provide: &#39;OPEN_AI&#39;,    useFactory: () =&gt; new OpenAI() }<br>// ...<br>],</pre><p>If you assign your API key to an environment variable named OPENAI_API_KEY it will automatically be picked up. I also had to add my organisation ID too which is OPENAI_ORG_ID .</p><p>With this in place it is possible to setup a conversation much like you’d have with ChatGPT, but with some added context you’ve given it in the first message with the system role.</p><pre>const response = await this.openai.chat.completions.create({<br>  model: &#39;gpt-3.5-turbo&#39;,<br>  temperature: 0.5,<br>  max_tokens: 150,<br>  messages: [<br>    {<br>      role: &#39;system&#39;,<br>      content:<br>        &#39;You are an assistant for a website that advises on mechanical and bodywork repairs, and car servicing. Try to answer the user’s question as best as you can.&#39;,<br>    },<br>    ...messages,<br>  ],<br>});<br>return {<br>  response: response.choices[0].message.content,<br>  messages,<br>};</pre><p>The result goes back to the client via an API call which is a pretty standard NestJS endpoint.</p><p>Note that I didn’t need to define any activation functions, weights, biases or any other mathematical constructs — i.e. no PhD needed here. It is all abstracted away, it’s just standard software engineering, calling an API. max_tokens and temperature (which is basically randomness) are two of several options you can configure, how deep you go is up to you.</p><h3>The front end code</h3><p>While calls through Postman were enough to verify the backend was working, I wanted to also expose this on the front-end of my app. I decided not to build a component from scratch, instead I used the <a href="https://fredrikoseberg.github.io/react-chatbot-kit-docs/">react-chatbot-kit</a>, a library written by Fredrik Oseberg. While I had a few issues getting up and running, it was largely intuitive and plenty good enough.</p><p>I was able to hack the header a little and add some React Bootstrap code into this (React Bootstrap is a React component library from Netlify), rather than just plain text, so I could add a close icon. Since react-chatbot-kit is well documented I will skim over this a bit.</p><pre>&lt;ReactChatbotKit<br>  saveMessages={(messages) =&gt; this.saveMessages(messages)}<br>  messageHistory={this.state.messages}<br>  config={config}<br>  messageParser={MessageParser}<br>  actionProvider={ActionProvider}<br>  runInitialMessagesWithHistory={true}<br>  headerText={<br>    (<br>      &lt;&gt;<br>        &lt;Container&gt;<br>          &lt;Row&gt;<br>            &lt;Col style={{ textAlign: &#39;left&#39; }}&gt;<br>              Unbreakmycar Virtual Assistant<br>            &lt;/Col&gt;<br>            &lt;Col className=&quot;col-auto&quot;&gt;<br>              &lt;CloseButton<br>                onClick={() =&gt; this.setState({ showBot: false })}<br>              /&gt;<br>            &lt;/Col&gt;<br>          &lt;/Row&gt;<br>        &lt;/Container&gt;<br>      &lt;/&gt;<br>    ) as any<br>  }<br>/&gt;</pre><p>The end result looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*dkfddrP4BqNm5guFbkv_YQ.png" /><figcaption>The chatbot in action</figcaption></figure><h3>Summary</h3><p>Here I’ve shared my experience integrating OpenAI’s API for a chatbot. I’ve described, briefly, the process of setting up an account, obtaining an API key, and provide backend code using NestJS. The code demonstrates how to interact with ChatGPT to generate responses without the need for complex mathematical constructs.</p><p>I also covered the frontend implementation using the react-chatbot-kit, a library by Fredrik Oseberg. The end result is a simple ChatGPT-powered chatbot that can give advice on a particular topic.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4eab7b4c1041" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaffolding a backend CRUD app using NestJS and Postgres]]></title>
            <link>https://medium.com/@garry.passarella/scaffolding-a-backend-crud-app-using-nestjs-and-postgres-6da6a97c5853?source=rss-826782ece220------2</link>
            <guid isPermaLink="false">https://medium.com/p/6da6a97c5853</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[nestjs]]></category>
            <dc:creator><![CDATA[GarryPas]]></dc:creator>
            <pubDate>Fri, 07 Jul 2023 16:22:44 GMT</pubDate>
            <atom:updated>2023-07-07T17:11:56.380Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*UEqwNvoLUe94uIm_F3x0hA.png" /></figure><h3>Introduction</h3><p>As part of an ongoing side-project, I’ve been building a backend service. This service does the one of the things you’ll often find in many enterprise applications —CRUD (creating, reading, updating and deleting) operations.</p><p>There are a multitude of options for doing this in the Node ecosystem — ExpressJS was the de-facto standard for a long time. Then came frameworks like IBM’s Loopback which promised to reduce the amount of work required to get up and running by providing tools to solve common problems so developers didn’t need to reinvent wheels (or waste lots of time trying to glue together disparate components). Frameworks are generally quite opinionated in how software engineering should be done, and this leads us on to NestJS.</p><h3>NestJS</h3><p>Nest (NestJS) is a framework for building efficient, scalable NodeJS server-side applications (from their <a href="https://docs.nestjs.com/">website</a>). For polyglots like myself who have worked in the Java ecosystem in the past you will immediately find concepts that are very familiar; NestJS heavily leverages ‘decorators’ (‘annotations’ to you Java devs, ‘attributes’ if you come from a C# background). To me it feels a lot like Spring.</p><p>The NestJS website has some extremely comprehensive documentation, and for the most part you can copy and adapt their code samples to suit your requirements. See <a href="https://docs.nestjs.com/first-steps">first steps</a> for more info. NestJS fully supports Typescript, and the setup gives you a Typescript project.</p><h3>My application</h3><p>My application will be a classic three tier application — a web front-end, a NodeJS backend and a Postgres database. At this stage I have no intention of complicating matters with message queues or anything like that, the problem I’m addressing (top secret at the moment 😁) doesn’t call for these solutions yet. I plan to host the front-end on <a href="https://vercel.com/">Vercel</a> and the backend in AWS, but this is still up in the air.</p><p>Central to my app is the concept of “listings” — these are something users create and then they appear on the web app.</p><p>I’ve used a popular ORM (typeorm) for connecting to a Postgres database (running in a Docker container locally). NestJS supports this natively along with others such as Sequelize. NestJS pushes you towards using the repository pattern, and greatly reduces the code you’d need to write if you were using something like Express (of course with this comes a loss of control, a usual trade-off we have to consider in software engineering).</p><h3>AWS Cognito for authentication and authorisation</h3><p>In a previous post I wrote about <a href="https://medium.com/@garry.passarella/setting-up-a-basic-sign-up-sign-in-flow-with-amazon-cognito-c50c0fc46237">integrating Cognito into a front-end React application</a>. A JWT issued from my Cognito user pool is sent to the backend application. The user must call the listing endpoints with a valid JWT, which is enforced using a <a href="https://docs.nestjs.com/guards">NestJS guard</a> — this is the authentication step.</p><pre>import { CanActivate, ExecutionContext, Injectable } from &#39;@nestjs/common&#39;;<br>import { CognitoJwtVerifier } from &#39;aws-jwt-verify&#39;;<br>import { CognitoJwtVerifierSingleUserPool } from &#39;aws-jwt-verify/cognito-verifier&#39;;<br><br>@Injectable()<br>export class CognitoGuard implements CanActivate {<br>  private readonly verifier: CognitoJwtVerifierSingleUserPool&lt;{<br>    userPoolId: string;<br>    clientId: string;<br>    tokenUse: &#39;access&#39;;<br>  }&gt;;<br>  constructor() {<br>    if (!process.env.COGNITO_USER_POOL_ID) {<br>      throw new Error(&#39;COGNITO_USER_POOL_ID environment variable was not set&#39;);<br>    }<br><br>    if (!process.env.COGNITO_CLIENT_ID) {<br>      throw new Error(&#39;COGNITO_CLIENT_ID environment variable was not set&#39;);<br>    }<br><br>    this.verifier = CognitoJwtVerifier.create({<br>      userPoolId: process.env.COGNITO_USER_POOL_ID,<br>      clientId: process.env.COGNITO_CLIENT_ID,<br>      tokenUse: &#39;access&#39;,<br>    });<br>  }<br><br>  async canActivate(context: ExecutionContext): Promise&lt;boolean&gt; {<br>    const request = context.switchToHttp().getRequest();<br>    const token = request.headers?.authorization;<br>    if (!token) {<br>      return false;<br>    }<br><br>    try {<br>      const result = await this.verifier.verify(token);<br>      return !!result?.username;<br>    } catch (err) {<br>      return false;<br>    }<br>  }<br>}</pre><p>When operations such as update or delete are attempted I also needed to check that the user calling the endpoint is really allowed to update/delete the record (authorisation). Otherwise we’d have users modifying each-other’s listings. For this I added a simple decorator using some of the code from the guard.</p><pre>import { ExecutionContext, createParamDecorator } from &#39;@nestjs/common&#39;;<br>import { User } from &#39;@user/user.types&#39;;<br>import { CognitoJwtVerifier } from &#39;aws-jwt-verify&#39;;<br><br>let verifier = null;<br>const getVerifier = () =&gt; {<br>  if (!verifier) {<br>    if (!process.env.COGNITO_USER_POOL_ID) {<br>      throw new Error(&#39;COGNITO_USER_POOL_ID environment variable was not set&#39;);<br>    }<br><br>    if (!process.env.COGNITO_CLIENT_ID) {<br>      throw new Error(&#39;COGNITO_CLIENT_ID environment variable was not set&#39;);<br>    }<br><br>    verifier = CognitoJwtVerifier.create({<br>      userPoolId: process.env.COGNITO_USER_POOL_ID,<br>      clientId: process.env.COGNITO_CLIENT_ID,<br>      tokenUse: &#39;access&#39;,<br>    });<br>  }<br>  return verifier;<br>};<br><br>export const CurrentUser = createParamDecorator(<br>  async (data: any, ctx: ExecutionContext): Promise&lt;User&gt; =&gt; {<br>    const request = ctx.switchToHttp().getRequest();<br>    const token = request.headers?.authorization;<br>    if (!token) {<br>      throw new Error(<br>        `authorization header undefined. This shouldn&#39;t happen. Ensure CognitoGuard is also applied.`,<br>      );<br>    }<br>    const payload = await getVerifier().verify(token);<br>    return {<br>      userId: payload.username,<br>    };<br>  },<br>);</pre><p>These can be applied like this</p><pre>@Controller(&#39;/api/v1/listing&#39;)<br>@UseGuards(CognitoGuard)<br>export class ListingController {<br>  ...<br>  @Put(&#39;/&#39;)<br>  async updateListing(<br>    @Body() listing: Listing,<br>    @CurrentUser() currentUser: User,<br>  ) {<br>  ...<br>  }<br>}</pre><h3>Testing</h3><p>NestJS uses Jest and <a href="https://www.npmjs.com/package/supertest">Supertest</a> under the hood, and the framework gives you a template to get up and running with both unit and end-to-end tests. The most interesting part of the unit tests is probably the setup, which allows you to test through controllers and bootstrap all the dependencies using the Nest testing module, which takes care of the dependency injection.</p><pre>beforeEach(async () =&gt; {<br>  const app: TestingModule = await Test.createTestingModule({<br>    controllers: [ListingController],<br>    providers: [<br>      ListingService,<br>      {<br>        provide: &#39;LISTING_REPOSITORY&#39;,<br>        useFactory: () =&gt; ({<br>          find: async () =&gt; [listing1],<br>          create: async (listing: ListingBase) =&gt; ({<br>            ...listing,<br>            id: &#39;FE7862C1-CBEB-45E3-A538-787956E1C026&#39;,<br>          }),<br>          save: async (entity: ListingEntity) =&gt; entity,<br>          update: async (entity: ListingEntity) =&gt; ({<br>            affected:<br>              entity.id === &#39;00000000-0000-0000-0000-000000000000&#39; ? 0 : 1,<br>          }),<br>          findOne: async () =&gt; ({<br>            createdBy: &#39;garry&#39;,<br>          }),<br>        }),<br>      },<br>    ],<br>    imports: [EnvModule],<br>  }).compile();<br><br>  listingController = app.get&lt;ListingController&gt;(ListingController);<br>});</pre><p>The LISTING_REPOSITORY is injected as a mock, with methods stubbed out as I don’t want unit tests going to a real database. As the code grows this mock could be moved to a helper, but for now this works OK. NestJS likes you to use it’s IoC container (whereas NodeJS devs has many other approaches, such as monkey-patching stubbed imports at runtime) — fighting the framework’s preferred way of doing things will lead you down rabbit-holes.</p><p>Tests are pretty simple and don’t need any elaboration, for example:</p><pre>const result = await listingController.updateListing(listing, {<br>  userId: &#39;garry&#39;,<br>});<br>expect(result).toEqual(listing);</pre><p>End to end-tests are far more interesting, and you have a bit more freedom here. First of all I setup my application…</p><pre>beforeAll(async () =&gt; {<br>  const moduleFixture: TestingModule = await Test.createTestingModule({<br>    imports: [AppModule],<br>  }).compile();<br><br>  app = moduleFixture.createNestApplication();<br>  await app.init();<br>});</pre><p>…and tests look like this:</p><pre>  it(&#39;should return 200&#39;, () =&gt; {<br>    return request(app.getHttpServer())<br>      .get(&#39;/health/ping&#39;)<br>      .expect(200)<br>      .expect({ status: &#39;OK&#39; });<br>  });</pre><p>But what happens under the hood?</p><p>Well I created a pretty elaborate approach which would connect to a real Postgres database that is spun up with docker-compose each time end-to-end tests are run. The process goes like this:</p><ol><li>Before the tests run docker-compose spins up a Postgres database in a container.</li><li>The end-to-end test suit is started.</li><li>Using a Jest test setup I wait until the database is ready.</li><li>A seeding script is run. Migrations may be happening, so it will retry until successful.</li><li>Once everything is ready in the database layer the end-to-end tests run.</li><li>The docker container is taken down on completion.</li></ol><p>AWS Cognito authentication is stubbed out, but everything else is very close to what I’d be testing if I used Postman.</p><h3>Other notes</h3><p>NestJS has a config module (@nestjs/config) which I found myself bypassing quite often. Since this wraps a variant of the dotenv library, it basically is just loading environment variables. Often dependency injection doesn’t really work for this — for example I wanted environment variables for a guard and dependency injection isn’t supported for guards. I’m unclear if this module really has any value.</p><p>I also added Husky to run both tests and e2e tests before committing. This is nice as it prevents you committing broken code. The checks can be skipped using --no-verify if you want to (for example) commit and push work-in-progress.</p><h3>Discussion</h3><p>NestJS allowed me to get an enterprise-standard backend application up and running very quickly. It bootstraps a lot of stuff like Jest, Supertest and prettier which you’ll likely find yourself doing anyway if you don’t use a framework. It is pretty opinionated, so you’ll want to try and find the “Nest way” of doing things rather than trying to make it work differently. The heavy use of decorators and IoC are two examples.</p><p>In this post I also showed how I could get a pretty high-fidelity suite of end-to-end tests up and running using a real database. Doing this gives me a lot of confidence that I haven’t introduced regressions without having to resort to lots of manual testing.</p><p>Overall I’d probably look to use NestJS on more projects. It works and it works well for building high-quality apps quickly. It might introduce a bit of bloat, but in most organisations the greatest cost is development time. If we optimise for this then NestJS is a very good tool to have in the box.</p><p>Code: <a href="https://github.com/garrypas/medium-post-nestjs-backend">https://github.com/garrypas/medium-post-nestjs-backend</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6da6a97c5853" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up a basic sign-up/sign-in flow with Amazon Cognito]]></title>
            <link>https://medium.com/@garry.passarella/setting-up-a-basic-sign-up-sign-in-flow-with-amazon-cognito-c50c0fc46237?source=rss-826782ece220------2</link>
            <guid isPermaLink="false">https://medium.com/p/c50c0fc46237</guid>
            <category><![CDATA[cognito]]></category>
            <dc:creator><![CDATA[GarryPas]]></dc:creator>
            <pubDate>Thu, 29 Jun 2023 20:56:29 GMT</pubDate>
            <atom:updated>2023-06-29T20:58:37.496Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p3sWU_3IqUPgYjF4oNAQFQ.png" /></figure><h3>Introduction</h3><p>I’m currently looking for an authentication provider for my side-project, to avoid having to manage user profiles, passwords etc. One such service is Cognito, available from AWS (Amazon Web Services).</p><p>I’ve done a spike to try and get the sign-up and sign-in flows working, just with username and password for now. The application is a web app built with React. I thought I’d share my learnings here.</p><h3>User Pools</h3><p>User pools are a user directory service that allows you to easily manage and scale user registration, sign-in, and authentication for your applications. Auth is usually pretty generic and not something you want to reinvent for every application, so it makes sense to offload some of this to a third party, even for a small fee.</p><p>The first step towards getting setup was to login to the AWS Management Console and search ‘Cognito’. From here there is an option called ‘Create user pool’.</p><p>You are presented with a wizard which has quite a bewildering number of options, including things like SES (Simple E-mail Service) integration, necessary for things like forgotten password use cases. For this exercise I just wanted the simplest configuration possible. Luckily AWS have built a tool that simplifies this sort of thing — The AWS Amplify toolchain.</p><h4>AWS Amplify</h4><p>I used AWS amplify, which allowed me to quickly get my web app code up and running. I used the @aws-amplify/cli tool to create the User Pool on the backend, and the aws-amplify library for the code in my web app.</p><pre>npm install --save aws-amplify<br>npm install --save-dev @aws-amplify/cli<br>npm install --save @aws-amplify/ui-react</pre><p>These commands initialised the files needed.</p><pre>mkdir amplify &amp;&amp; cd amplify<br>npx amplify init<br>npx amplify add auth<br>npx amplify push</pre><p>A bit of explanation as to what is going on here; amplify has created a Cloud Formation Stack which is used to manage resources such as the User Pool, Lambdas and quite possibly some things you don’t need. Even if you don’t use it for a ‘real’ product it’s a great way to get started and build a good mental model before going off and trying to Terraform everything</p><h4>React UI — Sign Up</h4><p>To tell the amplify code where your User Pool lives and how to interact with it, you need to create an amplify config document. I used this template from the <a href="https://docs.amplify.aws/lib/auth/getting-started/q/platform/js/">amplify docs</a> and adapted accordingly. I called configureAuth() from my application entry point.</p><pre>import { Amplify, Auth } from &#39;aws-amplify&#39;;<br><br>Amplify.configure({<br>  Auth: {<br>    // REQUIRED - Amazon Cognito Region<br>    region: &#39;eu-west-1&#39;,<br><br>    // OPTIONAL - Amazon Cognito User Pool ID<br>    userPoolId: &#39;XX-XXXX-X_abcd1234&#39;,<br><br>    // OPTIONAL - Amazon Cognito Web Client ID (26-char alphanumeric string)<br>    userPoolWebClientId: &#39;a1b2c3d4e5f6g7h8i9j0k1l2m3&#39;,<br><br>    // OPTIONAL - This is used when autoSignIn is enabled for Auth.signUp<br>    // &#39;code&#39; is used for Auth.confirmSignUp, &#39;link&#39; is used for email link verification<br>    signUpVerificationMethod: &#39;code&#39;, // &#39;code&#39; | &#39;link&#39;<br><br>    // OPTIONAL - Configuration for cookie storage<br>    // Note: if the secure flag is set to true, then the cookie transmission requires a secure protocol<br>    cookieStorage: {<br>      // REQUIRED - Cookie domain (only required if cookieStorage is provided)<br>      domain: &#39;.yourdomain.com&#39;,<br>      // OPTIONAL - Cookie path<br>      path: &#39;/&#39;,<br>      // OPTIONAL - Cookie expiration in days<br>      expires: 365,<br>      // OPTIONAL - See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite<br>      sameSite: &#39;strict&#39; | &#39;lax&#39;,<br>      // OPTIONAL - Cookie secure flag<br>      // Either true or false, indicating if the cookie transmission requires a secure protocol (https).<br>      secure: true,<br>    },<br><br>    // OPTIONAL - Manually set the authentication flow type. Default is &#39;USER_SRP_AUTH&#39;<br>    authenticationFlowType: &#39;USER_PASSWORD_AUTH&#39;,<br><br>    // OPTIONAL - Manually set key value pairs that can be passed to Cognito Lambda Triggers<br>    clientMetadata: {myCustomKey: &#39;myCustomValue&#39;},<br><br>    // OPTIONAL - Hosted UI configuration<br>    oauth: {<br>      domain: &#39;your_cognito_domain&#39;,<br>      scope: [<br>        &#39;phone&#39;,<br>        &#39;email&#39;,<br>        &#39;profile&#39;,<br>        &#39;openid&#39;,<br>        &#39;aws.cognito.signin.user.admin&#39;,<br>      ],<br>      redirectSignIn: &#39;http://localhost:3000/&#39;,<br>      redirectSignOut: &#39;http://localhost:3000/&#39;,<br>      responseType: &#39;code&#39;, // or &#39;token&#39;, note that REFRESH token will only be generated when the responseType is code<br>    },<br>  },<br>});<br><br><br>// You can get the current config object<br>export const configureAuth = () =&gt; Auth.configure();</pre><p>Then I implemented the sign-up code:</p><pre>export const register = async (username: string, password: string) =&gt; {<br>  try {<br>    const { user } = await Auth.signUp({<br>      username,<br>      password,<br>      attributes: {<br>        email: username,<br>      },<br>      autoSignIn: {<br>        enabled: false,<br>      }<br>    });<br>  } catch (error) {<br>    console.log(&#39;error signing up:&#39;, error);<br>  }<br>}</pre><p>And I wired this up to a React form I knocked up previously (I’d been using miragejs to mock out the backend up until now).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*eu2_zOGsmgyeQSs0lfbJZA.png" /></figure><p>After signing up I check in the Cognito user pool to ensure the user had been created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jNRlDJ7t1z1qj2kskBlL5A.png" /></figure><p>One further step is required; confirm your email. As you can see from the confirmation status, the user is not yet confirmed. Luckily Cognito gives you an option where it can send up to 50 emails a day without requiring you to set up SES. After that many registrations in 1 day you probably need a rest anyway! I checked my inbox and the email was there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*BHJCDtiD-aUAaXhgN32wZQ.png" /></figure><p>Unfortunately I had no way to verify yet. The code to do this is pretty trivial (see below).</p><pre>export const confirmRegistration = async ({ username, code}: { username: string, code: string }) =&gt; {<br>  try {<br>    await Auth.confirmSignUp(username, code);<br>  } catch (error) {<br>    console.log(&#39;error confirming sign up&#39;, error);<br>  }<br>};</pre><p>…and we also the ability to re-send the code…</p><pre>export const resendSignUpInfo = async ({ username}: { username: string }) =&gt; {<br>  try {<br>    await Auth.resendSignUp(username);<br>  } catch (error) {<br>    console.log(&#39;error resending registration code&#39;, error);<br>  }<br>};</pre><p>Again a fairly simple React component was needed to wire up to this code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*9NghiKQzgiCHXOtXHE55sQ.png" /></figure><p>When I did this first time around I got the error USER_PASSWORD_AUTH flow not enabled for this client. Turns out user/password auth isn’t enabled by default. I needed to go into the AWS management console and enable the ALLOW_USER_PASSWORD_AUTH authentication flow for the web client app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dhFESalaPLHbhLvYAXPshA.png" /></figure><h4>React UI — Sign-in</h4><p>Now was the time to look at sign-in. The code for this was pretty simple and the UI was again another simple forms component.</p><pre>    try {<br>      await Auth.signIn(username, password);<br>    } catch (error) {<br>      console.log(&#39;error signing in&#39;, error);<br>    }</pre><p>What happens next isn’t well documented; if you take a look in dev tools in Chrome you’ll see some cookies have been set automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*UHBu_3NDCWTt0x2ug3ovkQ.png" /></figure><p>With the username, client ID and other stuff in the name they look a bit unwieldy — how do you get the token out? To get to the values you might be tempted to use document.cookies and some parsing, but the correct way is instead like this.</p><pre>const { accessToken, refreshToken } = await Auth.currentSession();</pre><p>The accessToken can now be used to make API calls to your backend.</p><h4>Making a call</h4><p>The following code can now be used to verify that a token is authentic on the backend using theaws-jwt-verify library (see <a href="https://github.com/awslabs/aws-jwt-verify">here</a>).</p><pre>const verifier = CognitoJwtVerifier.create({<br>  userPoolId: &#39;eu-west-1_TlRHxqHgs&#39;,<br>  clientId: &#39;3bho05m8t5ihern2eov5togtv2&#39;,<br>  tokenUse: &#39;access&#39;,<br>});<br>const result = await verifier.verify(token);</pre><p>An error is thrown if it fails, you can catch this and handle accordingly (e.g. return a 403 response code).</p><h4>Conclusions</h4><p>I used the AWS Amplify toolchain to get up and running. I suspect the CLI commands to spin up the Cloud Formation stack has created some resources I don’t really need; I see Identity Pools in there, and I’ll need to investigate further why these were created as I don’t think I need these for my simple use case (the backend will just be a monolith for now). So why use AWS Amplify at all?</p><p>The tool allowed me to get setup extremely fast — this exercise was completed in a few hours in an evening (including a couple of UI components and some extra routes in react-router). When I think about how much work is needed to build your own auth service with something like PassportJS this is definitely a good option if you just need to get something off the ground quickly, and probably a serious candidate beyond this too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c50c0fc46237" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting up and running with Github Packages for the side project]]></title>
            <link>https://medium.com/@garry.passarella/getting-up-and-running-with-github-packages-for-the-side-project-3532ae98c18a?source=rss-826782ece220------2</link>
            <guid isPermaLink="false">https://medium.com/p/3532ae98c18a</guid>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[github-package]]></category>
            <dc:creator><![CDATA[GarryPas]]></dc:creator>
            <pubDate>Wed, 21 Jun 2023 17:50:09 GMT</pubDate>
            <atom:updated>2023-06-21T17:51:48.460Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RyLQgE8RzIDbvTAdyroGFA.png" /></figure><h3>Introduction</h3><p>Most popular programming languages have the concept of a package manager; .NET has NuGet, Node has NPM, Java has Maven and Gradle etc, however sometimes you don’t want to push your code into a public repo, or maybe you just long for something simple that can quickly allow you to break re-usable modules out into libs. Without this convenience developers may decide that certain potentially re-usable pieces of code aren’t worth the hassle of moving, and will leave them where they are. Making it easy to break code out into libs encourages better re-use and higher code quality.</p><p>Often developers will take shortcuts such as using Github itself as a package source. While this is simple it can lead to very slow install times as Github may rate limit requests. It might not be an option in the ecosystem you’re working in either (it is in Node). A large project I’ve been working on recently went from 120 seconds to around 10 seconds just by using Github Packages instead of references the source code repo!</p><p>Github Packages promises easy integration into your repo in Github. I’ve been using this for a while and thought I’d write about it. The <a href="https://github.com/features/packages#pricing">free tier</a> offers 500 MB storage and 1 GB transfer per month.</p><h3>Scenario</h3><p>I have some very simple validation code for a forms application. It’s exactly the sort of code that will be useful in other projects, so the last thing I want to do is have it sitting in my application repo - It’s calling out to be broken out into a lib, so that’s what I’ve done for this exercise. Obviously there are literally hundreds of such libraries around, so in the interests of not re-inventing the wheel you might decide to go with one of those. But this is a side project and I fancied rolling my own…</p><h3>Defining a workflow</h3><p>The first step is to create a repo in Github. Once this is created you’ll want to add a workflow. For this project I added 2; the first to build the project (basically run tests) and the second is responsible for publishing the package. The publish package code looks like this:</p><pre>name: &quot;Publish package&quot;<br><br>on:<br>  push:<br>    tags:<br>      - &#39;v*.*.*&#39;<br><br>jobs:<br>  build:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - name: Checkout project<br>        uses: actions/checkout@v3<br><br>      - name: Setup Node<br>        uses: actions/setup-node@v3<br>        with:<br>          node-version: 18<br><br>      - name: Install node_modules<br>        run: npm ci<br><br>      - name: Run tests<br>        run: npm test<br><br>  publish-gpr:<br>    needs: build<br>    runs-on: ubuntu-latest<br>    permissions:<br>      packages: write<br>      contents: read<br>    steps:<br>      - name: Checkout project<br>        uses: actions/checkout@v3<br><br>      - name: Setup Node<br>        uses: actions/setup-node@v3<br>        with:<br>          node-version: 18<br>          registry-url: https://npm.pkg.github.com/<br><br>      - name: Set NPM package version to match tag<br>        env:<br>          NODE_AUTH_TOKEN: ${{secrets.GITHUB_TOKEN}}<br>        run: |<br>          npm ci<br>          PACKAGE_VERSION=$(echo &quot;${GITHUB_REF_NAME}&quot; | sed &quot;s/^v//g&quot;)<br>          echo &quot;NPM Package version: ${PACKAGE_VERSION}&quot;<br>          npm version ${PACKAGE_VERSION} --no-git-tag-version<br>          npm publish</pre><p>So what’s going on here? The first interesting piece of code is this:</p><pre>on:<br>  push:<br>    tags:<br>      - &#39;v*.*.*&#39;</pre><p>This ensures that the jobs will only run when I create a tag v*.*.* - i.e. when I create a new release. There is some code for setting up Node, installing dependencies, and running tests. This completes the build job and allows the publish job (which depends on it completing successfully) to kick in.</p><p>Most of the publish job is pretty straight forward; the interesting part is this bit here:</p><pre>PACKAGE_VERSION=$(echo &quot;${GITHUB_REF_NAME}&quot; | sed &quot;s/^v//g&quot;)<br>echo &quot;NPM Package version: ${PACKAGE_VERSION}&quot;<br>npm version ${PACKAGE_VERSION} --no-git-tag-version<br>npm publish</pre><p>This takes the tag name, strips off the “v”, then sets the NPM package.json version property to the value (so you don’t need to do this in the source code). Finally the code is published to Github Packages. We can see it in the Github UI here, and there is the code to install it when you drill in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*cYQDLUO3Mrh4TYu0wd6qzQ.png" /></figure><h3>Other steps</h3><p>Some other things to note; you need the scope alias in the package.json for this to work. This will be your organisation/username — in my case I had to change…</p><pre>&quot;name&quot;: &quot;gp-validator&quot;</pre><p>…to…</p><pre>&quot;name&quot;: &quot;@garrypas/gp-validator&quot;</pre><h3>Discussion</h3><p>Github Packages makes it simple for you to get the sort of tools for managing software artefacts that you get at work, but for your side projects. Getting up and running with Github Packages is pretty straight forward, there are lots of code samples online. This can help to encourage you to re-use more of your code rather than rewriting it, having to pull it across from other projects, or referencing it from the Github source code repository.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3532ae98c18a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability engineering with Open Telemetry]]></title>
            <link>https://medium.com/@garry.passarella/observability-engineering-with-open-telemetry-e9d542a5924d?source=rss-826782ece220------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9d542a5924d</guid>
            <category><![CDATA[opentelemetry]]></category>
            <dc:creator><![CDATA[GarryPas]]></dc:creator>
            <pubDate>Mon, 19 Jun 2023 21:38:05 GMT</pubDate>
            <atom:updated>2023-06-19T21:38:05.664Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zFg4S3_vW1AJJHQJ.png" /></figure><h3>Overview</h3><h4>Introduction</h4><p>In today’s technology landscape microservices have emerged as a de-facto standard architecture for complex software systems. Promising close alignment with the principles of Agile, good separation of concerns, and rapid deployment, microservices have garnered significant attention and adoption. However the positivity belies a new set of challenges that make observability a maintenance imperative.</p><p>In traditional system architectures (seen 5–10 years ago), the “monolith” approach was commonly used. Although this approach has become less popular, it offers several advantages over microservices. One key benefit was that monoliths had more predictable and straightforward behaviour. In microservice architectures, procedural calls often involve traversing networks and (increasingly) asynchronous communication. Microservice architectures have had the side-effect of producing systems that are more complex to reason about and debug. As a result, traditional debugging tools may prove ineffective in identifying and understanding defects within this new landscape. This is where observability can help out.</p><h4>Observability</h4><p>There are three main pillars of observability; logging, tracing and metrics. Logging is self-explanatory and familiar to most developers and has existing as long as software systems themselves. Metrics typically give numbers at a point in time that can be used to track things like resource usage. Tracing, meanwhile, can be used to gain insights into the performance down to the level of function calls within code. A sequence of operations are linked together by spans, and these can be correlated across services using distributed tracing.</p><h4>Challenges</h4><p>This need for more sophisticated tooling has seen the rise of various providers who offer tooling to assist with instrumenting, with as-a-service management of the backends used to collate and present data (e.g. New Relic). Unfortunately this has often led to organisations heavily coupling to specific providers, and efforts to switch can become a arduous undertaking. During one such project a couple of years ago while working for a major retailer in the UK, I came across Open Telemetry, and I was keen to learn more. This is the subject of this article.</p><h4>Open Telemetry</h4><p>At this point you might be asking “what is Open Telemetry?” or “why should I be using Open Telemetry?”. Most telemetry providers already offer tooling for pumping metrics, traces etc into their systems, why not use those? Why would we want to complicate things by using this framework?</p><p>The first point to note here is that by adding a vendor’s tooling into your code you immediately couple yourself to that vendor. In some cases the code is pretty rudimentary, and swapping it for another vendor’s library wouldn’t be complicated. But now consider the situation in a large enterprise; thousands of codebases, some of which are coupled to a vendor in strange ways. Now image you have Open Telemetry sitting between. Now you have a proxy layer and you can change the provider much more easily.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0s5Y0zG2sN3tulve2yzg5A.png" /><figcaption>Above: Shows how the Open Telemetry collector works with New Relic. Other providers could be used in addition or instead (<a href="https://newrelic.com/blog/best-practices/opentelemetry-concepts">source</a>).</figcaption></figure><p>It’s worth noting that Open Telemetry is gaining a lot of traction lately. All of the major cloud providers are on board, as are major observability service providers (e.g. DataDog and New Relic). Organisations don’t want vendor lock-in, and providers are listening.</p><p>Open telemetry can be deployed as a gateway or an agent; a gateway is a standalone service, an agent runs on the same host as the service. I used a gateway for this project, with the collector taken from <a href="https://github.com/open-telemetry/opentelemetry-demo">this example</a> running in Docker.</p><h4>Scenario</h4><p>For this exercise I created a microservices architecture for an auction backend (like EBay); users can create auctions and other users can place bids on those auctions. It uses Kafka for raising events. These events are streamed into a second application (the consumer, or worker) which writes them to a Mongo data store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uHkuT9t8HvHtb3Vqu4GdHg.png" /><figcaption>Above: The application. Essentially there are 3 asynchronous operations — one to create an auction in Kafka, the second to persist it, and a third to send a confirmation back to the client who subscribes via a web-socket. This is the sort of scenario where debugging issues would be very tricky in a live application using traditional tooling.</figcaption></figure><h3>Getting started</h3><p>A common approach is to put all the tracing configuration in a single file, which is referenced when the application starts. This can be done adding an import statement to the start of the entry point file. A cleaner approach is to use the --require flag in the start script. For example:</p><pre>node -r ./dist/observability/index.js ./dist/app-api/index.js</pre><h3>Tracing</h3><h4>Distributed tracing</h4><p>Currently the traces in the API and consumer applications live in isolation. We really need a way to correlate the two sets of traces if we’re going to get the most out of our observability tooling; to do this we can utilise Open Telemetry’s distributed tracing capabilities.</p><p>Within the second application, we take the trace context (which consists of a trace ID and a span ID) and extract the trace ID part; then create a child span in the second application and use the extracted trace ID. This is tricky because we’re passing messages via Kafka. Fortunately Open Telemetry does a lot of heavy-lifting for us with the opentelemetry-instrumentation-kafkajs NPM package.</p><h4>Setup</h4><p>With the collector setup, the next step was to add tracing to an application. First some pre-requisits I installed.</p><pre>npm install --save @opentelemetry/sdk-node<br>npm install --save @opentelemetry/resources<br>npm install --save @opentelemetry/semantic-conventions<br>npm install --save @opentelemetry/exporter-trace-otlp-http<br>npm install --save @opentelemetry/sdk-trace-base<br>npm install --save @opentelemetry/api<br>npm install --save @opentelemetry/sdk-trace-node<br>npm install --save @opentelemetry/auto-instrumentations-node<br>npm install --save opentelemetry-instrumentation-kafkajs</pre><p>My file lived in ./src/observability/index.ts</p><pre>import { NodeSDK } from &#39;@opentelemetry/sdk-node&#39;;<br>import { Resource } from &#39;@opentelemetry/resources&#39;;<br>import { SemanticResourceAttributes } from &#39;@opentelemetry/semantic-conventions&#39;;<br>import { OTLPTraceExporter } from &#39;@opentelemetry/exporter-trace-otlp-http&#39;;<br>import { AlwaysOnSampler } from &#39;@opentelemetry/sdk-trace-base&#39;;<br>import { DiagConsoleLogger, DiagLogLevel, diag } from &#39;@opentelemetry/api&#39;;<br>import { RandomIdGenerator } from &#39;@opentelemetry/sdk-trace-node&#39;;<br>import { getNodeAutoInstrumentations } from &#39;@opentelemetry/auto-instrumentations-node&#39;;<br>import { KafkaJsInstrumentation } from &#39;opentelemetry-instrumentation-kafkajs&#39;;<br><br>diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.INFO);<br><br>const serviceName = process.env.SERVICE;<br><br>const traceExporter = new OTLPTraceExporter({<br>  url: &#39;http://localhost:4318/v1/traces&#39;,<br>});<br><br>const sdk = new NodeSDK({<br>  sampler: new AlwaysOnSampler(),<br>  traceExporter,<br>  idGenerator: new RandomIdGenerator(),<br>  instrumentations: [<br>    getNodeAutoInstrumentations({<br>      &#39;@opentelemetry/instrumentation-fs&#39;: {<br>        enabled: false,<br>      },<br>    }),<br>    new KafkaJsInstrumentation({<br>      enabled: true,<br>    }),<br>  ],<br>  resource: new Resource({<br>    [SemanticResourceAttributes.SERVICE_NAME]: serviceName,<br>  }),<br>});<br>sdk.start();</pre><p>Then I executed a Postman request that would cause my app and consumer to interact, took the traceId, and looked in Jaeger…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KRgGRFdegdjI_G8u3jJF7w.png" /></figure><p>…and I can immediately see if there are any performance bottlenecks in there that I need to address. Jaeger is an open source tool, but the major vendors all offer similar schematic views and a lot more.</p><h3>Logging</h3><h4>Setup</h4><p>To setup logging we need to install the logging provider (I used Winston)</p><pre>npm install --save @opentelemetry/instrumentation-winston</pre><p>…then include this in our auto instrumentations (example below).</p><pre>&#39;@opentelemetry/instrumentation-winston&#39;: {<br>  enabled: true,<br>  logHook: (span: Span, record: Record&lt;string, any&gt;) =&gt; {<br>    span.setAttribute(&#39;log.level&#39;, record.level);<br>    span.setAttribute(&#39;log.message&#39;, record.message);<br>    span.setAttribute(&#39;log.meta&#39;, JSON.stringify(record.meta));<br>    record[&#39;resource.service.name&#39;] = serviceName;<br>  },<br>},</pre><p>The standard Winston logging configuration required no changes.</p><h4>Running it</h4><p>When the application is run again we find that logs have some additional attributes, such as trace_id . This is pretty central to the OpenTelemetry way of doing logging (see <a href="https://opentelemetry.io/docs/specs/otel/logs/">here</a>), that logs should be easy to correlate with traces and metrics.</p><pre>{<br>  &quot;level&quot;:&quot;info&quot;,<br>  &quot;message&quot;:&quot;sending confirmation for&quot;,<br>  &quot;name&quot;:&quot;auction-confirmed&quot;,<br>  &quot;payload&quot;:{<br>    &quot;auctionId&quot;:&quot;30e70924-9ad2-4cfc-87ed-b1dd4fa70bac&quot;,<br>    &quot;sellerId&quot;:&quot;c4ce1bd6-16e2-45b5-87f9-9bf6e1bb856e&quot;<br>  },<br>  &quot;resource.service.name&quot;:&quot;auction-app&quot;,<br>  &quot;span_id&quot;:&quot;25f8bd85c3715582&quot;,<br>  &quot;trace_flags&quot;:&quot;01&quot;,<br>  &quot;trace_id&quot;:&quot;1a030b3ada87e18694768ca95215aa39&quot;<br>}</pre><h3>Metrics</h3><p>Finally we also want to get some metrics for our applications; we need to define an exporter (where to send the metrics, i.e. the OTel Collector) and a reader to read the metrics. I used the basic PeriodicExportingMetricReader.</p><p>First I installed the pre-requisites:</p><pre>npm install --save @opentelemetry/exporter-metrics-otlp-http<br>npm install --save @opentelemetry/exporter-trace-otlp-http</pre><p>Then I added the code to the observability setup file</p><pre>const metricExporter = new OTLPMetricExporter({<br>  url: &#39;http://localhost:4318/v1/metrics&#39;,<br>});<br><br>const metricReader = new PeriodicExportingMetricReader({<br>  exporter: metricExporter,<br>});</pre><p>Finally the metrics reader was added to the NodeSDK config setup</p><pre>const sdk = new NodeSDK({<br>  ...<br>  metricReader,<br>  ...<br>})</pre><p>Metrics can be view in Prometheus, for example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NcFuQaPdQgEMhL6jLNyMQw.png" /><figcaption>Some raw metrics in Prometheus</figcaption></figure><p>It’s also possible to instrument services like Kafka; I was running this in Docker and added the standard Java agent</p><pre>ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.26.0/opentelemetry-javaagent.jar /tmp/opentelemetry-javaagent.jar<br>RUN chmod go+r /tmp/opentelemetry-javaagent.jar</pre><p>…then…</p><pre>ENV KAFKA_OPTS=&quot;-javaagent:/tmp/opentelemetry-javaagent.jar -Dotel.jmx.target.system=kafka-broker&quot;</pre><h3>Discussion</h3><p>Here I’ve given a really brief overview of my experience with Open Telemetry. There are some things to note. Firstly, some parts of it still feel a little incomplete, especially logging. The NPM packages are very messy, there are lots of deprecated packages (and as a result, out of date articles that might have been helpful); the devs appear to have changed course several times while implementing these. You’ll also notice a large number of packages needed to be installed in this exercise.</p><p>The Open Telemetry collector would also need to be managed, which isn’t mandatory, but I found scant documentation for the agent-based approach. In any case it felt that the collector maximised the benefits. This adds complexity.</p><p>Overall this is definitely a promising set of tools well worth keeping your eye on of the next couple of years. It looks set to gain more prominence in the near future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9d542a5924d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS App Mesh — adding a service mesh to ECS]]></title>
            <link>https://medium.com/@garry.passarella/aws-app-mesh-adding-a-service-mesh-to-ecs-eefee4414a82?source=rss-826782ece220------2</link>
            <guid isPermaLink="false">https://medium.com/p/eefee4414a82</guid>
            <category><![CDATA[app-mesh]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[GarryPas]]></dc:creator>
            <pubDate>Tue, 28 Feb 2023 01:28:34 GMT</pubDate>
            <atom:updated>2023-02-28T01:28:34.343Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS App Mesh — adding a service mesh to ECS</h3><h3>Background</h3><p>App Mesh is AWS’s managed service mesh for performing infrastructure functionality such as routing and mutual TLS. Service meshes have become a bit of a buzzword in recent years as organisations increasingly adopt microservice architectures for their systems and struggle with a set of repeated tasks that generalise to all their services. There’s a nice <a href="https://youtu.be/1UDRGlmbiZA">video from AWS Summit 2019</a> giving an intro to the subject.</p><p>Traditionally many architectures have an API gateway, which can be implemented in a variety of ways. In a simplified model this might look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*V0EqZAHNLo1I5ewDvjAnTA.png" /><figcaption>Figure 1: ECS cluster with two services, using an API Gateway for inter-service communication</figcaption></figure><p>When the API Gateway wants to talk to Service-1 or Service-2, this approach works fine. It is also sufficient for a monolith; problem arise when you want Service-1 and Service-2 to talk to each-other. One solution to this is use the API Gateway; when Service-1 wants to call Service-2 it goes the Gateway first. Logically this is a good solution since the API Gateway knows where to find Service-2, however it introduces an extra network hop and modern services often want to respond quickly to incoming requests; it would be better if Service-1 could directly call Service-2. You could define a reference to Service-2 within Service-1 (even caching to reduce calls to the gateway), but this incurs a maintenance overhead; and what happens if something changes? As more services are added (and inter-dependencies increase) these problem becomes more acute. Then there is the problem of communication standards.</p><p>Service meshes offer a potential solution where you have a lot of inter-service communication (so-called “east-west” traffic”, as opposed to “north-south” which refers to ingress/egress). Gateway services are used to manage calls into the cluster, and a service mesh can manage communication between services within the cluster. Note that ECS has it’s own service discovery mechanism which leverages CloudMap, and AppMesh builds on this.</p><p>In the world of Kubernetes a mesh is increasingly becoming a standard. ECS — Amazon’s managed solution for running services in containers — offers <em>App Mesh integration</em>. For every service an Envoy <a href="https://dzone.com/articles/sidecar-design-pattern-in-your-microservices-ecosy-1">sidecar</a> container runs as a separate task.</p><h3>Objectives</h3><p>For this exercise I aimed to setup an ECS cluster with App Mesh enabled. Typically this sort of project would be scripted, following an infrastructure-as-code approach (using a tool such as Terraform), however because this is a PoC, and for simplicity, everything was done using the AWS management console. I wanted to do the following:</p><ul><li>Define two services, with service-1 talking to z. I also wanted to be able to call service-2 independently of service-1.</li><li>Enable AppMesh integration on each service.</li><li>Try out routing to send a percentage of traffic to a different implementation of service-2 (called service-2b).</li><li>Enable X-ray tracing on the services in the cluster.</li><li>Optional — get a virtual gateway setup, and auto-register it with an ALB (application load balancer).</li></ul><h3>Step by step</h3><h4>Setting up ECR and pushing images</h4><p>The first step is to create docker images to be used in the PoC. I created two Node projects; service-1 and service-2 (Figure 2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*iT-R25OY8n6M4MkjfqTRpQ.png" /><figcaption>Figure 2: Folder structure of the project from which I’d create images of service-1 and service-2 (and later service-2b)</figcaption></figure><p>service-1 was pretty simple, essentially:</p><pre>app.get(&#39;/&#39;, async (req, res) =&gt; {<br>  const url = `${process.env.SERVICE_2_ADDR}`;<br>  let body;<br>  let status = 200;<br>  try {<br>    console.log(&#39;Making a request to&#39;, url);<br>    const response = await fetch(url, { method: &#39;GET&#39; });<br>    console.log(&#39;Received response from&#39;, url);<br>    body = await response.text();<br>  } catch (err) {<br>    console.error(&#39;Error&#39;, err.message);<br>    body = {<br>      status: &#39;error&#39;,<br>      message: err.message,<br>      stack: err.stack,<br>    }<br>    status = 500;<br>  }<br>  res.send(body).status(status);<br>});</pre><p>While service-2 would just output whatever message had been defined in an environment variable</p><pre>app.get(&#39;/&#39;, (req, res) =&gt; {<br>  res.send({ status: process.env.MESSAGE || &quot;ok&quot; }).status(200);<br>})</pre><p>I then went into my AWS account and created an ECR repository for each project. In the ECR repo there is a <em>View push commands</em> option which lists all the commands required to push a Docker image to that repository. These save you having to craft these by hand — I took them and put them in an shell script, which I wired up to an NPM script which could be executed with npm run publish:ecr . The AWS CLI must be installed for this to work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lMc7a-453nRe6fLYu2opQA.png" /><figcaption>Figure 3: One of the repositories as viewed in the AWS management console</figcaption></figure><p>Once the Docker image for each project was published it appeared in the ECR (Figure 4):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rS6F4kPZbosnptnvp5LPiA.png" /><figcaption>Figure 4 — shows an example of the ECR in the AWS management console with a Docker image</figcaption></figure><h4>IAM</h4><p>As a pre-requisite, a role was set up for ECS called ecsTaskExecutionRole with the following permissions policies.</p><pre>SecretsManagerReadWrite<br>AmazonECSTaskExecutionRolePolicy<br>EC2InstanceProfileForImageBuilderECRContainerBuilds<br>AWSAppRunnerServicePolicyForECRAccess<br>AWSAppMeshEnvoyAccess</pre><h4>Network</h4><p>I created private subnets in two AZs (availability zones), eu-west-1a and eu-west1b. The CIDR ranges were 10.0.0.0/24 and 10.0.1.0/24, giving each 256 IPs — many more than necessary for this exercise.</p><h4>App Mesh</h4><p>Before setting up an ECS cluster, it is worth getting the App Mesh configured. This is pretty straight forward and was needed when creating task definitions later (Figure 5).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mX2pjBm2mdHIeeTcN0XgGg.png" /><figcaption>Figure 5: Setting up a new mesh</figcaption></figure><p>Then virtual nodes for each ECS service (Figure 6). Most of the settings can be left as defaults, but I selected the DNS service discovery type, specifying a DNS hostname for each service. In the ‘Listener configuration’ section I ensure ports were set for ingress traffic (Figure 7).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oJhX4LnGzFLo9lTXsMmSwg.png" /><figcaption>Figure 6: Virtual node configuration</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*w9HESD-GumH2vunFY6ePSg.png" /><figcaption>Figure 7: Listener configuration</figcaption></figure><p>Then I created routers for each service, each with a route to the single endpoint I’d defined (Figure 8).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uzEHb2D8RSaHexr2p5aIog.png" /><figcaption>Figure 8: route configuration for service-1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*npBBx8aPVsZHp7TnfnU_aw.png" /><figcaption>Figure 9: Target configuration for service-2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lBLtdceIFXh87KqeHv4r3A.png" /><figcaption>Figure 10: Match configuration for service-2</figcaption></figure><p>Then I created services and mapped them to the routers (Figure 11)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DveLz4_Ck8AUi3t-4fF4xw.png" /><figcaption>Figure 11: Service configuration, with the service provider set as a virtual router</figcaption></figure><p>Finally I created a gateway, and added a gateway route configuration for service-1 and service-2 (Figure 12).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*n9x_Pw3X8jDdvS5jwy21_g.png" /><figcaption>Figure 12: Gateway route configuration.</figcaption></figure><p>Any node that will call another service will need that service defined as a backend; I’d set service-1 up to call service-2, so this would need service-2.local as a backend. One final note, I was intending to setup my gateway using the Fargate launch type, and in this case <a href="https://docs.aws.amazon.com/app-mesh/latest/userguide/troubleshooting-setup.html">the port number needs to be greater than 1024</a>.</p><h4>ECS cluster</h4><p>The next step was to create an ECS cluster. I used an existing VPC for this. Once this is done I created new task definitions for each of the new services using the <em>Fargate</em> launch type, with the ecsTaskExecutionRole task role and <em>Linux</em> as the operating system.</p><p>I copied the URL from the ECR for each app task I wanted to run (Figure 13). I also defined an environment variable in service-1 to let it know where to go for service-2 (service-2.local).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o5K154VwgEzeBrH3mP-BGA.png" /><figcaption>Figure 13: Adding a container to the task definition</figcaption></figure><p>It’s important to note at this point that when doing this through the AWS management console, there is a usability option to use the <em>New ECS Experience</em>. However certain options are not yet (as of Feb 2023) available when setting up task definitions, and one of these was App Mesh integration. This was a real ‘gotcha’, so if you find any options are missing, be sure to toggle the <em>New ECS Experience </em>option to ‘off’.</p><p>With the Enable App Mesh integration checked I selected the relevant fields to associate virtual nodes with application containers (Figure 14).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*jfOPkqykro6EWJFK78AS7w.png" /><figcaption>Figure 14: Enabling App Mesh integration</figcaption></figure><p>Once the task definitions were created, I created create services for each, using the two private subnets created earlier and a new security group. The security group configuration is a bit tricky, ideally you only want to allow outbound requests to the ECR (to pull Docker images) over the public internet. A public IP can be assigned to a service, but this exposes it to the public Internet, which you might not be desirable. If not you need to create a NAT Gateway in a public subnet, add route table rules to forward Internet-bound traffic from the private subnet to the NAT Gateway, and ensure you have an Internet Gateway in your VPC with routing rules to forward egress traffic from your public subnet to the Internet Gateway. Security groups must also be configured to allow egress traffic and to ensure that no ACLs are blocking. This is a bit tricky, so for a PoC assigning a public IP might be preferable.</p><p>When creating the ECS service I also enabled service discovery (Figure 15).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QixnB-uJAPjpknnFqtd00Q.png" /><figcaption>Figure 15: Enabling service discovery for an ECS service</figcaption></figure><p>Within a few seconds of creating the services the tasks will begin to get started (Figure 16).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*oCxHPdJEk3EiKWm9UlgjaQ.png" /><figcaption>Figure 16: service-1 and the envoy proxy in a running state</figcaption></figure><p>Then I repeated most of this process for service-2, but remembering to update the task definition of service-1 to pass an environment variable with the DNS for service-2 (Figure 17).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bXfYAoOCHseH_NCeLSzcvg.png" /><figcaption>Figure 17: Telling service-1 the host name for service-2</figcaption></figure><h4>Implementing a virtual gateway</h4><p>There are a number of ways to create an API Gateway. The approach I used was to create an ALB (application load balancer), and then an ECS service that auto-registers with the target group associated with the ALB. I created a new target group with a target type of <em>IP addresses</em>. ECS services can auto-register with a desired target group, so it isn’t necessary to add any IP addresses to the target group.</p><p>For creating a gateway the scant documentation that explains how this is done; I created a task definition with no containers specified, then enabled <em>App Mesh integration</em> with a target of <em>Virtual gateway</em> (Figure 18).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_TRVRc0cdeNMenvHBl-27Q.png" /><figcaption>Figure 18: Setting up an API Gateway ECS service</figcaption></figure><p>When you apply this you notice that an envoy container has been added to the task definition.</p><p>Then I setup a service for this gateway task definition and registered it with an ALB to expose it to the public internet (Figure 19). Using the management console this is pretty straight forward — ECS will register the service with the target group of the ALB once the service is up and running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sZqTWxcJAfe5Mpe0e20-aQ.png" /><figcaption>Figure 19: Registering the api-gateway service with an ALB</figcaption></figure><p>Logging can also be enabled on the envoy virtual gateway by checking the <em>Auto-configure CloudWatch logs</em> checkbox, which will get logs into CloudWatch from the envoy proxy (Figure 20). This might be useful for debugging as there is otherwise little to indicate why an Envoy task failed to start.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jytpK1n0h1ucPOZP0q8btQ.png" /><figcaption>Figure 20: Enabling logging on the ECS gateway service</figcaption></figure><h4>Trying it out</h4><p>With everything setup it was time to try a few features out. The first thing I looked at was routing; AppMesh allows you to setup routing rules with <em>virtual routers</em>, which includes the ability to route a % of traffic between services. Figure 21 shows how I routed 50% of traffic to service-2b. This opens up options around traffic management to individual endpoints, A/B testing etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ClpuUr-bvyw0J16MCuL29g.png" /><figcaption>Figure 21 routing traffic using a virtual router</figcaption></figure><p>The ALB gives a publicly available DNS, and this will route to the virtual gateway; from there the request is passed to the service-2 virtual service; the routing rules kick in and route the request to one of the two virtual nodes. I set a different response message for each app and verified the load is being distributed between them (Figure 22).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xEGW8ShvwOqKiJk1q8f0CQ.png" /><figcaption>Figure 22: The response from the second ECS service, with traffic distributed 50/50 between two apps</figcaption></figure><p>Next I looked at setting up <em>X-ray</em> tracing which was pretty straight forward; first I modified my <em>Express</em> app, using the code sample AWS provided when I’d initially set <em>X-ray</em> tracing up.</p><pre>const AWSXRay = require(&#39;aws-xray-sdk&#39;);<br><br>// ...express routes...<br><br>app.use(AWSXRay.express.openSegment(&#39;gateway&#39;));</pre><p>Next I create a Dockerfile; AWS provide an error-riddled sample through the management console —my advice would be to ignore it. Refer to <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ecs.html">this documentation</a> instead, or just use the standard Amazon X-ray Docker image; <a href="https://hub.docker.com/r/amazon/aws-xray-daemon">amazon/aws-xray-daemon</a> — I used the standard image. The Envoy container definition needs to be modified specifying port 2000 (the X-ray port) via XRAY_DAEMON_PORT and ENABLE_ENVOY_XRAY_TRACING (Figure 23).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*JLe-H9M278c-J2QckfwpIQ.png" /><figcaption>Figure 23: Environment variables for the Envoy container</figcaption></figure><p>The agent container definition needs port 2000 mapped to the host port of 0 (UDP), which needs to be done in the JSON for the task definition. The task execution role also needs the AWSXrayWriteOnlyAccess policy at a minimum.</p><p>The nice schematic shown in Figure 24 shows how the services are interacting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*6N2kVf6WICZqb2iUROABpQ.png" /><figcaption>Figure 24: X-ray illustrating calls to services</figcaption></figure><p>Many organisations will have their own distributed tracing solutions; for these it seems to be a case of bypassing Envoy by configuring egress filters.</p><p>Although I didn’t explore it in this exercise (largely because it’s a topic worthy of a blogpost of it’s own), the end-to-end encryption options that AppMesh offers look very interesting; in many system architectures TLS-termination will have occurred once a request is inside the DMZ; AppMesh offers options to encrypt traffic between services. One option is <a href="https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret">Envoy’s Secret Discovery Service (SDS)</a> which claims to simplify the often-arduous process of certificate management.</p><h3>Discussion</h3><p>My overall impression of AppMesh was that it would entail a lot of overhead in terms of complexity for benefits that many organisations won’t need. Advanced routing may be useful in some organisations, but that itself may add complexity to service-to-service communication. The value of this feature depends on the situation at hand.</p><p>Virtual gateways are pretty simple to setup, although examples of how to do this were few and far between.</p><p>X-ray integration was extremely simple in ECS with AppMesh enabled, and gave nice visualisations as to what is going on.</p><p>Finally the TLS options offered look very interesting, and may give a lot more weight to a case for AppMesh in organisations who see this as desirable.</p><p>The end result of my looked something like Figure 25.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*Vwme_EQ2_tA9Ln2ty5qW2g.png" /><figcaption>Figure 25: The final result from the PoC</figcaption></figure><h3>Conclusion</h3><p>In this exercise I setup an ECS cluster using <em>Fargate</em> with <em>AppMesh integration</em> enabled. I explored how AppMesh adds functionality (routing, TLS etc) to features already available in ECS. Overall I felt AppMesh had things to offer to teams with specific requirements (routing, TLS for inter-service communication etc) who were prepared to take on the initial setup cost, although in many cases the overhead would not be justified.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eefee4414a82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a Kafka sandbox in Docker]]></title>
            <link>https://medium.com/@garry.passarella/creating-a-kafka-sandbox-in-docker-c8d122afab36?source=rss-826782ece220------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8d122afab36</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[zookeeper]]></category>
            <dc:creator><![CDATA[GarryPas]]></dc:creator>
            <pubDate>Sat, 05 Mar 2022 00:29:11 GMT</pubDate>
            <atom:updated>2023-01-01T22:41:41.638Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*mFfBqj-eCgsEFT39kFFlUg.png" /></figure><h3>Introduction</h3><p>The purpose of this exercise (aside from learning about Kafka configuration) was to setup a sandbox locally that could be used to experiment with Kafka/Zookeeper setups, that could be easily ripped down and recreated, but more importantly to understand how to setup a Kafka cluster from scratch, and therefore learning more about how it all works under the hood (to answer “what happens if I change this thing here” type questions). Spinning up VMs seemed like the wrong approach. So I looked at containerisation.</p><p>To do this I crafted Docker images of Kafka and Zookeeper nodes from scratch; each container acting as if it were a node in a cluster/ensemble; it is generally advised to run a single instance of Kafka on dedicated hosts/VMs¹ so this is a reasonable representation of how a setup would look in a production scenario .</p><p>Note, that if you’re looking to run Kafka in Docker for testing applications etc rather than learning about administration you’re probably not looking to reinvent wheels and go through the pain that follows; <a href="https://hub.docker.com/r/bitnami/kafka/">Bitnami Kafka</a> and <a href="https://hub.docker.com/r/wurstmeister/kafka/">Wurstmeister kafka-docker</a> are two existing solutions I’m aware of that might meet your needs.</p><h3>Dockerisation</h3><p>Dockerisation was a means rather than an end-goal of this project; during this exercise I was able to come up with an architecture that was broadly similar to an architecture that might exist in a ‘real’ setting (Figure 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*PH95B9IDy7cL9OOQjJf-BA.png" /><figcaption>Figure 1: Schematic of the Docker topology used for this exercise</figcaption></figure><p>Figure 1 clearly needs some more explanation.</p><p>Zookeeper is a service used by Kafka for configuration, service synchronisation etc. When multiple Zookeeper instances exist in a cluster as illustrated in Figure 1 this is known as an <em>ensemble².</em></p><p>In this setup Zookeeper uses the standard client port 2181 internally, but uses 228<em>n</em> externally where <em>n</em> is incremented from 1. This was necessary when publishing ports from Docker locally to prevent port conflicts; the standard port could be used if Zookeeper ran on dedicated hosts.</p><p>A 3-node Kafka cluster is also illustrated; ports 910m are used (again to prevent conflicts when publishing ports from Docker), where <em>m</em> is an incremental number starting a 1. Again this is not necessary when running a single instance of Kafka on each host.</p><p>Finally two networks (one for Zookeeper, one for Kafka) are used with subnets given a CIDR (Classless Inter-Domain Routing) range of 24, providing 256 IP addresses each; plenty for most local testing purposes, even allowing for allocating one to a gateway etc.</p><p>Some configuration of both Zookeeper and Kafka is needed for this to work, which we’ll go into next.</p><p>Note that no security is implemented here. Kafka security is a whole topic in itself, so I’ve considered this out of scope.</p><h3>Zookeeper</h3><p>The first logical step is to focus on getting Zookeeper up and running; this is because Zookeeper is independent of Kafka (the reverse is not true, although <a href="https://developer.confluent.io/learn/kraft/">KRaft promises a Zookeeperless future</a>³). Once this is done we can move on to Kafka.</p><p>I built my Docker image on Alpine, which is a lightweight Linux distro. The Dockerfile is given below.</p><pre>FROM alpine:3.14</pre><pre>RUN apk update</pre><pre>WORKDIR tmp</pre><pre>RUN apk update</pre><pre>RUN apk add bash</pre><pre># Install Java</pre><pre>RUN apk add openjdk11-jre</pre><pre>ENV JAVA_HOME &quot;/usr/bin&quot;</pre><pre># Required ps for Zookeeper (otherwise zkServer shell command gives spurious errors because Alpine uses BusyBox Commands)</pre><pre>RUN apk --no-cache add procps<br># Install Zookeeper</pre><pre>RUN wget https://dlcdn.apache.org/zookeeper/stable/apache-zookeeper-3.7.1-bin.tar.gz</pre><pre>RUN tar -zxf apache-zookeeper-3.7.1-bin.tar.gz</pre><pre>RUN cp -R apache-zookeeper-3.7.1-bin /usr/local/zookeeper/</pre><pre>RUN mkdir /app</pre><pre>COPY ./start-zookeeper.sh /app/</pre><pre>RUN chmod +x /app/start-zookeeper.sh</pre><pre>RUN mkdir /var/lib/zookeeper</pre><pre>COPY ./zoo.cfg /usr/local/zookeeper/conf/zoo.cfg</pre><pre>RUN rm -rf apache-zookeeper-3.6.3-bin.tar.gz &amp;&amp; rm -rf apache-zookeeper-3.6.3-bin</pre><pre>WORKDIR /usr/local/zookeeper/bin</pre><pre>ENTRYPOINT [ &quot;bash&quot;, &quot;/app/start-zookeeper.sh&quot; ]</pre><p>Here I defined my own shell script as the ENTRYPOINT. The advantage of doing this is it gave me the ability to specify a MyID² (the Zookeeper ID, an integer which must be unique for each node in the ensemble) dynamically at runtime rather than baking it into the Docker image, i.e.</p><pre>#!/bin/bash</pre><pre>set -e</pre><pre>echo $ZK_MYID &gt; /var/lib/zookeeper/myid</pre><pre>/usr/local/zookeeper/bin/zkServer.sh start-foreground</pre><p>The script then goes on to start Zookeeper.</p><p>I also copy a zoo.cfg file into the container, which uses the standard template aside from the last 3 lines where I’ve hardcoded the servers (peer port, 2888, and leader election port, 3888, are standard).</p><pre>server.1=zookeeper_1:2888:3888</pre><pre>server.2=zookeeper_2:2888:3888</pre><pre>server.3=zookeeper_3:2888:3888</pre><h3>Kafka</h3><p>Again this Docker image was build on an Alpine image.</p><pre>FROM alpine:3.14</pre><pre>RUN apk update</pre><pre>RUN apk add bash</pre><pre>WORKDIR tmp</pre><pre># Install Java</pre><pre>RUN apk add openjdk11-jre</pre><pre>ENV JAVA_HOME &quot;/usr&quot;</pre><pre># Install Kafka</pre><pre>RUN wget https://dlcdn.apache.org/kafka/3.3.1/kafka_2.13-3.3.1.tgz</pre><pre>RUN tar -zxf kafka_2.13-3.3.1.tgz</pre><pre>RUN cp -R kafka_2.13-3.3.1 /usr/local/kafka/</pre><pre># Configure Kafka</pre><pre>COPY ./server.properties /usr/local/kafka/config/server.properties</pre><pre>RUN mkdir /tmp/kafka-logs</pre><pre>RUN mkdir /app</pre><pre>COPY ./start-kafka.sh /app/</pre><pre>RUN chmod +x /app/start-kafka.sh</pre><pre>WORKDIR /usr/local/kafka/bin</pre><pre>ENTRYPOINT [ &quot;/app/start-kafka.sh&quot; ]</pre><p>A custom shell script is given here, which allows me to parameterise configurations in my server.properties file (more on that in a bit).</p><pre>#!/bin/bash</pre><pre>set -e</pre><pre>sed -i &quot;s/\$KAFKA_BROKER_ID/$KAFKA_BROKER_ID/g&quot; /usr/local/kafka/config/server.properties</pre><pre>sed -i &quot;s/\$KAFKA_EXTERNAL_PORT/$KAFKA_EXTERNAL_PORT/g&quot; /usr/local/kafka/config/server.properties</pre><pre>sed -i &quot;s/\$KAFKA_EXTERNAL_HOST/$KAFKA_EXTERNAL_HOST/g&quot; /usr/local/kafka/config/server.properties</pre><pre>sed -i &quot;s/\$KAFKA_ZOOKEEPER_CONNECT/$KAFKA_ZOOKEEPER_CONNECT/g&quot; /usr/local/kafka/config/server.properties</pre><pre>/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties<br></pre><h3>Kafka configuration</h3><p>In Kafka is is possible to define multiple host/port sets for Kafka to listen on; for this exercise this feature is extremely useful as the Kafka brokers can use internal docker network hostnames to communicate with one-another (on the standard Kafka port 9092, which is set by inter.broker.listener.name), but can advertise a different hostname for use externally, exposed via advertised.listeners. Below is a snippet showing the configuration in server.properties.</p><pre>listeners=INSIDE://:9092,OUTSIDE://:$KAFKA_EXTERNAL_PORT</pre><pre>listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT</pre><pre>advertised.listeners=INSIDE://kafka_$KAFKA_BROKER_ID:9092,OUTSIDE://$KAFKA_EXTERNAL_HOST:$KAFKA_EXTERNAL_PORT</pre><pre>inter.broker.listener.name=INSIDE</pre><p>The sed commands in the previous section makes the $xxx placeholders dynamically settable on startup.</p><p>A consumer just needs to boostrap to $KAFKA_EXTERNAL_HOST:$KAFKA_EXTERNAL_PORT</p><p>The Kafka broker also needs a unique ID; this is set dynamically too</p><pre>broker.id=$KAFKA_BROKER_ID</pre><h3>Running the environment</h3><p>I setup a docker-compose.yml file, which lacks a bit of flexibility but is simple, however it is possible to spin-up the environment using docker CLI commands.</p><p>To setup a Docker network for Zookeeper</p><pre>docker network create \</pre><pre>  -d bridge \</pre><pre>  --gateway 192.168.50.1 \</pre><pre>  --subnet 192.168.50.0/24 \</pre><pre>  network_zookeeper</pre><p>…and to build the image and run Zookeeper instances in this network (in this example we’d run this command 3 times, replacing $ZK_MYID with a 1, 2 or 3).</p><pre>docker build -t zookeeper_example .</pre><pre>docker run \</pre><pre>  --name &quot;zookeeper_$ZK_MYID&quot; \</pre><pre>  -h &quot;zookeeper_$ZK_MYID&quot; \</pre><pre>  -h &quot;zookeeper_$ZK_MYID&quot; \</pre><pre>  -p &quot;228$ZK_MYID:2181&quot; \</pre><pre>  -e ZK_MYID=$ZK_MYID \</pre><pre>  --network network_zookeeper \</pre><pre>  -l zookeeper \</pre><pre>  zookeeper_example</pre><p>Next to setup the Kafka network</p><pre>docker network create \</pre><pre>  -d bridge \</pre><pre>  --gateway 192.168.51.1 \</pre><pre>  --subnet 192.168.51.0/24 \</pre><pre>  network_kafka</pre><p>…and to build the image and run containers (again replacing $KAFKA_BROKER_ID with a 1, 2 or 3).</p><pre>docker run \</pre><pre>  --name kafka_$KAFKA_BROKER_ID \</pre><pre>  -h &quot;kafka_$KAFKA_BROKER_ID&quot; \</pre><pre>  -p &quot;910$KAFKA_BROKER_ID:910$KAFKA_BROKER_ID&quot; \</pre><pre>  -e KAFKA_BROKER_ID=$KAFKA_BROKER_ID \</pre><pre>  -e &quot;KAFKA_EXTERNAL_PORT=910$KAFKA_BROKER_ID&quot; \</pre><pre>  -e KAFKA_EXTERNAL_HOST=localhost \</pre><pre>  -e KAFKA_ZOOKEEPER_CONNECT=172.17.0.1:2281,172.17.0.1:2282,172.17.0.1:2283 \</pre><pre>  -l kafka \</pre><pre>  --network network_kafka \</pre><pre>  kafka_example</pre><h3>Testing</h3><p>To test we need a topic to produce events to, and to consumer them from.</p><p>By default topics are created automatically when a producer tries to send events to a topic that doesn’t exist, however some administrators prefer to turn this off (via the auto.create.topics.enable Kafka property). If this is the case you can issue a command to one of the Docker containers to create the topic manually. Note that kafka-topics.sh also allows things like replication factor and number of partitions to be configured.</p><pre>docker exec kafka_1 bash /usr/local/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9101 --create --topic test-topic</pre><p>With a topic created we can smoke test everything; first setup a consumer to listen for new events:</p><pre>docker exec kafka_1 bash /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic</pre><p>Then open a second terminal and send some events via the CLI producer:</p><pre>docker exec -it kafka_1 bash /usr/local/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic</pre><p>Note the first terminal writing the events as they are received.</p><p>The following repo has some sample consumers and producers in Java and Node JS to do the same test outside of the Docker network, and for reference.</p><p><a href="https://github.com/garrypas/kafka-zookeeper-docker">https://github.com/garrypas/kafka-zookeeper-docker</a></p><h3>Summary</h3><p>Here I described an exercise to Dockerise Kafka for the purposes of creating a local sandbox environment for experimentation and learning. This sort of sandbox environment is useful to ensuring your understanding of concepts is correct by experiment, or for quickly trying different configurations locally in throw-away environment.</p><h3>References</h3><p>1: Shapira, G., Palino, T., Sivaram, R. and Petty, K. (2022), ‘Kafka - the definitive guide’, p34, Sebastopol, CA: O’Reilly Books.</p><p>2: Apache Zookeeper, ‘ZooKeeper Administrator’s Guide’. Available at: ‘<a href="https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_zkMulitServerSetup">https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_zkMulitServerSetup</a>.</p><p>3: Confluent developer, ‘KRaft: Apache Kafka Without ZooKeeper’. Available at <a href="https://developer.confluent.io/learn/kraft/">https://developer.confluent.io/learn/kraft/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8d122afab36" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Typescript-based gaming back-end, with Loopback, AKS and Terraform.]]></title>
            <link>https://medium.com/@garry.passarella/building-a-typescript-based-gaming-back-end-with-loopback-aks-and-terraform-b533c9485e80?source=rss-826782ece220------2</link>
            <guid isPermaLink="false">https://medium.com/p/b533c9485e80</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[GarryPas]]></dc:creator>
            <pubDate>Tue, 01 Sep 2020 21:14:05 GMT</pubDate>
            <atom:updated>2020-09-03T07:53:51.967Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Typescript-based card game back-end, with Loopback, AKS and Terraform.</h3><h3>Introduction</h3><p>This started as a summer side-project — build an online version of the popular Italian card game Scopa which I called the <em>Scoparella project</em>. Most importantly it was a chance to get hands-on with some technologies I’d been eager to explore in more depth, in particular Azure Kubernetes Service (AKS) and Loopback, a framework for building microservices with Typescript.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i2tkMod3-YYcredTaCBstw.png" /><figcaption>Figure 1: Components used for this project — AKS, SQL Server and Key Vault hosted in Azure</figcaption></figure><p>Scopa is a traditional Italian card game played with a 40-card deck of Italian cards (also playable with the more familiar 52-deck cards by removing the 8, 9 and 10s). The rules are fairly simple, and can be easily found on Google so I won’t digress here. To facilitate this I wrote an <a href="https://github.com/garrypas/scoparella.engine">engine for the game in Typescript</a>. The API for the engine library is fairly rudimentary:</p><pre>const game = new ScoparellaGame({numberOfPlayers: 2});<br>game.addPlayer(new ScoparellaPlayer(&quot;player1&quot;));<br>game.addPlayer(new ScoparellaPlayer(&quot;player2<em>&quot;</em>));</pre><p>This sets up a game with 2 players…</p><pre>game.tryPlayCards(cardToPlay, cardsToTake, playersHand);</pre><p>…attempts to play a card and take 0…many cards, while playersHand indicates the player playing the hand. Validation is done under the hood to ensure any move is legal before committing it.</p><h3>Loopback</h3><p>Loopback is a rich API framework created by StrongLoop and has no less an entity than IBM backing it. With lots of features you’d expect from any enterprise-grade framework such as the ability to easily protect endpoints that require authentication, all the tools needed to build custom authorisation decorators easily, a CLI tool for scaffolding controllers, services and repositories etc. Loopback allowed me to quickly get productive and setup an API that would allow players to play online!</p><p>The first step is to install the Loopback CLI tool:</p><p>npm install -g @loopback/cli</p><p>Then simply run lb4, follow the instructions, and Loopback will scaffold all of the bits needed for a microservice. Once this was done I added some services, which was achieved using lb4 service from the root of the application folder. The structure of the src folder where the code lives is given in Figure 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*Z46mdw06SpDw99yR4DhpDw.png" /><figcaption>Figure 2: The src folder of a Loopback application with services added</figcaption></figure><p>The patterns used here are fairly standard for a CRUD API — controllers, talking to services, which talk to repositories which talk to data stores… the first question one might ask is why can’t I just create this by hand and avoid the need to learn the Loopback framework? As mentioned before Loopback gives you a lot out of the box — it has an ORM framework with wrappers around common data commands and queries so you can interact with a database without having to write a line of SQL; so to update the player in the game with the id of id with the player indicated by player we could do:</p><pre>await <em>super</em>.updateAll({player1:player}, {<br>  and: [{id}, {playerId: null}],<br>});</pre><p>It can also create entities POJOs (or should than be POTSOs?) using the CLI tool… it adds ES-linting, scaffolds tests, sets up NPM scripts and wires up Swagger, has it’s own IoC framework… and a lot more.</p><h3>OAuth2</h3><p>In order to play an online game you first need to establish some sort of identity. Traditionally this might be achieved by having a database of users that new users can register with, setting a password and verifying their email address using a one-time password (OTP). This is arguably the simplest to develop and test, however more and more service providers from all walks of life are delegating responsibility to well-establish third parties such as Google and Facebook. You’ve probably seen this at some point — you join a site whose services you want to access and they allow you to sign up with one of a select list of providers. There are a number of benefits to this approach — the user journey is simplified, you no longer need to concern yourself with storing potentially sensitive data in a database, users don’t have to manage yet-another-set-of-credentials, and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*rKmdPJyAnOobx9mK_JDCbw.png" /><figcaption>Figure 3: An example of the auth prompt shown to a user attempting to authenticate with OAuth2</figcaption></figure><p>For this project I set up OAuth2 with both Google and Facebook, which involves a bit of work with those providers such as explicitly defining redirect URIs.</p><p>OAuth2 providers require a callback URL once the client has given their consent, and for a developer this would then entail writing an webhook to handle the callback, authenticating with the provider using a token that was issued to access protected resources (such as the user’s name and email address)... which doesn’t sound like a lot of fun. Luckily For NodeJS developers a library for handling a vast range of common authentication scenarios already exists — <a href="http://www.passportjs.org/">passportjs</a>.</p><p>Passport strategies (as they are know) do not work out of the box with Loopback (it targets Express), but can be made to work using an adapter pattern that involves wrapping native passport strategies in a Loopback middleware that calls down to native passport authentication; I took inspiration from <a href="https://github.com/strongloop/loopback-next/tree/master/examples/passport-login">this example</a>. This middleware is then invoked on callback and the correct strategy used to authenticate. A signed JWT is issued which can be used to take part in games by providing it in the Authentication header as a bearer token. Because it is a JWT no further interaction with the provider is necessary while the token is valid, and it is signed so the claims cannot be tampered with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*q7XtzVRQAL2ajMbM0h0Nyg.png" /><figcaption>Figure 4: Loopback/Passport adapter</figcaption></figure><h3>Making a move</h3><p>The data store is fairly basic — a SQL Server table stores all games in progress and a serialised copy of the state of a game. When a player makes a move, the game is pulled out of the database, deserialised (using the Scopa engine I’d built), the move is played, the game is serialised and saved back into the database. The Engine acts as a state machine and will only allow valid moves (for example a player can only move when it is their turn).</p><h3>Testing</h3><p>The application can be spun up locally and a game can be played given valid JWT tokens. For automated testing I stubbed out the auth providers and used docker compose to create a) the application and b) the database, within a local Docker network where acceptance tests can be run.</p><h3>Hosting in Azure</h3><p>As a side-project one of my goals is to get hands-on with technologies that are interesting or likely to be valuable in my day-to-day work. As such, I decided to host the application in Azure on AKS (other options are available in Azure, notably <a href="https://azure.microsoft.com/en-gb/services/app-service/containers/">Web App for Containers</a>). I would need to setup a Kubernetes cluster in Azure, which costs a few pounds a week to keep running. Luckily my current employer gives me some free Azure credits every month as part of my MSDN subscription, so I was able to use these.</p><p>I decided to use Terraform, a tool for infrastructure-as-code that is gaining popularity in the world of devops. These days creating infrastructure using point-and-click user interfaces is frowned upon in most scenarios for good reasons that are beyond the scope of this article. Scripts can also accomplish these tasks, but lack the state-tracking of Terraform, which is a powerful tool for ensuring changes to code can easily be reflected by the provisioned resources.</p><p>To give you an example of it’s succinctness, this is all it takes to spin up a Kubernetes cluster comprising 2 nodes:</p><pre>resource &quot;azurerm_kubernetes_cluster&quot; &quot;scoparella-kube&quot; {<br>  name                = &quot;${var.environment}-scoparella-aks1&quot;<br>  location            = var.location<br>  resource_group_name = var.resource_group_name<br>  dns_prefix          = &quot;${var.environment}scoparellaaks1&quot;<br>  node_resource_group = &quot;${var.environment}-scoparella-aks-rg&quot;<br>  <br>  default_node_pool {<br>    name       = &quot;agentpool&quot;<br>    node_count = 2<br>    vm_size    = &quot;Standard_B2s&quot;<br>  }</pre><pre>  identity {<br>    type = &quot;SystemAssigned&quot;<br>  }</pre><pre>  lifecycle {<br>    prevent_destroy = false<br>  }</pre><pre>  tags = {<br>    environment = var.environment<br>  }<br>}</pre><p>Terraform resources can be organised into modules, and I also had modules for resources such as KeyVault which was used to store the database password among other things.</p><p>In addition to terraform I also defined the application deployment using the standard Kubernetes YAML format; I did experiment with defining these natively in HashiCorp Configuration Language (HCL — the configuration language of Terraform), but opted to use YAML instead as it was more familiar to me, and better supported and documented. Database objects (DBOs) were applied using sqlcmd .</p><p>I faced several challenges during the process of scripting up the infrastructure. The biggest obstacle was finding a way to access secrets held in KeyVault. Kubernetes has its own approaches to secrets management, however I chose to use a project called <a href="https://github.com/Azure/aad-pod-identity/">AAD Pod Identity</a> which allows Kubernetes applications to access resources in Azure (such as KeyVault) by authenticating with Azure Active Directory and assuming a role with the privileges required. I created roles and identities using terraform, but applied the deployment of the identity pods using Kubernetes YAML and some bash to extract the relevant UUIDs dynamically. On application startup the Scopa API application it will pull the secrets it needs from KeyVault and cache them locally.</p><p>Finally I applied a cloud-network load balancer, binding a public IP address (on port 80) to the internal IP address of the service (on port 3000). With this in place I could verify liveness over the public internet using curl to a /ping endpoint, confirming the cluster was setup correctly.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: &quot;scoparella-api-service&quot;<br>spec:<br>  type: LoadBalancer<br>  selector:<br>    app: scoparella-app<br>  ports:<br>    - protocol: TCP<br>      port: 80<br>      targetPort: 3000</pre><h3>API Management Services</h3><p>Clearly accessing a backend via it’s public IP address over an insecure connection wasn’t a desirable end-state. I used API Management Services to act as a reverse proxy to the load balancer service (over HTTPS). To do this I needed to add the API Management service to the AKS VNet, allocate an address space for it, and select the subnet from the VNet configuration of the API Management Service (Figure 5). There are challenges doing this at present with Terraform so I was forced to use the Azure UI. Scripting this up is on my todo list, even if just in a bash script.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*d2TLgiEeCQDqxkAHSZOz-A.png" /><figcaption>Figure 5: Pointy-clicky for setting API Management VNet Subnet</figcaption></figure><h3>Azure Devops</h3><p>Finally for CI/CD I chose to use Azure Devops. This is a tool we are using extensively in my current role, and it has the advantage of being fully managed and it integrates well with the rest of the Azure ecosystem. Pipelines are defined using YAML, and a base image can be selected to fit particular requirements. I chose Ubuntu, and setup an Azure pipeline to checkout the code, run tests inside a container, terraform, and apply deployments using kubectl.</p><p>These are defined as “tasks” in an Azure Devops pipeline. An example of one is given below:</p><pre>- task: Bash@3<br>  condition: or( eq(&#39;${{ parameters.task }}&#39;, &#39;*&#39;), eq(&#39;${{ parameters.task }}&#39;, &#39;terraform&#39;) )<br>  inputs:<br>    targetType: &quot;inline&quot;<br>    script: |<br>      wget <a href="https://releases.hashicorp.com/terraform/0.12.29/terraform_0.12.29_linux_amd64.zip">https://releases.hashicorp.com/terraform/0.12.29/terraform_0.12.29_linux_amd64.zip</a><br>      unzip terraform_0.12.29_linux_amd64.zip<br>      ./terraform init<br>      ./terraform plan<br>      ./terraform apply -auto-approve<br>      workingDirectory: &quot;./resources/${{ parameters.stage }}&quot;<br>    displayName: &quot;Terraform data/secrets&quot;</pre><p>While this is just some bash, Devops also has templates for common task, for example building and pushing a Docker image:</p><pre>- task: Docker@2<br>  displayName: Build and Push<br>  inputs:<br>    containerRegistry: &quot;dockerscoparella&quot;<br>    repository: &quot;garrypassarella/scoparella&quot;<br>    command: &quot;buildAndPush&quot;<br>    Dockerfile: &quot;Dockerfile&quot;<br>    tags: |<br>      scoparella_api_${{ variables.env }}<br>      condition: or( eq(&#39;${{ parameters.task }}&#39;, &#39;*&#39;), eq(&#39;${{ parameters.task }}&#39;, &#39;docker&#39;) )</pre><p>One caveat is that Azure Devops requires an identity in Azure Active Directory with the privileges necessary to provision resources in Azure; my organisation do not grant me the ability to create these, so I was forced to do an Azure login whenever I wanted to run a build — a bit clunky, but something I wouldn’t face if I owned the subscription. For similar reasons I was unable to use Azure Container Registry (ACR) but I got around this by using Docker Hub and setting up a service connection.</p><h3>Next steps?</h3><p>Unfortunately my Azure credits ran out after 2 weeks (implying I might need to swat up on cost optimisation next, or that Azure is just very expensive). I still have some things I’d like to do to this project, in particular script out the API Management parts. In the meantime I am looking at building a web-based front-end for the game using React. That might be the subject of another article in a few months…</p><p>The full code for the API is here: <a href="https://github.com/garrypas/scoparella.api">https://github.com/garrypas/scoparella.api</a></p><p>For the Scoparella Engine here: <a href="https://github.com/garrypas/scoparella.engine">https://github.com/garrypas/scoparella.engine</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b533c9485e80" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analysing active galaxies with the Sloan Digital Sky Survey]]></title>
            <link>https://medium.com/@garry.passarella/analysing-active-galaxies-with-the-sloan-digital-sky-survey-f0c2151eed11?source=rss-826782ece220------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0c2151eed11</guid>
            <dc:creator><![CDATA[GarryPas]]></dc:creator>
            <pubDate>Tue, 16 Jun 2020 16:45:00 GMT</pubDate>
            <atom:updated>2020-06-16T21:33:40.936Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*lXf5vTJr5Oc3186rco4YNw.png" /></figure><h3>Introduction</h3><p>In my day job I’m an enterprise developer, in the evenings I’m studying for a degree in astronomy and physics with the OU, for which I’m in the final year. I recently worked with a team of 7 others on a project to produce a characteristic spectrum for active galaxies (also called quasars, or QSOs for ‘quasi-stellar objects’); these are galaxies at high redshifts (i.e. they are very old) with compact, energetic point like nuclei known as active galactic nuclei (AGN).</p><p>Active galaxies are interesting because they allow astronomers to essentially look back in time, sometimes to just a billion or so years after the big bang. <a href="http://doi.org/10.1093/mnras/stw3168">There is strong evidence that the quasar luminosity function is variable</a>-there are fewer quasars as redshift becomes smaller, implying that quasars are a stage of galactic evolution that many galaxies-including our own Milky Way-may have gone through in a bygone epoch. Active galaxies may also be the source of the ionisation of neutral hydrogen observed in the intergalactic medium.</p><h3>The goal</h3><p>The goal of the project was to collect data from multiple spectra emitted by AGN, to normalise and aggregate the data, such that we would be able to present a high signal-to-noise ratio (SNR) combined spectrum.</p><p>An example of such a spectrum is given below from <a href="https://doi.org/10.3847/0004-6256/151/6/155">Harris et al. (2001)</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ePhH8bkvGcFEVbWi5yCQ0A.png" /><figcaption>Figure 1: A characteristic combined spectrum from ~100k quasars</figcaption></figure><p>The plot in Figure 1 shows distinct peaks at key wavelengths that can be used to infer the chemical composition of QSOs, and also to determine the velocity dispersion of the AGN; note that the lines are wide— the explanation for this is that the gas at the centre of an active galaxy is moving at relativistic speeds. Measuring velocity dispersion would be one of the key indicators that we had correctly measured quasars and not some other class of object (e.g. a normal galaxy, like the Milky Way), and this is achieved using the simple equation △<em>v=</em>△<em>λc</em>/<em>λ</em>, where<em> λ</em> is the measured wavelength (the x-axis in figure 1) and <em>c</em>≈300000 km s⁻¹ is the speed of light; the full-width at half-maximum was applied.</p><h3>Methods</h3><p>We aimed to download data obtained by the Sloan Digital Sky Survey’s <a href="https://www.sdss.org/instruments/">2.5 m altitude-azimuth telescope</a> located at the Apache Point Observatory in New Mexico, from a huge, freely available, online database. We used data from Data Release 12 (<a href="https://www.sdss.org/dr12/">DR12</a>) collected through July 2014, which can be queried using T-SQL (the back-end database is Microsoft SQL Server) through a web portal called <a href="http://skyserver.sdss.org/dr12/en/home.aspx">SkyServer</a>.</p><p>We chose a redshift (z) range of 1.5 ≤ z ≤ 3.1 which, as you can see from Figure 2, covers the majority of quasars found by the Baryon Oscillation Spectroscopic Survey (BOSS) survey (<a href="https://link.springer.com/article/10.1007%2Fs00159-014-0081-z">QSO densities may not drop at higher redshifts</a> as is implied here).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KlIau-anl7cMEZjWK7c37w.png" /><figcaption>Figure 2: QSO density as a function of redshift (age)</figcaption></figure><p>First we aimed to select representative spectra, and this meant trying to obtain a sample that is spread broadly across the wavelength range we’d agreed. The first step was to select some good sample objects. This was done by executing ad-hoc SQL queries against the SDSS database using <a href="http://skyserver.sdss.org/dr12/en/tools/search/sql.aspx">SkyServer</a>. An example finding 30 spectra, meeting a set of quality constrains - such as low SNR and no warning flags- is given below:</p><pre>SELECT TOP 35<br> bestObjId,<br> z, <br> — A URL you can paste into something like word and click <br> ‘<a href="http://skyserver.sdss.org/dr12/en/tools/explore/summary.aspx?id=&#39;+CAST(bestObjId">http://skyserver.sdss.org/dr12/en/tools/explore/summary.aspx?id=&#39;+CAST(bestObjId</a> as varchar) as url, <br> — Generate some deterministic randomness <br> REVERSE(SUBSTRING(CAST(specObjId as varchar), 1, 5)) indexer<br> FROM SpecObj <br> — We want quasars <br> WHERE class = &#39;QSO&#39;<br> — only select from BOSS<br> AND survey=&#39;BOSS&#39;<br> — Redshift range<br> AND z &gt;= 1.7 AND z &lt; 1.9 <br> — 0 means all is well <br> AND zWarning = 0 <br> — Signal to noise median over all good pixels <br>AND snMedian &gt;= 15<br> — Some of these come back as zero, don’t know why and finding out is a distraction. Plenty of others, just exclude them <br> AND bestObjId &gt; 0 <br> — Bitmask flag for no warnings (see schema docs for flags) <br> AND zWarning_noqso=0x00000000 <br> — Redshift error <br> AND zErr&lt;0.0005<br> — Important — ensure we’re not picking up rogue objects<br> AND sourcetype=&#39;QSO&#39;<br> — For getting a selection of items using our “random” indexer <br> ORDER BY indexer ASC</pre><p>This would select 35 objects (though not download their spectra - that would be the next step), that could be downloaded as in a CSV file. With 8 team members doing this for a range of z=0.2 each this would give us 280 spectra spanning the whole range. Mine was 1.7≤z≤1.9.</p><p>With objects selected the next step was to download their spectrographic data; doing this through the web-based UI was a laborious process that would need to be repeated hundreds of times, and may need to be repeated several times over if issues were found, so being a developer my immediate instinct was to automate the process. This was achieved in the form of a fairly rudimentary <a href="https://gist.github.com/garrypas/d7751eded6a763f71432a0c7e4328e13">Python script</a> that took the CSV file downloaded and grabbed the spectra from the SDSS website.</p><p>Some spectra are extremely intense while others are faint, and this often has nothing to do with their intrinsic luminosity, but is, instead, a function of their distance from the observer, position in the sky, obscuring dust and gas etc. To compensate for this we used a process of normalisation, whereby all spectra were normalised to a baseline value using a region of the continuous spectrum where no prominent emission lines were present. This process is complicated somewhat by the range of redshifts involved; because SDSS observes in the near-ultraviolet to far-infrared, and the data is subsequently corrected to represent the intrinsic emission wavelengths, spectra ranges vary depending on redshift; one spectrum might begin at 100 nm and end at 400 nm, another at 150 nm and end at 600 nm. To normalise to a common baseline it greatly simplifies things if the baseline is common across all spectra. Luckily there were several continuous ranges meeting this criteria, and we selected the range 220–230 nm. A further Python script would churn through the spectra calculating normalisation factors.</p><p>With spectra and normalisation factors in hand, we used an OU-provided tool called SpecCombine that would combine the spectra.</p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V7-kurEqs3WZ-HkEIT0lvg.png" /><figcaption>Figure 3: How our results compared to several examples selected from academic literature</figcaption></figure><p>Figure 3 shows how our results compared to other examples from the academic literature - including the aforementioned sample from Harris et al. (2016) - showing a high level of consistency.</p><p>We also compared our SNR with those from the academic literature, and this too gave impressive results considering the relatively small sample size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aeYys340zdQG0cX6SCdx4A.png" /><figcaption>Figure 4: SNR data comparison with academic literature</figcaption></figure><p>Calculations for velocity dispersion using prominent emission lines (such as Lyα) also gave reassuring results - of the order ~10⁴ m s⁻¹, consistent with expectations of relativistic velocity dispersions; we had successfully produced a high-quality composite spectrum.</p><h3>Summary</h3><p>This project was part of the final year of an astrophysics degree, but is the sort of project anyone can undertake as the data and the tools are freely available (SpecCombine would be trivial to implement in code), although some research is required to attach meaning to the data. Researching the nature of QSOs and understanding how our data could be interpreted gave me a deeper understanding of astrophysical concepts and the methods for processing and analysing scientific data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0c2151eed11" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>