<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ryan Baker on Medium]]></title>
        <description><![CDATA[Stories by Ryan Baker on Medium]]></description>
        <link>https://medium.com/@ryan-baker?source=rss-12d9f22091d9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6KbrEpDAg5WHhkvQD5kvJA.jpeg</url>
            <title>Stories by Ryan Baker on Medium</title>
            <link>https://medium.com/@ryan-baker?source=rss-12d9f22091d9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 05:26:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ryan-baker/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Leverage this AWS feature to securely share S3 files with your users]]></title>
            <link>https://medium.com/singularity-energy/leverage-this-aws-feature-to-securely-share-s3-files-with-your-users-d853171b0c40?source=rss-12d9f22091d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d853171b0c40</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Ryan Baker]]></dc:creator>
            <pubDate>Tue, 03 Mar 2026 15:01:02 GMT</pubDate>
            <atom:updated>2026-03-03T15:01:02.405Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://singularity.energy/">Singularity Energy</a> we build analytical tools to help utilities understand and maximize their renewable power. Most of the time, our tool creates complex and highly customized reports. Because the reports don’t have a strict structure, we store them as files and put them into an <a href="https://aws.amazon.com/s3/">AWS S3</a> bucket. We don’t want to build and manage a proxy server, so we serve those files to our users through <a href="https://aws.amazon.com/cloudfront/">AWS Cloudfront</a>. However, this led us to a natural question:</p><p><strong>How do we ensure customers from organization A are the only users who have access to the reports for organization A?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oKXiQHwZs3JQ7iHbrNC6NQ.jpeg" /></figure><p>There were a couple things we wanted to avoid doing ourselves along the way:</p><ul><li>Avoid writing our own CDN (yuck! plenty of great options exist already)</li><li>Avoid any “hooks” that have the CDN call our servers to authenticate</li></ul><p>Essentially, we wanted to take full advantage of a CDN’s features of availability and variable traffic handling without putting any pressure on our own application servers.</p><p>In our deployment, we use AWS S3 to handle file storage and AWS CloudFront to handle the CDN features. From a quick search, we learned about <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html">presigned URLs</a>. This feature allows us to create URLs for specific objects in the S3 bucket which allow access to that object.</p><p>This would work fine. However, it means that we need to create a URL for every file that is requested. That’s not very practical when we want to request dozens of files at once. For example, when we finish a report with ID 100 which belong to customer with ID ABC, the data for that report is found in ABC/report-data/100/* . This folder will contain many different files that contain various slices of the report’s output.</p><p>After a little more searching, we found that we could use <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-cookies.html">Signed Cookies</a> in Cloudfront to allow access with a wildcard match.</p><h4>Setting up Cloudfront and our code</h4><p>First, set up your Cloudfront distribution as normal. We have a couple origins and behaviors to reverse proxy to our internal services. We also have the “private” S3 bucket as an origin; the one that we want to restrict access to based on the ID of the customer to which the user belongs.</p><p>In the behavior that routes requests to that origin, check “Yes” under “Restrict viewer access”. Then select (or create) a trusted key group. When you do that, make sure you store the key ID and signing key somewhere that your application code can access.</p><p>When you send requests from your frontend to that origin, you will need to include 3 things as cookies: CloudFront-Policy , CloudFront-Signature , and CloudFront-Key-Pair-Id . In our system, we made an API route to generate and return these cookie values prior to the frontend sending the request to S3:</p><pre>from botocore.signers import CloudFrontSigner<br>from cryptography.hazmat.backends import default_backend<br>from cryptography.hazmat.primitives import hashes, serialization<br>from cryptography.hazmat.primitives.asymmetric import padding<br>from flask import current_app<br><br><br>def _rsa_signer(msg: str | bytes):<br>    key = str(current_app.config[&quot;PRIVATE_KEY&quot;]).encode(&quot;utf-8&quot;)<br>    private_key = serialization.load_pem_private_key(<br>        key,<br>        password=None,<br>        backend=default_backend(),<br>    )<br>    return private_key.sign(msg, padding.PKCS1v15(), hashes.SHA1())<br><br><br>def make_cloudfront_cookies(<br>    app_url: str, s3_access_directory: str, expires_at: datetime | None = None<br>) -&gt; Dict[str, str | bytes]:<br>    key_id = current_app.config[&quot;PUB_KEY_ID&quot;]<br>    cloudfront_signer = CloudFrontSigner(key_id, _rsa_signer)<br>    url = f&quot;{app_url}/{s3_access_directory}/*&quot;<br>    expires = expires_at or in_n_days(1)<br>    policy = cloudfront_signer.build_policy(url, expires).encode(&quot;utf-8&quot;)<br>    policy_64 = cloudfront_signer._url_b64encode(policy).decode(&quot;utf-8&quot;)<br>    signature = _rsa_signer(policy)<br>    signature_64 = cloudfront_signer._url_b64encode(signature).decode(&quot;utf-8&quot;)<br><br>    return {<br>        &quot;CloudFront-Policy&quot;: policy_64,<br>        &quot;CloudFront-Signature&quot;: signature_64,<br>        &quot;CloudFront-Key-Pair-Id&quot;: key_id,<br>    }<br><br></pre><p>The important part here is the s3_access_directory . This is the folder (or key) which you will allow access to for the users’ requests. In our implementation, the s3_access_directory is the ID of the user’s customer. Before this function runs, we grab the customer ID off of the user and pass that in as the s3_access_directory string. In other parts of our platform, we make sure to upload objects to the bucket that follow the pattern of s3://bucket_name/CUSTOMER_ID/path/to/object . This way, we ensure that we have a robust control over who has access to those objects.</p><h4>Conclusion</h4><p>While it’s a relatively straightforward implementation, you could see how you can get more complex with it. Our solution is simple on purpose, but I wrote this article because I couldn’t find any documentation for how to do this in Python. To whoever finds this article, I hope it helps you!</p><p>Sound interesting? Think you can do it better? <a href="https://wellfound.com/l/2BLRDe">We’re hiring</a> and we’d love to hear from you 🌱</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d853171b0c40" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/leverage-this-aws-feature-to-securely-share-s3-files-with-your-users-d853171b0c40">Leverage this AWS feature to securely share S3 files with your users</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Handling Real-time Updates in your React App]]></title>
            <link>https://medium.com/singularity-energy/handling-real-time-updates-in-your-react-app-0a87f9ad8d66?source=rss-12d9f22091d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a87f9ad8d66</guid>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[redux]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[cleantech]]></category>
            <dc:creator><![CDATA[Ryan Baker]]></dc:creator>
            <pubDate>Thu, 07 Aug 2025 17:35:35 GMT</pubDate>
            <atom:updated>2025-08-07T17:35:35.737Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Also — </em><a href="https://wellfound.com/jobs/3334613-software-engineer?utm_campaign=startup_share&amp;utm_content=startup_share_module&amp;utm_medium=social&amp;utm_term=singularity-energy"><em>we’re hiring</em></a><em>!</em></p><p>A few years ago I wrote about <a href="https://medium.com/@ryan-baker/building-an-elixir-cluster-on-ecs-d969faa442c2">how to build a websocket cluster</a> using <a href="https://www.phoenixframework.org/">Phoenix</a> and <a href="https://aws.amazon.com/">AWS</a>. It may not come as a surprise that we’ve been using that cluster in production for over 4 years now. What I didn’t mention was how we handle the “receiving end” of that websocket cluster. That is, how our frontend handles new incoming messages from the connection.</p><p>As you can imagine, we’ve had four years to iterate on how we handle incoming events.</p><figure><img alt="An unrelated, but relaxing photo of a boat resting in a calm bay with clear water on a sunny day." src="https://cdn-images-1.medium.com/max/1024/1*OLghbTUZfcJQc5J7UCE9zg.jpeg" /></figure><h4>Setup the connection</h4><p>The first thing to do is setup the connection code. It should be pretty straightforward, as it’s independent of React:</p><pre>import { Socket } from &quot;phoenix&quot;;<br>// whatever your hostname is for your websocket cluster<br>import { WS_URL } from &quot;shared/constants/resources&quot;;<br><br>export let socket: Socket | null = null;<br><br>interface IConnectParams {<br>  // this is a token to authenticate the user when connecting to your WS cluster<br>  wsToken: string,<br>  channels: string[],<br>}<br><br>function handleMessage(event: any, payload: unknown) {<br>  // TODO: implement this later<br>  console.log(&#39;event&#39;, event, &#39;payload&#39;, payload);<br>  return payload;<br>};<br><br>export function connect({wsToken, channels}: IConnectParams) {<br>  if (!socket) {<br>    // initialize the socket connection<br>    socket = new Socket(`${WS_URL}/socket`, { params: { token: wsToken } });<br>  }<br><br>  // reconnect if called again and we got disconnected<br>  if (!socket.isConnected()) {<br>    socket.connect();<br>  }<br><br>  socket.onError(() =&gt; socket?.disconnect());<br>  channels.forEach(channelName =&gt; {<br>    const channel = socket?.channel(channelName);<br>    // if we&#39;ve already connected, do nothing<br>    if (!channel) return;<br><br>    channel.join()<br>      .receive(&#39;ok&#39;, () =&gt; console.log(&#39;joined topic&#39;, channelName))<br>      .receive(&#39;error&#39;, () =&gt; {<br>        console.log(&#39;Failed to join topic&#39;, channelName);<br>        channel.leave();<br>      });<br><br>    channel.onMessage = handleMessage;<br>}</pre><p>And there we have it! We’ll come back later to fill out the logic of handleMessage , which will do the heavy lifting.</p><p>Then, you need to connect to your channels once you are able. In our app, this happens immediately after the app loads. We call this process “bootstrapping”. There is an initial fetch to get the data necessary for the app to load, and then the app can display itself once that is done.</p><pre>import { getCurrentUser, getIsBootstrapFailed, getIsBootstrapLoading, getIsBootstrapNotStarted } from &#39;./shared/store/user/selectors&#39;;<br>import { kickoffBootstrap } from &#39;./shared/store/user/slice&#39;;<br>import { useAppDispatch, useAppSelector } from &#39;./store&#39;;<br><br>function App() {<br>  const dispatch = useAppDispatch();<br>  const router = createRouter(routes);<br>  const shouldBootstrap = useAppSelector(getIsBootstrapNotStarted);<br>  const loading = useAppSelector(getIsBootstrapLoading);<br>  const user = useAppSelector(getCurrentUser);<br>  const failed = useAppSelector(getIsBootstrapFailed);<br><br>  useEffect(() =&gt; {<br>    if (shouldBootstrap) {<br>      dispatch(kickoffBootstrap());<br>    }<br>  }, [shouldBootstrap, dispatch]);<br><br>  if (loading) {<br>    return &lt;div className=&quot;app-bootstrap--loading&quot;&gt;<br>      &lt;LoadingOverlay visible={true} /&gt;<br>    &lt;/div&gt;;<br>  }<br><br>  // if bootstrap failed, and there is no user, redirect to login<br>  if (failed &amp;&amp; !user) {<br>    bootToLogin();<br>    return &lt;div className=&quot;app-bootstrap--loading&quot;&gt;<br>      &lt;LoadingOverlay visible={true} /&gt;<br>    &lt;/div&gt;;<br>  };<br><br>  // we use react-router here<br>  return &lt;RouterProvider router={router} /&gt;;<br>}<br><br>export default App;</pre><p>As you can guess, kickoffBoostrap handles the actual logic. It’s implemented as a <a href="https://redux-toolkit.js.org/">redux toolkit</a> <a href="https://redux-toolkit.js.org/api/createAsyncThunk">thunk</a>.</p><pre>type BootstrapStatus = &#39;FAILED&#39; | &#39;IN_FLIGHT&#39; | &#39;NOT_STARTED&#39; | &#39;SUCCESS&#39;<br><br>interface UserState {<br>  byId: Record&lt;string, IUser&gt;,<br>  bootstrap: {<br>    status: BootstrapStatus,<br>    userId: string | null,<br>    customer: ICustomer | null,<br>  },<br>}<br><br>const initialState: UserState = {<br>  byId: {},<br>  bootstrap: {<br>    status: &#39;NOT_STARTED&#39;,<br>    userId: null,<br>    customer: null,<br>  },<br>};<br><br>export const kickoffBootstrap = createAsyncThunk&lt;{data?: IFetchBootstrapResponse, status: BootstrapStatus} | undefined, undefined, {<br>  dispatch: AppDispatch,<br>  state: RootState,<br>}&gt;(<br>  &#39;user/kickoffBootstrap&#39;,<br>  async (_, {dispatch, getState}) =&gt; {<br>    if (getState().user.bootstrap.status === &#39;IN_FLIGHT&#39;) {<br>      // short circuit to prevent double bootstrapping<br>      return undefined;<br>    }<br>    dispatch(userSlice.actions.updateBootstrapState(&#39;IN_FLIGHT&#39;));<br><br>    try {<br>      // this is just a simple API call<br>      const {data} = await fetchBootstrap();<br>      return {data, status: &#39;SUCCESS&#39;};<br>    } catch (err) {<br>      return {status: &#39;FAILED&#39;};<br>    }<br>  },<br>);</pre><p>And then in the slice we have a listener for a successful bootstrap finishing:</p><pre>extraReducers: (builder) =&gt; {<br>  builder.addCase(kickoffBootstrap.fulfilled, (state, action) =&gt; {<br>    // short circuited because a request was in-flight<br>    if (!action.payload) return;<br><br>    state.bootstrap.status = action.payload.status;<br><br>    if (action.payload.data) {<br>      const user = action.payload.data.user;<br>      state.bootstrap.userId = user.id;<br>      state.byId[user.id] = user;<br>      state.bootstrap.customer = action.payload.data.customer;<br>      connect({<br>        wsToken: action.payload.data.ws_token,<br>        channels: [`user:${user.id}`, `customer:${user.customer_id}`],<br>      });<br>    }<br>  });<br>}</pre><p>And now we have an app that connects to our websocket cluster when it finishes loading!</p><h4>Handling new events</h4><p>We’re halfway there: we now connect and listen, but we don’t have any logic that handles incoming events. For our app, we handle events by dispatching the payload to the correct part of the store after validating the payload and event name. Let’s first add the validation logic to our handleMessage function:</p><pre>interface IWSEventPayload {<br>  sent_at: string // ISO8601 datetime string<br>  data: Record&lt;string, unknown&gt; // the data of the payload<br>}<br><br>enum WSEventTypes {<br>  USER_UPDATE = &#39;user_update&#39;,<br>}<br><br>function validateEventAndPayload(event: any, payload: unknown): null | {<br>  eventType: WSEventTypes,<br>  payload: IWSEventPayload<br>} {<br>  if (typeof event !== &#39;string&#39; || !event.includes(&#39;:&#39;)) {<br>    return null;<br>  }<br>  const [se_ws_event, eventType] = event.split(&#39;:&#39;);<br>  // &quot;se_ws_event&quot; -&gt; &quot;singularity energy websocket event&quot;<br>  // all of our application event names should have the format<br>  // &quot;se_ws_event:&lt;event_type&gt;&quot; and we should check that to avoid<br>  // handling internal events such as healthcheck<br>  if (se_ws_event !== &#39;se_ws_event&#39;) {<br>    return null;<br>  }<br>  if (!Object.values(WSEventTypes).includes(eventType as WSEventTypes) {<br>    return null;<br>  }<br>  if (typeof payload !== &#39;object&#39; || payload === null) {<br>    return null;<br>  }<br>  // we want to make sure that everything is valid<br>  if (<br>    !(&#39;data&#39; in payload) ||<br>    !(&#39;sent_at&#39; in payload) ||<br>    typeof payload.data !== &#39;object&#39; ||<br>    payload.data === null ||<br>    typeof payload.sent_at !== &#39;string&#39; ||<br>    isNaN(Date.parse(payload.sent_at))<br>  ) {<br>    return null;<br>  }<br>  return {<br>    payload: payload as IWSEventPayload,<br>    eventType: eventType as WSEventTypes,<br>  };<br>};</pre><p>While typescript does a lot for us when we check function calls in our own code, we have to — and should — do a lot of work to validate unknown data types that might appear at runtime.</p><p>And last, we want to have a way to dispatch the data we receive into the store so our app updates correctly:</p><pre><br>import { receiveUser } from &quot;shared/store/user/slice&quot;;<br><br>type CallbackActionCreator = typeof receiveUser<br>type ActionCreatorParams = Parameters&lt;typeof receiveUser&gt;[0]<br><br><br>interface IUserUpdatePayload {<br>  user: IUser;<br>}<br><br>const isUserUpdatePayload = (payloadData: Record&lt;string, unknown&gt;): payload is IUserUpdatePayload =&gt; {<br>  if (&#39;user&#39; in payloadData) {<br>    if (typeof payloadData.user !== &#39;object&#39;) return false;<br>    if (payloadData.user === null) return false;<br><br>    const user = payloadData.user;<br><br>    if (!(&#39;id&#39; in user) || typeof user.id !== &#39;string&#39; || !looksLikeUUID(user.id)) {<br>      return false;<br>    }<br>    if (!(&#39;customer_id&#39; in user) || typeof user.customer_id !== &#39;string&#39;) {<br>      return false;<br>    }<br>    if (!(&#39;email&#39; in user) || typeof user.email !== &#39;string&#39;) {<br>      return false;<br>    }<br>    if (!(&#39;name&#39; in user) || typeof user.name !== &#39;string&#39;) {<br>      return false;<br>    }<br>    return true;<br>  }<br>  return false;<br>};<br><br>const getEventsToCallback = (event: WSEventTypes): {<br>  actionCreator: CallbackActionCreator,<br>  payloadValidator: (payloadData: Record&lt;string, unknown&gt;) =&gt; boolean<br>} | undefined =&gt; {<br>  switch (event) {<br>    case WSEventTypes.USER_UPDATE:<br>      return {<br>        actionCreator: receiveUser,<br>        payloadValidator: isUserUpdatePayload,<br>      };<br>  }<br>}</pre><p>Finally, it’s time to revisit handleMessage :</p><pre>function handleMessage(event: any, payload: unknown) {<br>  const validateResult = validateEventAndPayload(event, payload);<br>  if (validateResult === null) {<br>    return payload;<br>  }<br><br>  const {<br>    actionCreator,<br>    payloadValidator,<br>  } = getEventsToCallback(validateResult.eventType);<br><br>  if (payloadValidator(payload)) {<br>    const action = actionCreator(payload as ActionCreatorParams);<br>    store.dispatch(action);<br>  }<br>  return payload;<br>};</pre><p>And there we have it!</p><h4>Conclusion</h4><p>Hopefully, this has made it clear how to listen to a websocket connection’s events and handle them gracefully in your React app. It may surprise you that most of the code is validation — while we do have types with typescript, we can’t control the data our application is given at runtime and we want to make sure it’s valid before introducing it to our application.</p><p>Have any questions? Have an improvement to suggest? Reach out! <a href="https://wellfound.com/l/2BtmKG">We’re hiring</a> 🌱</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a87f9ad8d66" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/handling-real-time-updates-in-your-react-app-0a87f9ad8d66">Handling Real-time Updates in your React App</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elegant data fetching & handling in TypeScript]]></title>
            <link>https://medium.com/singularity-energy/elegant-data-fetching-handling-in-typescript-9211ff0a4a75?source=rss-12d9f22091d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/9211ff0a4a75</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Ryan Baker]]></dc:creator>
            <pubDate>Thu, 30 Jan 2025 14:58:39 GMT</pubDate>
            <atom:updated>2025-01-30T14:58:39.191Z</atom:updated>
            <content:encoded><![CDATA[<p>As somebody who comes from a background of primarily Python, Java and JavaScript, learning TypeScript felt simple but had some quirks to it. One thing that I never got a good grasp on was making sure my code was organized with the standard object oriented principle of <a href="https://en.wikipedia.org/wiki/Encapsulation_(computer_programming)">encapsulation</a>.</p><p>In this blog post, I’m going to talk about how we started structuring our code in our TypeScript apps to improve readability and organization — specifically when fetching and manipulating data from a known source (such as an API).</p><figure><img alt="A calm swan sits in a lake close to view. Behind it is a quaint town bordering the lake. Further behind it is a snow capped mountain with sunlight streaming in from above." src="https://cdn-images-1.medium.com/max/1024/1*dps7uLf7x12Sireq2A6Ogw.jpeg" /><figcaption>Be as serene as this swan when you think about how you’ll be handling data by the end of this article.</figcaption></figure><h3>The Scenario</h3><p>First, let’s set up our scenario. Imagine that you’re working in a TypeScript codebase that fetches data from an API. Here is an example of some code that you might find in your repository:</p><pre>export interface ICustomer {<br>  id: number<br>  name: string<br>  parent_id: string | null;<br>  status: &#39;active&#39; | &#39;inactive&#39; | &#39;suspended&#39;;<br>  created_at: string;<br>  updated_at: string;<br>  customer_type: string;<br>  // let&#39;s pretend we have ICustomerAttribute defined somewhere<br>  attributes: ICustomerAttribute[];<br>}<br><br>// then, in another file, where you need the data<br>axios.get&lt;ICustomer&gt;(&#39;https://api.example.com/v1/customers/123&#39;);</pre><p>Pretty normal, right? The Axios repository and documentation don’t give specific guidance on typing the .get function call, but the first page of Google results were blog posts that either used an interface or just a plain type (50/50 split between 6 articles).</p><p>Next, let’s imagine that we have some code elsewhere that performs actions based on the ICustomer response data. Perhaps this is choosing to render things differently in the UI or processing data differently if done on the backend. Whatever it is, you might have some utility functions that take an ICustomer as input and return some value:</p><pre>export const isUtilityCustomer = (customer?: ICustomer) =&gt; {<br>  if (!customer) {<br>    return false;<br>  }<br>  // in a real project, we would probably define<br>  // this string as an enum or constant somewhere<br>  return customer.customer_type === &#39;utility&#39;;<br>}<br><br>function getCustomerProperty&lt;T&gt;(customer: ICustomer, prop: string) {<br>  return customer.attributes.find(attr =&gt; attr.name === prop)?.value as T | undefined;<br>}<br><br>export const getCustomerLogo = (customer: ICustomer) =&gt; getCustomerProperty&lt;string&gt;(customer, &#39;logo_url&#39;);</pre><p>So far, this is all pretty standard behavior for a TypeScript repository.</p><h3>The Problem</h3><p>What’s the issue with the above code?</p><h4>Encapsulation is broken.</h4><p>This code breaks encapsulation. The data and logic are tightly coupled, but not co-located. Even if they were in the same file, you’re <em>describing </em>data with one structure then, outside of that structure, changing <em>behavior</em> based on the internals of that structure. In short: data and logic are separated, but they should not be.</p><h4>So what?</h4><p>Why should you care if encapsulation is broken? What difference does it make?</p><p>Ultimately, breaking encapsulation doesn’t matter if:</p><ul><li><strong>You don’t care: </strong>if you don’t care about code organization and code structure, totally fine. This doesn’t make a big difference in that case.</li><li><strong>You are on a small team: </strong>if your team is small (say, 3 people or less) then code structure might not matter all that much. In fact, it might even slow you down — debating code structure when everybody knows the project intimately just adds cycles without any benefit.</li><li><strong>Your codebase is small: </strong>if your codebase is small (say, under 50 files) then you probably don’t need much organization. Scaling before you need to is another waste of time because you won’t get many benefits from organizing only a few files.</li></ul><p><strong>Breaking encapsulation matters when code organization matters.</strong></p><p>If you have a large team, are in a large company, or have a large codebase; organization matters. It’s going to make a real difference to new hires who onboard to your projects or to colleagues who join your team. It will help them understand the logic and data more quickly and prove to be more productive.</p><h3>The Solution</h3><p>Ok, so how do we solve this? How do we introduce encapsulation in a language and library that doesn’t seem to do it very easily for us? In Java or Python it might be simpler, but the solution here is to:</p><ol><li>Create a class</li><li>Marshall data</li></ol><h4>Create a class</h4><p>In addition to our interface, let’s create a class that will have methods (instead of utility functions that exist outside of the class).</p><pre>export interface ICustomer {<br>  id: number<br>  name: string<br>  parent_id: string | null;<br>  status: &#39;active&#39; | &#39;inactive&#39; | &#39;suspended&#39;;<br>  created_at: string;<br>  updated_at: string;<br>  customer_type: string;<br>  attributes: ICustomerAttribute[];<br>}<br><br>export class Customer extends ICustomer {<br>  id: number<br>  name: string<br>  parent_id: string | null;<br>  status: &#39;active&#39; | &#39;inactive&#39; | &#39;suspended&#39;;<br>  created_at: string;<br>  updated_at: string;<br>  customer_type: string;<br>  attributes: ICustomerAttribute[];<br><br>  private getPropertyValue&lt;T = string&gt;(propertyName: string) {<br>    return this.attributes.find(({name}) =&gt; name === propertyName)?.value as T | undefined;<br>  }<br><br>  get logoUrl() {<br>    return this.getPropertyValue(&#39;logo_url&#39;);<br>  }<br>}</pre><h4>Marshall data</h4><p>Next, let’s make sure we instantiate an instance of that class when we receive the data that conforms to the interface.</p><pre>axios.get&lt;ICustomer&gt;(&#39;https://api.example.com/v1/customers/123&#39;)<br>  .then(res =&gt; {<br>    const customer = new Customer();<br>    customer.id = res.data.id;<br>    customer.name = res.data.name;<br>    customer.status = res.data.status;<br>    // and so on...<br>    return customer;<br>  });</pre><p>Done!</p><p>We’ve successfully marshalled our data into a class and kept encapsulation intact.</p><p>However, you might notice this code is somewhat ugly. Here are a couple things you might have noticed:</p><ul><li><strong>re-declaring the properties</strong>: even though the class extends the interface, we need to re-declare the properties on the class. In Python or Java, we would just <em>get </em>those properties through inheritance. In TypeScript, we need to declare them again.</li><li><strong>assigning every property: </strong>in the response then() call, we need to assign every property from the response to the new customer instance. This isn’t terrible, but it is verbose. Also, if the customer properties ever change, we need to update the code in every place that we marshall the data.</li></ul><p>How can we clean this code up?</p><h3>Declaration merging</h3><p>In TypeScript, <a href="https://www.typescriptlang.org/docs/handbook/declaration-merging.html">Declaration Merging</a> is a language feature that <em>merges </em>any <em>declarations</em> of the same name. Here is the simple example from their documentation:</p><pre>interface Box {<br>  height: number;<br>  width: number;<br>}<br><br>interface Box {<br>  scale: number;<br>}<br><br>let box: Box = { height: 5, width: 6, scale: 10 };</pre><p>In this case, the box variable has all valid properties of a Box interface because TypeScript merged the two declarations of the interface. Let’s try using declaration merging on our own code to clean it up:</p><pre>// keep this interface around to describe the data structure<br>// that is returned from the API call<br>export interface ICustomer {<br>  id: number<br>  name: string<br>  parent_id: string | null;<br>  status: &#39;active&#39; | &#39;inactive&#39; | &#39;suspended&#39;;<br>  created_at: string;<br>  updated_at: string;<br>  customer_type: string;<br>  attributes: ICustomerAttribute[];<br>}<br><br>// &quot;Customer&quot; here will have a red underline<br>export interface Customer extends ICustomer { }<br>// &quot;Customer&quot; here will have a red underline<br>export class Customer {<br>  private getPropertyValue&lt;T = string&gt;(propertyName: string) {<br>    return this.attributes.find(({name}) =&gt; name === propertyName)?.value as T | undefined;<br>  }<br><br>  get logoUrl() {<br>    return this.getPropertyValue(&#39;logo_url&#39;);<br>  }<br>}</pre><p>Beautiful! We’ve cleaned up our code. But we get a warning (or error, depending on your linting settings). What is the warning?</p><p>TSLint will tell you that it is <em>unsafe </em>declaration merging between a class and interface (<a href="https://typescript-eslint.io/rules/no-unsafe-declaration-merging/">link</a>). Why? Consider this example:</p><pre>interface Foo {<br>  nums: number[];<br>}<br><br>class Foo {}<br><br>const foo = new Foo();<br><br>foo.nums.push(1); // Runtime Error: Cannot read properties of undefined.</pre><p>Oof, very valid. The properties are not automatically initialized just because we made an instance of the class. Let’s make sure that doesn’t happen in our code:</p><pre><br>// keep this interface around to describe the data structure<br>// that is returned from the API call<br>export interface ICustomer {<br>  id: number<br>  name: string<br>  parent_id: string | null;<br>  status: &#39;active&#39; | &#39;inactive&#39; | &#39;suspended&#39;;<br>  created_at: string;<br>  updated_at: string;<br>  customer_type: string;<br>  attributes: ICustomerAttribute[];<br>}<br><br>// eslint-disable-next-line @typescript-eslint/no-unsafe-declaration-merging<br>export interface Customer extends ICustomer {}<br>// eslint-disable-next-line @typescript-eslint/no-unsafe-declaration-merging<br>export class Customer {<br>  // declaration merging is safe because we assign all attributes<br>  // in the constructor<br>  constructor(data: ICustomer) {<br>    Object.assign(this, data);<br>  }<br><br>  private getPropertyValue&lt;T = string&gt;(propertyName: string) {<br>    return this.attributes.find(({name}) =&gt; name === propertyName)?.value as T | undefined;<br>  }<br><br>  get logoUrl() {<br>    return this.getPropertyValue(&#39;logo_url&#39;);<br>  }<br>}</pre><p>In this case, we are intentionally ignoring the warning for the line it affects. Normally, I hate doing this. In this case, it’s a valid warning that any user should be aware of. Also in this case, we intentionally avoid that from happening in our code by requiring the ICustomer data in the constructor.</p><p>We will also change the code that marshalls our data from Axios:</p><pre>axios<br>  .get&lt;ICustomer&gt;(&#39;https://api.example.com/v1/customers/123&#39;)<br>  .then(res =&gt; new Customer(res.data));</pre><h3>In Summary</h3><p>Practice encapsulation, it will help keep our code clean and easy to understand. We should marshall data into classes to keep logic close to the structure that describes them. We may get this for free in other languages, but it must be an intentional practice in TypeScript.</p><p>See anything wrong? Have a suggestion for an improvement? We’d love to hear from you at <a href="https://singularity.energy/">Singularity Energy </a>— drop us a line! 🌱</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9211ff0a4a75" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/elegant-data-fetching-handling-in-typescript-9211ff0a4a75">Elegant data fetching &amp; handling in TypeScript</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deciding on a language for your web service]]></title>
            <link>https://medium.com/singularity-energy/deciding-on-a-language-for-your-web-service-c6f73fb9d448?source=rss-12d9f22091d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6f73fb9d448</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Ryan Baker]]></dc:creator>
            <pubDate>Mon, 08 Jul 2024 17:02:08 GMT</pubDate>
            <atom:updated>2024-07-08T17:02:08.910Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://singularity.energy/">Singularity</a>, we primarily use Python for our web services. On the frontend, we use Typescript. We also use Elixir for our real-time push server. We recently started a project that requires very high I/O throughput and were concerned that Python may not be the best tool to serve us in that use case. So we asked the question:</p><h4>Which language should we use for a high-throughput web service?</h4><p>Based on our in-house experience on our team, we had a few candidate languages that we wanted to benchmark. As a team we decided to try testing:</p><ul><li>NodeJS</li><li>Java</li><li>Rust</li><li>C++</li></ul><p>We decided to devote about 2 hours to standing up each language with a framework that seems to have a large community support and thorough documentation. The web servers would each interact with the same database in a docker-compose network with the PostgreSQL DB. The web servers would have 2 endpoints that only do: fetch item by ID, paginate list of items with page/offset parameters.</p><h4>The Testing</h4><p>The languages and frameworks we actually tested were:</p><ul><li>Python (Flask/SQLAlchemy/psycopg2)</li><li>NodeJS (express/TypeORM)</li><li>Rust (Actix Web/tokio_postgres)</li><li>Async Python (FastAPI/SQLAlchemy/asyncpg)</li><li>Elixir (Phoenix/Ecto)</li></ul><p>Although we had agreed to also try out C++ and Java, I had trouble setting them up, even for a simple case. Researching them took far too long, lacked (public) resources and community support that I could find, and was unable to get a JSON API → PostgreSQL db service working in a modest amount of time. I’m open to trying again with those at another time, but the difficulty in approaching them turned me off of them for this set of tests.</p><p>For all of the frameworks and languages listed, I followed the docs’ recommended practices for creating a production-ready docker image. I then ran those images as containers in a docker network on my laptop that also has a PostgreSQL 16 docker container running, so they all hit the same database. I ran the tests using <a href="https://locust.io/">Locust</a> locally.</p><p>This article <strong>will not</strong> dive into the <em>why </em>of each result set, that would make the research and article much longer. Additionally, the point of this exercise is to see what can be learned the quickest for the highest IO throughput of a server.</p><p>For each round of results, we’ll go through the number of concurrent users and how the different frameworks handled them.</p><h4>Under 1,000 users</h4><p>A set of tests for 1, 10, and 100 concurrent users that each send up to 2 requests per second.</p><p><strong>Python</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pgLpmbfG4AcKoIL5vjLmzQ.png" /><figcaption>Python: Flask/SQLAlchemy/psycopg2</figcaption></figure><p><strong>NodeJS</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JOVAjV0k6iq1r-MA15TJGA.png" /><figcaption>NodeJS: express/typeORM</figcaption></figure><p><strong>Elixir</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCTvCmzwfA6Fgdg31P4OIA.png" /><figcaption>Elixir: Phoenix/Ecto</figcaption></figure><p><strong>Rust</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W2ccwSsi2ZXPrX3kk3_n0Q.png" /><figcaption>Rust: Actix Web/tokio_postgres</figcaption></figure><p><strong>Async Python</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OZcZAxBWu_d5-oPBEbroAA.png" /><figcaption>Async Python: FastAPI/SQLAlchemy/asyncpg</figcaption></figure><p>For these set of tests, we see what we would expect. Almost all setups handle the ~200 concurrent requests just as they come in and are served. The only exception is Python using the async/await syntax. Unfortunately, it had an error rate of about 50%. When looking at the error that was in the logs, it appeared as:</p><p>sorry, too many clients already</p><p>Propagated from the asyncpg driver. Looking this error up shows that it came from the PostgreSQL db directly and was not an error in application code. I’m sure that there could be some tweaks and configuration changes that I could do to make this setup work well, but that would’ve defeated the purpose of the exercise.</p><p>All subsequent tests will exclude the <strong>Async Python</strong> setup.</p><h4><strong>1,000 Users</strong></h4><p><strong>Python</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ybxvdNOKmTtOCgtwFoZ9NA.png" /><figcaption>Python: Flask/SQLAlchemy/psycopg2</figcaption></figure><p><strong>NodeJS</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tiqnkrP2kgTO5U2nGyXrKA.png" /><figcaption>NodeJS: express/typeORM</figcaption></figure><p><strong>Elixir</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uuHctrSf0pGa0RcjmZkEsg.png" /><figcaption>Elixir: Phoenix/Ecto</figcaption></figure><p><strong>Rust</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RGqEM97NWnQa0TE7vYyGDw.png" /><figcaption>Rust: Actix Web/tokio_postgres</figcaption></figure><p>Here, we are asking our setups to keep up with roughly 2,000 requests per second. All frameworks make pretty light work of it except Python. Why can’t Python keep up? Why does it cap at around ~200 reqs/s? What is my exact setup?</p><p>All good questions, I’m curious too, but not the point of the exercise. However, since we didn’t see errors happen, I’ll continue to include Python in the last set of tests.</p><h4><strong>2,000 Users</strong></h4><p><strong>Python</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yMB95UCnS0FY5Xpbzw54WQ.png" /><figcaption>Python: Flask/SQLAlchemy/psycopg2</figcaption></figure><p><strong>NodeJS</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*20fzB4gjtmVMXRw_L1Th3A.png" /><figcaption>NodeJS: express/typeORM</figcaption></figure><p><strong>Elixir</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jmFVWe7jYch0fUQ3aStRUg.png" /><figcaption>Elixir: Phoenix/Ecto</figcaption></figure><p><strong>Rust</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nIEZ2_PLvhacW5v5jIMYQg.png" /><figcaption>Rust: Actix Web/tokio_postgres</figcaption></figure><p>Here, we are asking our setups to keep up with roughly 4,000 requests per second. All frameworks that try to reach that level clearly have a tough time keeping up. However, because every setup (excluding Python) seems to have very similar behavior, I’m tempted to believe that there is a bottleneck on the laptop I’m using to run the tests and not a bottleneck with the setup itself. That may or may not be true.</p><h3>Conclusion</h3><p>For workloads with high throughput, where IO is the bottleneck, it appears clear that you won’t go wrong choosing and learning any of these setups aside from Python.</p><p>I won’t dissuade anybody from using Python, we have used it for years, enjoy its ease-of-setup and syntax, and will continue to use it, especially for data analysis and manipulation workloads.</p><p>However, we did this test because we will soon have a new product with a different workload than our previous products. We wanted to see if we should switch up our stack to better serve the new product.</p><p>Curious what the new product is? Interested in our stack? Disagree with the results?</p><p>We’re hiring a software engineer! <a href="/service/mailto:jobs@singularity.energy">Email us</a> and tell us your thoughts, we’d love to hear from you 🌱</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6f73fb9d448" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/deciding-on-a-language-for-your-web-service-c6f73fb9d448">Deciding on a language for your web service</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons from ingesting 500M database records at once]]></title>
            <link>https://medium.com/singularity-energy/lessons-from-ingesting-500m-database-records-at-once-88425b70700c?source=rss-12d9f22091d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/88425b70700c</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Ryan Baker]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 15:21:15 GMT</pubDate>
            <atom:updated>2024-01-03T15:21:15.996Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://singularity.energy/">Singularity Energy</a>, we started building one of our products to handle much larger scale than we have previously supported. Our primary product’s data store has about 2 billion data points. However, that was ingested over a few years’ time. For this new product, we wanted to see where our system would fail under higher-than-expected strain. To do this, we tried ingesting 500 million data points at once.</p><p>As <a href="https://medium.com/singularity-energy/invisibly-migrating-billions-of-database-records-91686aa8130">we’ve mentioned previously</a>, we queue all of our processing so it can be done asynchronously. We added about 500,000,000 messages to that queue. We watched closely to see which parts succeeded at their jobs and which failed.</p><p>In the rest of this post, I’ll go over what already worked well, what needed to change, and what changes we will make ahead of our next large scale data ingestion.</p><h3>What worked already</h3><p>Before starting this workload, we already had a system where work would be pulled off of a queue and worked on by processes that we called “workers”. As the workers picked up work, they started inserting lots of rows into the database. From here, it was obvious that we did not give the database enough storage space for all the data it would create. However, we had already set up something that would protect against that:</p><h4><strong>Auto-scaling RDS storage space</strong></h4><p>Whenever our RDS instance was low in storage space, it would automatically add more storage space (without downtime) and allow work to continue. Luckily, this feature in AWS is as easy as checking a box in the RDS configuration page.</p><h4>Modeling large workloads as “jobs” in our database</h4><p>In our system, we use <a href="https://docs.celeryq.dev/en/stable/index.html">Celery</a> to handle asynchronous work. We like to keep our “tasks” as small as possible (for multiple reasons that I won’t get into here). For this reason, we decided to model larger workloads as a “jobs” table in our database, that have a status, configuration, cursor, and percentage complete. This allowed us to see how the jobs were doing, as well as check the logs for any specific or problematic jobs we observed.</p><figure><img alt="A screenshot of an internal dashboard for Singularity Energy that shows jobs in a table format, including their status and percentage complete." src="https://cdn-images-1.medium.com/max/1024/1*eaoSjuoltO8iCCZVkznwUg.png" /><figcaption>What we see in our internal dashboard</figcaption></figure><h4>A separate “slow-lane” worker queue</h4><p>As you can imagine, we have a lot of work that our workers do in the background. For this reason, we configured two sets of workers to read from two different queues: normal workers and latency-insensitive workers.</p><p>The “latency insensitive” workers will do work that typically takes a longer time to complete and/or work that is resilient to system delays.</p><p>This allowed our “jobs” to be completed and take up most of the throughput of the workers without interrupting the normal workers. If we did not have that, then a normal system function would have been severely delayed. For example, our “reset your password” email is sent via a worker. If we have 500,000,000 messages on the queue and we add a single message to the end of the queue in order to send the password reset email, the user would likely not receive that email until after the other 500,000,000 messages were take off the queue. This would result in a significant delay for the user.</p><h3>What we changed to work better</h3><p>During the data ingest, we updated a few things about our system in order to get through the 500 million messages in a timely manner. We prioritized these based on the issues and errors that we saw in our system.</p><h4>Provide workers direct database access</h4><p>Typically, we have our workers send a request to the API to access objects in the database. This is roundabout and unnecessary, but it helped us by limiting what resources could access the database and guaranteeing a log line at every data request (by having the API log every incoming request).</p><p>However, we quickly found that the workers were choking the API:;</p><figure><img alt="An image showing the CPU and memory utilization of the API. The memory hovers around 25%, but CPU is almost always at 100% for a 12 hour window." src="https://cdn-images-1.medium.com/max/1024/1*Gjt0JXUu9fZ1U6ApQiF_Nw.png" /><figcaption>Resource utilization for our API showed they were consistently holding 100% CPU</figcaption></figure><p>We identified the requests causing high resource utilization and found that they were doing lots of DB operations and would actually exceed the “healthy” threshold for a response. Instead of fixing the issue by making the database faster (which showed healthy metrics at this point), we moved the DB access for these high-latency operations from the API to the workers.</p><p>This relieved pressure from the API and removed what probably should never have been an operation the API was doing in the first place.</p><h4>Upsize the database based on the workload</h4><p>After fixing the previous point, the system was humming along nicely. However, it seemed that it was going very slow and would take a couple weeks to finish the data ingestion. We decided to increase the number of workers in order to give them higher throughput. Naturally, we made sure to watch the database when we did this.</p><p>Sure enough, the database CPU jumped from ~20% to about 80%. This was well past the allowance for our RDS instance size, and the CPU credits for our instance started depleting. In response, we decided to upsize the database during the period that we would be doing this work.</p><p>It worked like a charm! Dropping the CPU back to healthy levels allowing high throughput of the jobs, as well as supporting the application usage needs for the database.</p><h4>Add the ability to pause and restart a worker job</h4><p>In certain cases, we had a problematic job that needed to be fixed. We decided to implement a way for a Singularity employee to pause and restart a worker job in order to effectively re-run the job upon deploying a bug fix.</p><p>This worked out very nicely for us and prevented us from having to restart the entirety of the 500M record ingest, and just restart a smaller portion of it.</p><h3>Things we’ll do for next time</h3><p>Overally, the process was relatively smooth. The least smooth parts about it were that we had to watch and monitor it the whole time. However, with a few tweaks, we could comfortably kick off another job of this size and rest assured that the system would take care of itself. With a few small changes:</p><h4>Auto-scale database resources</h4><p>As part of the work we did during the ingest, we upsized the database. We could, instead, apply an auto-scaling group to the database based on its CPU utilization and allow it to upsize itself if it needed to.</p><p>This would allow us to not have to manually interfere with changing the configuration of the database and allow our system to instead respond to changes in its workload.</p><h4>Minimize payloads sent over SQS</h4><p>We learned that AWS SQS has a built-in message limit. We knew that this existed, but felt we would not hit it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L1t0vxmqEtHrnsJ_aisaYg.png" /><figcaption>An error in our Sentry dashboard that we saw in our QA environment</figcaption></figure><p>We were wrong.</p><p>Next time, instead of sending large payloads over SQS, we will store them somewhere first, and provide the Celery worker a reference to the large payload. That way, we can provide arbitrarily large jobs or configurations without having to stretch the limit of SQS.</p><h4>Auto-scale logging cluster disk space</h4><p>Unfortunately, we ran out of space in our logging cluster during our ingest. We had so many more logs than normal that our OpenSearch cluster ran out of space to store them! In the console, we manually added 2 more data nodes to the logging cluster and bumped their disk space by 20%.</p><p>This added more cost, but solved a critical observability issue for us. In the future, we will add an auto-scaling policy to the logging cluster and allow it to grow and shrink as needed to fit our needs.</p><p>—</p><p>In conclusion, we learned a lot from this ingestion and have a good list of projects to keep us accelerating our scale of data.</p><p>Comments? Thoughts? See anything we missed or did wrong? Send it over to jobs@singularity.energy, we’d love to chat.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88425b70700c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/lessons-from-ingesting-500m-database-records-at-once-88425b70700c">Lessons from ingesting 500M database records at once</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Single-line Onboarding Using Docker Compose]]></title>
            <link>https://medium.com/singularity-energy/single-line-onboarding-using-docker-compose-971aa6ce8a?source=rss-12d9f22091d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/971aa6ce8a</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Ryan Baker]]></dc:creator>
            <pubDate>Thu, 24 Aug 2023 15:07:34 GMT</pubDate>
            <atom:updated>2023-08-24T15:07:34.141Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How to speed up engineering onboarding with this one weird trick!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kr7o6kUINSYptFCqplrPtg.jpeg" /><figcaption>Breeze through onboarding by using Docker locally</figcaption></figure><p>All my career, I’ve worked at various startups that have had a very similar onboarding strategy. Because the company (and product) is small enough, it’s reasonable to ask the engineers to run the entirety of the product locally on their machine for development. If this sounds familiar, the following steps might also sound familiar:</p><ol><li>Download the (correct version) of the backend language</li><li>Download the (correct version) of NPM/Node for the frontend</li><li>Download the package manager for the backend</li><li>Download the package manager for the frontend</li><li>Download the (correct version) of the DB to run locally</li><li>(optionally) download the other non-primary storage DBs (e.g. ElasticSearch, Redis)</li><li>Install the frontend and backend dependencies</li><li>Start the frontend server</li><li>Start the backend server</li><li>Start other backend processes (e.g. async workers), if needed</li><li>Debug failures and undocumented dependencies and return to step 6</li></ol><p>This process is tiring, but expected. Setting up a development environment is something that takes between a few hours and a few days and should not have to be repeated. Unfortunately, it’s also difficult to help others because computers, languages and versions change over time so they may run into errors you didn’t encounter or you might have forgotten the errors you did encounter when you set up your environment 6 months prior. Surely there is a better way.</p><h4>The Shortcut: Docker Compose</h4><p>About a year ago I took <a href="https://www.udemy.com/course/docker-mastery/?kw=docker+kubernetes&amp;src=sac">a course on Udemy that covered Docker and deployment orchestration</a>. While I’ve used Docker pretty extensively in my roles, it’s almost always been for the purpose of managing deployed environments (typically via AWS ECS).</p><p>Docker Compose was new to me, and I hadn’t tried to learn much about it before the course. I found the official documentation daunting and too unclear to sell me on its benefits. The course, however, quickly displayed how powerful Docker Compose could be with a relatively simple YAML file.</p><p>Instead of needing to install the various dependencies on my host machine, I could define a few containers with the same versions of libraries and languages used in production. I could then run one command to spin up all the containers that I needed:</p><pre>docker compose up</pre><p>The result would be a terminal shell that shows the output of building several images and running them as containers. At Singularity, we have quite a few services to run locally for a single application, so here is an example of one of our docker-compose.yml files:</p><pre>version: &#39;3.7&#39;<br><br>services:<br>  db:<br>    image: postgres:12.11<br>    restart: always<br>    environment:<br>      - POSTGRES_PASSWORD=app_db_password<br>      - POSTGRES_USER=app_db_user<br>      - POSTGRES_DB=app_db<br>    ports:<br>      - 5432:5432<br>    volumes:<br>      - db-data:/var/lib/postgresql/data<br><br>  app_api:<br>    restart: always<br>    build:<br>      context: ./backend<br>    env_file: secrets.env<br>    volumes:<br>      - ./backend:/usr/src/app/<br>    ports:<br>      - 5010:5010<br>    entrypoint: flask<br>    command:<br>      - run<br>      - --host<br>      - &#39;0.0.0.0&#39;<br>    environment:<br>      - &#39;MQ_BROKER_URL=amqp://admin:mypass@rabbitmq:5672&#39;<br>      - FLASK_APP=autoapp.py<br>      - FLASK_RUN_PORT=5010<br>      - DATABASE_URI=postgresql://app_db_user:app_db_password@db/app_db<br>      - FLASK_DEBUG=1<br>    depends_on:<br>      - db<br><br>  frontend:<br>    restart: always<br>    build:<br>      context: ./frontend<br>    volumes:<br>      - ./frontend:/usr/src/app/<br>    ports:<br>      - 3000:3000<br>    entrypoint: yarn<br>    command:<br>      - start<br>    depends_on:<br>      - app_api<br><br>  rabbitmq:<br>    restart: always<br>    image: rabbitmq:3-management<br>    environment:<br>      - RABBITMQ_DEFAULT_USER=admin<br>      - RABBITMQ_DEFAULT_PASS=mypass<br>    ports:<br>        - 5673:5672<br>        - 15673:15672<br>    volumes:<br>      - rabbitmq-data:/var/lib/rabbitmq<br><br>  worker:<br>    restart: always<br>    build:<br>      context: ./backend<br>    env_file: secrets.env<br>    volumes:<br>      - ./backend:/usr/src/app/<br>    entrypoint: celery<br>    command:<br>      - -A<br>      - worker.tasks<br>      - worker<br>      - -Q<br>      - worker-queue<br>    environment:<br>      - &#39;MQ_BROKER_URL=amqp://admin:mypass@rabbitmq:5672&#39;<br>      - &#39;SERVICE_API_HOST=http://app_api:5010&#39;<br>    depends_on:<br>      - rabbitmq<br>      - app_api<br><br>volumes:<br>  db-data:<br>  rabbitmq-data:</pre><p>You might have noticed something interesting with the URLs: Docker Compose will automatically create a Docker bridge network and attach all the services (“app_api”, “worker”, etc.) to the network. Additionally, the containers attached to that network will have a network alias that is the service name. This is why the hostname of the DATABASE_URI is just db , for example.</p><h4>Expanding on it</h4><p>You might later want to add something to your infrastructure. For example, we wanted to try adding tracing via OpenTelemetry using Jaeger as the backend. To do this, we instrumented our code according to the <a href="https://opentelemetry-python-contrib.readthedocs.io/en/latest/">OpenTelemetry docs</a>, and we added Jaeger as a service in our docker-compose.yml :</p><pre>diff --git a/docker-compose.yml b/docker-compose.yml<br>index 9dc64ca..4f1d459 100644<br>--- a/docker-compose.yml<br>+++ b/docker-compose.yml<br>@@ -112,6 +112,14 @@ services:<br>    depends_on:<br>      - rabbitmq<br>      - app_api<br> <br>+  jaeger:<br>+    restart: always<br>+    image: jaegertracing/all-in-one:1.44<br>+    ports:<br>+      - 16686:16686<br>+      - 4317:4317<br>+    environment:<br>+      - COLLECTOR_OTLP_ENABLED=true<br> <br> volumes:<br>   db-data:</pre><p>Adding the code was still tedious, but adding this one service for Docker Compose allowed me to test the OTel implementation without needing to deploy anything (even to the development environment)!</p><p>Seeing an extra component of infrastructure being added this easily made me feel unencumbered to try more things: cacheing via Redis, search via ElasticSearch, even trying out a timeseries database instead of PostgreSQL.</p><h4>The Caveats</h4><p>Of course, nothing is simple and that includes changing a local development environment. Of the 11 steps that I highlighted above, the first 4 are still required to have any sort of integrated development environment (where you want your code editor to resolve the installed libraries and help with text completion and code suggestions).</p><p>Additionally, developers will have to download docker and have some (at least rudimentary) understanding of using docker and docker compose in order to fluently manage their containers. This might seem reasonable for a backend, full stack, or devops engineers, but could be asking a lot of a frontend engineer.</p><p>Ultimately, don’t change your tooling just to change it, but carefully understand what it would mean to do so. Decide with your team if it’s worth it. Perhaps try it out with yourself and a couple others before rolling it out more broadly.</p><p>Interested in this type of work? Have suggestions for how we could do it better? <a href="/service/mailto:jobs@singularity.energy">Email us</a>! <a href="https://wellfound.com/company/singularity-energy">We’re hiring</a> 🌱</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=971aa6ce8a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/single-line-onboarding-using-docker-compose-971aa6ce8a">Single-line Onboarding Using Docker Compose</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Tech to Sales]]></title>
            <link>https://medium.com/tinycadence/from-tech-to-sales-d1377bc0c176?source=rss-12d9f22091d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1377bc0c176</guid>
            <category><![CDATA[bootstrapping]]></category>
            <category><![CDATA[sales]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Ryan Baker]]></dc:creator>
            <pubDate>Mon, 21 Aug 2023 21:07:12 GMT</pubDate>
            <atom:updated>2023-08-21T21:07:12.003Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Documenting the start of my journey from building things to selling things.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P6XmvIacmGnfF0yImOWISA.jpeg" /><figcaption>This is roughly how confident I feel right now.</figcaption></figure><p>For the last six or so months, I’ve worked on building a SaaS. It’s nothing big or groundbreaking: it’s <a href="https://www.tinycadence.com/">an email sequence tool</a>. Think Outreach, Salesloft, or HubSpot Sequences but for smaller businesses (<a href="https://www.getweflow.com/blog/outreach-vs-salesloft">and for 1/10th the price</a>). It integrates with HubSpot’s CRM, so it’s a direct competitor to HubSpot Sequences. We’ve gotten the product to the point where it’s usable and solves a real problem that exists.</p><p>Now we just have to sell it.</p><p>For the past 10 years, I’ve worked at various tech startups in some form of a software engineering role. That may be tech lead, senior engineering manager, or just software engineer. I’ve joined security review calls, software demo calls, vendor calls (on the side of the buyer), and some prospective sales calls. I worked at a marketing technology company that was sales-oriented. I know the words, the terms, the tools in the space.</p><p>I have no idea how to sell.</p><p>Conventional startup wisdom says to:</p><ul><li>focus on a niche</li><li>focus on a user’s pain point and solve for that</li><li>validate that your solution solves the pain</li><li>ask for money</li></ul><p>However, we’ve validated that this is a pain that exists and our solution solves the problem (otherwise the established players wouldn’t exist).</p><p>So instead, we’ll start by focusing on specific people and users in a target segment. Let’s define them by their problems:</p><ul><li><strong>budget constrained</strong>: they can’t afford HubSpot’s <a href="https://www.hubspot.com/pricing/sales/starter?products=sales-hub-starter_1&amp;term=monthly">$500/mo price tag</a> ($100/user/month with a 5 user minimum)</li><li><strong>time constrained</strong>: they have more contacts to reach out to than time or people to reach out to them. They deeply feel the pain of not automating their email outreach.</li><li><strong>routine, predictable script</strong>: they have an outbound process that works. It’s the same email(s) with some light customization to everybody that enters the funnel.</li></ul><p>Based on these problems, we can get at some other attributes about potential customers that we could use as a proxy for a good customer fit:</p><ul><li><strong>small business</strong>: once you have a large team, you likely have deep pockets. For this reason, you can probably afford Outreach or Salesloft and would not be interested in a cheaper solution.</li><li><strong>early stage business</strong>: once you’re established, you likely have a software vendor process that would eliminate us due to security requirements. We have built products with and understand SOC2 certification, but that doesn’t mean we have it (yet).</li><li><strong>somewhat technical</strong>: because we’re early, we’ll be iterating with our customers. We’ll benefit from customers who “get” software product development. They can look past some minor bugs and get excited when we launch features they’ve been asking for.</li></ul><p>But I’ll say it again: we have no idea what we’re doing.</p><p>If you have any sales or prospecting advice (e.g. how do I get a warm intro? how do I get email addresses for prospects? etc.) I’d love to hear from you! Shoot me an email: (hello [at] ryanbaker [dot] dev)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1377bc0c176" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tinycadence/from-tech-to-sales-d1377bc0c176">From Tech to Sales</a> was originally published in <a href="https://medium.com/tinycadence">TinyCadence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Invisibly Migrating Billions of Database Records]]></title>
            <link>https://medium.com/singularity-energy/invisibly-migrating-billions-of-database-records-91686aa8130?source=rss-12d9f22091d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/91686aa8130</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Ryan Baker]]></dc:creator>
            <pubDate>Wed, 10 May 2023 15:39:13 GMT</pubDate>
            <atom:updated>2023-05-10T15:39:13.418Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How we upgraded a database with well over a billion records without downtime.</em></p><p>We handle lots of requests. We handle even more data that we keep up to date. On top of all that, we also had a very old database. We recently upgraded our primary database. We had multiple things to upgrade with our database. We wanted to make sure we were able to upgrade our database with no negative customer impact. In this article, I will go over the methods we used at Singularity Energy to achieve this and the lessons we learned along the way.</p><figure><img alt="Men and women gathered at a table staring at a laptop while laughing and cheering. The corporate joy is palpable. One woman is wearing a Microsoft Vista themed sweater." src="https://cdn-images-1.medium.com/max/1000/0*aE_6oa7XigAWy9x5" /><figcaption>Live shot of the Singularity team happily watching our database upgrade.</figcaption></figure><h4>Reasons to upgrade</h4><p>We had a lot of reasons to upgrade our primary database. For us, though, it came down to 2 reasons:</p><ol><li>It was over 3 years old and was running a version of PostgreSQL that was not going to be supported soon.</li><li>It was originally the MVP database and was missing security and redundancy parameters.</li></ol><p>Since we use AWS’ Relational Database Service (RDS), we get a lot of upgrading support for free. Out of the box, they offer minor version upgrades with configurable maintenance and backup windows. We were upgrading from PostgreSQL 10 to 14 (the latest version offered by RDS at the time of the upgrade). Based on the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html#USER_UpgradeDBInstance.PostgreSQL.MajorVersion">AWS documentation</a>, we needed to upgrade the minor version to be able to upgrade to the latest 14.x version. Luckily, it’s simple to upgrade the minor version without downtime in AWS. Once we did that, we were ready to upgrade the major version.</p><h4>Steps to upgrade</h4><p>In order to make sure there was be no data loss, the upgrade would consist of 4 parts:</p><ol><li>Dual write</li><li>Create a secondary DB from the snapshot</li><li>Upgrade secondary DB</li><li>Cutover</li></ol><p>Steps #2 and #3 meant that the secondary database (the one we upgraded) was not reachable. Therefore, it is important that #1 is implemented in a way that can wait until the upgrade is complete before writing to the new database.</p><p>Additionally, we wanted to make sure there was minimal code change for step #4 in order to minimize the chance of failure from a coding error.</p><h4>Dual writing</h4><p>When we cut over to using the new database, we wanted to make sure that it had all the data from the old database. For this reason, we took a snapshot of the old database and created a new one from the snapshot.</p><p>However, what happens in the time between taking the snapshot and creating a new database from the snapshot? Our system processes millions of events per day, and we can’t just stop it because we want to upgrade our database.</p><p>The solution here is to “dual write” the data to both databases. There is a catch, though. Since we wanted to dual write before the new database existed, we needed to capture what we <em>would </em>write to the database if it existed. For our architecture, it seemed fitting to create a queue that would hold copies of the data that was written to the old database. We then needed to write that data to the new database in order to catch up. Implementing this architecture ended up looking like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UfBcOiKhWAZPQScaMPm6qw.png" /></figure><p>When the API wrote to the database, it also needed to send to the queue. The code for that looked like this:</p><pre>from contextlib import contextmanager<br>import json<br><br>import boto3<br><br><br>@contextmanager<br>def get_sqs_client():<br>    sqs_client = boto3.client(&#39;sqs&#39;)<br>    try:<br>        yield sqs_client<br>    finally:<br>        sqs_client.close()<br><br><br>def get_queue_url(https://medium.com/sqs_client,%20queue_name):<br>    queue_url = sqs_client.get_queue_url(https://medium.com/QueueName=queue_name)[&#39;QueueUrl&#39;]<br>    return queue_url<br><br><br>def send_dual_write_msg(operation: str=&#39;insert&#39;, table: str, data: dict):<br>    with get_sqs_client() as sqs:<br>        sqs.send_message(<br>            QueueUrl=get_queue_url(https://medium.com/sqs_client=sqs,%20queue_name=&#39;dual-write-db&#39;),<br>            MessageBody=json.dumps({<br>                &#39;operation&#39;: operation, # insert, update, or delete<br>                &#39;table&#39;: table,<br>                &#39;data&#39;: data,<br>            })<br>        )</pre><p>Then using that code in a model class of ours looked like this:</p><pre><br>class CRUDMixin(object):<br>    &quot;&quot;&quot;Mixin that adds convenience methods for CRUD operations.<br><br>    all of our SQLAlchemy model classes inherit from this.&quot;&quot;&quot;<br><br>    def _add_to_session_and_commit(self):<br>        with get_session() as session:<br>            session.add(self)<br>            session.commit()<br>            session.refresh(self)<br>            return self<br><br>    @classmethod<br>    def create(cls, **kwargs):<br>        &quot;&quot;&quot;Create a new record and save it the database.&quot;&quot;&quot;<br>        instance = cls(**kwargs)<br>        # all of our SQLAlchemy classes have __tablename__<br>        # class variable declared<br>        send_dual_write_msg(&#39;insert&#39;, cls.__tablename__, kwargs)<br>        instance._add_to_session_and_commit()<br>        return instance<br><br>    ...</pre><h4>Restore from snapshot &amp; upgrade the database</h4><p>In order to copy the majority of the data, we <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html">created a snapshot</a> of our database in AWS. After creating a snapshot, AWS made it pretty simple to <a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/encrypt-an-existing-amazon-rds-for-postgresql-db-instance.html">encrypt the snapshot</a> (in the case that the database was not encrypted), and then <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html#USER_RestoreFromSnapshot.Restoring">create a database from that snapshot</a>.</p><p>Luckily for us, we didn’t use any deprecated or removed features between the major versions of PostgreSQL. So we simply followed this AWS guide on <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html#USER_UpgradeDBInstance.PostgreSQL.MajorVersion.Process">upgrading the major version</a> once the database was live.</p><h4>Cutover</h4><p>This was the most delicate step in the process. Before we cut the traffic over to the new database, we had to make sure that there was no interruption or data loss. We carefully combed through the database records to ensure the data was the same between the two databases.</p><p>The next step was making sure the cutover was invisible. To do that, we simply changed the config variable that represented the database URL. We started new tasks that used the new database and slowly rotated them into use and rotated the old ones out.</p><p>We now had an architecture that looked like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b9Bm-dNlcpFxBDBOch0-Zg.png" /></figure><p>For the next few days we switched the writer worker to write to the old database, in the event that we needed to switch back. Luckily, nothing bad happened and we were able to use the new DB without issue!</p><h4>Conclusion</h4><p>While nerve-wracking, the upgrade was able to be carried out without a hitch! This blog post breaks it down somewhat simply, but I can’t advocate enough for having a documented plan. Share that plan with the team, and ask one more team member to be on duty to help troubleshoot if something bad happens.</p><p>Interested in problems like these? Thoughts on how we could do it better? <a href="/service/mailto:jobs@singularity.energy">Reach out</a>! We’d love to hear from you 🌱</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91686aa8130" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/invisibly-migrating-billions-of-database-records-91686aa8130">Invisibly Migrating Billions of Database Records</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Simple AWS Security Tips]]></title>
            <link>https://medium.com/singularity-energy/5-simple-aws-security-tips-a114e5f94b2f?source=rss-12d9f22091d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a114e5f94b2f</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Ryan Baker]]></dc:creator>
            <pubDate>Fri, 02 Dec 2022 14:13:55 GMT</pubDate>
            <atom:updated>2022-12-02T14:13:55.986Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’re building a business in the AWS cloud that needs to be secure (i.e. any business these days), there are some easy, but non-obvious things you can do to make sure your security is buttoned down and ready to sell confidently to enterprise customers.</p><p>Here are five simple things to do to make sure you are not opening yourself to any unnecessary risks without having to change much (or any!) of your application code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kAIjuvBx6bshKUh43DDgGQ.jpeg" /></figure><h4>1. Drop “NET_RAW” requirement in your ECS task</h4><p>As with all things in AWS, networking is difficult to manage and easy to get wrong. This is one of those things that can leave you very vulnerable to an attack but is easy to patch. This permission is available by default in your task definitions, so you need to explicitly forbid it.</p><p><strong>What is does</strong>: this makes sure that your containers that run in the same docker network cannot craft raw packets and send them to each other. If you have one container that is taken over by a bad agent, you want to make sure they cannot communicate with another container in the network.</p><p><strong>How to do it:</strong> simply add the following line to your container definitions inside of your task definition:</p><pre>...<br>&quot;linuxParameters&quot;: {<br>  &quot;capabilities&quot;: {<br>    &quot;drop&quot;: [&quot;NET_RAW&quot;]<br>  }<br>},<br>...</pre><h4>2. Docker containers should not use the root user</h4><p>In the same vein, you should limit the permissions of a container that is compromised. For example, removing the ability to install various packages via the package manager of the container. Removing the ability to run arbitrary commands for a container allows you to limit the attack vectors.</p><p><strong>What is does</strong>: this makes sure that your containers cannot run arbitrary commands if they become compromised. The same way you would not want to give root access to any instance in your cloud, you do not want to give root access to any container in your fleet.</p><p><strong>How to do it:</strong> first, you need to make sure a non-root user exists. Add the following couple lines to your Dockerfile when building the image:</p><pre>FROM python:3.11<br><br>...<br># your logic<br>...<br><br><br>RUN groupadd -r your_username &amp;&amp; useradd --no-log-init -r -g your_username your_username<br>RUN mkdir -p /home/your_username &amp;&amp; chown -R your_username /home/your_username &amp;&amp; chown -R your_username $APP_DIR<br># assumes APP_DIR is where your application code is</pre><p>Then, you need to tell the container definition in the task definition to use that user when running the container:</p><pre>...<br>&quot;user&quot;: &quot;singularity&quot;,<br>...</pre><h4>3. SQS queues should have server-side encryption</h4><p>You will never be as secure as AWS. That being said, there are options in AWS to make your AWS cloud <em>more</em> secure than they are by default. One of those options is to tell SQS to be encrypted at rest.</p><p><strong>What is does</strong>: this makes sure that when the messages on the queue are stored (i.e. between the queue receiving the message and the queue providing the message to a consumer), they are encrypted at rest without any performance hit. Encryption at rest is an important checkbox for SOC2 compliance.</p><p><strong>How to do it:</strong> In the AWS console, find the queue you want to encrypt, click “edit”, then configure your queue to look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0DrutK21Y9NTGQhvcnSRQA.png" /><figcaption>KMS is the best option, but SSE is the simplest option that checks the “encrypted at rest” box</figcaption></figure><p>Alternatively, if you use terraform, <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue#server-side-encryption-sse">check the docs</a> for having your queue use SSE.</p><h4>4. S3 bucket server-side encryption should be enabled</h4><p>For the same reason you encrypt the SQS queues, you also want to enable encryption of the S3 buckets.</p><p><strong>What is does</strong>: this makes sure that the new objects in the bucket are encrypted before being stored on disk and decrypted when being retrieved from disk. Encryption at rest is an important checkbox for SOC2 compliance.</p><p><strong>How to do it:</strong> In the AWS console, find the bucket you want to have SSE. Click on the bucket name, click on the “Properties” tab, scroll down until you see “Default Encryption”, then click “Edit”. Then configure your settings to be the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gh4reQHIfCZKz_oQDGaKxg.png" /><figcaption>KMS is the best option, but SSE is the simplest option that checks the “encrypted at rest” box</figcaption></figure><p>Alternatively, if you use terraform, <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_server_side_encryption_configuration">check the docs</a> for having your bucket use SSE.</p><h4>5. RDS instances should be encrypted</h4><p>For the same reason you encrypt the SQS queues, you also want to enable encryption of the RDS instances.</p><p><strong>What is does</strong>: this makes sure that the contents of the disk on the RDS instance are encrypted before being written to disk and decrypted when being retrieved from disk. Encryption at rest is an important checkbox for SOC2 compliance.</p><p><strong>How to do it:</strong> Unfortunately, if you did not enable encryption when you created the RDS instance, there is no way to modify the instance to enable encryption. <a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/encrypt-an-existing-amazon-rds-for-postgresql-db-instance.html">AWS has a doc</a> on how to migrate to an encrypted database if you have an unencrypted instance running. If your application allows downtime, that’s a good path to use. Otherwise, you will have to do a dual-write methodology where your application code will need to write to both databases and switch to read from the encrypted one once it’s up to date.</p><p>In this post, we’ve covered 5 simple, but not obvious, security holes you can easily cover. We learned about these holes by undergoing a third party audit, but thought that everybody would benefit from knowing about them. Interested in more of what we’re doing? Think we’re amateurs and want to show us a thing or two? <a href="https://angel.co/company/singularity-energy/jobs">Stay up to date with when we’re hiring</a>⚡️</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a114e5f94b2f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/singularity-energy/5-simple-aws-security-tips-a114e5f94b2f">5 Simple AWS Security Tips</a> was originally published in <a href="https://medium.com/singularity-energy">Singularity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Elixir cluster on ECS]]></title>
            <link>https://ryan-baker.medium.com/building-an-elixir-cluster-on-ecs-d969faa442c2?source=rss-12d9f22091d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d969faa442c2</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[elixir]]></category>
            <dc:creator><![CDATA[Ryan Baker]]></dc:creator>
            <pubDate>Wed, 06 Jan 2021 13:47:17 GMT</pubDate>
            <atom:updated>2021-01-06T13:49:30.715Z</atom:updated>
            <content:encoded><![CDATA[<p>Last month I needed to create a cluster of elixir nodes. There seems to be extensive literature on this topic. Unfortunately, since the solution is extremely dependent on your infrastructure needs and cloud provider, none of them worked for me. Here are the constraints I had:</p><ul><li><strong>runs on AWS ECS:</strong> our entire infrastructure was using AWS ECS. I didn’t care to invest in switching to Kubernetes (though that’s more popular and arguably something I should learn — I’ll learn on my own another time).</li><li><strong>dynamic cluster size</strong>: I wanted to be able to spin up and down nodes in the cluster as needed. Many solutions suggested allocating specific servers and IP addresses and having each node in the cluster know about each other from static files commit to git.</li><li><strong>minimize changes</strong>: ideally, the changes to create a cluster would be mostly configuration. For this reason, articles that had lots of elixir code or involved creating new infrastructure like a Redis cache didn’t appeal to me.</li></ul><p>Before we talk about <em>how </em>to set up a cluster, let’s talk about <em>why </em>we would need to set up a cluster.</p><p><em>disclaimer: this article assumes knowledge of AWS ECS, AWS networking, and EPMD. If you’re not familiar with these topics, </em><a href="https://start.jcolemorrison.com/aws-vpc-core-concepts-analogy-guide/"><em>here</em></a><em> is a good primer on AWS networking; </em><a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html"><em>here</em></a><em> is a good primer on AWS ECS; and </em><a href="https://elixir-lang.org/getting-started/mix-otp/distributed-tasks.html"><em>here</em></a><em> is a primer on distributed Elixir with EPMD.</em></p><h3>Why do we need a cluster?</h3><p>Let’s imagine that we want to push live updates to our website users. One of the approaches is to use a websocket so that the site visitor stays connected to the server and the server can “push” updates as needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*YByzyKlQZxl5ZgyWKk7hiw.png" /></figure><p>This solution is pretty straightforward. Once a user connects to the server, the server can send updates to the user through the live connection. Here, we show some other service that’s telling the websocket server to send data to the user.</p><p>Eventually (hopefully), we’ll get enough users to need more than one server. When you need more than one server, you’ll likely use a load balancer to route requests equally to the different servers you have. Here is an example of what can go wrong in that situation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*I2cbrPGEYGN1ikpYSVJ6HA.png" /><figcaption>The servers can’t communicate about the changes if they don’t know about each other</figcaption></figure><p>While the load balancer is helpful in routing requests, here it’s actually hurt us. It prevented the “other service” from pushing data to the user. The solution here is not to remove or change the load balancer. If we want to send new data to users who subscribe to a feed of that data, it shouldn’t matter which server they’re connected to. The “other service” should be able to tell a single websocket server about the update and all users who care should get the new data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*L1btRakGx7VUiNhow5M34w.png" /><figcaption>The ideal state includes all websocket servers knowing about each other</figcaption></figure><p>For this reason, we need all the websocket servers to know about each other. When multiple servers communicate like this, it’s called a <em>cluster. </em>Each server in the cluster is called a <em>node</em>. In a cluster, each node would communicate changes to the other nodes in a cluster. In this way, the “other service” can push data to a user no matter which server either are connected to.</p><p>So the question is now:</p><h3><strong>How do we create a cluster?</strong></h3><p>Sure, we understand why we need one. But how do we actually make one? It can seem daunting, but I went through the trial and error of many different attempts so you don’t have to. Here are the different pieces and how they fit together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*szgIaHerGwYg36z3958xnQ.jpeg" /></figure><p>Since we’re using ECS, we’ll first need to create a task definition:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa7f8f61fd505ce4ff4237f888000f2b/href">https://medium.com/media/aa7f8f61fd505ce4ff4237f888000f2b/href</a></iframe><p>Then create a new service in your ECS cluster. Make it a FARGATE service and give it a few subnets in a VPC. Make sure that a new ENI is created for each task and the service discovery is enabled for the service based on the IP (since each ENI will get its own IP address).</p><p>Next, we need to have a way to start <a href="http://erlang.org/doc/man/epmd.html">EPMD</a> in a running server, give the server a node name, and connect it to another node (or nodes). I did this directly in elixir code with a Phoenix controller:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e62b94b9c6cace8c79ed553f3b0ec455/href">https://medium.com/media/e62b94b9c6cace8c79ed553f3b0ec455/href</a></iframe><p>When a new task launches, we can now:</p><ul><li>start EPMD</li><li>start the node</li><li>connect the node to the cluster</li></ul><p>Finally, we need something to do those three things (as they don’t happen automatically when a task is launched). I accomplished this with a small Python script that runs on AWS Lambda every few minutes. Here is the script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4eea289f7449bd0948228c03b48a99eb/href">https://medium.com/media/4eea289f7449bd0948228c03b48a99eb/href</a></iframe><p>And that’s it!</p><p>You now have an elixir cluster that will dynamically register and deregister nodes as you create or de-provision them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d969faa442c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>