<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Bluecore Engineering - Medium]]></title>
        <description><![CDATA[Go behind the scenes of the retail marketing platform. Interested in working with us? Check out our careers page here: https://www.bluecore.com/careers - Medium]]></description>
        <link>https://medium.com/bluecore-engineering?source=rss----4407d22b34ea---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Bluecore Engineering - Medium</title>
            <link>https://medium.com/bluecore-engineering?source=rss----4407d22b34ea---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:11:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/bluecore-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Adopting React Query at Bluecore]]></title>
            <link>https://medium.com/bluecore-engineering/adopting-react-query-at-bluecore-183c29bcdcd8?source=rss----4407d22b34ea---4</link>
            <guid isPermaLink="false">https://medium.com/p/183c29bcdcd8</guid>
            <category><![CDATA[react-query]]></category>
            <category><![CDATA[react-query-adoption]]></category>
            <category><![CDATA[react-query-migration]]></category>
            <category><![CDATA[redux]]></category>
            <category><![CDATA[bluecore]]></category>
            <dc:creator><![CDATA[Sarfaraz Ahmed]]></dc:creator>
            <pubDate>Thu, 29 Feb 2024 15:17:33 GMT</pubDate>
            <atom:updated>2024-02-29T15:17:32.868Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>React Query (now TanStack Query) is a popular data synchronization library for React based applications. We, at Bluecore, recently adopted the React Query library to manage our server state used by the client application. This blog post talks about how we approached our migration to React Query and provides some code examples to highlight a few important features/benefits of React Query.</p><h3>Why React Query?</h3><p>We had heard positive reviews about React Query across various tech communities for some time so there was absolutely no doubt on its usefulness, in general. The only question for us was to figure out if there were strong reasons to start using it here at Bluecore. We started by looking at our code and made some important observations:</p><ol><li>Quite a lot of our application components relied heavily on the Redux state selectors. These selectors were simply fetching the loading state of the API and the cached response.</li><li>Components relying on the Redux state were dispatching actions that would eventually call the required APIs and set the API loading state.</li><li>Some of the API responses and their loading states were being used in multiple components.</li></ol><p>With these observations, it was straightforward for us to see how ReactQuery could simplify our data synchronization process and reduce the (server-side) state management. The other nice thing about React Query is that it is perfectly capable of working alongside Redux where data fetching responsibilities are delegated to React Query and management of client state is delegated to Redux.</p><p>The decision to use React Query was also driven by another factor. It simplified the process of data fetching through automatically generated React Query clients. This was particularly beneficial during our transition from a monolithic to a microservices-based backend architecture. We adopted Protobuf for microservices communication, creating .proto files that could generate client and server code. These same .proto files could also generate OpenAPI specs, enabling the creation of React Query clients using <a href="https://orval.dev/">third-party libraries</a>. This automation provided developers with pre-filled data for data fetching calls, making it easier to integrate React Query into applications.</p><p>Traditionally, when using React Query, developers need to manually create clients for data fetching, specifying unique keys for cache identification and types for type safety. Managing these can be time-consuming and error-prone, as they must align with the server counterparts. However, with generated React Query clients, the keys and types are automatically handled. Developers can use the query clients without the need for extensive configuration, streamlining the adoption of React Query in applications.</p><h3>Planning the migration</h3><p>The first question in front of us was to decide on a migration approach. We considered both incremental and overhaul approaches and chose the incremental approach primarily because it was less risky and required less dev effort to start with. The choice between the two approaches depends on the specific needs of the project, the size of your codebase, and the team’s familiarity with both Redux and React Query. Incremental migration can be a safer choice for larger, more complex projects (like ours), while a complete overhaul may be more suitable for smaller projects or when you want to fully embrace React Query’s capabilities.</p><p><strong>Incremental Approach<br></strong>Pros:</p><ol><li><strong>Reduced Risk:</strong> Allows you to minimize risks by not disrupting the entire application at once. You can continue to use Redux in some parts of the application while transitioning to React Query in others, which is especially valuable for larger projects.</li><li><strong>Familiarity:</strong> Developers who are already comfortable with Redux can continue using it in existing code while learning and adopting React Query incrementally. This approach can ease the learning curve.</li><li><strong>Incremental Improvements:</strong> You can gradually improve the performance and maintainability of your application by tackling one feature or module at a time. This makes it easier to monitor the impact of each change.</li></ol><p>Cons:</p><ol><li><strong>Complex Integration: </strong>Managing both Redux and React Query concurrently can add complexity to your codebase, potentially leading to increased maintenance overhead.</li><li><strong>Potential Confusion:</strong> Developers might get confused when dealing with two different state management systems, especially when trying to synchronize data between them.</li></ol><p><strong>Overhaul Approach<br></strong>Pros:</p><ol><li><strong>Clean Slate: </strong>Starting with React Query from scratch provides a clean slate for your application, eliminating the need to maintain and refactor Redux code gradually.</li><li><strong>Full Benefits:</strong> You can fully leverage React Query’s features and benefits without any compromises or integration complexities.</li></ol><p>Cons:</p><ol><li><strong>High Initial Effort: </strong>A complete overhaul can be time-consuming and resource-intensive. One will need to rewrite and retest significant portions of your codebase.</li><li><strong>Learning Curve:</strong> Developers familiar with Redux may need time to learn React Query, potentially slowing down development initially.</li><li><strong>Risks:</strong> A full-scale migration carries more significant risks, as you’re replacing a core piece of your application’s architecture all at once, which could lead to unforeseen issues.</li><li><strong>Maintenance Overhead:</strong> Maintenance overhead of two applications as we still need to support the old application while development for the new application with react query is in progress.</li></ol><p>Like with any other migration, it was expected that we would run into some unforeseen issues at various stages of the migration. We brainstormed as a team to figure out a way that will help us to roll back React Query related changes without much risk if the need were to arise.</p><p>We opted not to make direct modifications to the existing files to be migrated but chose to create a parallel structure and update the new files as per the requirement. The strategy was to use <a href="https://martinfowler.com/articles/feature-toggles.html">feature toggles</a> to have control over the activation of this new path and when the feature stability had been proven then remove the old code. This decision was bolstered by the fact that the entirety of the code for the targeted route for migration resided neatly within specific files and directories (discovered during earlier code analysis), which simplified the cloning process and facilitated subsequent cleanup. This strategic move spared us from littering our current code with numerous conditional statements. Following this approach, we embarked on a systematic migration, focusing on one API at a time.</p><h3>React Query In Action</h3><p>React Query simplifies complex Redux state setups by automating caching, offering built-in state management, and providing tools for automatic invalidation and refetching. This leads to more concise and maintainable code while achieving the same data management goals. Now, let’s see some examples which demonstrate a React Query-based data fetching setup and other code simplifications, in comparison to Redux.</p><p><strong>Data Fetching Setup<br></strong>Let’s assume you have a Redux-based data fetching setup like this:</p><pre>// Redux reducer<br>const todoListReducer = (state = initialState, action) =&gt; {<br>  switch (action.type) {<br>    case &#39;FETCH_TODOS_REQUEST&#39;:<br>      return { ...state, isLoading: true };<br>    case &#39;FETCH_TODOS_SUCCESS&#39;:<br>      return { ...state, data: action.payload, isLoading: false };<br>    case &#39;FETCH_TODOS_FAILURE&#39;:<br>      return { ...state, isLoading: false };<br>    default:<br>      return state;<br>  }<br>};<br><br>// Redux thunk to perform async data fetching<br>const fetchTodosAsync = () =&gt; {<br>  return (dispatch) =&gt; {<br>    dispatch({ type: &#39;FETCH_TODOS_REQUEST&#39; });<br>    fetch(&#39;/todos&#39;)<br>      .then((response) =&gt; response.json())<br>      .then((data) =&gt; {<br>        dispatch({ type: &#39;FETCH_TODOS_SUCCESS&#39;, payload: data });<br>      })<br>      .catch((error) =&gt; {<br>        dispatch({ type: &#39;FETCH_TODOS_FAILURE&#39; });<br>      });<br>  };<br>};</pre><p>Now, let’s migrate this to React Query:</p><pre>import { useQuery } from &#39;react-query&#39;;<br><br>// Function to fetch todos (same as before)<br>const fetchTodos = async () =&gt; {<br>  const response = await fetch(&#39;/todos&#39;);<br>  const data = await response.json();<br>  return data;<br>};<br><br>// React Query usage in your component<br>const TodoListComponent = () =&gt; {<br>  const { data, isLoading, isError } = useQuery(&#39;todos&#39;, fetchTodos);<br>  if (isLoading) {<br>    return &lt;div&gt;Loading...&lt;/div&gt;;<br>  }<br>  if (isError) {<br>    return &lt;div&gt;Error fetching data&lt;/div&gt;;<br>  }<br><br>  // Render component based on &#39;data&#39; from React Query<br>  return data.map(<br>    (todo) =&gt; &lt;Todo <br>      key={todo.id}<br>      title={todo.title}<br>      description={todo.description}<br>   /&gt;)<br>};</pre><p>Here’s what we did:</p><ol><li>Imported useQuery from React Query to define data fetching logic directly within the component. To maintain a clean folder structure, one can also define the data fetching logic outside the component code.</li><li>Used the same fetchTodos function for data fetching as in the Redux example.</li><li>Used the useQuery(&#39;todos&#39;, fetchTodos) to declare a query named todos that fetches data using the fetchTodos function. React Query handles caching, loading, and error states automatically.</li><li>Checked the isLoading and isError states provided by React Query to render loading and error messages as needed.</li></ol><p>After successfully migrating the data fetching to React Query and verifying that it was working as expected, the rest of the Redux related code, including actions, reducers, and Redux store configuration that were specific to this data fetching operation could be removed.</p><p><strong>Automatic Caching<br></strong>With React Query, you don’t have to define a separate reducer, and actions, and manually manage caching. React Query automatically caches and manages data, thus reducing boilerplate code.</p><p><strong>Redux:</strong></p><pre>// Redux state<br>const initialState = {<br>  data: [],<br>  isLoading: false,<br>};<br><br>// Redux reducer<br>const todoListReducer = (state = initialState, action) =&gt; {<br>  switch (action.type) {<br>    case &#39;FETCH_TODOS_REQUEST&#39;:<br>      return { ...state, isLoading: true };<br>    case &#39;FETCH_TODOS_SUCCESS&#39;:<br>      return { ...state, data: action.payload, isLoading: false };<br>    case &#39;FETCH_TODOS_FAILURE&#39;:<br>      return { ...state, isLoading: false };<br>    default:<br>      return state;<br>  }<br>};</pre><p><strong>React Query:</strong></p><pre>import { useQuery } from &#39;react-query&#39;;<br><br>const fetchTodos = async () =&gt; {<br>  const response = await fetch(&#39;/todos&#39;);<br>  const jsonResponse = await response.json();<br>  return jsonResponse;<br>}<br><br>// Component using React Query<br>const TodoListComponent = () =&gt; {<br>  const { data, isLoading, isError } = useQuery(&#39;todos&#39;, fetchTodos);<br>  // No need to manually manage caching - React Query handles it<br>  if (isLoading) {<br>    return &#39;Fetching&#39;;<br>  }<br><br>  if (isError) {<br>    return &#39;Error&#39;;<br>  }<br><br>  return data.map(<br>    (todo) =&gt; &lt;Todo <br>      key={todo.id}<br>      title={todo.title}<br>      description={todo.description}<br>   /&gt;)<br>};</pre><p><strong>Built-in State Management<br></strong>React Query provides built-in state management through the useQuery hook. Thus, there is no longer a need to define Redux actions and reducers for data fetching states like loading or success; React Query simplifies this process.</p><p><strong>Redux:</strong></p><pre>// Redux actions and reducer (as shown in Step 1)<br>// Reducer<br>const todoListReducer = (state = initialState, action) =&gt; {<br>  switch (action.type) {<br>    case &#39;FETCH_TODOS_REQUEST&#39;:<br>      return { ...state, isLoading: true };<br>    case &#39;FETCH_TODOS_SUCCESS&#39;:<br>      return { ...state, data: action.payload, isLoading: false };<br>    case &#39;FETCH_TODOS_FAILURE&#39;:<br>      return { ...state, isLoading: false };<br>    default:<br>      return state;<br>  }<br>};<br><br>// Action<br>const fetchTodos = async (dispatch) =&gt; {<br>  try {<br>    dispatch({type: &#39;FETCH_TODOS_REQUEST&#39;});<br>    const response = await fetch(&#39;/todos&#39;);<br>    dispatch({type: &#39;FETCH_TODOS_SUCCESS&#39;, payload: response});<br>  } catch (e) {<br>    dispatch({type: &#39;FETCH_TODOS_FAILURE&#39;})<br>  }<br>}<br><br>import {useEffect} from &#39;react&#39;;<br>import {useSelector, useDispatch} from &#39;react-redux&#39;;<br><br>// Component<br>const TodoListComponentWithRedux= () =&gt; {<br>  const data = useSelector((state) =&gt; state.data);<br>  const isLoading = useSelector((state) =&gt; state.isLoading);<br>  const dispatch = useDispatch();<br><br>  // Trigger data fetching when the component mounts<br>  useEffect(() =&gt; {<br>    dispatch(fetchTodos());<br>  }, []);<br><br>  // Render component based on &#39;data&#39; and &#39;isLoading&#39; from Redux state<br>  if (isLoading) {<br>    return &#39;Fetching&#39;;<br>  }<br><br>  return data.map(<br>    (todo) =&gt; &lt;Todo <br>      key={todo.id}<br>      title={todo.title}<br>      description={todo.description}<br>   /&gt;)<br>};</pre><p><strong>React Query:</strong></p><pre>import { useQuery } from &#39;react-query&#39;;<br><br>const fetchTodos = async () =&gt; {<br>  const response = await fetch(&#39;/todos&#39;);<br>  const jsonResponse = await response.json();<br>  return jsonResponse;<br>}<br><br>// Component using React Query<br>const TodoListComponentWithReactQuery = () =&gt; {<br>  const { data, isLoading } = useQuery(&#39;todos&#39;, fetchTodos);<br><br>  // Render component based on &#39;data&#39; and &#39;isLoading&#39; from React Query<br>  if (isLoading) {<br>    return &#39;Fetching&#39;;<br>  }<br><br>  return data.map(<br>    (todo) =&gt; &lt;Todo <br>      key={todo.id}<br>      title={todo.title}<br>      description={todo.description}<br>   /&gt;)<br>};</pre><p><strong>Automatic Invalidation and Refetching<br></strong>React Query simplifies cache invalidation and refetching by providing tools like useMutation and queryCache , making it easier to handle data synchronization when data needs to be refreshed.</p><p><strong>Redux:</strong></p><pre>// Redux actions and reducer (as shown in Step 1)<br>// Reducer<br>const todoListReducer = (state = initialState, action) =&gt; {<br>  switch (action.type) {<br>    case &#39;FETCH_TODOS_REQUEST&#39;:<br>      return { ...state, isLoading: true };<br>    case &#39;FETCH_TODOS_SUCCESS&#39;:<br>      return { ...state, data: action.payload, isLoading: false };<br>    case &#39;FETCH_TODOS_FAILURE&#39;:<br>      return { ...state, isLoading: false };<br>    default:<br>      return state;<br>  }<br>};<br><br>// Action<br>const fetchTodos = async (dispatch) =&gt; {<br>  try {<br>    dispatch({type: &#39;FETCH_TODOS_REQUEST&#39;});<br>    const response = await fetch(&#39;/todos&#39;);<br>    dispatch({type: &#39;FETCH_TODOS_SUCCESS&#39;, payload: response});<br>  } catch (e) {<br>    dispatch({type: &#39;FETCH_TODOS_FAILURE&#39;})<br>  }<br>}<br><br>import {useEffect, Fragment} from &#39;react&#39;;<br>import {useSelector, useDispatch} from &#39;react-redux&#39;;<br><br>// Component<br>const TodoListComponentWithRedux= () =&gt; {<br>  const data = useSelector((state) =&gt; state.data);<br>  const isLoading = useSelector((state) =&gt; state.isLoading);<br>  const dispatch = useDispatch();<br><br>  // Trigger data fetching when the component mounts<br>  useEffect(() =&gt; {<br>    dispatch(fetchTodos());<br>  }, []);<br><br>  // Manually trigger a refetch<br>  const handleRefresh = () =&gt; {<br>    dispatch(fetchTodos());<br>  };<br><br>  if (isLoading) {<br>    return &#39;Fetching&#39;;<br>  }<br><br>  // Render component with a refresh button<br>  return (<br>    &lt;Fragment&gt;<br>      {<br>        data.map(<br>          (todo) =&gt; &lt;Todo <br>            key={todo.id}<br>            title={todo.title}<br>            description={todo.description}<br>        /&gt;)<br>      }<br>      &lt;button onClick={handleRefresh}&gt;Refresh&lt;/button&gt;<br>    &lt;/Fragment&gt;<br>  )<br>};</pre><p><strong>React Query:</strong></p><pre>import {Fragment} from &#39;react&#39;;<br>import { useQuery, useMutation, queryCache } from &#39;react-query&#39;;<br><br>const fetchTodos = async () =&gt; {<br>  const response = await fetch(&#39;/todos&#39;);<br>  const jsonResponse = await response.json();<br>  return jsonResponse;<br>}<br><br>// Component using React Query<br>const TodoListComponentWithReactQuery = () =&gt; {<br>  const { data, isLoading } = useQuery(&#39;todos&#39;, fetchTodos);<br>  const refreshData = useMutation(() =&gt; {<br>    // Manually trigger a refetch<br>    return queryCache.invalidateQueries(&#39;todos&#39;);<br>  });<br><br>  if (isLoading) {<br>    return &#39;Fetching&#39;;<br>  }<br><br>  // Render component with a refresh button<br>  return (<br>    &lt;Fragment&gt;<br>      {<br>        data.map(<br>          (todo) =&gt; &lt;Todo <br>            key={todo.id}<br>            title={todo.title}<br>            description={todo.description}<br>        /&gt;)<br>      }<br>      &lt;button onClick={refreshData}&gt;Refresh&lt;/button&gt;<br>    &lt;/Fragment&gt;<br>  )<br>};</pre><p><strong>Unit Testing<br></strong>React Query can significantly change the unit testing process for applications by simplifying data fetching and state management. Here’s a detailed before-and-after migration example to illustrate these changes.</p><p><strong>Redux:<br></strong>Suppose there is a component that fetches and displays a list of todos using Redux for state management. Testing this component typically involves mocking Redux actions, and reducers, and handling asynchronous behavior:</p><pre>// Redux reducer<br>const todosReducer = (state = { data: [], isLoading: false }, action) =&gt; {<br>  switch (action.type) {<br>    case &#39;FETCH_TODOS_REQUEST&#39;:<br>      return { ...state, isLoading: true };<br>    case &#39;FETCH_TODOS_SUCCESS&#39;:<br>      return { ...state, data: action.payload, isLoading: false };<br>    case &#39;FETCH_TODOS_FAILURE&#39;:<br>      return { ...state, isLoading: false };<br>    default:<br>      return state;<br>  }<br>};<br><br>// TodoListComponentWithRedux.test.js<br>import React from &#39;react&#39;;<br>import { render, screen } from &#39;@testing-library/react&#39;;<br>import { Provider } from &#39;react-redux&#39;;<br>import configureStore from &#39;redux-mock-store&#39;;<br>import TodoListComponentWithRedux from &#39;./TodoListComponentWithRedux&#39;;<br><br>// Mock the fetch function<br>jest.mock(&#39;fetch&#39;, () =&gt; ([<br>  { id: 1, title: &#39;Item 1&#39;, description: &#39;Item 1 Description&#39; },<br>  { id: 2, title: &#39;Item 2&#39;, description: &#39;Item 2 Description&#39; }<br>]));<br><br>// Create a mock Redux store<br>const mockStore = configureStore([]);<br>const store = mockStore({<br>  data: [],<br>  isLoading: false,<br>});<br><br>beforeEach(() =&gt; {<br>  // Clear mock calls and set initial store state before each test<br>  store.clearActions();<br>});<br><br>// Testing the component<br>it(&#39;renders todos correctly&#39;, async () =&gt; {<br>  render(<br>    &lt;Provider store={store}&gt;<br>      &lt;TodoListComponentWithRedux /&gt;<br>    &lt;/Provider&gt;<br>   );<br><br>  // Assertions based on your component&#39;s behavior<br>  expect(screen.getByText(&#39;Item 1&#39;)).toBeInTheDocument();<br>  expect(screen.getByText(&#39;Item 2&#39;)).toBeInTheDocument();<br><br>  // Expect that the fetchData action was dispatched once<br>  expect(store.getActions()).toHaveLength(2);<br>  const expectedPayload = [<br>    { type: &#39;FETCH_TODOS_REQUEST&#39; },<br>    { type: &#39;FETCH_TODOS_SUCCESS&#39;, payload: [<br>      { id: 1, title: &#39;Item 1&#39;, description: &#39;Item 1 Description&#39; },<br>      { id: 2, title: &#39;Item 2&#39;, description: &#39;Item 2 Description&#39; }<br>    ]}<br>  ]<br>  expect(store.getActions()).toEqual(expectedPayload)<br>});</pre><p><strong>React Query:<br></strong>Migrating to React Query simplifies the testing process by reducing the need to mock actions, reducers, and Redux boilerplate. Here’s how the testing process changes:</p><pre>import { render, screen } from &#39;@testing-library/react&#39;;<br>import { QueryClient, QueryClientProvider } from &#39;react-query&#39;;<br>import TodoListComponentWithReactQuery from &#39;./TodoListComponentWithReactQuery&#39;;<br><br>const queryClient = new QueryClient();<br>it(&#39;renders todos correctly&#39;, async () =&gt; {<br>  render(<br>    &lt;QueryClientProvider client={queryClient}&gt;<br>      &lt;TodoListComponentWithReactQuery /&gt;<br>    &lt;/QueryClientProvider&gt;<br>  );<br><br>  // Assert that the component renders todos correctly<br>  const todoElement = await screen.findByText(&#39;Example Todo&#39;);<br>  expect(todoElement).toBeInTheDocument();<br>});</pre><p>Key Changes and Benefits:</p><ol><li><strong>Simplified Testing Setup:</strong> Before React Query, you had to set up complex Redux mocks, actions, and reducers. With React Query, the testing setup becomes much cleaner, focusing on the core functionality of the component.</li><li><strong>Focus on Component Behaviour:</strong> React Query allows you to concentrate on testing the behavior of the component rather than managing the intricacies of Redux state management.</li><li><strong>Realistic Testing:</strong> Tests using React Query more closely resemble real-world scenarios because you’re using React Query’s actual data fetching logic. This leads to more realistic and reliable test results.</li><li><strong>Efficiency and Readability:</strong> React Query tests are often more efficient to write and easier to read. They closely resemble how components interact with data in the application, making it simpler for other developers to understand and maintain the tests.</li><li><strong>Testing Loading and Error States:</strong> React Query simplifies testing loading and error states as well. You can wait for data to load or simulate error responses easily, making it straightforward to validate how components handle these states.</li></ol><p>Overall, React Query enhances the testing and QA process by simplifying data fetching and state management, resulting in cleaner, more focused, and more realistic tests. This, in turn, improves the efficiency and reliability of testing and QA efforts.</p><h3>Conclusion</h3><p>React Query provides a modern alternative to do efficient data management in React based applications. It helped us simplify our code, made it efficient, and improved the developer/QA experience. We recommend migrating to React Query, especially for cases where significant data fetching is involved. It is advisable to look at the existing code/implementation in detail before undertaking the migration. Doing so, not only validates some of the assumptions one might have about the code but can also reveal things that you can use to plan your migration better. React Query and Redux work well together and can be used to get the best of both worlds — easier data handling and client state management while also benefiting from the reduction of boilerplate code for data fetching in Redux.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=183c29bcdcd8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluecore-engineering/adopting-react-query-at-bluecore-183c29bcdcd8">Adopting React Query at Bluecore</a> was originally published in <a href="https://medium.com/bluecore-engineering">Bluecore Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simplified API Test Automation —  Introducing Block Framework]]></title>
            <link>https://medium.com/bluecore-engineering/simplified-api-test-automation-introducing-block-framework-b1a1e8b54b5c?source=rss----4407d22b34ea---4</link>
            <guid isPermaLink="false">https://medium.com/p/b1a1e8b54b5c</guid>
            <category><![CDATA[bluecore]]></category>
            <dc:creator><![CDATA[Karan Agarwal]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 21:21:18 GMT</pubDate>
            <atom:updated>2024-01-03T21:21:18.498Z</atom:updated>
            <content:encoded><![CDATA[<h3>Simplified API Test Automation — Introducing Block Framework</h3><blockquote><em>Real-world testing challenges go beyond technical complexities. We must innovate and strategize to deliver optimal results. It’s not just about adopting new tools, but fostering a mindset that welcomes creative solutions.</em></blockquote><p>API test automation might seem like a walk in the park for those with experience. After all, there are plenty of tools available. Creating custom code for API checks is often straightforward. This isn’t always the case, though. Sometimes, the established methods just don’t fit with the unique demands of a project, or the bigger picture in the business world.</p><p>At Bluecore, we’ve noticed that the usual approaches can be quite limiting, particularly when scaling up or integrating smoothly into existing systems in preview of dynamic requirements. We think outside the box by taking inspiration from LEGO® blocks — known for their simple yet flexible nature, we aim to bring these qualities into our automation framework. This article explores the<strong> </strong>tangible hurdles one may encounter during API test automation, and puts forth a solution to mitigate them effectively.</p><h4><strong>Available Strategies to Approach API Testing Automation Projects</strong></h4><p>How do test automation leaders typically respond when faced with a new API testing project? Generally, they consider two main strategies:</p><ul><li><strong>Utilizing Industry Standard Tools</strong>: This involves making use of well-recognized and adaptable tools, like Postman. Such tools are reliable, well-supported, and well-documented, making development and troubleshooting smoother. Plus, they’re easy to learn and use, which speeds up onboarding and script writing. This strategy is usually adopted if the business approves and there are no security or regulatory concerns with the tool.</li><li><strong>Developing an In-House Framework</strong>: This strategy involves leveraging open-source technologies and frameworks to create a customized solution. It is particularly favored when there is a need for customization and scalability, especially if there are budget constraints or a preference for in-house development.</li></ul><p>As we delve into the above strategies at Bluecore, we discuss the actual requirements, which might echo some challenges you may encounter. Below is a sample conversation:</p><p><strong><em>Program Manager</em></strong><em>:</em> We’re upgrading to the latest Python version and need to test over 100+ APIs. We need to cover positive and negative test cases for each API. We’re thinking about a phased approach during development sprints. Your thoughts?</p><p><strong><em>QA Lead:</em></strong> A phased approach is wise. For testing, are we considering standard tools or something more tailored?</p><p><strong><em>Program Manager:</em></strong> That’s the question. Standard tools are straightforward, but can they handle this scale?</p><p><strong><em>QA Lead:</em></strong> Postman could manage the basics, but for this scale, a custom framework might offer the flexibility we need. It could be more scalable.</p><p><strong><em>Program Manager:</em></strong> Interesting. And how about managing multiple test environments? They need to be auto-purged regularly.</p><p><strong><em>QA Lead:</em></strong> A custom framework could integrate environment management seamlessly. We’d have control over data and environment specifics.</p><p><strong><em>Program Manager:</em></strong> And there’s the challenge of testing end-to-end workflows, which includes handling API logs in BigQuery and/or in-memory objects.</p><p><strong><em>QA Lead:</em></strong> That’s a complex requirement. Licensed tools might fall short there. Our custom solution could be designed to handle such intricacies.</p><p><strong><em>Program Manager:</em></strong> Okay. Let’s also think about the team. We have two automation testers and two functional testers. One is keen on API testing.</p><p><strong>QA Lead:</strong> We have a solid foundation. Let’s build a team strategy that leverages their strengths and fosters skill development.</p><p>Standard approaches to API automation, despite being tried-and-tested, sometimes fall short in offering the desired level of flexibility. This leads to obstacles in achieving seamless integration and scalability. Using our example project from above, the usual methods (such as making separate libraries for each API client, or setting up integration tests with pytest) may be slow and difficult, especially with the given challenges:</p><ul><li><strong>Scalability</strong>: test management of over 100 APIs during a backend upgrade to the latest Python version.</li><li><strong>Customization</strong>: the need for a tailored approach, accommodating phased automation and test planning during development sprints.</li><li><strong>Integration</strong>: seamless integration into existing systems across multiple, dynamically changing test environments.</li><li><strong>Data Management</strong>: efficient management if data in a manner that is both environment- and data-agnostic, while also handling in-memory objects and database interactions.</li><li><strong>Third-Party Integrations</strong>: handling interactions with large-scale databases like BigQuery, and in-memory objects like Redis or Spanner.</li><li><strong>Resource Limitations</strong>: a compact team with specialized skills, not all equipped with automation testing expertise.</li><li><strong>Skill Development</strong>: fostering skill development for team members eager to expand their knowledge in API testing.</li></ul><h4><strong>Reinventing Strategies with the </strong>LEGO® <strong>Block Abstract Concept</strong></h4><figure><img alt="An image depicting LEGO blocks arranged to represent abstract concepts connected as single, reusable units within a framework." src="https://cdn-images-1.medium.com/max/995/1*4XLGfb67o47IdHB9EO9cSg.png" /><figcaption>The modularity of LEGO® blocks allows for adaptable and flexible frameworks.</figcaption></figure><p>Faced with these challenges, we sought a more adaptable solution, finding inspiration in LEGO® blocks. These toys, known for their simplicity and flexibility, mirrors what we need in our projects — a method that is both straightforward and adaptable.</p><p>Like creating a structure with LEGO®, the Block Framework allows us to build our project with individual, standardized units. These units can easily fit together, making the whole process more straightforward without sacrificing any of the details.</p><h4>What are Blocks in the Block Framework?</h4><p>Okay, here’s the scoop — in the Block Framework, a “block” is like a little testing superhero with its own superpower of functionality. These blocks are capable of executing specific actions, such as initiating API requests, validating responses, and even interacting with other blocks seamlessly.</p><p>Think of backend services like RestRequest and GRPCRequest as simple, interchangeable blocks, each with standard input and output features. This setup makes orchestrating complex workflows easy, allowing the blocks to work together seamlessly, and manage the intricacies of API testing effectively.</p><h4>Block Configuration: Everything You Need to Write Test Cases</h4><p>The Block Configuration serves as your fundamental toolkit in the testing process, where you are in charge of shaping the rules and structure. It facilitates the organization of individual blocks, each designated with a unique role, enabling you to tailor your tests according to your precise preferences.</p><p>Here is an example to illustrate the functionality of a standard RESTRequest block:</p><figure><img alt="A diagram and code block inclusive of a LEGO-style diagram that demonstrates the inputs from the endpoint through validations and outputs." src="https://cdn-images-1.medium.com/max/1024/0*FQ5usCFhlF_K8GWp" /><figcaption>The Block Configuration demonstrates the metadata, input values, the validations necessary for the test, and the output from the test completion</figcaption></figure><ul><li><strong>Metadata</strong>: this section assists in defining the goal of each block. With fields like block name, description, and other attributes, it enables the customization of block execution. It also includes tagging blocks facing application issues and linking them to the respective Jira tasks, facilitating a comprehensive and customized report.</li><li><strong>Input</strong>: this section houses all the vital configurations necessary for the block to start processing. The details within this section will fluctuate based upon the block type.</li><li><strong>Output Configurations</strong>: this facilitates text and response attribute validations to streamline the verification of crucial elements and response attributes, respectively. Additionally, it provides a secure space for storing key data elements, enhancing workflow coherence.</li></ul><h4>Block by Block: Enhancing Efficiency with Scriptless Testing</h4><p>After shaping these components into robust classes, the aim was to foster straightforward collaboration between blocks using uncomplicated, light configurations. To implement this, a data-driven strategy was adopted, leveraging the light and readable JSON to orchestrate block configurations, simplifying test case management and maintenance, particularly with git repositories.</p><figure><img alt="Multiple code blocks paired with a diagram demonstrate an init section, followed by the building blocks to create a combined multi-block end-to-end test." src="https://cdn-images-1.medium.com/max/1024/0*OnKBE29B_Xzks-gf" /><figcaption>Multiple blocks combine together to generate a JSON end-to-end test</figcaption></figure><p>Each JSON file configuration functions as a distinct test case, encompassing one or more building blocks that serve as test steps. These blocks are designed to address specific business objectives. As a result, we optimize the testing process while maintaining a level of flexibility and user-friendliness.</p><p>This progress has made the technical aspects simpler, allowing for a smoother testing process and increased confidence in delegating API testing to functional testers. This approach helps broaden our automation initiatives, with functional testers leading the way in an adaptable and scalable testing framework.</p><h4>Addressing Data and Environment Dependencies</h4><p>To create a testing environment that is unaffected by particular data or settings, we established a central test data management system. This system leverages dynamic data pulled from static files, ensuring uninterrupted operation — even when environmental data is deleted. Whether these central data files are generated manually or automatically, the integrity of the remaining workflow stays preserved. This is a common pain point for many test teams.</p><p>As the framework developed, we employed it to automatically generate these data files, enhancing efficiency and cohesiveness. Utilizing the same JSON configuration files, we documented output data directly within these files, which were then accessed by multiple test cases until deletion.</p><h3>Putting it all together</h3><p>As illustrated in the diagram below, the framework integrates several vital components, each with specific roles, to create a streamlined testing process:</p><figure><img alt="A diagram demonstrates which engineer type manages test configurations and cases, and which engineer maintains the block factory." src="https://cdn-images-1.medium.com/max/1024/0*7B-0I64nUPuQxNXW" /><figcaption>The Block Framework can be easily integrated and managed by large test teams and across multiple facets of work.</figcaption></figure><ul><li><strong>Test Driver:</strong> this is the <strong>front gateway</strong> for integrating JSON files into the framework. The driver excels at managing either individual or numerous test cases at the same time, promoting streamlined test execution.</li><li><strong>Data Configuration</strong>: a centralized <strong>data repository</strong> harboring both the test cases and the associated test data. Crafted and managed by test engineers, this module dictates the test workflows and permutations, ensuring seamless operation.</li><li><strong>Block Executor</strong>: this is the framework backbone that synthesizes data configurations with JSON templates, orchestrating the workflow according to the specifications laid out in each block configuration. It acts as a mediator, ensuring harmonious interaction between blocks, while managing data configurations seamlessly from INI (initialization) files to JSON configurations.</li><li><strong>Block Factory</strong>: This outlines the core capabilities in the form of various functional blocks like REST, gRPC, Spanner, etc. It provides a JSON interface for necessary actions within each component, thereby facilitating smooth functionality</li><li><strong>Common Utils</strong>: These are <strong>reusable</strong> libraries that sustain the framework, encompassing shared logic and functionalities like assertions, data handling, and logging and reporting mechanisms. As a result functionality and efficiency of the entire configuration is enhanced.</li></ul><p>By integrating these components seamlessly, we lay the groundwork for a simple yet robust testing framework, promising efficiency and adaptability at every stage of the testing process.</p><h3>Meeting Expectations: A Seamless Delivery</h3><p>In just a single quarter, our team integrated over 400 test cases using the Block Framework. What’s remarkable isn’t just the volume, but the approach. These tests form part of our progressive automation strategy: dynamic, adaptable, and requiring minimal maintenance.</p><figure><img alt="A line chart indicates test case count (gradually increasing over time), the functional effort involved (stays flat throughout), while also showing the automation effort involved (a gradually decreasing effort per sprint over time)" src="https://cdn-images-1.medium.com/max/770/0*2-cEpt-6OjaNA204" /><figcaption>With an initial high investment of automation works, the Block Framework pays off over time for efficiency of the test team, particularly when we start to add a large amount of test cases.</figcaption></figure><p>Usually, handling this many tests would be a tough job. Thanks to the Block Framework, we’ve made this process simpler. We are reducing the time and effort needed for automation gradually, making it easier for our functional team to expand the automation suites as needed.</p><h3>Navigating Challenges: The Bluecore Expedition</h3><p>As the sample project phase wraps up, it’s clear that the new method we used was quite successful. The new approach significantly streamlined the team’s workflow. We were able to tackle complex tasks without getting tangled in technical complexities. Moreover, the automation framework could scale independently, reducing the constant need for oversight from automation engineers.</p><p>Our functional testers adapted quickly, taking charge of both creating and executing test cases. This not only made processes more efficient, but also provided a rich learning experience for the team. It empowered the team to independently familiarize themselves with the nuances of the automation framework.</p><p>The team is now well equipped to take full responsibility for maintaining the framework. Initially developed for this project, the positive outcome has encouraged us to consider using this approach in other areas as well.</p><h3>Learning Curve and Anticipated Challenges</h3><p>The journey towards adopting the Block Framework comes with its fair share of hurdles that may impact the ease of transition and adoption. Key anticipated challenges include:</p><p><strong>Adoption of JSON: </strong>the reliance on JSON for configuration and test case definitions may not be intuitive for everyone, especially for those primarily acquainted with scripting or programming.</p><p><strong>Codeless Test Cases: </strong>seasoned testers might find the shift to a codeless setup mundane, affecting their enthusiasm or ownership. However, this framework is designed to streamline testing, allowing for enhanced coverage with reduced effort. This frees up time to quickly tackle Block Framework tasks, and move on to additional framework design needs.</p><p><strong>Adapting to Block Structure: </strong>mastering the block structure, in defining and configuring blocks for varied testing scenarios, is crucial for leveraging the framework effectively. We do find there is a small learning curve involved.</p><p><strong>Transitioning Mindset: </strong>a shift in mindset is required to transition from traditional scripting to a structured, modular block-based approach. This may be challenging for some team members.</p><p>Additionally, if there are minimal APIs to test, using standard tools (like Postman) might be more practical. It’s essential to weigh the project’s unique needs and context when considering adopting this framework.</p><p>Bluecore’s adoption of the Block Framework, while challenging, was strategically aligned with our long-term goals and scalability needs. Block Framework makes these challenges relatively minor in the broader context. Overcoming these challenges truly depend on the requirements at hand. We made the necessary trade-offs, keeping in mind the ultimate efficiency and adaptability benefits that this framework offers.</p><h3>Reflecting on the Approach</h3><p>In conclusion, real-world challenges in any testing project go beyond technical complexities. They prompt us to innovate and strategize effectively, and deliver optimal results under given circumstances. The journey isn’t merely about adopting new tools, but fostering a mindset that welcomes creative solutions.</p><p>As shown, the Block Framework has proven to be a great problem-solver for our API testing needs. It shows us the importance of thinking strategically, encouraging us to keep tweaking our methods, to move swiftly and wisely in the ever-changing world of automation testing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1a1e8b54b5c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluecore-engineering/simplified-api-test-automation-introducing-block-framework-b1a1e8b54b5c">Simplified API Test Automation —  Introducing Block Framework</a> was originally published in <a href="https://medium.com/bluecore-engineering">Bluecore Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Load Testing a Recommendation Service]]></title>
            <link>https://medium.com/bluecore-engineering/load-testing-a-recommendation-service-971198b09c6a?source=rss----4407d22b34ea---4</link>
            <guid isPermaLink="false">https://medium.com/p/971198b09c6a</guid>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[recommendations]]></category>
            <category><![CDATA[bluecore]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Tiago Alves]]></dc:creator>
            <pubDate>Tue, 24 Jan 2023 14:22:25 GMT</pubDate>
            <atom:updated>2023-01-24T14:22:25.484Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u3u3oQQ7cUzpaA6v" /><figcaption>“Cargo ship unloading colorful containers in port”, Kelly via <a href="https://www.pexels.com/photo/cargo-ship-unloading-colorful-containers-in-port-6595779/">Pexels</a></figcaption></figure><p>It’s that time of the year again. Great deals everywhere, everybody wants to spend their savings, and all retailers can think about is letting shoppers know that they have the best prices of the year. That also means a huge volume of emails being sent every hour, which means a huge volume of requests to the recommendation service for our team. We see a 77% increase in volume during the period of Black Friday and Cyber Monday (BFCM) compared to a regular week, and this year wasn’t any different.</p><p>Every personalized email that is sent generates multiple requests to the recommendation service in order to get the recommended products for each shopper. To handle these requests, the service is based in Go and most of the time it will fetch precomputed recommendations according to the parameters, either from Bigtable or ElasticSearch, with an LRU cache for ElasticSearch requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*dGNgTpOJhfoI9CZEGbLbhw.png" /><figcaption>Figure 1 — The Recommendation Service will fetch recommendations either from Bigtable or ElasticSearch, according to the requested type.</figcaption></figure><p>With that in mind, our main concern is “<em>How many requests can we make to Bigtable concurrently? What about ElasticSearch?</em>” We actually started preparing for BFCM around August, during which we run rigorous tests and identify the biggest bottlenecks and pain points in our system.</p><h3>The Experiments</h3><h3>Recommendation Service Load Test Tool</h3><p>For this year’s load testing we used the same tool that was used last year with a few minor updates. The tool is composed of two parts: a server running in the same cluster as the recommendation service, and a client running on our local machines.</p><p>The client part functions as a driver, giving the server instructions and settings on which requests we want to make, at what rate, how many concurrent requests should be made at the same time, etc. The server, given the instructions from the client, will download a file with the requests and send them to the recommendation service, following the settings as much as possible. Once the test is over, the server returns the stats and the client compiles them, generating user-friendly logs and histograms. We also apply a stat_fetcher tool to monitor the Prometheus metrics of the service during the test period.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5fOi0s-x4C-hyc2l" /><figcaption>Figure 2 — Multiple tools are combined around the recommendation service to run the tests.</figcaption></figure><p>With the load test tool ready, the next step was to define which experiments we wanted to run, and what the input and expected results would look like for them. An experiment can be defined by the following attributes:</p><ol><li>A set of requests (or test file)</li><li>How long it should take</li><li>Number of requests per second</li><li>Number of requesting instances (load test tool servers)</li><li>Number of threads per server</li><li>Expected number of requests (#2 x #3 x #4)</li><li>Actual number of requests</li><li>p99 of the latency</li></ol><p>We can divide the experiments in two groups: Bigtable and ElasticSearch. There’s also one extra experiment with mixed requests, hitting both data sources. For Bigtable we defined the attributes to test with:</p><ol><li>64 distinct requests to use as a baseline on a simple scenario.</li><li>Over 200,000 distinct requests to break internal caches and simulate a close to reality scenario.</li><li>Complex requests only (requests that have a lot of filters or require a lot of products) to see how the system responds under a lot of stress.</li><li>The replication of a real campaign that was underperforming to better understand why it was underperforming.</li></ol><p>For ElasticSearch, the following tests were defined:</p><ol><li>A single unique request to use as a baseline.</li><li>Many simple requests (requests that don’t have a lot of filters or only have common filters), as they are the most common type.</li><li>Over 200,000 distinct requests to see how the system responds with low cache hit rates.</li></ol><p>With all the files created and experiments defined, we are ready to start hitting the recommendation service with all we got. Our goal is to reach at least 50,000 requests per second with a p99 latency of 200ms.</p><h3>Running Gamedays</h3><p>Before starting to run the load tests, we upscaled the systems to mimic traffic levels close to what we would expect to get during BFCM. There were between 80–120 instances of the recommendation service running and around 40 load-tester instances. We scheduled the gamedays for times when the regular load was low to avoid interfering with production traffic.</p><p>We used a separated <em>shadow</em> deployment of the recommendation service that was identical to production but with a different endpoint. This ensured that we wouldn’t affect regular requests by spamming the service and breaking the cache. However, we still needed to be careful because we were hitting the same Bigtable and ElasticSearch instances.</p><p>The first attempts were really frustrating. There were too many knobs to turn. Sometimes the driver file would break the test because of unexpected errors and even when we were able to run a complete experiment, it would not reach the expected number of requests. A few retries, pull requests, and resource adjustments later, we started to see some real traffic going into the recommendation service shadow instances.</p><p>We reserved one-hour for each gameday session, just to make sure we had enough time to not just run the experiment but also to downscale systems after the experiment was complete. Every experiment ran for a short period of time, usually around five minutes. In some special cases, we let it run for 15 minutes to get enough traffic to all the pods and to warm up all the local caches. During that session, at least two engineers would stay connected in a virtual room: one of them would be responsible for running the client load tester and collecting the results, while the others would monitor the system metrics to ensure everything was going as expected.</p><h3>What We Found</h3><p>Once we got over those initial hurdles, the experiments were really successful. Our target was more than<em> 50,000</em> requests/second and less than 200ms p99 latency. The simple Bigtable requests experiments showed very fast results reaching up to <em>120,835</em> requests/second with a p99 latency of 120ms. The simple ElasticSearch requests were also very successful, going up to <em>52,399 </em>requests/second with a p99 latency of 100ms<em> </em>while the test with many requests was even faster, getting <em>112,149 </em>requests per second with 200ms of p99 latency!</p><p>However, the complex Bigtable experiments didn’t go well. Just like above, our target for these experiments was more than<em> 50,000</em> requests/second and less than 200ms p99 latency. The experiment with over 200,000 distinct requests got at most <em>14,010 </em>requests per second with 1400ms p99 latency. The complex requests test was a little bit better, with 23,938 requests/second and 510ms latency, but that wasn’t good enough. Finally, the real campaign recreation didn’t give great results either, reaching 16,831 requests/second and 410ms p99 latency. During these tests we also noticed some errors reported by Bigtable, meaning that there was something going wrong with these requests.</p><p>The results are compiled in Table 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*PfjJ5r2wBAOuNEclB6RYWQ.png" /><figcaption>Table 1 — Experiment results in requests per second and p99 latency.</figcaption></figure><p>From the first rounds of the experiments, we were able to conclude that we were ready for BFCM on the ElasticSearch side, mainly because of our caching system, but not quite ready when it comes to Bigtable requests. So we shifted our focus to figuring out what went wrong in these experiments.</p><h3>Fixing the Bottleneck</h3><p>To understand what was wrong in our requests, we analyzed the queries that were being made. We quickly noticed that there was a correlation between one type of recommendation, co-recommendations (co-recs), and the errors in Bigtable. Co-recs are recommendations based on the interaction between shoppers and products, such as products viewed together or products bought together. Even though this is a personalized recommendation type, it takes products as input instead of shoppers’ identifiers, making it slightly different from the other personalized recommendation types. We then separated the requests of this recommendation type and reran the experiments, confirming that the problem was indeed in these queries.</p><p>After more investigation, we discovered that the difference in serving this type of recommendation and the rest is the amount of products we need to process during the request. Since co-recs are generated from input products, the service has to fetch recommendations for each of these products, mix them, and return a single set of recommendations, where with other types it usually only needs to handle a single input identifier. This is why it would take longer to process these queries and that it needs to fetch more data from Bigtable. Once a lot of these requests were made, the pipeline started to get overwhelmed and caused latency spikes and the Bigtable errors we saw.</p><p>Recommendations are unique which means Bluecore doesn’t serve personalized recommendations for the same shopper more than once in a few hours. Because of this, we never thought about caching them (unlike ElasticSearch based recommendations that are less personalized and thus much more cacheable), but after a brainstorming session we concluded that since co-recs are generated from products that shoppers interacted with (browsed, bought, etc.), it is very likely that the majority of the requests would ask for recommendations for the most popular products, which means that in this particular case, we were fetching the same keys over and over again from Bigtable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*x7ROrXZaQSZuWfkJtemsCg.png" /><figcaption>Figure 3 — Co-recs Bigtable recommendations are also cached now.</figcaption></figure><p>We decided to test that hypothesis by analyzing the query logs and checking the ratio of distinct input product IDs and the number of requests, and we were surprised to discover that over a short period of time, this ratio was actually considerably low, meaning that many requests were asking for the same input product IDs.</p><p>The solution was right in front of us: cache Bigtable co-recs recommendations, as shown in Figure 3. It’s important to note that this assumption was not true for personalized recommendations, so caching them would be pointless and unnecessarily consuming a lot of memory.</p><p>Since we were adding another in-memory cache, we also decided to fine tune the computational resources for the service, allowing pods to request a higher CPU limit and increasing the memory available from 8GB to 12GB for each instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*QF3fO7ua5cq4rvT3PWu6wQ.png" /><figcaption>Table 2 — Experiment results in requests per second and p99 latency after deploying the new LRU cache and increasing computational resources.</figcaption></figure><p>Once everything was implemented, we reran the experiments and this time we got amazing results. The new caching sped up the slow requests by reducing the number of calls to the original data source. This caching, along with increased resources allowed us to respond faster, ultimately increasing the number of requests per second of all the experiments, as shown in Table 2.</p><p>Experiments that benefit from caching, such as the Bigtable 64 unique requests and the ElasticSearch ones became extremely fast, reaching up to <em>317,474 </em>requests per second, with only 60ms p99 latency, way more than what we were aiming for. Our biggest surprise was the replication of the real campaign, which would barely complete before, now got up to <em>258,640</em> requests per second and <em>75ms</em> of p99 latency. That’s because this campaign is mainly composed of co-recs, so after a few requests, almost all recommendations were in the in-memory cache, avoiding hitting Bigtable and responding much faster. Needless to say, we no longer experienced Bigtable errors.</p><p>The only experiment that didn’t reach the expected performance was the Bigtable one with over 200,000 distinct requests, even though there was a significant increase, going from <em>14,010 </em>requests per second and 1400ms latency to <em>40,269 </em>requests per second (287%) and 240ms (583%). This is mainly due to the fact that this experiment is composed of many different recommendation types, so the speed up came from caching co-recs, but we were still making a lot of concurrent requests to Bigtable. Even though we didn’t reach the expected performance, we decided that we would still be fine, since we were aiming much higher than what we expected to see during the peak load, and the cost of improving the performance of this experiment was too high at the moment.</p><h3>Conclusion</h3><p>We ran experiments to understand the performance of our system, identified the bottlenecks, and analyzed the usage pattern to reach a working solution. After validating and implementing the solution, we reran the experiments and noticed a considerable improvement in the performance.</p><p>November came to its end, and with it BFCM arrived. We saw the volume increase in load, reaching to a maximum of 35.29 million requests in a single hour (an average of 9802 requests per second) and the recommendation service was able to handle all of it with 91ms p99 latency, so it’s safe to say that we achieved our goals.</p><p><em>Tiago Alves is a Senior Software Engineer who has been at Bluecore for a year, mostly focused on infrastructure for data science.</em></p><p>Interested in helping scale our platform as we continue growing? We’re <a href="https://www.bluecore.com/careers/">hiring</a> at Bluecore!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=971198b09c6a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluecore-engineering/load-testing-a-recommendation-service-971198b09c6a">Load Testing a Recommendation Service</a> was originally published in <a href="https://medium.com/bluecore-engineering">Bluecore Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Not Another Virtual Event: How Hackathon Made a Cultural Impact]]></title>
            <link>https://medium.com/bluecore-engineering/not-another-virtual-event-how-hackathon-made-a-cultural-impact-8aa7bf2dd16c?source=rss----4407d22b34ea---4</link>
            <guid isPermaLink="false">https://medium.com/p/8aa7bf2dd16c</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[culture-impact]]></category>
            <category><![CDATA[bluecore]]></category>
            <dc:creator><![CDATA[Courtney Seto]]></dc:creator>
            <pubDate>Mon, 21 Nov 2022 14:47:10 GMT</pubDate>
            <atom:updated>2022-11-21T14:47:10.569Z</atom:updated>
            <content:encoded><![CDATA[<p>Building community in a virtual, hybrid world is a daunting task. On that side of the calendar invite, right before the send button is clicked, there is a world of logistics, planning, and aspirations that most people don’t think about. I know this feeling all too well, not only having coordinated various virtual events and meetings since the beginning of the pandemic, but also as a planner for this year’s hackathon. A couple of thoughts haunted me as we set out to plan for this event: <em>How are we supposed to drum up excitement about working on extra projects in teams from their individual home desks? How do we connect folks spread out in different organizations and time zones across the world?</em></p><p>I’m currently in that mid-point between the lingering triumph of our last hackathon, and already thinking about planning for the next one — and I sit here full of reflections of what surprisingly worked really well, and what I learned to do better as we think about the next event.</p><h3><strong>The Event Dynamics and Challenges</strong></h3><p>The executive leaders had a few specific thoughts around this hackathon: a hybrid event that is mostly remote but with some in-person opportunities, thoughtful inclusion of our global teams, and an invitation for new ideas to create a more robust event than we have in the past. Oh — and we also designed this particular hackathon, traditionally an Engineering event, to be opened up to the entire company across all departments for the very first time. Challenge accepted.</p><p>I’ll admit, we started with grandeur plans. There was the hope of a recruiting event, a notion of inviting external executives for a Q&amp;A panel, the fun idea of mini events and games throughout the hack week. Spoiler alert: none of those things happened. What actually happened: paring it back to the core of what hackathon is all about — being able to brainstorm and build out big ideas, while being able to also introduce hackathon to newbies, with some added extras along the way.</p><h3><strong>New and Improved!</strong></h3><p>Though we ended up focused on simplicity more than anything else, there was still room for designing a virtual event that felt exciting. Again, this was the first time Bluecore made hackathon inclusive to everyone in the company, so we went beyond the straightforward announcements at town halls and also got creative with releasing video promotions over Slack leading up to the sign-ups. We invited all interested hackathon participants to pitch their ideas to over 100 attendees for a full 60 minutes. Teams were formed by a simple Google Sheet, with people from all different parts of the organization volunteering to be a part of projects solely because they like the ideas. Swag was designed for all active participants, virtual backgrounds were created, and everything was branded from the visuals to the lingo. During the four-day <em>Hack Week</em> (the four days were also a new thing, by the way — typically Bluecore’s hackathon is half the length of time), we encouraged people to use the time they needed to work together in their teams and also attempted to bring everyone together by asking a themed <em>Question of the Day</em> in Slack — a Bluecore Engineering tradition in the earlier pandemic days. We also invited reputable external judges outside of Bluecore for final presentations which added an exciting layer to the finishing teams.</p><h3><strong>The Result</strong></h3><p>Though the planning committee clearly enjoyed the more creative components to the event, the best and most consistent feedback I heard after hackathon was over was that people had fun. Fun working in their teams, fun working on creative solutioning for problems they cared about, fun having the time and opportunity to innovate. Overall we had 125 total participants which was around 40% of the company and 31 project submissions to review. Since this was an ambitiously cross-collaborative event and the first of its kind, I was floored to see that 31% of the event participants were from non-Engineering teams (as in, not engineering or product) with the make-up of 32% of the teams involving three or more people from different teams. This goes to show that all the branded swag in the world cannot replace how energizing it is to come together to work on creative solutions — the core spirit of what hackathon is all about.</p><h3><strong>What We Learned</strong></h3><p>While we can call this hackathon a success, there were certainly things we could have done differently. We were overwhelmed by the participation and number of submissions and can streamline the process in the future by giving the judges plenty of time to review and narrow down their favorites (which proved to be an incredibly difficult task!). We received feedback that some of the logistics and communications around sign-ups were confusing and could be clearer next time. There were a few shipping delays around the swag. All-in-all, nothing that cannot be fixed or tweaked to improve when we plan the next event. What we truly took away from all of this was that creating this cross-collaborative event impacted the company culture in more ways than imagined. With the great ideas sprouted from all across the organization, by some teams and people working together for the very first time and by choice, to the executive leadership team who were incredible partners and promoters for hackathon, and down to the planning committee who spent months organizing and meeting on a weekly basis — we experienced what it means to be <em>as simple as possible, but as powerful as necessary</em> — a Bluecore value and the very theme for this 2022 hackathon. An event’s theme that is woven into and exemplified with its very being…well, I think we nailed it.</p><p><em>Interested in helping scale our platform as we continue growing? We’re </em><a href="https://www.bluecore.com/careers/"><em>hiring</em></a><em> at Bluecore!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8aa7bf2dd16c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluecore-engineering/not-another-virtual-event-how-hackathon-made-a-cultural-impact-8aa7bf2dd16c">Not Another Virtual Event: How Hackathon Made a Cultural Impact</a> was originally published in <a href="https://medium.com/bluecore-engineering">Bluecore Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons Learned Spelunking]]></title>
            <link>https://medium.com/bluecore-engineering/lessons-learned-spelunking-4f9a769291fd?source=rss----4407d22b34ea---4</link>
            <guid isPermaLink="false">https://medium.com/p/4f9a769291fd</guid>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[legacy-code]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[organizational-culture]]></category>
            <category><![CDATA[teams-and-teamwork]]></category>
            <dc:creator><![CDATA[Mike Hurwitz]]></dc:creator>
            <pubDate>Tue, 23 Aug 2022 11:08:19 GMT</pubDate>
            <atom:updated>2022-10-04T16:59:06.333Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6yO33n-PxUMjWKLbHu6rYQ.jpeg" /><figcaption>“Rocky formation with rough surface and painting”, ArtHouse Studio via <a href="https://www.pexels.com/photo/rocky-formation-with-rough-surface-and-painting-4328432/">Pexels</a></figcaption></figure><p>Every organization I’ve ever worked for, no matter how large or small, no matter how functional or dysfunctional, has had something that everyone was afraid to touch. Sometimes I’ve been asked to take ownership of such code. Sometimes I’ve been asked to advise.</p><p>Recently, I was asked to dig into our file imports pipeline, learn what I could, and provide recommendations to the teams involved. Knowing I wasn’t going to have a long-term relationship with the systems in question meant that I couldn’t follow my normal path — get the basics now and deep-dive opportunistically. I needed more of a plan.</p><h3>Setting goals</h3><p>Sometimes I feel like a giant imposter who can’t do anything. Sometimes I feel like a great hero poised to conquer. Sustainability and continuity require a team, and I am definitely just one person. So the first job is setting person-sized goals.</p><h4>Identify problems, suggest solutions</h4><p>This was my explicit mandate. Success means learning and applying my experience. Success can’t mean implementing because that is bigger than one person.</p><p>Position power can be dangerous. What if I make a mistake? Empowering the teams that are going to own these problems and solutions is the only way to create a margin of error in my thinking. In cases like this, my preferred mode is to use position power to get in front of the right people and appeal to a higher authority if there is a problem later in the process. Flexing whatever position power I have can poison future interactions and creates unnecessary risk.</p><h4>Make consensus possible</h4><p>Any group of smart, opinionated people is going to have some differences of opinion. With the time I had, reaching consensus on the myriad of issues was unrealistic. Instead, I wanted to get to a point where the base assumptions were aligned enough that we could unblock decisions for the future. It sounds like a little thing, but everyone’s got a point-of-view and it’s easy to get entrenched. It doesn’t matter how good my ideas are if we can’t move the organization to implement them.</p><h4>Communicate</h4><p>I’m supposed to provide a recommendation. There isn’t any position power for me to wield here, so I need to convince the whole organization that what I’m recommending fits the problem, provides a path to an improved situation, and is something this company can do. Those three things can feel at odds when the problem is big.</p><p>Picking the medium is important, too. In a small organization you can get away with just talking, but we’re not that small anymore. I needed to produce an artifact. If it’s a smaller problem space with a hard decision, presentations can be great. In this case, where it’s a huge problem space with a million little decisions, a document felt better.</p><h4>Get others to communicate</h4><p>Our file import system touches a lot of teams. Part of the problem we’ve had so far is that those teams aren’t talking to each other, or at least not hearing each other. If I can put some people together, maybe those relationships can help long-term sustainability. At least I can at least get everyone speaking the same language.</p><h3>How did we get here?</h3><h4>Legacy code</h4><p>A long time ago, I had a friend who had a note on the side of his monitor that read, “Don’t write legacy code.” He explained that legacy code was something that people were afraid to change. Maybe there was a lack of documentation, poor test coverage, no comments, or no contract. Especially in a start-up environment, this can be the norm rather than the exception. If you don’t know that your code is going to survive a month, don’t waste time acting like it’s going to be there in 7 or 8 years. But it might…</p><h4>Undeclared inbound dependencies</h4><p>We already know we don’t have a contract, so we’re in trouble. There may also be quirks in declared behavior that something else relies on. Those quirks may not be correct anymore given how the business and other systems have evolved, or may have never been correct in the first place. The worst part is, you can’t know.</p><h4>Death by a million tiny forks</h4><p>At Bluecore, we serve several hundred clients. Some have been with us for years (thank you!), some just signed up (thank you, too!). Our file import systems have evolved over that time, sometimes in ways that might break older functionality while adding new and improved versions. The solution to this problem has been feature flags that allow <em>legacy</em> clients to use the legacy code paths, making for an easier transition. It’s an effective solution to maybe the most common class of problem we face — changing the wheels on a moving car. However, the result is that the legacy code persists, complexity increases, and it is nearly impossible to know what is still running and what isn’t.</p><h3>How to proceed</h3><h4>Who to talk to</h4><p>The easy answer here is <em>everyone</em>, but that’s impractical. Instead, find subject-matter experts, keeping use cases for the whole system in mind. In my case, there were a few camps of people involved:</p><ul><li>Platform Engineering: They own the back-end code. There were factions that had focused on different parts of the problem.</li><li>Forward-Deployed Engineering (FDE): They write automations using this system, which makes them both consumers of the back-end and the owners of many thousands of lines of code. If the system is clunky to use, it’s FDE who pays most of the price.</li><li>Product: They own requirements and the future direction. What we build can’t do <em>everything</em> or we’ll never finish writing it. Their job is to add good features and cut bad ones.</li><li>Design: There are some UX and DX questions that Product is raising, which brought in the design team. Knowing it’s one of my weak spots, I tried to listen carefully to what the designer and the product people were trying to do in this area.</li><li>Engineering Management: Cost (both monetary and human), reliability, and sustainability are their primary concerns. That goes beyond just having a system that people aren’t afraid to touch.</li><li>Senior Leadership: It’s a big deal to our clients, it’s a big deal to finance, it’s a big deal in terms of allocated engineers. I didn’t have the CEO or CFO on this, but our SVP of Engineering had a real interest. He’s also my manager, so that meant he’d be in the loop whether he wanted to be or not.</li><li>Me: I’m supposed to be defining a point of view, not just acting as Chief Software Historian. That means that my architectural goals for the company and all my biases will be represented. Better to do it consciously.</li></ul><p>In each of these groups, I was able to find someone to talk to: I needed to get in front of the #1 expert for each functional area. That’s not always possible. The secret is to find several people to talk to. Maybe someone with more availability. Maybe someone where I didn’t have to worry about how my early ignorance impacts my ability to be convincing later. Maybe someone who can just be less formal so we can stumble into new dark corners together. Two invaluable examples:</p><p>I have a friend who is a bit of an expert on the code. I sat down with him for two hours to get a dump of <em>his</em> frame for this whole mess. That was a great start that helped carry me through the whole project. I couldn’t take just his opinions wholesale, but I could take his facts and consider his opinions going forward. What he learned had come through a failed project, so having a comparatively untainted messenger was a win for him, too.</p><p>Shortly before the start of the project, I happened to do some work with an FDE on an unrelated issue. She isn’t the #1 subject-matter expert with the FDE team, but she knows what she’s doing and was able to explain the FDE workflow. Seeing her pain points helped sharpen my thinking about what a better system might look like. Giving her a sneak-preview of some proof-of-concept code helped validate my assumptions. When it came time to talk to the leadership of the FDE team, I was much better prepared.</p><h4>Meeting planning</h4><p>I am lucky that there were not significant political obstacles in this case. Disagreements about tactics and priorities or technical decisions are always there, but I didn’t have to worry about keeping certain people from being in the same room (or Zoom). But even in this mostly collegial environment, there are still rules.</p><p>Early meetings should be as small as possible. People will be more open, it’s easier to schedule, and you can worry less about a strict agenda. But the more honest reason is that I don’t need 10 people watching me not know what I’m talking about when I’m learning from an 11th. I want people to be heard, but I don’t want them to feel like they need to sit through hours of meetings where 90% isn’t covering the parts they care about. I feel like I’ve only got a few <em>meeting tokens</em>, so I try to spend them wisely.</p><p>The focus on file imports spurred some other people to have their own conversations on the topic. I was in several, but led none of them. I learned a ton! I didn’t have to burn any of my meeting tokens, which was good. I didn’t have control over the agenda, so it didn’t always feel as time-efficient as I might like. In the later meetings like this, I was already nudging the conversation toward my plan, and especially toward my vocabulary (see below).</p><h4>Vocabulary</h4><p>Regardless of the medium, I needed a frame. I needed groups to start using the same words to talk about requirements, functions, and failures. Even before I was done with discovery, I started sneaking my chosen vocabulary into the conversation, being as specific as I could. If someone said, “when we make pasta” I’d interject, “spaghetti?” and then let them either correct me or continue. This gave me a chance to get more and more specific, but also allowed any of the experts to tell me if my words were wrong.</p><p>I defined some new terms. This was the only <em>secretive</em> part of the process. It was vital that I didn’t accidentally establish words I didn’t want people to use going forward! I struggled. I wrote, erased, rewrote, and erased again. I still don’t like everything I came up with, but it’s good enough. Here were the rules I gave myself:</p><ul><li>Terms cannot run contrary to their generally-accepted meaning. I don’t want people to have to unlearn to understand.</li><li>Terms should not have a specific meaning in another part of our system. Words like “apply” are going to show up everywhere, but that’s ok as long as it isn’t capital-A “Apply” anywhere.</li><li>Avoid codewords and acronyms — we’ve got enough of both to last a lifetime. Jargon creates an in-group and pushes people away.</li></ul><p>For this project, my <em>new</em> terms for our file imports pipeline were receive, prepare, transform, split, apply and publish. I’m not going to define them here, but if I tell you that everything we receive becomes a series of events, your guess about the meanings of those terms is probably pretty close.</p><p>In my document, I made sure to cast both the existing solution and the recommended changes into those categories. Reinforcing terms at every opportunity is the best way to maximize your chance for adoption.</p><h4>Is there truth? There is code</h4><p>Recognizing the value of language is a good step, but it only matters if you’ve got something to say. <em>Consultant Mike</em> gathered all this information so that <em>Architect Mike</em> can come up with something. That involved a lot of listening, observing, and talking. But there’s one more member of this team of Mikes that has been on the bench until now: <em>Programmer Mike</em>. I am lucky that all of the components were written with languages and tools I know.</p><p>File imports are way too big for me to internalize in the few weeks I had. That’s part of the problem I’m trying to solve! But I’d be ignoring the most objective source I’ve got if I didn’t take time to read some code. Which code? How much time to spend on this?</p><p>I’d like to give the world a perfect rubric, but I don’t think it’s possible. To be honest, I read a ton of code. In this four week, part-time project, I spent at least 20 hours just reading. The majority of that time was spent writing my document. Because a big portion of the document was spent describing the functional units of the existing systems, I stopped to read each significant portions of each component just before I wrote it. This was not efficient, but it was effective. I was able to link my statements about the component to the implementation, removing doubt. In at least one case, a code owner told me that the implementation of the component I was working from wasn’t the one used in production, providing an important correction.</p><h4>How to talk smack and alienate people</h4><p>Not all decisions are great. Not all implementations are great. Maybe the best way to ensure you don’t get good information is to tell everyone involved that they’re stupid, that their work is terrible, and that you’re going to show them how to design better systems, write better code, eat their soup without slurping, and generally be better people. If you think this is actually the case, someone should be working on their résumé rather than on fixing systems.</p><p>Instead, try to figure out why decisions were made. As an example, I noted above that my friend who gave me the broad overview had been involved in a failed project in this functional area. There was a lot to learn here, including the assumptions that lead a bunch of smart, well-intentioned people to somehow build the wrong thing. In this particular case, the assumption was that there was a reliability problem in one part of the pipeline. There wasn’t, so the additional complexity and cost weren’t justified. But that’s great to know!</p><p>There was part of the code that bothered me. It was inefficient and poorly tested. I rage-wrote a replacement. I knew that the author was no longer with the company, so I couldn’t ask him what was up. I was absolutely furious that we would let something like this anywhere near production. I asked around and learned two very important facts: first, the author was an intern who did not get an offer at the end of his internship. Second, this code path was never turned on — the <em>legacy</em> implementation was still handling all production traffic. It should be deleted, but at least it wasn’t hurting anyone.</p><p>Lead with empathy. Reserve judgment. You can always get mad later.</p><h3>Putting it all together</h3><p>I wrote a document. The reason I chose a document over a presentation is that readers can skip around, click links, and rely on summary paragraphs. Presentations have to be more linear than that. A big reason this mattered to me is that my audience is at all different levels of detail (upper management down to individual contributors), different levels of technical skill, and focused on different areas of this broad system. I would have needed to produce many presentations to serve all these audiences.</p><p>My document had an introduction to explain the problem and define my terms. The very next section was a TL;DR (too long; didn’t read) guide. Knowing that some readers wouldn’t hang on my every word and needed to know what the important parts were, I didn’t want them guessing what the important parts were. The TL;DR guide explicitly told them what to read and what to skim or skip.</p><p>The conclusions were specific and written as a bulleted list. There was some commentary in that list, but it was as tight as I was comfortable making it in the time I had. It’s important to answer the “so now what?” question as clearly as possible. If I didn’t, someone else would and I might not like what they choose.</p><p>Because we use Google Docs, it was easy to share to everyone with comment access. Lots of tools support this kind of thing. I wanted feedback, both private and public, on what I’d written. Most were requests for clarification, which I was happy to give — if one person doesn’t understand something, odds are there are more.</p><h4>Release</h4><p>I had planned a big reveal. I was going to invite principals from all of the stakeholder groups and unveil my masterpiece like some kind of magic trick. However, I also wanted to share the document with some of my experts before release so I could get their feedback.</p><p>It leaked.</p><p>I was a little let down that I didn’t get to pull the rabbit out of my hat or saw someone in half, but it was worth it. Everyone was bought in before the general <em>release</em>, so no theatrics were necessary.</p><p>What you have to do here depends completely on your audience.</p><h4>Follow up</h4><p>I am not a consultant. The document represents the end of a stage of analysis, but I’ve still got a responsibility to make it happen. Many co-workers of mine are impacted by this work and will continue to be for months. Helping engineering teams understand is part of it, but the more important role is probably in advocacy. None of this will matter if we don’t have the time, space, and focus to implement. It’s always hard to keep organizational focus on long-term things and the only antidote I know is for there to be a tenacious person who steps up. At the start, I am that person. Handing-off that role properly is at least as important as handing off the technical information. Failing to do so is a guarantee that we never get to the end. I have to find a new advocate and be that advocate until I do.</p><h3>Summary</h3><ul><li>Understand why nobody wants to touch that thing</li><li>Define the goals for your project</li><li>Do you just need the answer and you can drive the solution, or do you need to bring others in to help drive it?</li><li>Choose your medium based on your audience(s)</li><li>Be honest and humble about what you don’t know</li><li>Be intentional about who you talk to and how you talk to them</li><li>Early meetings can be small and less formal</li><li>Be empathetic about prior decisions</li><li>Make everyone speak the same language; invent new vocabulary if necessary</li><li>Be ready to advocate for your solutions, even after your analysis is over</li></ul><p><em>Mike Hurwitz is a Principal Software Engineer who has been at Bluecore for four years, mostly focused on infrastructure for data science. He is about to celebrate 24 years as a professional developer.</em></p><p>Interested in helping scale our platform as we continue growing? We’re <a href="https://www.bluecore.com/careers/">hiring</a> at Bluecore!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f9a769291fd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluecore-engineering/lessons-learned-spelunking-4f9a769291fd">Lessons Learned Spelunking</a> was originally published in <a href="https://medium.com/bluecore-engineering">Bluecore Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[lazylru/generic — Proving It]]></title>
            <link>https://medium.com/bluecore-engineering/lazylru-generic-proving-it-8a4250e5946a?source=rss----4407d22b34ea---4</link>
            <guid isPermaLink="false">https://medium.com/p/8a4250e5946a</guid>
            <category><![CDATA[generic-programming]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[generics]]></category>
            <category><![CDATA[caching]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Mike Hurwitz]]></dc:creator>
            <pubDate>Wed, 30 Mar 2022 13:59:12 GMT</pubDate>
            <atom:updated>2022-03-30T13:59:12.250Z</atom:updated>
            <content:encoded><![CDATA[<h3>lazylru/generic — Proving It</h3><figure><img alt="Cans containing various flavors of generic soda. I hope you appreciate that this is the thinnest possible metaphor for this post." src="https://cdn-images-1.medium.com/max/1024/1*Vy5qoY7nr0FrDSUW_dc1aQ.jpeg" /><figcaption>Taste the generic flavor!</figcaption></figure><p>Way back in the long-forgotten days of June 2021, I put up a <a href="https://medium.com/bluecore-engineering/lazylru-laughing-all-the-way-to-production-19d2a053c3cb">post</a> about <a href="https://github.com/TriggerMail/lazylru">LazyLRU</a>, a least-recently-used cache that focuses on reducing lock contention. The <a href="https://tip.golang.org/doc/go1.18">release of Go 1.18</a>, and with it <a href="https://tip.golang.org/doc/go1.18#generics">generics</a> spurred a revisit of the library. Today I put out v0.2.0, which contains a submodule with a generic-enabled version of the cache.</p><h4><strong><em>A note about using the generic version</em></strong></h4><p>While many developers have been champing at the bit for the 1.18 release, the <a href="https://go.dev/blog/survey2020-results">2020 Go Developer Survey</a> highlighted that very few organizations jump to every new release. It appears that ~90% of those who have some kind of cadence for their upgrades will be on a new release within 6 months. This release contains the <em>string keys and interface values</em> Go 1.17 and below version in the root module. There is also a generic module below it that contains the generic version.</p><h3>Preserving the interface</h3><p>Arguably the greatest strength of the Go ecosystem has been the <a href="https://go.dev/doc/go1compat">promise of compatibility</a>. In the spirit of that, the new generic versions of the LazyLRU cache are compatible with the old interface completely.</p><pre>import (<br>    &quot;time&quot;<br>    lazylru &quot;github.com/TriggerMail/lazylru/generic&quot;<br>)</pre><pre>lruOld := lazylru.New(10, time.Minute)<br>lruNew := lazylru.NewT[string, string](10, time.Minute)</pre><p>The lruOld variable will contain a *lazylru.LazyLRU[string, interface{}]. That has exactly the same interface as the non-generic version. And like the old version, the caller will need to cast whatever is returned by Get() before using it. The lruNew version, however, will always return strings.</p><p>What this means is that all existing code should work without modification.</p><h3>But why?</h3><p>If channels are a <a href="https://muppet.fandom.com/wiki/A_Cookie_is_a_Sometime_Food">sometimes food</a>, then generics should be a special, once-in-a-blue-moon treat. Go went fifteen years without generics, ten years since 1.0 was released. <a href="https://en.wikipedia.org/wiki/Java_version_history#:~:text=February%202013.%5B26%5D-,Java%20SE%205,-%5Bedit%5D">Java SE5</a> brought generics ten years after the initial release; <a href="https://en.wikipedia.org/wiki/.NET_Framework_version_history#.NET_Framework_2.0">.NET 2.0</a> did it in four. In other words, you’d better have a good reason to bring generics into your code, especially in the public interface of a library.</p><h4><strong>A cleaner interface</strong></h4><p>It’s hard to imagine a case when a caller will be satisfied with interface{}. The caller is pretty much guaranteed to want something else. This is a sanitized excerpt from some production code using the previous version of LazyLRU:</p><pre>func Fetch(hashKey string) (*customers_pb.GetResponse, error) {<br>    if ires, ok := ls.cache.Get(hashKey); ok {<br>            if res, ok := ires.(*customers_pb.GetResponse); ok {<br>                    return res, nil<br>            }<br>    }<br>    // fetch from underlying source<br>}</pre><p>And this is the same code using the generic version:</p><pre>func Fetch(hashKey string) (*customers_pb.GetResponse, error) {<br>    if res, ok := ls.cache.Get(hashKey); ok {<br>            return res, nil<br>    }<br>    // fetch from underlying source<br>}</pre><p>That seems nicer.</p><h4><strong>More functionality</strong></h4><p>A nice feature of Go is that maps can be keyed on most types. Before generics, it was next to impossible to create containers that gave the caller a choice of types. There are some packages, like <a href="https://pkg.go.dev/container/heap">container/heap</a>, that use function calls over other containers to get the job done, but still end up returning interface{}.</p><p>Using the <a href="https://go.dev/ref/spec#:~:text=The%20predeclared%20interface%20type%20comparable">comparable type constraint</a> on keys and no constraint on values gives us a lot of flexibility. Just like with maps and slices, the type is declared when making the object and the compiler does the rest.</p><pre>// factory method<br>NewT[K comparable, V any](maxItems int, ttl time.Duration) *LazyLRU[K, V] {}</pre><pre>// functions on *LazyLRU[K, V]<br>Get(key K) (V, bool)<br>Set(key K, value V)<br>MGet(keys ...K) map[K]V<br>MSet(keys []K, values []V) error</pre><p>By using the generic interface, it’s possible to build on top of the cache in ways that would have been really ugly. As an example, I was able to implement sharding on top of the LazyLRU cache, giving the caller the ability to provide a number of shards and a sharding function (func (key K) int64)). We don’t lose flexibility in the keys, don’t have to cast to interface{}, and the compiler can make sure that we do the right thing.</p><h4><strong>Higher performance</strong></h4><pre>100% Read<br>------------------------------------------------<br>flavor      time         bytes alloc num. alloc<br>---------- ------------- ----------- -----------<br>interface   137.1 ns/op       0 B/op 0 allocs/op<br>generic     140.8 ns/op       0 B/op 0 allocs/op</pre><pre>100% Write                                    <br>------------------------------------------------<br>flavor      time         bytes alloc num. alloc<br>---------- ------------- ----------- -----------<br>interface   175.7 ns/op      32 B/op 2 allocs/op<br>generic     143.7 ns/op       8 B/op 1 allocs/op</pre><p>While the interface is cleaner on the read path, the performance difference between the interface-based version and the generic version on the read path is within the margin of error. The cost of that extra casting, the extra conditional, was zero. This was surprising, but the benchmarks don’t lie. The numbers below were recorded on the 100% read trial with string for keys and []int for values. There were no bytes allocated, so that has been removed for simplicity. The interface version is even a little faster!</p><p>However, when you move into writes, things get more interesting. The generic version is not only <strong>18% faster</strong>, the allocations are much lower.</p><p>The fact that we’re using []int as the type of our values is a little strange. The reason for it is that plain int values resulted in no difference at all. Even creating pointers to structs didn’t make much difference.</p><pre>100% Read<br>----------------------------------------------------<br>flavor    struct time        bytes alloc num. alloc<br>--------- ------ ----------- ----------- -----------<br>interface array   70.88 ns/op     0 B/op 0 allocs/op<br>generic   array   71.11 ns/op     0 B/op 0 allocs/op<br>interface struct  69.63 ns/op     0 B/op 0 allocs/op<br>generic   struct  71.27 ns/op     0 B/op 0 allocs/op<br>interface value   77.60 ns/op     0 B/op 0 allocs/op<br>generic   value   71.20 ns/op     0 B/op 0 allocs/op</pre><pre>100% Write<br>----------------------------------------------------<br>flavor    struct time        bytes alloc num. alloc<br>--------- ------ ----------- ----------- -----------<br>interface array  172.9 ns/op     32 B/op 2 allocs/op<br>generic   array  145.4 ns/op      8 B/op 1 allocs/op<br>interface struct 173.6 ns/op     48 B/op 1 allocs/op<br>generic   struct 167.9 ns/op     48 B/op 1 allocs/op<br>interface value  141.9 ns/op      0 B/op 0 allocs/op<br>generic   value  143.7 ns/op      0 B/op 0 allocs/op</pre><p>The lesson to draw here is that the interface{} codepaths in the Go runtime are really well optimized. You may have a performance win, you may not.</p><h3>Conclusions</h3><ul><li>Generics in Go are limited, but still powerful. Think about your users before choosing to use them</li><li>I was expecting a bigger performance win on the read side. As is always the case with performance-sensitive code: measure, measure, measure</li><li>Releasing generic-enabled open-source code is going to be an interesting policy question. Modules are great, but the “<a href="https://donatstudios.com/Go-v2-Modules">2.0 problem</a>” has lead to some strange choices</li></ul><p>Is this your idea of a good time? Come work with me and other great engineers at <a href="https://www.bluecore.com/careers/">Bluecore</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a4250e5946a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluecore-engineering/lazylru-generic-proving-it-8a4250e5946a">lazylru/generic — Proving It</a> was originally published in <a href="https://medium.com/bluecore-engineering">Bluecore Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running a Successful Remote Hackathon]]></title>
            <link>https://medium.com/bluecore-engineering/running-a-successful-remote-hackathon-c782a5d21825?source=rss----4407d22b34ea---4</link>
            <guid isPermaLink="false">https://medium.com/p/c782a5d21825</guid>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[remote-working]]></category>
            <category><![CDATA[bluecore]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Rasna Tomar]]></dc:creator>
            <pubDate>Tue, 05 Oct 2021 12:02:34 GMT</pubDate>
            <atom:updated>2021-10-05T12:02:29.723Z</atom:updated>
            <content:encoded><![CDATA[<p>The shift in the working paradigm due to COVID has revolutionised the way we collaborate with each other. The remote-first work environment has made it possible for teams across the globe to come together and learn from each other more so than before. Last year was our first remote hackathon at Bluecore and this year we applied learnings as we held another season of our remote hackathon.</p><p>Hackathons have always been an integral part of Bluecore culture so that people across teams can come together and collaborate through idea sharing and brainstorming to accomplish common goals. Many products that are productionized have evolved from our hackathons as we identify solutions that complement existing products and services. Better yet, it’s one of the events that uplift the company culture and bring inclusiveness to the organisation.</p><p>Hackathons at Bluecore are time-boxed events: two days where teams work on conceptualizing the idea into proof of concept to check for functionality, feasibility, and alignment to its goal. One day is dedicated to hacking and the other for demoing projects. Nearly 37 ideas were transformed into 15 projects, including Smart A/B testing, ML Models to generate complete outfits, Quantum Recommendations, Demand forecasting using BQML, Open-time personalization, Caching and request merging reverse-proxy for ElasticSearch, to name a few. These projects are tangible and have impact on our organization and our customers by improving existing services, and by unlocking the potential to bring Bluecore’s personalization intelligence across customer’s platforms for a consistent multi-channel marketing experience. Check out our detailed blog post on last year’s Remoticon2020<a href="https://medium.com/bluecore-engineering/remoticon2020-bluecores-first-remote-hackathon-e85a3644aafe"> here</a>.</p><p>A successful hackathon requires a ton of effort, organizing, and planning. Depending on the scope of the event, it can take anywhere from four to seven weeks, so we started planning two months prior to the event. The backbone of any event is the team of volunteers who make it viable in the end. There are different roles that volunteers take up such as Orchestrators and a Master of Ceremony (MC). Orchestrators do the task of planning, guiding the judges, designing and communicating details of the hackathon, whereas the MC hosts the entire event on the demo day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JvYHh6I9shHrab8D" /><figcaption>Gino Knodel, Staff Data Scientist explains quantum recommendations.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C5otoJJT1vZ1LvUz" /><figcaption>Zahi Karam, Director of Data Science explains a model to generate an outfit.</figcaption></figure><p><strong>Setting up a Theme</strong></p><p>Hackathons must have a theme or a goal in mind. So, the first step in planning a hackathon is to come up with a brilliant theme around which the whole event revolves — it should be comprehensive so that prospective participants can swiftly come up with the ideas about what they would like to build during the event and that encourages ideas to naturally gravitate towards a goal that is critical for the company. In the past, we focused on themes: cutting cost, developer velocity, product reliability, “vision” (increase insights and analytics into our systems). Our 2021 hackathon theme was titled “Imagine the Possibilities” to honor one of Bluecore’s new cultural values. Thematically this meant: “Bring your moonshot ideas and see what you can make happen. When building something we should not confine ourselves to the current state of the problem we are trying to solve; instead we should stretch our boundaries and embrace the possibilities that come with it.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n3aTzXb-sq-U4dzT" /><figcaption>Michael Hurwitz, Principal Engineer explains how to reduce duplicate requests to ElasticSearch.</figcaption></figure><p><strong>Building Anticipation</strong></p><p>There are different ways to convey information about an event, be it on slack channel and/or over email and/or during a Zoom meeting that is likely to be attended by the majority of people in a company such as R&amp;D meeting, sprint review meeting, or town hall. You need to make sure that you are able to promote your hackathon to all the potential contestants in an exciting way. You can create an official Slack channel where people who wish to participate can share ideas. You can also put together resources related to previous hackathons such as ideas and presentation videos to give hackers an idea about how hackathons work. This influences people, but also encourages them to participate. Without participants there is no hackathon. In these resources, communicate the hackathon timeline like dates of the event, officially kick-off time, presentation date, and time well in advance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6M_CMnaVouuIDvaH" /><figcaption>Charles Wheeler, Forward Deployed Engineer explains how to detect code reuse in partner repos.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J2TXlYxfvrx69gxy" /><figcaption>Daniel Mantica, Data Insights Analyst explains demand forecasting using BQML.</figcaption></figure><p><strong>Forming Teams and Brainstorming Ideas</strong></p><p>The objective of hosting a hackathon is that everyone should participate, not just engineers. Research and Development (R&amp;D), Product, Forward Deployed Engineers (FDEs) and even Customer Success Managers (CSM) can come together to work on creative ideas. This helps to increase sign ups and encourage teammates to create a marketplace of ideas so that anyone can pick up any potential idea. It is advised to potential participants to start looking for teammates soon after the hackathon announcement and start pitching your idea to them. Anywhere between one and five members is a good number for a team size. We discouraged solo teams as the purpose of the hackathon is to innovate and collaborate with other Bluecore employees.</p><p>Once you have received a fair number of sign ups, organize a brainstorming session to come up with ideas that can turn into potential projects. Here, you can also educate newcomers about what the hackathon is. Facilitate a structured ideation process to welcome ideas from all teammates, not just from those who signed up to participate or product managers. Customer empathy is the key driving factor in brainstorming ideas for a hackathon. You can hypothesize ideas without restricting yourselves to what can be transformed into a project and whatnot can be transformed into a project. This process helps in building a repository for ideas. This not only increases the number of ideas but also helps to improve projects that can become MVP for a company or even production ready project. As we know, everyone is creative but not everyone is good at coming up with ideas, so finding inspiration from the marketplace of ideas assists those who are looking for a right idea to indulge in without being certain of what idea they can suggest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kjvPLDgcVP3mPYfj" /><figcaption>Andrew Klein, Senior Software Engineer explains open time personalization.</figcaption></figure><p><strong>Defining Judging Criteria and Selecting Judges</strong></p><p>After you have set up a theme, officially announced the event and facilitated team formation, the next step is to define judging criteria. This is important in deciding on what basis the winner will be chosen. It lays the ground rule clear among participants that what will be the expectations of judges while deciding the finalists. There are different criteria for judgement such as relevance to the theme, innovation, technical feasibility, business value, completeness, execution, design, extensibility etc. At Bluecore, we defined three criteria based on which judges assigned a score from 1–10 to choose the best projects.</p><ol><li>Relevance to the theme — how well the project is aligned to the theme?</li><li>Impact — how much impact will the solution make?</li><li>Execution — how well executed the project is during demo?</li></ol><p>Once you have set the judging criteria, then define a judges rubric that helps to guide the judges on what qualifies as a winning hack and enables them to vote easily and come to a conclusion.</p><p>Judging is not an easy task and choosing judges can be tricky. The best strategy is to have a panel of stakeholders from diverse fields within your company. You can have assorted experts from an engineering, product, marketing, sales, customer success, and finance background. The number of judges that you choose depends on the scale of the hackathon, but anywhere between three and six is a good number. All the judges should be given clear instructions about the event. It is also important for them to arrive on time in the event to carry out their responsibilities swiftly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1xiq6mIzyfq7YNIJ" /><figcaption>Selam Tegegn, Software Engineer explains an idea to modernize precomputed attribute rank.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*23lrbooqUmTKaHRt" /><figcaption>Noam Soam, Software Engineer explains an idea to modernize precomputed attribute rank.</figcaption></figure><p><strong>Kicking off the Hackathon</strong></p><p>As we’ve said before, promote the hackathon as much as you can — encourage people to find inspiration from a repository of ideas to join an already formed team or pick up a completely new project that aligns with their skills. When the hackathon kicks off, it is essential for organizers to assist the participants if they need help in something or they are blocked due to technical issues. During offline mode, there are a lot of planned activities to entertain the participants. Whereas, in a remote-first working mode, it is difficult to plan these activities. Despite these challenges, Bluecore remote hackathons do not lose their charm as engineers are equally excited in virtual hackathons and hack in over Zoom calls to build their prototype. To induce fun, at the end of the presentations while judges were deciding winners of the hackathon we had a little chit chat session to break the silence. We also asked the audience about how they felt about the whole event in general.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/0*AFzTzpwGtI3hyq4E" /><figcaption>Ankit Kumar, Software Engineer explains smart A/B testing.</figcaption></figure><p><strong>Demo Day and Awards</strong></p><p>The most exciting part of the hackathon is demonstrating work to the panel of judges. It is the chance for all the teams to see what others have accomplished. The demos usually take around two to three hours, depending on the number of teams that participate. Since this is a time-bound event, every team is assigned some reasonable time to present their project, explain the impact and relevance. It is the job of the MC to keep track of time and invite each team one by one to present their work. After all teams have presented, it’s time for the announcement of the results.The panel of judges can choose the best solutions by referring to the judges rubric, and name the winners across different categories such as best theme, highest total score, highest impact, and people’s choice — just to name a few.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h-rhtYEklQegQyn0J6Ic5g.png" /><figcaption>Rasna Tomar, Data Scientist explains customer segmentation using BQML.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zVBqmKqD95G2XFt2" /><figcaption>Alexa Griffith, Software Engineer explains the TODO app.</figcaption></figure><p><strong>Key Takeaways</strong></p><p>Any event that happens with perfection is a myth, if it turns out to be the perfect one then learning stops then and there only and barring any scope of improvement. Continuity is the key to success, so find ways to improve the next hackathon, have a retro, make lists and things to consider so that next time you don’t have to reinvent the wheel and you won’t forget any details. Something like: “Much like a theatrical play, you need to anticipate the needs of the audience and make sure you have enough staff to handle things behind the scenes. You should show up to the hackathon date with very little left to do other than carry out with the responsibilities you planned on having and addressing issues as they arise.” This year’s hackathon gave us plenty of opportunities to learn as learning is a continuous process, and we are ready for the next one.</p><p>We have prepared a cheatsheet from our experiences and learnings that you can use when organizing a hackathon:</p><p><strong>Cheatsheet for Running a Hackathon:</strong></p><ul><li>Pick a theme and dates</li><li>Set boundaries (limiting team sizes, days hackathon will be on, and how long the hackathon will be)</li><li>Announcement of event and pre-hackathon meetings (email, Slack, and company meetings)</li><li>Create a market to exchange ideas (Google sheets or equivalent)</li><li>Elevator pitch event to encourage Ideas and creating teams</li><li>Have teams come up with team names so it’s easier for people to remember what projects are</li><li>Select judges</li><li>Decide prizes (get finance involved early)</li><li>Create method for judges to vote and come to a conclusion</li><li>Templatize the host’s slides for hackathon presentation</li><li>Have all participants come prepared with all of their environments and setups ready prior to hackathon</li><li>Host the event and declare winners</li><li>Have a hackathon retro</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RGmAlMhimCgeoS0j" /><figcaption>Arthur Herbout, Data Scientist explains object detection on product images.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rwOu5n2FehsXQD0q" /><figcaption>Michael Myung, Forward Deployed Engineer and Extensibility Engineer explains how to make personalised product recommendations exportable.</figcaption></figure><p>Keep hacking folks :)</p><p>Interested in helping scale our platform as we continue growing? We’re <a href="https://www.bluecore.com/careers/">hiring</a> at Bluecore!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c782a5d21825" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluecore-engineering/running-a-successful-remote-hackathon-c782a5d21825">Running a Successful Remote Hackathon</a> was originally published in <a href="https://medium.com/bluecore-engineering">Bluecore Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bluecore’s Frontend North Star]]></title>
            <link>https://medium.com/bluecore-engineering/bluecores-frontend-north-star-eb7d2a04167a?source=rss----4407d22b34ea---4</link>
            <guid isPermaLink="false">https://medium.com/p/eb7d2a04167a</guid>
            <category><![CDATA[react]]></category>
            <category><![CDATA[bluecore]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[frontend]]></category>
            <category><![CDATA[typescript]]></category>
            <dc:creator><![CDATA[Matthew Gerstman]]></dc:creator>
            <pubDate>Tue, 10 Aug 2021 18:12:52 GMT</pubDate>
            <atom:updated>2021-08-10T18:12:52.444Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bluecore’s Frontend North Star — Building Frontend at a Newly Minted Unicorn</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QCFDDzRri5TbYnRWefOV9w.jpeg" /></figure><p>At Bluecore, our mission is to help retailers find their best customers and keep them for life. We do that by providing data science as a service so our customers can personalize their content for their users.</p><p>Over the years, we’ve invested heavily in our data science and infrastructure teams, while building a relatively light frontend. This got us pretty far, we recently <a href="https://techcrunch.com/2021/08/05/bluecore-lands-125m-series-e-on-1b-valuation-as-e-commerce-personalization-grows/">announced our Series E</a>, raising $125 million at a $1 billion valuation.</p><p>As we approach our next chapter, we need to enable more complex workflows for our customers. In service of this, we wrote a frontend north star for our web application. This post contains our internal frontend north star document that we’re using to guide frontend development going forward. It covers several things:</p><ul><li>Terminology for engineers from other domains</li><li>Component types and code organization</li><li>An interface for an API Client</li><li>State Management</li><li>Testing Guidelines</li></ul><p>We think it’s pretty good, so we decided to share it with the world. If this type of work gets you excited, <a href="https://jobs.lever.co/bluecore/299d0d1b-2b57-4cd6-9dec-726988c9361c?lever-via=hqR3sDGa5y">we’re hiring</a> people to build it.</p><h3>Introduction</h3><p>This document details the architecture for a modern web application built on top of React. It is opinionated with regard to system design, but not with implementation details or open source libraries.</p><p>It recommends a series of primitives and abstractions to make it easy to build new pages and flows. Some layers of abstraction build on top of others, however it’s designed such that we can ship individual pieces as soon as they’re ready. This allows us to provide incremental value to ongoing feature development while marching towards our goal.</p><p><em>Some notes:</em></p><ul><li><em>We will determine an execution strategy after we have aligned on a design strategy.</em></li><li><em>This document details our in-browser architecture. We will document portions that live on the server in a future RFC.</em></li></ul><h3>Terminology</h3><h4>Pure Functions</h4><p>This document frequently mention pure functions or impure functions. Pure functions are functions whose inputs always have a 1:1 mapping to their output and have no side effects. Impure functions maintain some kind of internal state, trigger a side effect, or depend on state not passed in as an argument.</p><h4>Components</h4><p>Components are the primary mechanism provided by React to power applications. Components are (sometimes) stateful nodes in a tree that power an application. Many types of components are detailed in this document.</p><p>React provides two types of components, function components and class components. Most of our components will be built on function components. This provides a clean, declarative, programming experience.</p><p>React applications are trees of components rendering each other. Function components are stateful (impure) functions that handle business logic, render layout, and display UI. State is typically passed up and down the React tree through props and callbacks.</p><h4>Props</h4><p>Props are the arguments passed when calling the function that creates the Component. React will automatically rerender or update a component when its props change. Passing a callback as a prop allows a child component to send data back up to its parent.</p><p>React has an internal render loop that separates DOM mutations from the code that describes them. When this was first released it was a remarkable development in fronted engineering.</p><h4>Hooks</h4><p>Hooks are a primitive provided by React to share business logic by using stateful functions. These functions occasionally have system-wide side effects. React maintains internal hook state by remembering the order they were called in.</p><p>As a convention, all hooks begin with the word use like useEffect or useState. Developer tools will assume that any function that starts with use is a hook.</p><h4>Context</h4><p>React provides a primitive called context that allows users to pass data up and down the tree without threading callbacks through every component. Internally, React will update the relevant components when a context changes that it depends on.</p><blockquote>Think of them like Non-Fungible Tokens 😝.</blockquote><p>More seriously, context has two parts, a provider and a consumer. The provider is a component that takes an initial state prop from its parent and provides a bit of state anyone can go looking for. The consumer will often be a hook that returns a getter and a setter for the state.</p><h4>Design System</h4><p>This is a component library filled with any UI elements that are displayed on a page. This library contains components such as: buttons, modals, icons, loading spinners — you name it.</p><p>The design system should be a tool that developers and designers use as leverage with each other. Sketch, Invision, and Figma all have a notion of components (sometimes called symbols), and we can provide a 1:1 mapping of those components to a real React component. This creates an environment where designers and developers are speaking the same language and ensures our product is consistent across products/teams.</p><p>The design system can also feature layout components and importable styles (although most styling should be done in the design system itself).</p><pre>ls<br>- src/<br>- src/components/<br>- src/layout/<br>- src/styles/</pre><p>A good rule of thumb is anything that renders to the DOM, belongs in the design system. To avoid being overly prescriptive, we should follow this rule 80% of the time.</p><h4>Core Design System</h4><p>This is the layer where we convert the components in Sketch/InVision into actual code. These will likely be written manually (as opposed to exporting them from the design application).</p><p>These components will map 1:1 with a component/symbol in our actual designs and will be focused on rendering to the DOM. The prop types for this component should allow a user to pass anything to the component’s root node. We can strictly type this.</p><h4>Magic Components</h4><p>These are components that we will build that will be tightly coupled to one or more of our APIs. They’ll allow us to reuse business logic for things like typeaheads, search bars, and data tables.</p><p>These will require a strongly typed API and a robust API library. Another name for these would be <em>widgets</em>, but I like Magic Components so much better.</p><h4>Product (Application) Components</h4><p>Product code will be composed of magic components, layout components, and design components. For custom components written for one feature, we should optimize for simple interfaces like the ones above so that they can be promoted to more generic magic components.</p><h4>Testing and Mock Data</h4><p>Our design system will use Storybook, a popular tool for demoing and manually testing individual components. We will write a set of strongly typed mock input data that we use for both our manual tests and automated tests (below).</p><p>It is easier to reproduce and test errors manually since we’ve maintained the same data set for both automated and manual testing</p><p>Magic components will leverage the API Mock Data described below.</p><h3>API Library</h3><p>We will have a base API class that supports auto-generated routes. This client will be able to take a path to a route, a typed object of options.</p><h4>bluecoreApi Example Call</h4><pre>bluecoreApi(<br> ‘/api/v1/create_campaign’,<br> {<br>   request_arguments: {<br>     campaign_name: ‘Frontend’<br>   },<br>   headers: {<br>     auth: “TOKEN”<br>   }<br> } <br>)</pre><h4>Implementation Detail</h4><p>Typescript supports keying a type on a string, which allows a strictly typed auto-generated API. Below is sample code of how TypeScript supports function overloading based on a string argument.</p><pre>type PathOneResponse = {one: true};<br>type PathTwoResponse = {two: true};<br>function bluecoreApi(path: &#39;/path_one&#39;): PathOneResponse;<br>function bluecoreApi(path: &#39;/path_two&#39;): PathTwoResponse;<br>function bluecoreApi(path: string) {<br>  if (path === &#39;/path_one&#39;) {<br>    return {one: true};<br>  }</pre><pre>  if (path === &#39;/path_two&#39;) {<br>    return {two: true};<br>  }</pre><pre>return undefined;<br>}<br>const responseOne: PathOneResponse = bluecoreApi(&#39;/path_one&#39;);</pre><pre>// ts(2741): Property &#39;two&#39; is missing in type &#39;PathOneResponse&#39; but required in type &#39;PathTwoResponse&#39;<br>const responseTwo: PathTwoResponse = bluecoreApi(&#39;/path_one&#39;);</pre><pre>// error: The call would have succeeded against this implementation, but implementation signatures of overloads are not externally visible.<br>const responseThree: PathTwoResponse = bluecoreApi(&#39;invalid_path&#39;);</pre><h4>API Hooks</h4><p>We will provide access to the API directly via react hooks to power magic components. These hooks will be strongly typed against the API client described above.</p><p>The hook will take the request arguments as function arguments. The example below shows how types can propagate up to the React layer.</p><pre>type APIPaths = &#39;/path_one&#39; | &#39;/path_two&#39;;<br>type APIResponseMap = {<br>  &#39;/path_one&#39;: PathOneResponse;<br>  &#39;/path_two&#39;: PathTwoResponse;<br>};</pre><pre>type APIResponse&lt;T extends APIPaths&gt; = APIResponseMap[T];<br>function useApi&lt;Path extends APIPaths, Response = APIResponse&lt;Path&gt;&gt;(<br>  path: Path,<br>  // TODO: Implement APIArgs<br>  apiArgs: APIArgs&lt;Path&gt; = {}<br>): Response {<br>  // Authentication will be an implementation detail of the API.<br>  return bluecoreApi(path, apiArgs);<br>}</pre><h4>Promise Wrapper</h4><p>Each API hook will be wrapped in a function that returns an easy to consume object as shown below.</p><pre>function MagicComponent() {<br>  // useSpecificApi (or useApi) will be as simple as possible on the<br>  // outside, and as powerful as necessary on the inside to create a<br>  // state machine that abstracts a promise into a simple response.<br>  const {response, isLoading, error} = useSpecificApi();<br>  <br>  if (isLoading) {<br>   return &lt;LoadingComponent /&gt;<br>  }<br>  <br>  if (error) {<br>   return &lt;ErrorComponent /&gt;<br>  }</pre><pre>if (response) {<br>   return &lt;UIComponent /&gt;<br>  }<br>}</pre><h4>Data Fetching</h4><p>Application data will be fetched with the bluecoreApi module. The interface will provide EntityTypes rather than ResponseTypes.</p><p>We will provide an interface for product components to consume Entities. These interfaces will employ strictly pure functions that have no side effects. They will usually (but not always) take a single ResponseType and return an EntityType.</p><pre>function transformCampaign(response: CampaignResponse): CampaignEntity;</pre><pre>function useCampaign(campainId): Campaign {<br> const {response} = useSpecificAPI(campaignId);<br> if (!response) {<br>  return;<br> }<br> <br> return transformCampaign(response);<br>}</pre><h4>Client-Side Caching</h4><p>We will include some form of client-side caching for APIs. This will only include API state and previous response data. Caching is kept as an abstract and only available to the API Library itself. We may decide to implement tools around polling or refetching of data. It is also possible we find a library that does this for us.</p><h4><strong>Mutations</strong></h4><p>Mutations are writes to the server that often have a corresponding UI change. These are supported inside of the API Library.</p><pre>async function writeCampaign(<br>  campain: Campain<br>) {</pre><pre>  // Optionally supports optimistic rendering   <br>  writeCampaignToCache(campain);<br>  return writeCampaignToServer(campain);<br>}</pre><h4>Optimistic Rendering</h4><p>Optimistic rendering is when we update the client before the server has had a chance to respond, allowing interactions to appear faster to the user. The potential tradeoff is that the server might provide a different response than we expect, in which case we now have to undo it and display an error.</p><p>Users want most things to process instantly, but they expect some things like a bulk import to take a moment. For toggles and typeaheads, we’ll usually do things as fast and optimistically as possible. For anything that requires trust, we’ll want to display some form of animation to show the customer what we’re doing.</p><p><em>Aside: This was a fun article about what to do when your writes are too fast. </em><a href="https://www.fastcompany.com/3061519/the-ux-secret-that-will-ruin-apps-for-you"><em>https://www.fastcompany.com/3061519/the-ux-secret-that-will-ruin-apps-for-you</em></a></p><h4>Testing and Mock Data</h4><p>Every API should have at least one checked in sample request/response. We should use these sample requests/responses both in our integration tests against the API and as inputs to tests for Magic Components and Product Code.</p><h3>Product Code</h3><p>Product code leverages the primitives and interfaces detailed above to build new features, workflows, and functionality quickly. These features will be maintained by the product teams working on them. These are the teams that will work directly on shipping products and interfaces to the users.</p><h3>Application State</h3><h4>Forms</h4><p>Forms are historically a nightmare to implement in React. Fortunately, there are a wide variety of form libraries we can employ to do this job.</p><h4>Transition State</h4><p>Legacy code will continue to use Redux. We will likely move away from Redux towards libraries with better ergonomics. Another option is to invest in the modern Redux Toolkit.</p><p>For the unfamiliar, Redux is a pub-sub event dispatcher that maintains an internal data store. However, it often looks remarkably like trying to implement your own database in the client. The nice thing with Redux is it puts all your application state in one place for better debugging.</p><h4>Application Shell</h4><p>At the root of the (in-browser) application will live an application shell that handles page routing, metadata, and context providers. It will take an initial payload from the server, usually in the form of a token used for API authentication.</p><h3>Miscellaneous</h3><h4>Routing</h4><p>We will continue to use React router for client-side routing. It is the standard tool for the job.</p><h4>Logging</h4><p>We will work with our data team to determine requirements for client-side metrics. We may decide to use bluecoreApi against a single logging route, or decide to use a lighter weight implementation.</p><h4>Feature Gating</h4><p>We will continue to use our existing feature gating system, however we may build new hook and component based interfaces for simpler React integration.</p><h4>Types</h4><p>We will rely heavily on TypeScript for maintaining contracts between parts of the system.</p><h4>Unit Tests</h4><p>We will use Jest, a popular test-running for headless unit testing. This will mostly be on a per component basis.</p><h4>Integration Tests</h4><p>We will use mock API responses and write integration tests in the same system we use for unit tests (Jest).</p><h4>End-to-End Tests</h4><p>We will continue to use Cypress for our end-to-end tests.</p><h4>Mock Data Strategy</h4><p>Every layer of our application should have robust mock data to quickly generate testing states. For example, we may want to auto-generate a state that logs a user in and skips them to a specific step of a flow.</p><h4>In-Page Debug Tools</h4><p>We will provide a debug panel in a corner of the page (likely lower-right). This will only be available in the development/Qa environment. This debug panel will be used as a surface for triggering state changes or automatically dropping the developer into a specific workflow for testing.</p><h4>Bug Reporter</h4><p>We will ship a bug reporter available in production to Bluecore employees. This bug will be an icon in a corner of a page that allows anyone to submit a bug about the page they’re on. This tool will create a JIRA ticket and include as much context about the current state of the application as possible.</p><p>We will only display this icon for Bluecore employees and we can hide it on demo surfaces.</p><h3>We’re Hiring!</h3><p>Wanna help build this system? We’re <a href="https://jobs.lever.co/bluecore/299d0d1b-2b57-4cd6-9dec-726988c9361c?lever-via=hqR3sDGa5y">hiring talented frontend engineers.</a> Come join us!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb7d2a04167a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluecore-engineering/bluecores-frontend-north-star-eb7d2a04167a">Bluecore’s Frontend North Star</a> was originally published in <a href="https://medium.com/bluecore-engineering">Bluecore Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learnings from building a mission-critical API]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/bluecore-engineering/learnings-from-building-a-mission-critical-api-43b1cd6c54ce?source=rss----4407d22b34ea---4"><img src="https://cdn-images-1.medium.com/max/1400/0*yhY2gHvreMnrfjWc" width="1400"></a></p><p class="medium-feed-snippet">8 key learnings you should always keep in mind when starting a new project</p><p class="medium-feed-link"><a href="https://medium.com/bluecore-engineering/learnings-from-building-a-mission-critical-api-43b1cd6c54ce?source=rss----4407d22b34ea---4">Continue reading on Bluecore Engineering »</a></p></div>]]></description>
            <link>https://medium.com/bluecore-engineering/learnings-from-building-a-mission-critical-api-43b1cd6c54ce?source=rss----4407d22b34ea---4</link>
            <guid isPermaLink="false">https://medium.com/p/43b1cd6c54ce</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[bluecore]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Alexa Griffith]]></dc:creator>
            <pubDate>Fri, 30 Jul 2021 16:19:05 GMT</pubDate>
            <atom:updated>2021-08-16T22:15:22.508Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Kubernetes Policy Enforcement with Open Policy Agent]]></title>
            <link>https://medium.com/bluecore-engineering/kubernetes-policy-enforcement-with-open-policy-agent-cd893f09a3ab?source=rss----4407d22b34ea---4</link>
            <guid isPermaLink="false">https://medium.com/p/cd893f09a3ab</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[bluecore]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[open-policy-agent]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Indu Subbaraj]]></dc:creator>
            <pubDate>Fri, 25 Jun 2021 15:34:51 GMT</pubDate>
            <atom:updated>2021-06-25T15:34:51.827Z</atom:updated>
            <content:encoded><![CDATA[<h4>Programatically enforcing best practices</h4><p>For the past year, Bluecore’s engineering team has been building our Kubernetes ecosystem as we migrate our core services from Google App Engine, Google’s serverless offering, to Google Kubernetes Engine. One of our guiding principles as we’ve made critical infrastructure decisions has been streamlining the Kubernetes developer experience. We want it to be extraordinarily simple for an engineer to spin up a new service, <a href="https://medium.com/bluecore-engineering/the-infrastructure-triumvirate-continuous-service-and-infrastructure-delivery-with-argo-a33a3f76dc06">deploy it to our clusters</a>, and monitor it with our observability tools. This not only improves developer velocity but also ensures our services are adhering to vetted infrastructure patterns, such as using <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity">Workload Identity</a> for service account authentication or utilizing internal tracing libraries. In addition to creating and promoting adoption of these development happy paths, we also wanted to adopt a programmatic approach to enforcing best practices. This led us to the world of policies and <a href="https://www.openpolicyagent.org/docs/latest/#overview">Open Policy Agent</a> (OPA), an open-sourced policy engine that we’ve integrated into our platform to enforce policies in our Kubernetes software development life cycle (SDLC).</p><h3>What Is a Policy?</h3><p>Best described <a href="https://www.openpolicyagent.org/docs/latest/philosophy/#policy">here</a>, policies are rules that govern software. Policies can be defined at any layer of a technical stack and exist because they allow us to build compliant, technically sound, and scalable systems. Example policies include mandating labels on Kubernetes resources, provisioning infrastructure only through Terraform, or requiring distinct hostnames on Ingress routes. The best way to enforce policies in a scalable manner is to define them as code.</p><h3>What Is Open Policy Agent (OPA)?</h3><p>The most prevalent open-sourced tool we found to implement policy as code was OPA. Simply, it is a policy engine that separates policy decisions from application code, allowing it to integrate at many points of a pipeline. OPA uses <a href="https://www.openpolicyagent.org/docs/latest/#rego">Rego</a>, a declarative language, to define its policies. The following diagram illustrates how a service (which can be anything from the Kubernetes API server to a CI/CD pipeline to a database) can query OPA to evaluate arbitrary input JSON against policies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nMZ46OZivfBJVb5N" /><figcaption><em>Source: </em><a href="https://www.openpolicyagent.org/docs/latest/"><em>https://www.openpolicyagent.org/docs/latest/</em></a></figcaption></figure><p>An example workflow may be as follows:</p><ol><li>The Kubernetes API server receives a request to create a Kubernetes deployment resource in the “default” namespace.</li><li>The API server sends a request to the OPA server to verify that the deployment adheres to all relevant policies (the request contains the deployment as JSON).</li><li>The server runs the deployment JSON against the policies in its store and finds that the JSON violates a policy mandating that resources must have a namespace other than “default.”</li><li>The OPA server returns a “deny” response to the API server.</li><li>The API server rejects the request and the Kubernetes deployment config is not applied.</li></ol><p>Check out <a href="https://www.openpolicyagent.org/docs/latest/">OPA documentation</a> for more details on how the tool works.</p><h3>Integrating OPA Into Our Kubernetes SDLC</h3><p>Our initial use case for OPA was enforcing policies around our Kubernetes clusters. We wanted to be able to enforce these policies at three distinct stages of the software development life cycle: during development (before the Kubernetes resource is created), during deployment (real-time as the resource is being created), and maintenance (audit for already existing resources). Below, we describe how we set up our infrastructure to achieve this.</p><h4>Create centralized policies repo</h4><p>We first created a central repository to contain all our policies. These policies are written in <a href="https://www.openpolicyagent.org/docs/latest/policy-language/">Rego</a>, OPA’s query language, and are all unit tested. The repo’s CI runs these unit tests on every commit.</p><p>A typical workflow of adding a Rego rule would be as follows:</p><ol><li>Create a new folder under policy/kubernetes/ and name it using the hyphenated format of my-policy-name.</li><li>Define your policy in src.rego. The package should be your policy name in the format of my_policy_name.</li><li>Test your policy in src_test.rego.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/748fe25bf5463ebc5971c2f5bc440097/href">https://medium.com/media/748fe25bf5463ebc5971c2f5bc440097/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ab5243550b9d3f828fd113b10f83ddd/href">https://medium.com/media/6ab5243550b9d3f828fd113b10f83ddd/href</a></iframe><p>We use <a href="https://github.com/open-policy-agent/conftest">conftest</a>, an open-sourced framework that allows us to test Kubernetes YAML against Rego policies, to push the policies to Google Container Registry as a step in our CI. From there, we can pull the policies when needed for evaluation.</p><h4>Integrate OPA with CI pipeline</h4><p>At Bluecore, Kubernetes resources are defined in central manifest repositories and deployed to our Kubernetes clusters via Argo CD. We use conftest to validate all Kubernetes manifests against our policies in a policy validation CI step.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba118c8a703764a69414503a261a63dd/href">https://medium.com/media/ba118c8a703764a69414503a261a63dd/href</a></iframe><p>These steps run on commits to every branch so changes that don’t pass our policy checks are surfaced early in the SDLC and before they are deployed to our clusters.</p><h4>Integrate OPA with Kubernetes</h4><p>While the majority of our Kubernetes resources are validated with conftest before being deployed via Argo CD, resources may still be applied directly to our Kubernetes clusters. We want to ensure our policies are applied to resources in this scenario as well. To achieve this, we use <a href="https://github.com/open-policy-agent/gatekeeper">OPA Gatekeeper</a>, an active open-sourced project, that allows us to enforce policies in Kubernetes clusters in real-time via an admission controller webhook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tp5XMM04uRd0YS0BPvY5Lw.png" /><figcaption><em>Source: (left) </em><a href="https://www.openpolicyagent.org/docs/latest/"><em>https://www.openpolicyagent.org/docs/latest/</em></a><em> (right) drawn by author</em></figcaption></figure><p><a href="https://open-policy-agent.github.io/gatekeeper/website/docs/">Gatekeeper uses OPA</a> under the hood to enforce policies and the diagram above explains how Gatekeeper is analogous to the OPA paradigm of policy decoupling.</p><p>Gatekeeper consumes Rego policies as <a href="https://github.com/open-policy-agent/frameworks/tree/master/constraint">Constraint and ConstraintTemplate Kubernetes objects</a>. A ConstraintTemplate declares a new constraint type and contains the Rego to evaluate the policy. A Constraint is an instance of a ConstraintTemplate. Here is an example of the same Rego policy we saw above but defined within Kubernetes Constraint and ConstraintTemplate resources.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/143ac9b00ebbb9ebfea0502d3e7214e6/href">https://medium.com/media/143ac9b00ebbb9ebfea0502d3e7214e6/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6fc0f0bb3927fc02fcbbc230e7591b87/href">https://medium.com/media/6fc0f0bb3927fc02fcbbc230e7591b87/href</a></iframe><p>Once we had installed Gatekeeper in our clusters, we needed a way of generating and deploying the Kubernetes ConstraintTemplate and Constraint objects whenever policies were added, deleted, or updated.</p><p>To reuse the Rego policies we already had defined and avoid maintaining two copies of the same policy, we used <a href="https://github.com/plexsystems/konstraint">konstraint</a>, an open-source tool that generates Kubernetes Constraints and ConstraintTemplates from .rego files. We wrote a custom go script that, on every merge to the main branch in the policies repo, runs konstraint create policy/ — output constraints/ — dryrun, outputs all the generated Constraint and ConstraintTemplate YAML to a single file, and commits the file to our Kubernetes manifests monorepo. ArgoCD then auto-deploys the updated Constraints and ConstraintTemplates to our clusters for Gatekeeper to use. This pattern allows us to rely on the raw Rego files as the single source of truth for policies.</p><p>We also use konstraint to auto-generate documentation for our policies.</p><h4>Audit existing resources with OPA Gatekeeper</h4><p>Many of our existing Kubernetes resources did not adhere to the new policies we were adding. We wanted a way to easily surface these violations so our engineers could fix them.</p><p>Gatekeeper has an <a href="https://open-policy-agent.github.io/gatekeeper/website/docs/audit">audit feature</a> that runs at a regular, configurable interval and reports which resources violate existing policies. The results are stored in the Constraint resources themselves. We added a custom Argo CD health check to mark the resource as degraded if any violations of the Constraint were found. However, this still wasn’t the most user-friendly way of surfacing violations and we explored some options of reporting violations on a UI.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b64d9d790f540e94f02f113595b7f5d/href">https://medium.com/media/5b64d9d790f540e94f02f113595b7f5d/href</a></iframe><p>Since Bluecore runs on Google Cloud Platform, we first tried using <a href="https://github.com/GoogleCloudPlatform/gatekeeper-securitycenter">gatekeeper-securitycenter</a> to surface these violations on the Google console. The limitations around search and filtering were less than ideal. We then deployed <a href="https://github.com/sighupio/gatekeeper-policy-manager">gatekeeper-policy-manager</a>, which provides a much cleaner UI to view audit violations reported by Gatekeeper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RrRzIU1_36U--7cP" /><figcaption><em>Gatekeeper-policy-manager UI</em></figcaption></figure><p>We designed this Kubernetes policy enforcement pipeline to make it simple to add new policies as we evolve and establish new engineering standards. It has already proven useful in ensuring we have properly labeled resources, are deploying to the right clusters, and enforcing memory and CPU limits. We will be adding policy support via OPA for other parts of our platform in the future as well; for example, <a href="https://forsetisecurity.org/">Forseti</a> is a tool built on top of OPA that allows policy enforcement of Google Cloud resources.</p><p>Interested in helping scale our platform as we continue growing? We’re <a href="https://www.bluecore.com/careers/">hiring</a> at Bluecore!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd893f09a3ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluecore-engineering/kubernetes-policy-enforcement-with-open-policy-agent-cd893f09a3ab">Kubernetes Policy Enforcement with Open Policy Agent</a> was originally published in <a href="https://medium.com/bluecore-engineering">Bluecore Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>