The project entailed weekly meetings between the students and the Data Science project lead and mentor to ensure guidance around project parameters and to create a sense of community and connectivity throughout. With a visit to our headquarters to attend one of our global town hall meetings and feel a sense of community, the community that we were founded on, 60 years ago!

At the conclusion of the project, the students shared back their findings internally with our tech and product organization and at our virtual hackathon. The article below highlights one of the team’s research around Image Generation with Stable Diffusion. Thank you to the students for their research and efforts, we enjoyed having you and look forward to welcoming future capstone students from Columbia again. Please read on to hear more about the project in their own words.

Betsey Corey, Director, Strategic Talent Programs & Partnerships

Image Generation with Stable Diffusion

One of our favorite parts of the WeightWatchers app is the recipe recommendation section, where one can browse 12,000+ delicious (and healthy) meals. Each recipe is accompanied by beautiful, high-resolution images, which are often professionally created by WW. We tried to generate such images through latent diffusion models by simply giving the model a prompt, for example, “pasta puttanesca”. The recent release and rapid progression of diffusion models inspired this project and we sought to use stable-diffusion-v2–1 to generate pictures that resembled professional images as closely as possible.

Implementing the model & initial results

Getting things set up and generating the first images is very straightforward. We found running the model on a local or cloud GPU to speed things up considerably, especially when generating many images. At its most basic implementation, stable-diffusion-v2–1 only needs a prompt with a random seed being generated every time the model is run.

When experimenting with different recipes and food items, the model generated pictures of widely differing quality. Some prompts, such as full meal names (e.g., “pasta puttanesca with basil, black olives and tomatoes”) worked considerably better than single foods (e.g., “an apple”). Further, since most models generate a random seed each time the model is run, the same prompt will lead to vastly different outputs. This makes a rigorous evaluation more challenging. Here are some initial examples of images we generated:

(results for “clean pink table with a white round plate of chopped salad with sweet red pepper garlic clove […] string beans in the middle”)

(results for “apple”)

While these results are not terrible, they are still far from the professional benchmark we are comparing them to. The most noticeable deficiency at this stage was the appearance of unwanted items, such as hands, fingers, or cutlery, appearing on pictures despite the fact that our prompts did not include them. This led us to explore options to constrain the model or further specify instructions.

Improving the output

After some research and quite a bit of trial and error, we implemented the following measures to improve the outputs. First, we fed a list of negative prompts to the model, i.e. items that should be excluded from appearing. In addition, we varied the degree of complexity or detail of the prompts. Simply prompting with “Apple” yields different results than “high-resolution, shiny, centered Apple”. We tried to systematically determine whether complexity and detail in the prompt improved the images. We varied the seed and complexity of prompts for a given food item and created numerous images. Below, results for “banana” and “olive oil” are shown. Note that the complexity scale is arbitrary and subjective. Low complexity is something like “banana” while level 5 complexity is more akin to: “DSLR food photograph of normally shaped croissant, in a circular blue plate, white napkin background, light coming from the side, natural lighting, masterpiece, 4K, 85mm f1.8”.

This analysis showed that depending on the food, differing complexity levels of the prompts yielded different image qualities. To generate a realistic Apple or Banana, giving details such as “DSLR food photograph” and “high-resolution” proved to be effective. For items such as olive oil or salt, simplicity in the prompt was key.

The last avenue we have so far pursued is to use ControlNet which allows one to input a sketch and prompt, based on which the model will output an SD-generated image. This allows us to specifically draw out the position and shape of foods, which helps to constrain the outputs. See below for an example:

Looking forward, we seek to test some further ideas for improved prompt quality, such as using models like BLIP, which generate a prompt given an image. By feeding very high-quality images to such a model, we might be able to gain insight into how to phrase the prompts better.

Lessons learned

While the images we are currently generating are still not quite where we want them to be, the work with stable diffusion has taught us some useful lessons:

• When coming up with prompts, no one size fits all. Depending on the food to be generated, the prompt needs to be customized in order to get the best results.
• Working with negative prompts makes life considerably easier! Especially when seeking to eliminate elements that keep on popping up.
• Things move incredibly fast and each day new models are being published and resources made available. Staying up to date on Reddit, Discord and Huggingface has proven incredibly helpful.

For inspiration, here are a few of the most recent images we generated:

Voice Controlled Food Tracking

Background

Developing a food-tracking habit is consistently shown to be the most important factor in weight loss. WW is constantly looking for ways to make food tracking easier and exploring the use of AI tools to create better food tracking experiences. In this capstone project, our team built a website where people can talk about the food they eat, and the website can extract the food entities from the audio, matching with the current WW database, and returning the points.

The current way to track food on the Weight Watcher app is to manually search for the food that you eat for every meal. However, this process could become time-consuming and inefficient. For instance, Sweetgreen is one popular option among current customers. However, when it comes to a customized bowl, it could easily contain more than 10 different ingredients. This kind of complex real-life scenario deters users from tracking food consistently and accurately when they need it most. Taking images of food is a standard industrial solution and also might be your first thought. However, images could encounter problems as well. A human or model may be able to identify a casserole from an image, but this does not provide enough information to determine how we should track its nutritional value, which can vary widely depending on the ingredients used.

On the other hand, talking is a low-friction activity. People love to talk about food and can describe a whole meal in a few seconds. We believe that voice is the lowest friction method to track food. We also believe that the latest advances in open source and multimodal AI models have put “voice-to-track” within reach. In the rest of the blog, we will introduce it to you in more detail.

App Structure

Let’s take a closer look at the app structure and tech stack. At a very high level, the app is mainly coded in Python. We utilized FastAPI, used state-of-the-art models from open source communities, which are mainly OpenAI and Hugging Face, and deployed with docker. For the front end, we used Streamlit, which is a free and open-source framework to rapidly build and share beautiful data science web apps.

To transcribe a user’s audio describing the food to text, we used the Whisper model from OpenAI. To extract food names from the transcribed text, we tried both food-based BERT and GPT API. Next, we encoded the found food names in embeddings with SBert and then compared their similarity with encoded food names in the WW database using FAISS. Finally, we return the most related food names in the database to the front end.

App Demo

The figure below is a screenshot of the user interface of our app. First, users would be asked to record their voice by clicking on the microphone icon. Once the recording is successful, users would click on the “Detect” button to initiate the audio transcription, food extraction and matching process. The final matchings along with their respective portions, units, and points would be returned to the user and the user would provide feedback on the accuracy of the results by either selecting “Yes!!” or “No, this is not what I ate!” to answer the question “Is this what you ate?”. If the latter is selected, the user could choose to manually type in what they ate in the text box and new food extractions as well as final matchings would be generated based on the input in the textbox. Given that the matching is correct, the user could then click on the drop down menu on the rightmost column to adjust the amount of food intake as well as the unit. In addition, we also allow users to choose OpenAI’s GPT 3.5 Turbo as an alternative for Food-Based BERT to perform food entity extraction. Users could switch to this option by selecting the “OpenAI GPT 3.5-turbo” button and inputting their OpenAI API key in the leftmost column. Overall, our clear and interactive user interface allows users to track their daily food intake in an efficient way and provides highly customized results.

User Expression Pattern Research

Having our model at hand, we also conducted research on the user side to gain a deeper understanding of how people would use our app. We collected 112 recordings of people describing what they ate on the Columbia University morningside campus and performed analysis on these data collected in an attempt to find patterns in user expressions. The insights we gained from these real-life user data are as follows:

• Insight 1: Users almost always use ‘a’ to specify a single item but they tend to use ‘some’ when they had more than one item rather than specifying the specific amount
• Insight 2: When people describe what they ate, the name of the restaurant/ store is often neglected
• Insight 3: “and” usually connects two food entities whereas “with” might connect two food entities (e.g. toast with tea) or might be used when the user wanted to provide a more detailed description of the ingredients inside a food (e.g. coffee with oat milk/ bagel with cream cheese)
• Insight 4: Nouns (mostly foods) occur most frequently in people’s descriptions, with an overall frequency of 0.24. Better prompts might reduce the percentage of redundant information such as adjectives and adverbs.

Results

The accuracy of food entity extraction was compared between OpenAI GPT method and Food-Based BERT method. Out of all 112 collected transcribed texts, the Food-Based BERT method correctly extracts all food entities from 56.8% of the sentences, while the GPT method’s performance is 81.1%. Across all sentences, the Food-Based BERT method correctly detects 76.6% of entities, compared to 93.4% using the GPT method.

Next Steps

Potential future improvements that could be made to our app include running Whisper model on GPU to increase speed, finding a better name entity recognition model in terms of accuracy and speed, and adding aesthetic elements to our user interface. The next step would be to integrate our model with the WW app to provide users with this time-saving alternative for food tracking.

Interested in joining the WW team? Check out the careers page to view technology job listings as well as open positions for other teams.

Data Science Capstone Projects was originally published in WW Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

When you embark on a journey to improve your health and wellness, making informed choices on which foods you consume to fuel your body is crucial. At WW, our behavior and nutrition science teams conduct clinical trials [ref] and constantly monitor the latest scientific literature to provide our members with the most current information on which nutrients and overall dietary pattern will maximize health benefits. For example, we know that consuming fiber is beneficial for heart health while protein provides the body with important building blocks. On the other hand, saturated fats are not beneficial for heart health, and consuming sugar, especially added sugar, can increase the risk of both obesity and diabetes [ref]. We take the complexity of all of this nutritional information and summarize it into a single value (known as Points®) to behaviorally nudge our members to an overall healthier dietary pattern using the most cutting-edge nutrition and behavior change science.

This is why we strive to have the most comprehensive and complete food database in the world. However, with hundreds of millions of foods from around the globe, this is a large challenge for our nutritionist team to handle manually. Therefore they are in constant and close collaboration with us, the Data Science and Machine Learning Engineering team at WW, to fill in the blanks and provide them with precise estimates of missing values. One of the contact points where this need becomes especially apparent is when our members use the points calculator in our app. There they can input the nutrition information listed on a food label to instantly receive the Points® value for any food that they consider consuming. Unfortunately, many countries do not require all information that has been identified by scientific research to be important to estimate the impact of a food on health and wellness to be reported on the label. For example, the US only started requiring listing added sugar content in 2020/2021 [ref]. Still, in many other countries currently added sugar is not required to be listed on the label, despite its known detrimental effects on health.

In this blog post, we discuss how the Data Science and Machine Learning Engineering team at WW built an ML model to estimate the added sugar content of a food, and how we made this information available to be consumed by the calculator within the WW app as well as our internal database. As a first step, we worked together with our nutrition science experts to review currently available methods of added sugar estimation. One of the most used approaches is based on a manual process [ref], which is very valuable as an internal tool for our nutritionist team when they review existing entries in the database. However, this manual process would not have been suitable for use within our member-facing points calculator considering the subjective evaluations involved. After in-depth discussions with the nutrition science team, we came to the conclusion that the problem could be approached as a machine learning problem. After trying out alternative approaches such as linear regression and KNN we decided to go ahead with building an XGBoost [ref] model, a method which has shown great success discovering nonlinear relationships in tabular data. In addition to the high accuracy of the resulting model, due to its scalability XGBoost also seemed like an ideal choice of model as we expected the final resulting endpoint to be able to handle individual requests from the calculator with fast response times, as well as being able to respond to batch requests for backfilling suggested values to be reviewed in the food database.

As a first iteration, we went ahead and collected all food data available in the US which already had entries for added sugar values. Using total sugar (i.e. all sources of sugar, including naturally occurring and added sugars), carbohydrates, fiber, total fat, sodium, calories, saturated fat as well as food category as features, we built an XGBoost model on 80% of the available data and performed hyperparameter optimization on it. During this initial exploration phase, we also learned that training on all foods, including the ones with no sugar present (where manual estimation of added sugar would automatically default to zero) yielded a more reliable model, presumably because it allowed XGBoost to learn relationships among all nutrients present. Using the resulting model to predict added sugar values on the 20% hold-out dataset, we saw very encouraging results, which we published in a scientific article [ref]. However, we did not think a model validated on only US foods would deliver sufficiently reliable results to our members, especially globally.

As a next step, we asked our global team of nutritionists to evaluate our model predictions on manually estimated added sugar values on international data (as due to the lack of added sugar information no ground-truth dataset is available). Unsurprisingly, the manual process identified that for many foods from our international food databases, the added sugar predictions were suboptimal, and that there were specific food categories in which this was the case, for example for sugary drinks and milk products. Our nutrition science team focussed on the areas where the model was most wrong, and in several iterations, we added newly verified international foods to our training and test datasets. As the model will mostly be used in markets that do not have added sugar on the label, i.e. outside the US, we made sure that at least 30% of the dataset consisted of non-US foods. We also learned that adding the food category as a feature increased model accuracy only marginally, so we decided to remove it from the model in order to increase usability and decrease guesswork for the members when using the calculator.

In order to gain a more intuitive understanding of how the XGBoost model uses feature values and their interactions to make predictions, we also looked at SHAP values, a measure of feature importance based on game theory [ref]. The figures below provide an example of SHAP plots; each dot represents one specific food, the horizontal axis shows the value of a feature (total sugar or fiber), and the vertical axis represents which direction the value of the feature changed the added sugar prediction for each food. The top figure shows that on average the model predicts higher added sugar values for a food with higher total sugar. Intuitively this makes sense — the more sugar a food contains, the more we expect it to contain added sugar. The bottom figure adds some nuance to this observation: here, each dot is colored by its sugar value, i.e. foods high in sugar are shown in red, while foods low in sugar are shown in blue, and the value of fiber is shown on the horizontal axis. For foods that are high in sugar, but also high in fiber, the model predicts lower values of added sugar. This makes sense as foods naturally high in fiber, such as fruits, tend to have little to no added sugar. Considering such summary plots provides transparency into how the model derives predictions. This understanding can then serve as a basis for discussions with our nutrition science team to pinpoint areas where the model is underutilizing specific information and should be retrained with better, more specific, or simply more data.

After several iterations, we were able to achieve an average mean absolute error of 0.8g (corresponding to 3.2 calories) at the default portion size on global foods. As there are an average of 180 calories in a portion, and the USDA suggests consuming not more than 10% of calories from sugar [ref], we felt that this was a reasonable starting point for a first launch into production, with the prospect of iterating on the model as more data becomes available.

At WW we believe that it is crucial that the Data Science and Machine Learning Engineering team itself is responsible for deploying models [ref] so iterations can be done seamlessly. To expose the model we set up an endpoint in FastAPI [ref], which is a Python Framework and set of tools that enables developers to use a REST interface to call commonly used functions to implement applications. By sending requests containing necessary ingredient information to the endpoint, the estimated added sugar result will be included in the response to be consumed both by the calculator to provide live initial estimates, and by our database to prefill an estimated added sugar field for review by the nutrition science team. Below you see the calculator as it appears in the US (where added sugar is available on the labels; “Added sugar” is a field required to be filled out by the member) and in the UK (where added sugar needs to be estimated; no “Added sugar” field available):

Given the global application of the added sugar model, the endpoint is able to handle requests that are slightly different among different markets and apply corresponding logic. For example, in the US fiber is included under total carbohydrates, while in the UK it is not; the endpoint automatically takes this into account when it interprets the incoming information from the nutrition label. Additionally, while it is often voluntarily disclosed by manufacturers, fiber information is not required on the label in the UK. If this optional information is missing, the endpoint will first apply a fiber estimation model, which we reviewed both independently and in conjunction with the added sugar model, to get the best estimation of the field and then pass the value to the added sugar estimation model.

This endpoint has been used to batch-backfill all foods in our current database with estimations to provide guidance for manual review and is being currently hit about 100 thousand times a day, providing our members with the information they need to make informed decisions on which foods they consume to support their health and wellness journey.

Interested in joining the WW team? Check out the careers page to view technology job listings as well as open positions for other teams.

Combining nutritionist expertise and ML to provide high-quality nutrition data worldwide was originally published in WW Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

X12 is a message formatting standard used for Electronic Data Interchange (EDI) documents. It’s used by trading partners to share business documents in an agreed-upon and standard format. X12 is the most common EDI standard used in the United States.

Our vendor was using EDI to send payment details to WW via an 835 file. A brief description of the 835 transaction is below:

The Electronic Remittance Advice (ERA), or 835, is the electronic transaction that provides claim payment information. These files are used by practices, facilities, and billing companies to auto-post claim payments into their systems.

The message flow was unidirectional — from the vendor to WW and only the 835 transaction was being sent to us.

Research was done to find 3rd party software that we could leverage to easily communicate using EDI. We decided to use the Transformer software provided by Trace Financials as it provided flexibility for parsing EDI 835 transactions as well as dozens of other EDI transactions.

Via the Transformer tool, we were able to parse the 835 transaction, and via scripting provided by the tool, extract the required fields from the transaction that were needed for processing. The Transformer tool enables you to create a service and export a jar file with callable APIs that can be used by an application.

Below is a code fragment that shows the service API being called from a route in Java code. Some items were removed or generalized for privacy reasons.

// Process 835 file to extract information and store it in the database
from(“direct-vm://process835”)
.routeId(“process835”).routeDescription(“process835”)
.pollEnrich().simple( /* file details were removed */ )
.convertBodyTo(String.class)
.log(“calling txfrmr”)
.to(“txfrmr://ServiceName/APIName”)
.log(“txfrmr called — ${body.size} records”)
.split(body()).unmarshal()
.jacksonxml()
.to(“direct-vm://insertDataToDatabase”);

We configured the Transformer tool to parse the 835 transaction and extract the data fields of interest. Via the tool, we generated a jar file with an API endpoint for the data mapping and used this jar with our deployed application. When an 835 file is sent from the vendor, our route calls the exposed API which parses the transaction and returns the data which we split, unmarshal, and process.

This past year, we started working with a new vendor that also uses EDI transactions. The interface for this vendor is more complex. The message flow is bidirectional and uses multiple EDI transaction types including acknowledgment transactions.

A high-level description of the transaction flow is given below:

• When an order is placed, WW constructs and sends an EDI 850 “purchase order” transaction to the vendor and they respond with an EDI 997 “acknowledgment” with a status indicating if it was successfully processed.
• When the vendor ships the item, they send an EDI 856 “advance shipping notification” transaction to WW and we process it and return an EDI 997 “acknowledgment”.
• At some point, the vendor sends an EDI 810 “invoice” transaction to WW and we process it and return an EDI 997 “acknowledgment”.

The Transformer tool was configured to parse the different EDI transactions. We created a jar with multiple API endpoints (one for each transaction’s data mapping) and used this jar with our deployed application.

In addition to the added complexity of using bidirectional EDI transactions along with acknowledgment transactions, we needed to generate a unique sequence number for each outgoing transaction as this was required by our vendor. Since our application uses a database to keep track of purchase orders, shipment details, invoice information, and other data — we used the database to generate a unique sequence number for any outgoing transactions.

Due to the sensitive nature of some of the data being sent between WW and our vendor, we are encrypting EDI message files sent to the vendor and the vendor is encrypting EDI message files sent to WW.

I hope I gave you a high-level idea of how we incorporated X12 EDI transaction processing into a few of our apps at WW. For details on X12, please visit x12.org.

Interested in joining the WW team? Check out the careers page to view technology job listings as well as open positions for other teams.

Using X12 EDI with Red Hat Fuse/Apache Camel Apps was originally published in WW Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

After a multi-leg train ride from Heathrow Airport, I was somewhere in the southwest of England waiting for a good friend to pick me up. I mistakenly took a pricier express train and was 45 minutes early, hoping Kristina had gotten my text. I hadn’t heard from her all morning, but I lucked out. I learned by chance she stepped out to check her text messages at a specific spot on the street. Kristina, her wife, Sarah, and her terrier pup, Lila, moved to this speck on the map for many reasons, one being a reset from city and work life after the pandemic.

I listened to the click-clack of horse hooves on cobblestones while watching some teenagers aboard a tractor drinking as one of them drove to the pub. It felt so otherworldly, light-years from our dual-monitor home offices in a fourth floor walk-up with a video doorbell monitoring the surrounding area or to the local Starbucks where our smartphones talk to smart cash registers to redeem a free coffee. In cities and suburbia, tech is inescapable and pervasive. Does rural life — so distant from ours — need access to technology as someone living in the big city?

Kristina yelled at Lila, “Don’t eat sheep poop!” The dog responded by pulling harder on the leash. The four of us strolled along a public path that traversed through a farm, woods, and grazing land, avoiding stinging nettles as we passed thigh-high grass, hedgerows, sheep, scampering quail, cows, the occasional human, and wild pheasants struggling to keep their distance from the terrier. We made our way to the top of a hill overlooking the town of Wiveliscombe (pronounced Wiv-el-is-cum) with a population south of 3,000. Pondering life with one of my best friends, we paused to admire our surroundings. I slid my phone out of my pocket — noticing my pants were splattered with brick red mud — and snap, snap, snap, I digitized the vista, opening Instagram without thought to get some likes. I quickly realized I was barless! Kris turned to me and said, “No service out here.” Without a choice, I quickly adapted to the limited tether and enjoyed the rest of our nature walks without the likes.

The question still remains: Why disrupt such tranquility with better access to tech?

One of my guiding principles in accessibility is “everything should be accessible. Don’t limit access to impose morals.” Accessibility is about choice. The example I always use is the soda vending machine. Should the vending machine be accessible? Someone could argue soda is bad, why make it easier to drink? If that’s the case, do it through laws, not by imposing your morals through lack of accessibility.

The case to build tech that loads fast with slow internet, and functions offline is the same. Access to tech in rural areas can mean checking for directions while meandering in the woods, calling for help where no people are within earshot, or a farmer at the market completing a digital transaction. Access to technology is about choice, and making tech work in rural areas is one more facet of accessibility.

Even potential pitfalls, like the unending itch to get likes or doomscrolling should not be used as an excuse to deny access to the technology (it’s a whole other problem that could be attacked using more thoughtful design, but that’s a topic for another blog post.)

On a side note, cutting-edge solutions like Starlink (satellite internet) will help to bridge our worlds, but it won’t solve the problem of a lack of high-speed internet or tech that is usable offline.

Our company data shows that one of the most used features of the WeightWatchers® app is the barcode scanner, which was designed so members can quickly know the Points® value of a packaged food while at a grocery store. The barcode scanner relies on a database in the cloud, so if our app can’t send a request to the server, the barcode scanner is useless. What creative ways could solve this problem? Thinking out loud, one solution could be when the member is online, downloading the necessary data of foods in their area so the app isn’t reliant on a server when offline. The point being, solutions are reachable and quite doable.

That’s just one example of how the WW app can be improved when offline. A very high percentage of popular apps today have any number of downfalls when it comes to load times and offline abilities. Why is that?

It’s the same reason everything isn’t accessible. The two big reasons: (1) Lack of representation. (2) It’s human nature to design for yourself, who you know, and where you live. Many tech companies generally are in San Francisco and New York City — cities with high speed internet and cell service, where everyone is using the latest, greatest, and fastest technology. These two reasons create an unconscious bias where people overlook the value of designing for poor people, or for people with slow or no internet access.

But there are solutions to these problems. Start by adding to your definition of what diverse hiring practices look like, including looking at people from rural areas and a range of income brackets. Companies can also build low internet, no internet practices into your product lifecycle — from research to design to testing.

Connecting everyone to the internet means more than not getting lost in the woods, it means climate refugees can find resources and shelters, it means anyone who relies on assistive technology can continue to do so wherever they are, and, of course, it can mean anyone at any grocery store can make healthy food choices.

I really enjoyed being disconnected in rural England but I also enjoyed sending my parents back home pictures once I was on Wi-Fi. My personal takeaway is to take more walks disconnected, but my professional takeaway is that we have an obligation to connect everyone and to continue to make everything accessible.

I really enjoyed being disconnected in rural England but I also enjoyed sending my parents back home pictures once I was on Wi-Fi. My personal takeaway is to take more walks disconnected, but my professional takeaway is that we have an obligation to connect everyone and to continue to make everything accessible.

The choice to be disconnected was originally published in WW Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Logs in JSON format

In order to get the logs in JSON format, we need to let the Log4j2 know about the layout and we will use a template for our logs which will look like this:

JsonTemplateLayout

Create a new file under the src/main/resources folder by the name — Log4jEventLayout.json.

You can find additional details about the template layout here.

Couple of things to note here:

1. You would see we are duplicating trace_id and span_id by adding trace.id and span.id. This is intentional as NewRelic does not understand trace_id and span_ids.
2. The service.name is externalized and will be replaced by the value stored in the environment variable OTEL_SERVICE_NAME variable.

Letting Log4j2 know about this configuration

This can be done as follows:

<?xml version=”1.0" encoding=”UTF-8"?>

<Configuration status=”WARN” monitorInterval=”30">

<Appenders>

<Console name=”ConsoleAppender” target=”SYSTEM_OUT” follow=”true”>

<JsonTemplateLayout eventTemplateUri=”classpath:Log4j2EventLayout.json”/>

</Console>

</Appenders>

<Loggers>

<Root level=”info”>

<AppenderRef ref=”ConsoleAppender” />

</Root>

</Loggers>

</Configuration>

At this point, all the logs spit out by our application should be in JSON format.

{“timestamp”:”2022–08–19T12:55:33.485–0400",”thread.name”:”restartedMain”,”log.level”:”INFO”,”logger.name”:”com.example.otelservice.OtelServiceApplication”,”message”:”No active profile set, falling back to 1 default profile: \”default\””,”service.name”:”wcp-otel-service”}

Some samples with the span and trace details.

{“timestamp”:”2022–08–19T13:14:56.615–0400",”thread.name”:”http-nio-8080-exec-1",”log.level”:”INFO”,”logger.name”:”com.example.otelservice.service.GreetingService”,”message”:”greet service invoked”,”trace_id”:”f05a3a2dfcc8d04d4f18beaa11f17909",”span_id”:”88477fb780ebf427",”trace.id”:”f05a3a2dfcc8d04d4f18beaa11f17909",”span.id”:”88477fb780ebf427",”service.name”:”wcp-otel-service”}

OpenTelemetry instrumentation

As per the instructions on the OpenTelemetry website, we need to add a java agent to your application. This is going to be a two-step process. First download and stage the instrument library at a known location, and then configure the spring boot maven plugin to use the java agent accordingly. This is accomplished using the maven-dependency-plugin. Refer to the pom.xml file for the project.

Introduction

Before diving into the implementation details, let’s take a look at the high-level architecture of the collector. The following diagram is from their website:

The OpenTelementry collector is based on a pluggable model with three primary components as can be seen above. You have the

1. Receiver
2. Processor
3. Exporter

Set up otel-collector

For this demo, we are going to use the docker image provided by opentelemetry-collector-contrib. We will start off by creating two files.

• otel-config.yaml
• docker-comopse.yaml

Please refer to the github project for references to these files.

Bringing up the docker instances

At this point, we are ready to take our application and OTel Collector for a spin. Before we bring up the Oel Collector, we need to export the NewRelic API key. Export your API key like so — export NEW_RELIC_API_KEY=<YOUR_API_KEY>

Now we can start the docker instances by firing up the following command: ./mvnw clean install && docker compose up –build

The first part of the command outputs the uber-jar and places the opentelemetry-javaagent.jar file under target/agents directory. The second part of the command first builds the docker instance for our service and brings up the collector along with it.

Try to call our service a bunch of times using the following curl command — curl http://localhost:8080/hello\?name\=OpenTelemetry

You should be able to see our new service under Service — OpenTelemetry

Head over to the “Transactions” from the left navigation menu and you should see something like this.

Click on one of the traces below you will be able to see one trace and a couple of spans.

Taking care of logs

Now that we have trace flow working, it would be nice to see the logs adjacent to our traces. This will help triage any issue we are trying to resolve. To get the logs exported from our application to NewRelic we will take a little help from FluentD. You can read more about FluentD and the underlying architecture here.

Log flow

At a high level, this is what the flow will look like.

Setup the Fluentd instance

The first step is to configure the Fluentd instance to receive incoming logs on port 24224. After receiving these logs, we will want to filter all the logs in JSON format and send it over to our OpenTelemetry forwardreceiver listening on port 8006.

Please refer to the fluentd.conf and the fluentd docker service in the docker-compose.yaml file in the github repo.

Final Results

Once we have established the Fluentd instance, two things should happen on the NewRelic portal. Under the wcp-otel-service, when you click on Logs in the left navigation menu, you should see your application logs.

Click on Open in the logs to get a view.

This allows you to see all the application logs in one place, but we can do better than that. Earlier we said it will be really helpful to associate the traces and the logs. If we head over to the Transactions menu, you will see the usual transactions.

Now if you click on one of the traces you should see something like this.

Interestingly, now the Logs aren’t showing 0, but 2 2. Heading over the logs you should see something like this.

Now we have all traces and spans. This should help to debug issues easier than before.

Next Steps

In the subsequent post, I would like to cover the following topics:

• Logging exception traces appropriately
• Exposing critical metrics and sending them to NewRelic via our OpenTelemetry Collector.

Interested in joining the WW team? Check out our careers page to view technology job listings as well as open positions on other teams.

Springboot/Java application observability with OpenTelemetry & NewRelic — Part 2 was originally published in WW Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Weaving OpenTelemetry, FluentD, and NewRelic for complete observability

Observability: more is less

Being able to track individual transactions and see application logs in one place is a requirement when we are dealing with a micro-services explosion. You can head over here to read more about observability and distributed traces. There are two parts to this puzzle though. We need the ability to generate traces and spans, usually achieved by instrumenting the method calls and tying application logs to these traces and spans.

We will walk through the entire process and, in the end, you will have a working example that would look something like this on NewRelic. The source code of the working example can be found here.

Trace and span view

Application logs associated with those traces and spans

Trace and span setup

Setting up traces and spans in a Springboot application is relatively simple. We will use

• Free account setup for NewRelic.
• Springboot application with Log4j2 logging in JSON format.
• Wrap up the application with OpenTelemetry instrumentation.
• Use Docker to bring-up OpenTelemetry Collector to talk to NewRelic.

NewRelic account setup

If you already have an account then simply move to the next step. Otherwise, follow along. You can sign up for a free NewRelic account using the following link.

API Keys

From Account Settings, head over to the API Keys as shown below.

Create a new ingest– (or license) key and provide a friendly name to the key on the following screen.

Take note of the key being generated. We need to pass on this information to the OpenTelemetry Collector later. In case you already created a key before and you want to reuse the same key you can get a key like so -

SpringBoot application with Log4j2

Head over to start.spring.io and create a new springboot project as per your liking or simply use this configuration of the spring initializr to download the projectused in this demo.

Additional dependencies

Modify the pom.xml file for the project you just created above by adding the following properties/dependencies:

Properties

Add the following property to the “properties” section of your pom.xml.

<opentelemetry.version>1.16.0</opentelemetry.version>

Dependencies

Head over to the dependencyManagement section and add the following snippet to your pom.xml.

<dependencyManagement>

<dependencies>

<dependency>

<groupId>io.opentelemetry</groupId>

<artifactId>opentelemetry-bom</artifactId>

<version>${opentelemetry.version}</version>

<type>pom</type>

<scope>import</scope>

</dependency>

</dependencies>

</dependencyManagement>

Additionally, add the dependencies for Log4j2 and OpenTelemetry, like so:

<! — Log4j2 →

<dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-log4j2</artifactId>

</dependency>

<dependency>

<groupId>org.apache.logging.log4j</groupId>

<artifactId>log4j-layout-template-json</artifactId>

<version>2.17.2</version>

</dependency>

<! — OpenTelemetry →

<dependency>

<groupId>io.opentelemetry</groupId>

<artifactId>opentelemetry-exporter-otlp</artifactId>

</dependency>

<dependency>

<groupId>io.opentelemetry</groupId>

<artifactId>opentelemetry-extension-annotations</artifactId>

</dependency>

Furthermore, we need to ensure that we are excluding Log4j version 1 libraries like so:

<dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-web</artifactId>

<exclusions>

<exclusion>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-logging</artifactId>

</exclusion>

</exclusions>

</dependency>

New controller and service

This is the most boring controller and service you will ever see, but it is good enough for the purpose of this demonstration.

GreetingService

Create a new service.

@Service

@Slf4j

public class GreetingService {

@WithSpan

public String greet(@SpanAttribute String name) {

Span.current().setAttribute(“param-name”, name);

log.info(“greet service invoked”);

return String.format(“Hello %s”, name);

}

}

GreetingController

Create a new controller.

@RestController(“/”)

@Slf4j

public class GreetingController {

private final GreetingService greetingService;

@Autowired

public GreetingController(GreetingService greetingService) {

this.greetingService = greetingService;

}

@GetMapping(“/hello”)

@ResponseBody

@WithSpan

public String greet(@RequestParam(“name”) String name) {

log.info(“Greet Controller invoked”);

return greetingService.greet(name);

}

}

At this point, we have our controller ready and it is communicating with our service. We can test this out by running the application either using the IDE or maven. The following curl should respond with “Hello OpenTelemetry” — curl http://localhost:8080/hello\?name\=OpenTelemetry

Springboot/Java application observability with OpenTelemetry & NewRelic — Part 1 was originally published in WW Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Betsey Corey, Director, Strategic Talent Programs & Partnerships

Personally, I believe that one of the best parts of the WeightWatchers program is the concept of ZeroPoint® foods. These are foods that members can eat without tracking them or deducting any Points® from their daily Points Budget, giving them even more flexibility in what they choose to eat in a day. Earlier this year, WeightWatchers enabled flexible ZeroPoint categories, which adds logic in our food database for determining which foods should be valued at 0 Points.

This posed a complex problem. Each food has to be reviewed by the nutrition team to calculate its composition and be assigned to the right ZeroPoint foods category. Then a discount value is applied so that tracking these foods won’t deduct Points from a member’s daily Budget if the foods are in the member’s ZeroPoint foods list. For example, if “nonfat yogurt” is a ZeroPoint food for the member, then “vanilla nonfat yogurt” should also be considered as a ZeroPoint food and take no Point off for that member. In addition, this deduction should be applied even if the ZeroPoint food is only partially present in the food. For example, if avocado is a ZeroPoint food for the member, then the corresponding Points for avocado should be deducted from the total points of Chicken Avocado Salad.

Currently, we have millions of foods in our database. Manually evaluating each food and then assigning it to a category can be a huge undertaking. However, we hypothesized that this is the exact type of complex problem that is actually well suited for machine learning. We set out to discover if our predictions may be true by having Shuning, our data scientist intern, take this on as her summer research project. Our objective was to evaluate and confirm if we can solve this issue using machine learning tools or not. Shuning was able to solve this by using a combination of features and advanced techniques of stacking multiple machine learning models to create a proof of concept that worked really well. So what were the data, models, and results from Shuning’s research? Let’s break it down further:

Data

Data used in this project came from multiple sources. WeightWatchers has a presence globally, but to keep things manageable, this project focused on the U.S. market, where we collected a dataset of more than 50,000 foods. Variables used are food type, category, nutrition information, and ingredient statements.

Model

The four machine learning models used were decision tree, random forest, XGBoost, and LGBM. Prediction outputs from the four models, harmonic mean of the prediction outputs from all four models (harmean 4), and harmonic mean of the prediction outputs from random forest, XGBoost, and LGBM (harmean 3) are evaluated.

The metrics used for evaluation were R-squared, RMSE, and accuracy based on different residual thresholds (0.05, 0.1, 0.15, and 0.2). R-squared finds how much of the spread of the data can be explained by the features. The closer to one, the better it is. RMSE gives the difference between predicted value and actual value on average. The closer to zero, the better it is. The prediction output is considered accurate if the difference between actual value and predicted value is within a threshold level that allows for small errors in the estimation.

Results

Metrics are compared among different models and prediction outputs. Overall, harmean 3 gives the best result. In addition, two sets of variables are run — one without the ingredient statement and one with the ingredient statement. This is done to check if adding ingredient statements will improve the performance as this feature increases model complexity and places additional demands on data quality. It turned out that ingredient statements are very helpful. There are improvements on all metrics. See the below table for the corn category as an example.

To test model performance on new data, 55 foods were selected for testing. In general, the prediction outputs are good from an outsider perspective, even though it can sometimes underestimate the percentage of ZeroPoint foods within a dish. But since there are many rules behind food adjustment, further evaluation by the nutrition team is needed.

Next steps

This research project has proven that the problem is solvable. The next step is to make modifications and put them into production.

Shuning Yang — Data Science Intern

Interested in joining the WW team? Check out our careers page to view technology job listings as well as open positions on other teams.

Predicting food composition using machine learning was originally published in WW Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

WHY

The legacy system relied heavily on a third-party service that acted as a middleman between data providers like Apple Health, Fitbit, Garmin, and our WeightWatchers services. They were responsible for integrating with those providers and normalizing the data before sending it to WeightWatchers. There were various drawbacks with the legacy system:

• Did the issue occur on the member’s device? The data provider? The third-party service? WeightWatchers services?
• The database design did not scale well because it wasn’t originally designed to support third-party data and it was hard to track updates to existing data.
• There was little auditability which made it difficult to debug.
• It was expensive to operate and maintain.

There was an opportunity to rebuild the application so it is reliable and correct to address those pain points and also support future use cases, we called it Health Tracking Service.

HEALTH TRACKING SERVICE (HTS)

HTS is responsible for integrating with third-party data providers, storing exercise data, and forwarding sleep and weight data to our respective microservices.

Since HTS was going to replace an existing system, we used that as an opportunity to deploy the service to the production environment in the background and send it production-level reads and writes which helped us discover issues and address scaling concerns without impacting members.
When HTS was ready, we would switch the traffic from the existing system to HTS without the impacting the member’s experience.

We started the design process by utilizing a diagramming tool that helped us reason about the flow of data and present our ideas and methods to the engineering council for review.

We ended up creating 26 (and counting) pages of diagrams from very high level to very low level and it helped communicate ideas effectively.

The Health Tracking System is denoted by the purple colored boxes. At the core of HTS is a Node.js HTTP server that is responsible for handling communication between external clients and internal systems. It is serving tens of thousands of requests per minute to support our WeightWatchers apps and data from third-party data providers like Fitbit. We recognized that providers will send large amounts of data into HTS continuously and processing synchronously would present some performance and scaling challenges. For that reason, we decided to use a streaming architecture where the core of HTS simply queues the provider’s data for asynchronous processing.

STREAMING ARCHITECTURE

We used Apache Kafka for our event-based streaming system, which allowed us to process data in near real time. That grants better auditability since each event is a single immutable event from which we can derive data in a deterministic and repeatable way. The event from each provider would be placed in a provider-specific Kafka topic with multiple partitions. Each partition would have its own consumer, which allowed us to scale each provider independently. Each consumer would be responsible for validating and processing that particular provider’s event.

Before the event would be placed in the topic, we recorded it into an event log database table that would be updated every time the event entered a new phase. The event log entry also helped deduplicate events in the case there was a network issue or the provider sent duplicate requests before the original request was processed.

We used a hash function on the event as the unique identifier to determine if there is a duplicate entry. This worked well for push events since it contained all necessary data, but presented challenges with ping events. A ping event indicates that there are updates that need to be fetched; so while the event payload is the same, the fetched data from the provider may have changed. In the case of ping events, we added an additional check to see if we’ve seen the hash and it has not been processed.

THE EVENT LOG

One important tool for debugging is the ability to trace data so all events are logged to a PostgreSQL table, which keeps track of the state of the event as it moves through the system.

It allowed us to have some insights about the system:

• Who is the event for?
• How was it initiated? (provider notification vs manual action)
• Was the event picked up for processing?
• Did the event succeed or fail?
• If it failed, what was the reason?
• How long does it take to process the event?

THE PROCESSING

In general, the processing boils down to:

• Fetch incoming data from the provider (in the case of ping events).
• Verify the integrity by validating the provider data against a known schema.
• Transform, filter, and aggregate the data.
• Fetch the member’s previously stored data to compare against the incoming data.
• Store the updated data.

The event might have information about the member’s exercise, sleep, and weight. HTS makes its best effort to process all the data in the event. If part of the event cannot be processed, it will skip that part and continue processing the rest of the event.

There are many systems that are involved in processing a single event, which means there is a large surface area for failures.

REASONS FOR FAILURES

Failure to process an event can have many culprits, including but not limited to:

• It failed validation because it expected a specific data structure and/or values.
• It failed to fetch data from external or internal systems; maybe there was a network fault, service didn’t respond fast enough, or the service was unavailable.
• It failed to read from the database.
• It failed to write to the database.

RETRY MECHANISMS

Because the system was designed with immutable events in mind, it guarantees — no matter if the event gets processed once or multiple times — that it will result in the same set of operations. This makes implementing a retry mechanism easier.

The retry mechanism consisted of detecting failures and inserting the event into a dead-letter queue, which was just another Kafka topic that would try to process the event again.

Replaying the event in order to try again is very easy, but it has one significant drawback: If the event failed because a rate limit has been exceeded, then it will quickly retry again. There is no concept of backoffs built in without impacting the overall throughput of the system. HTS retries failed events a couple of times before giving up.

In the case that all the retries executed during the outage time period, it would have consistently failed and given up. It means that once the external systems are restored, the events are not processed again. This is where our scheduled job comes in that runs every night during off-peak hours that is responsible for retrying failed events in the hope that the outage has been resolved by then.

The streaming architecture provided by Apache Kafka guarantees that events are delivered in order and have high throughput due to processing events in a distributed manner. The system is able to derive all state updates from a single event thus replaying the event makes it easier to recover from bugs and makes HTS more fault-tolerant.

APACHE KAFKA NEWBIES

For my team, this was the first time using Apache Kafka and node-rdkafka and we had made some assumptions about how they worked that were true until they weren’t.

“The bugs that cause these kinds of software faults often lie dormant for a long time until they are triggered by an unusual set of circumstances. In those circumstances, it is relieved that the software is making some kind of assumption about its environment — and while that assumption is usually true, it eventually stops being true for some reason.”

Source: Richard I. Cook: “How Complex Systems Fail,” Cognitive Technologies Laboratory, April 2000.

KAFKA: RUNNING HEADLESS

All Kafka consumers need a connection to the database so they can process an event and update state. If the database is unavailable, then the Kafka consumers are instructed to crash so they can be restarted. It turns out that node-rdkafka operates the consumers in a separate node thread. So when the database went down, it killed the main javascript thread, but the Kafka consumer stayed alive listening to new events and committing their offset, but never persisting the state into the database.

When the database encountered a fatal error, the code was calling process.exit(1), which wasn’t picked up by the Kafka consumers. Once we changed our implementation to process.kill(process.pid, “SIGINT”), node-rdkafka was able to detect the going down signal and crashed the consumer as designed.

KAFKA: RACE CONDITION #1

Our Quality Assurance (QA) team flagged an issue that sometimes the exercises are duplicated and it was very hard to replicate.

We noticed that when events were published to the Kafka topic they were assigned random partitions that would be processed by different consumers and create a race condition.

We addressed this issue by using a deterministic algorithm to place events from the same member in the same partition.

KAFKA: RACE CONDITION #1.1

The QA team observed that duplicate data did reduce in frequency, but did not eliminate the problem. The system guarantees that an event from the member will always be processed by the same consumer. However, each consumer was configured to process multiple events in parallel to maximize the throughput. When there were two or more back-to-back events from the same member, it created a race condition.

Once we understood that multiple events were being processed simultaneously in the same consumer, it revealed a well-known database problem due to the read-modify-write approach taken when processing events.

Two events for the same member are picked up for processing in parallel. Each processor makes a database query and arrives at the same conclusion that it needs to create a new record in the database. So both processors create a record resulting in duplicate data.

Since events from the same member are guaranteed to end up in the same consumer, we could detect back-to-back events using an in-memory hash map and throttle the processing so one event was processed at a time per member while still being able to process multiple members concurrently.

The event log gave us auditability, the streaming architecture gave us reliability and some fault tolerance, but we needed visibility into performance and availability. That’s where New Relic came into the picture.

NEW RELIC AND THE MEMORY LEAK

Integrating with New Relic allowed us to have visibility into the overall health of the system from a single dashboard. We could see the throughput of the systems, any errors raised, and set up alerts in case certain thresholds were exceeded. The integration of New Relic is a simple process of adding the agent to the application and setting up a configuration file. We added New Relic, confirmed it was working, and moved on.

After exposing our Kafka consumers to production level traffic, we observed that the consumers would consume all available memory and crash. Only to be restarted again and go through the cycle again.

We thought it was our least recently used caching strategy to avoid querying the database multiple times for the same member, which was somehow growing beyond the established limits.

We had many theories as to what could be causing the memory leak, but we struggled to find the culprit in the code. This haunted us for a week until an engineer from the platform team reached out to us and said our Kubernetes pods were running out of storage and it revealed that the New Relic agent log files were gigabytes in size. That led us to recheck all the New Relic agent settings and documentation. Under “New Relic Logging Level,” we saw

“Do not use debug or trace logging unless New Relic Support asks you to use them. These levels of logging can generate excessive overhead. For most situations, use info.”

https://docs.newrelic.com/docs/apm/agents/nodejs-agent/installation-configuration/nodejs-agent-configuration/#logging_config

During the implementation of the New Relic agent, we had set the logging level to “trace” and that made it into the production environment. Changing the logging level back to “info” brought our system back to good health and the requests per minute increased by 10,000.

DATABASE: CHANGE DATA CAPTURE

The PostgreSQL database used by HTS uses Debezium to stream changes directly from the database. This allowed other microservices within the WeightWatchers ecosystem to subscribe to HTS events by simply listening to a Kafka topic.

Only the data that is stored in the database is broadcasted, so if downstream services need to interpret the data, they will need to duplicate the business logic already owned by HTS. It was more performant and distributable to store the derived data along with the raw data so downstream services did not have to duplicate the business logic.

In order to store the derived data, we had to do some migrations and that presented some challenges:

DATABASE: SCHEMA MIGRATION

As the HTS requirements evolved, the database schema also changed. This means that some code releases required database schema migrations. These same migrations worked perfectly in our staging environment so we thought it would be safe to run it in production.

That’s when we relearned that some statements work perfectly fine and fast in small data sets, but become problematic when executed in large data sets. In one particular example, we had an alter table statement that added a column with a calculated value. This caused the entire table to be locked, which halted all processing of the system. Our database administrators were quick to jump in and help us unlock the table and come up with a better migration strategy.

DATABASE: SECONDARY REPLICA

The database migrations going forward resulted in spinning up a secondary database that follows the primary database. Run the migrations in the secondary database and then promote the secondary as the primary and drop the old primary.

This approach worked very well, but one time we were too eager to shut down the old primary and we did so before the replication could finish, so we ended up losing data. We were, however, able to recover the data by restoring the old primary and manually copying over missing records.

DATABASE: THE SEQUENCE

Spinning up the secondary database to do migrations has been very successful, but at some point during the evolution of the system, we made a decision to introduce a sequential primary key to the event logs table, which receives a lot of IOPS. This decision forced us to better coordinate the switchovers between old primary and secondary because we had to determine the rate at which the sequence increments over a period of time and then set the starting sequence in the secondary at a much higher level so the replication didn’t create conflicting entries. In retrospect, the only reason for the sequence was merely for comparing events age by looking at the IDs and to allow easy ordering of the set which all could’ve been done by the “created at” column.

DATABASE: MAX CONNECTIONS

The database selected for HTS started small and we increased it in size as we needed it. HTS started with the goal of supporting three providers and quickly grew to nine (and counting) different providers. Each provider has many consumers that all connect to the database.

When a new deployment is made, new Kubernetes pods are brought online and then traffic is slowly switched over to the new pods and the old pods are brought down.

For a brief period of time during the deployment, the database connections almost doubled. This caused us to reach the max connections allowed by the database so this meant that new deployments could not connect to the database and would be stuck in a crash loop and the deployment would fail.

The about 300 pods that help power HTS each maintained a connection pool of 10 and 90% of them were idle. We have bumped the max database connections allowed, reduced some unnecessary pods, and switched to pgbouncer, which is a connection pooler for PostgreSQL.

The switch to pgbouncer reduced the connections to PostgreSQL by 80%, which also reduced CPU usage by 30%, increased available memory by 25%, and reduced IOPS by 30%.

CONCLUSION

Over the last seven days, the system has averaged 41,900 rpm and had a peak of 85,600 rpm.

The official launch of the Health Tracking Service in early August 2022 went very smoothly and didn’t experience any significant issues because we battle tested the system with production traffic in the background. There were a lot of things we learned that we will keep in mind as we continue to grow and build new systems. This wouldn’t have been possible without the help of the amazing and talented people of WeightWatchers.

Replacing an existing backend service: A retrospective was originally published in WW Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

📖 ICYMI 👉 Check out part I & part II

First up, let’s hear from Ethan:

“This past summer, I worked as a Product Management intern at WeightWatchers. As part of the member education team, my project focused on determining the efficacy of various in-app messaging patterns, more specifically with educating users on how to successfully interact with WW’s application. I was pleasantly surprised that, at such a large company, I was able to do work that would be seen by real WeightWatchers members and would directly impact WW’s application.

The organized structure of this internship led directly to its impact and success. I was paired with an amazing manager who included me in important meetings, taught me about the product management role during weekly educational sessions, and answered any questions that I had. In addition, I learned a lot from different speaker sessions from high-level employees who were focused on educating the interns on various aspects of the company. Finally, all of WW’s employees were very approachable and welcoming. I connected with, networked, and learned from team members and fellow intern peers.

I have never learned as much as I have in such a short period of time as I had during this internship. In just 11 weeks, I enhanced my product management expertise and learned to effectively roadmap a product, lead a team of designers, engineers, and analysts, and use software such as Jira and Figma. I also improved my decision-making skills and learned how to leverage research and data to make informed decisions and successfully prioritize the decisions I make. Finally, I further developed my communication skills, learned how to listen and adapt to feedback, and experienced what it takes to cultivate a collaborative team environment. I truly valued my internship experience at WeightWatchers, and I know that the skills I learned throughout the program will help me throughout my life, and my career.

-Ethan Sims, Product Management Intern

Thanks for sharing, Ethan! Up next, we’ll hear from Ayomikun, our Mobile Engineering intern:

“This summer, I worked as a Mobile (Android) Engineering Intern on the growth team. My tasks throughout the program focused on projects that laddered up to the organization’s broader effort to modernize its codebase.

From the first day of my internship, my manager made me feel so comfortable and an important part of the team. I was provided with all of the resources I needed to succeed and most importantly, the team was always ready and willing to help troubleshoot any challenges I faced.

In my third week of WW’s internship program, I had already begun contributing to the codebase which was a very exciting moment for me. The work culture at WeightWatchers is something I have never experienced before. Everyone wants you to succeed.”

-Ayomikun Adegoke, Mobile Engineering Intern

Thank you, Ethan and Ayomikun for sharing your internship journey with us 🙌

To our readers, I hope that you enjoyed reading through our three-part mini-series which highlights some of our Technology and Product interns’ summer internship experiences and the tremendous impact they made throughout our 11-week program.

⭐ Interested in joining the WW team? Check out our careers page to view technology job listings as well as open positions on other teams.

The WW / WeightWatchers Internship Experience: Part III was originally published in WW Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Last week, we highlighted our awesome Summer Internship Program and kicked off our mini-series with Vineet, our Mobile Engineering intern (thanks, Vineet!) Up next, we’ll hear from Leonel, our Product Design Intern, and Reed, our Mobile Engineering Intern!

Leonel, take it away!

“My internship at WeightWatchers this past summer was truly amazing. As a Product Design Intern, I was challenged with several meaningful projects throughout the 11-week program that helped to hone my skills as a designer.

This summer, I had the opportunity to be a part of the Growth Product Design team where I conducted rapid experiments, and assisted with auditing and reconstructing the design system for our WW guest site. I also had the amazing opportunity to be part of our home page redesign.

Everyone I’ve met throughout the 11-week program was incredibly welcoming. From the moment I began my internship, my team made me feel as if we were part of a family. The people at WeightWatchers are very respectful and open-minded. I was encouraged to share my ideas and opinions and always felt comfortable voicing my thoughts and feedback.

WeightWatchers does an amazing job with their internship program. In addition to the impactful projects we were assigned, we also participated in networking opportunities with one another through Zoom Pods, engaged in team-bonding activities, and worked together on fun side projects. Because of this amazing program, I am leaving with new product design skills, amazing friends, and career references.

-Leonel Duran Jimenez, Product Design Intern

Thanks so much for sharing, Leonel! We’re now passing the mic to Reed, our Mobile Engineering intern! 🎤

“As an iOS Mobile Engineering Intern at WW / WeightWatchers, I was able to work on projects with other engineers and my own project that I presented at the conclusion of our internship program. A highlight of mine while working at WW was being able to engineer a feature using on-device machine learning. The process included researching new technology, creating a prototype, presenting my findings, implementing it in the app, and testing. It was both an exciting and engaging experience.

The people at WeightWatchers are incredible and the connections I made here are invaluable. Throughout my internship, I was able to lean on other engineers at the organization as resources and they were always happy to help. Managers and leaders of WW are very tactful and truly wish to see their coworkers succeed. The feedback I received on my code reviews helped me to become a better engineer and write cleaner code. Although I was remote throughout the internship, the daily meetings with my team helped me to feel and stay connected. The agile process at WW is empowered by the great scrum masters who effectively keep things organized and moving. The people at WW made my internship experience exceptional.”

-Reed Hunsaker, Mobile Engineering Intern

Thank you Leonel & Reed for sharing your incredible intern experience with us! Stay tuned for part III, next week!

⭐ Interested in joining the WW team? Check out our careers page to view technology job listings as well as open positions on other teams.

The WW / WeightWatchers Internship Experience: Part II was originally published in WW Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.