From Activity to Intent: Generating User Journeys with LLMs

Lin Zhu | Sr. Staff Machine Learning Engineer; Manan Kalra | Machine Learning Engineer II; Byung Eun Jeon | Sr. Machine Learning Engineer; Ye Liu | Staff Machine Learning Engineer; Xiangyi Chen | Sr. Machine Learning Engineer; Jaewon Yang | Principal Machine Learning Engineer; Jinwen Xu | Manager II, Machine Learning Engineering; Tingting Zhu | Sr. Manager, Engineering; Sudarshan Lamkhede | Director, Machine Learning Engineering

Pinterest is built to get inspired and then turn the inspiration into realization — a dinner, a renovation, a wedding, a new skill. That only works if we understand more than the Pin a user just clicked. The useful unit of personalization isn’t the thing someone just clicked, it’s what they’re trying to accomplish. Such goals unfold over weeks and months, across dozens of sessions, through hundreds of saves.

We call those goals journeys: the intersection of a user’s interests, intent, and context at a point in time. A journey is a sequence of interactions, often spanning sessions, centered on one interest and a clear intent. For example, “Summer dresses” are an interest. “Figuring out what’s in style for a summer wedding I’m attending in August” is a journey. Someone might be interested in five at once: a kitchen remodel, a vegetable garden, learning sourdough and the mix changes as life does.

If we can name those journeys, recommendations and notifications can help someone finish a project instead of bouncing between unrelated interests. This post is about how we infer them in production: a fine-tuned 4-billion-parameter large language model (LLM) that reads a chronological activity log and emits a ranked list of journey names, in the user’s locale.

We didn’t start there. The initial user journey system clustered keywords and attached attributes with separate machine learning models. This post is the follow-up which covers how we formulate journey inference as generation, how we distill frontier-model quality into a 4B student we can actually serve, how the serving stack holds up at notification scale, and what we’re building next. The recipe is domain-agnostic: any platform with sequential user activity and structured intent targets (e-commerce, streaming, search) can adopt the same activity-to-intent formulation, collapsing classical multi-stage recommendation pipelines into a single generative model, making generative journey inference a general recipe for personalization beyond Pinterest.

Why interests aren’t enough

Most recommenders optimize the next item. However, journeys on Pinterest are longer-horizon. Planning a kitchen is not the same as a one-off click on a cabinet Pin, and it isn’t well described by a durable interest like “Home Decor”. The cabinet Pin, the backsplash search, and the garden-bed save might be one project, two projects, or noise. Getting that distinction right changes what we retrieve, what we write in a notification, and when we should stay quiet.

So the output we want is not a taxonomy label. It’s a short, specific, ranked list of a user’s active journeys, with specific names they’d recognize.

The legacy clustering system

We had almost no labeled journeys when we started, so we stayed lean. We reused pretrained text embeddings, started from a few hundred human examples, and built a multi-stage pipeline:

1. Extract keywords from search queries, annotations and interests on engaged Pins, and Pins on owned boards.
2. Embed and cluster those keywords hierarchically. Each cluster is a journey candidate.
3. Name the cluster by ranking its keywords and taking the top one.
4. Rank and diversify the clusters with a pointwise model, then a similarity penalty so the top of the list isn’t five near-duplicates.
5. Classify stage with heuristics: evergreen vs. situational (roughly, does engagement persist past a few weeks?), and for situational journeys, ongoing vs. ended (time since last engagement).

That pipeline powered the first journey-aware notifications. Against interest-based notifications, it lifted email click rate by 88%, push open rate by 32%, and positive survey feedback by 23%.

Figure 1: User journey inference via the legacy clustering system

It also compounded its own errors. Every stage was locally reasonable; together they fragmented coherent goals.

• Fragmentation: Cabinets, countertops, and backsplash are one kitchen, not three journeys fighting for the same notification slot.
• Generic buckets: When a cluster was messy, the naming model fell back to the most frequent keywords — “Art”, “Recetas”, “Shopping”. Taxonomically true, useless for personalization.
• Incidental noise: A handful of meme Pins or celebrity clicks could occupy a top slot. Clustering doesn’t know which activity is a goal and which is a passing click.

Journey inference as generation

To solve these problems, we stopped treating journey inference as a chain of transformations and started treating it as one act of generation: read the activity, write the intent. The model gets a single input prompt containing a semi-static profile plus a chronological activity log, and returns strict JSON — a ranked list of journey names:

User Profile:
gender: female; language: en-US; country: US; age: 30-35
User Owned Boards: Kitchen Reno 2026; Weeknight Dinners; Garden Plans
User Activities:
2026-06-14 search shaker cabinets.
2026-06-14 save subway tile backsplash, kitchen backsplash; quartz countertop, white kitchen.
2026-06-12 click raised garden bed, vegetable garden; tomato trellis.
2026-06-09 search sourdough starter discard.
2026-05-30 save cast iron skillet cornbread, dutch oven recipes.
...

{"journeys": ["Modern Farmhouse Kitchen", "Raised Bed Vegetable Garden",
              "Sourdough Bread Making", "Cast Iron Recipes"]}

Notice what’s gone: no keyword extraction step, no embeddings, no clusters, no separate naming model. One model reads the log and writes the answer.

Some deliberate choices in the prompt:

• Actions collapse into a tiny vocabulary: search, save, like, click, comment, dislike. The model doesn’t need to distinguish a repin from a save-to-device; it needs to know both are strong positives. The prompt states the ordering explicitly: search/save > comment > like > click > dislike.
• Events are grouped by (date, action), with a semi-colon (;) separating Pins and a comma (,) separating tags within a Pin. That’s token economy, and it also gives the model a cleaner view of same-session bursts.
• Dislike is deliberately included in the input, as hides are evidence. The prompt says to suppress a journey whose support is predominantly dislikes unless there is evidence of significant related positive engagements.
• Temperature 0, reasoning tokens off, max 300 output tokens. We want a deterministic, replayable read of the same log, and with a fixed output schema for decoding purposes. Reasoning is turned off to reduce online inference cost.
• A 360-day lookback, up to 150 events. Journeys are long-horizon by definition, we experimented with different lookback windows and # of events to balance cost and relevance.

The output contract is deliberately simple: drop non-strings, dedupe case-insensitively, and the first occurrence wins. The output is parsed defensively because a generative system that occasionally emits prose where you expected JSON is a system that occasionally drops a user’s entire personalization payload.

The other thing worth saying out loud is that once you collapse the pipeline, the prompt becomes the ranking policy and the safety policy. Everything the staged system encoded in model weights and thresholds now has to be clearly stated. In production it asks for up to about twelve distinct journeys, each 1–5 words, in the user’s locale, ranked by:

1. Engagement strength (search/save over click)
2. Repeated support
3. Recency
4. Coherence
5. Specificity (“Mid-Century Coffee Table”, not “Furniture”)

It also encodes the failure modes we observed with the legacy journey clustering system: don’t merge unrelated topics; drop one-off outliers; treat auto-generated tags as noisy; lightly diversify the top of the list only among journeys that are already well supported. Empty output is allowed when the activity doesn’t support anything but inventing unsupported journeys is not.

These instructions aren’t just decorative. The diversity pass replaced a similarity-penalty diversifier. The dislike rule replaced a hand-tuned negative-signal filter. Few tag-noise instructions exist because auto-generated annotations are genuinely noisy, and the model will happily build a journey out of a classification artifact if you let it. Prompts like this deserve the same review discipline as model code, so we treat changes to them as launches.

Get Pinterest Engineering’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Clustering was the right cold-start system, and it still exists as a fallback. Production inference is now LLM-first: one model, one decode, structured output.

Figure 2: User journey inference system with feedback

Distilling a frontier teacher into a 4B student

Frontier models can infer coherent journeys without task-specific training, but they are too expensive to run for hundreds of millions of people on Pinterest regularly. Instead, we use them as teachers to generate synthetic training examples: each activity history is paired with a high-quality target journey. We also found that teacher outputs improved when the model was allowed to use additional reasoning tokens before producing its final structured response.

We fine-tuned open-weight Qwen3 students with 0.6B, 1.7B, 4B and 8B parameters. Supervised fine-tuning (SFT) teaches the model from the teacher-generated examples, while low-rank adaptation (LoRA) lets us update small adapters rather than every model parameter.

Quality improved rapidly during the first few thousand examples, then largely stopped increasing. Adding more similar examples provided little benefit; selecting diverse, high-quality examples — especially difficult consolidation and multilingual cases — was more valuable.

Model size presented a separate trade-off. The 0.6B and 1.7B students were faster, but produced less relevant and consistent journeys on the harder cases. Among the sizes we tested, the 4B model offered the best balance between output quality and serving throughput, so it became our production student.

Table 1: Same-user comparison of the staged clustering pipeline against the fine-tuned LLM. Fragments consolidated (Cabinets + Countertops → Modern Farmhouse Kitchen). Generic buckets got specific (Art → Penguin Art Drawing). Dropped intents came back (Cast Iron Recipes). Noise disappeared (Celebrity News, Memes). And user B got journeys in Spanish — something the keyword pipeline structurally couldn’t do, because it named journeys out of an English-annotation vocabulary.

Serving at notification scale

Scoring hundreds of millions of people on Pinterest is a batch problem with online-shaped latency. We run a full refresh when the algorithm changes, and a daily incremental pass over recently active users, so journeys move when behavior does.

We benchmarked the offline workflow against serving the same 4B model behind an online inference stack. Online serving won on throughput. The reason is a set of now-mature decode optimizations that a naive offline loop typically doesn’t get for free: PagedAttention, so the key-value cache isn’t wasted on padding; continuous batching, so a finished sequence immediately yields its GPU slot to the next user instead of waiting for the slowest prompt in a fixed batch. The serving stack is NVIDIA Dynamo in front of vLLM decode workers on L40S GPUs. Frontend and decode workers scale independently, and the frontend routes with key-value (KV) cache awareness so requests sharing prompt structure land on workers that already hold the relevant prefix, which matters because every request shares a long, stable system prompt. We enable prefix caching and chunked prefill and cap concurrent sequences per worker.

In production, a cluster on the order of 100 GPUs sustains about 775 requests per second end-to-end, with a median around 1.2 seconds and p95 around 3.0 seconds. The batch side is where the cheaper wins live. Inference runs as a daily fan-out over journey-eligible users, sharded into small per-job batches with bounded concurrency inside each batch, so a slow model response back-pressures one shard instead of the pipeline. The single most effective throughput lever we added wasn’t a GPU at all: we track when each user was last inferred and skip anyone processed within the last couple of days. Journeys don’t change hour to hour, and paying to re-derive an unchanged journey is the most expensive kind of waste — the kind that looks like progress on a dashboard.

If a decode fails, we retry that user rather than storing a partial list. An empty but valid JSON list is a successful inference; some users don’t have a supportable journey that day, and we’d rather send nothing journey-specific than a generic “Art”. We suppress inference of sensitive traits at multiple levels before anything is stored:

• Input keywords pass our notification content filter before reaching the prompt, so the model can’t build a journey on a keyword we wouldn’t have sent anyway.
• Output journey names pass that filter and our generative content safety filters, so names that are unsafe in a journey-inference sense are dropped even if they would pass a generic keyword list.
• The model is instructed never to infer or target topics related to health conditions, politics, religion, sexual orientation, gender identity, race, etc.
• Profile inputs are minimized, with ages bucketed at the low end and grouped, because the model needs a rough life stage rather than a birthday. Under-18 users in the EEA, Switzerland, and the UK aren’t eligible yet due to local laws.

Inference writes a ranked list of journeys per user into a store that notification serving reads at send time, and each journey name then becomes a retrieval query. We built embedding-based retrieval models to pull Pins for each user journey which are further ranked by our Pin ranking models. Journeys the user was recently notified about are fatigued, and journeys they’ve given explicit negative feedback on are suppressed outright. Only then does the notification framework decide whether to send at all, and on which channel.

Figure 3: Notification surfaces for User Journeys

How we know it’s better

There is no ground-truth label for a user’s latent goal. We evaluate at three levels, from inspection to a causal test.

• Same-user qualitative comparison: For journey-eligible users, we put clustering output and LLM output side by side on the same user activity. We keep both the strong cases and the honest misses: over-generalization (“Home Cooking” when the log is clearly “Spam Kimchi Fried Rice”) and under-generation (dropping a well-supported side interest). This is slow, but it’s the only way to see fragmentation and language quality that metrics flatten.
• Evidence-grounded diagnosis: A separate language model scores a sample of emitted journeys on axes we care about in production: relevance to the visible context, support from the profile and activity log (is this name faithful to what we showed the model?), diversity / non-redundancy across the list, and granularity. The support axis is the important one. A fluent, on-brand name that isn’t in the log is a failure, even if a user might have clicked it.
• Online experiment: Initial experiment ramped to the US and Canada users, with the clustering production system as control, showed lifts of 1.1% on email click-through and 1.3% on push opens, plus multiple statistically significant sitewide engagement and retention gains. A subsequent multilingual international rollout replicated the direction, and shipping both experiments increased the daily, weekly, and monthly active users on the platform.

What we’re building now

The fine-tuned Qwen-4B model is the production default. Everything in this section is under experiment — trained or rolling out, not what every user gets today.

Semantic IDs, so Pins aren’t a bag of tags

Tags mean every Pin costs a handful of tokens and drags along a dependency on Pin-to-text mappings. Semantic IDs (SIDs) are the alternative: each Pin is compressed by a residual-quantized variational autoencoder (RQ-VAE) into five levels of 11 bits each, which we pack into a single 64-bit integer and unpack at prompt-build time into special tokens:

<sid_begin><sid1_1263><sid2_973><sid3_725><sid4_793><sid5_1727><sid_end>

A general-purpose model has no idea what <sid3_725> means, so we bridge the gap with continued pre-training (CPT): extend the tokenizer with the SID vocabulary, then run next-token prediction over interleaved SID sequences, Pin titles, and tags until the model has mapped its world knowledge onto our catalog. SFT goes on top, so that given a SID sequence the model writes journeys.

SIDs only work better for users with dense engagement, so production is a fallback ladder: fewer than ten SID-carrying events and we fall back to tags for the whole prompt; a single event missing a SID falls back to tags for that event; and search queries always keep their literal text, because a typed query is the highest-fidelity signal we have. The bet is fewer input tokens, no dependency on annotations, and a shared item language between recommenders and LLMs. The cost is interpretability — a tag-based prompt can be read by a human mid-debugging session. A SID prompt cannot.

Offsite conversions as a complementary signal

Onsite search and saves remain the strongest intent signals, but they can be sparse for dormant or low-activity users. We therefore use recent offsite actions — such as page visits, add-to-carts, sign-ups, and checkouts — as a complementary signal, adding a compact, safety-filtered summary to capture timely shopping or planning intent beyond Pinterest.

Offsite evidence remains secondary: onsite outweighs offsite, and checkout is stronger than add-to-cart or a page visit. We only add it when onsite activity is sparse, cap its prompt footprint, and avoid ranking offsite-only journeys at the top. We are testing whether better prompting is enough or whether a fine-tuned offsite-aware model is needed.

One model, several products

Once inference is generative, adding an output is a prompt-and-fine-tune change rather than a new service.

• Multi-granularity journeys: Rather than a flat list of names, have one call emit each journey together with deeper cuts of it and adjacent topics that broaden discovery, so “Sourdough Bread Making” comes back with a specific next step into the interest and a sibling topic a user might jump to. That gives downstream retrieval a ladder of intent per user instead of a single rung.
• Copy titles: A good journey name isn’t always good notification copy, and we had engagement data showing which names underperform as titles (“Photography”, “Hairstyles”, “Quotes” — the generic ones again). One model now produces both, with a hard ordering constraint: finalize the journey names first, then write copy, and never reshape a name to make its copy sound better.

Closing the loop with User feedback

Distillation got us strong journeys, but it has a ceiling. The student is optimized to match the teacher, so its quality is bounded by the teacher’s sense of what users want. To go further, we need to optimize directly toward user preferences, using real feedback from users themselves.

Much of that signal already exists. Journey emails include a module asking “Are you interested in <journey>?” with Yes and No buttons. Each answer is logged, and a No immediately suppresses that journey for that user. Beyond these explicit answers, engagement inside journey-aware notification sessions gives us a much larger implicit signal: saves, clicks, and hides on the Pins we retrieved for a journey.

This feedback comes with a caveat: it is pointwise. Each response is a verdict on a single journey, while the model generates a ranked list. To bridge that gap, we split training into two stages. First, a mid-training stage applies preference optimization to single journeys across a large volume of feedback. Given a user’s activity, the model learns to make a journey that users confirm or engage with more likely, and a journey they reject less likely. Then, we formulated feedback-as-edit: the chosen response is the model’s list with the Yes journeys kept or the No journeys removed, and the rejected response applies the reverse edit. Starting from the mid-training checkpoint, fine-tuning restores list-level behavior: ranking, diversity, and the output format described earlier.

Improving the LLM with preference optimization and real user feedback is still an active area of experimentation. We are exploring how to weight explicit versus implicit feedback, and how to formulate a loss that is amenable to scaling, one whose gains follow a scaling law as feedback volume grows. Offline results are promising, and online A/B tests are now running. We hope to share more details in a follow-up blog post or publication.

Acknowledgements

We’d like to thank Justin Tran, Darien Boyd, Sarah Schwager, Shipeng Yu, Randy Tumalle, James Hernandez, Mengtian Hu, Dimitra Tsiaousi, Yaxin Li, Ying Wei, Sam Owens, Gloria Lee, Justin Pangilinan, Ye Tian, Rui Liu, Anya Trivedi, Binghui Gong, Tianqi Wang, Fangzheng Tian, Eric Tam, Mengying Yang, Lauren Fratamico, Andrew von Rosenbach, Ignacy Debicki, Zhenyu Tan, Jiajing Xu, and other contributors from Notifications, ATG Applied Science, ML Platform, GenAI Serving, and Content Safety teams for their insights and support!