Open Source Profiler on an Internet Scale Fleet: Pinterest and Intel Partnership on gProfiler

Ambud Sharma | Principal Engineer, Pinterest; Prashant Patel | Staff Engineer, Pinterest; David Westbrook | Sr. Engineering Manager, Pinterest; Rogerio Gelamo | Solutions Architect, Intel; Min Yeol Lim | SW Engineer, Intel ; Keith Sundberg | Solutions Architect, Intel ; Danil Korlovskiy | SW Engineer, Intel; Jason M Harper | SW Engineer, Intel; Kim McLaughlin | Account Executive, Intel

Pinterest serves 640 million monthly active users worldwide, and our backend services span hundreds of thousands of instances running code written in Java, Python, C++, Go, Node.js.

Developing observability on bottlenecks aka Profiling is a key step to driving data driving infrastructure efficiency optimization efforts for Pinterest. At our scale, even a 1% efficiency gain translates to a meaningful impact to our profit margins.

This post describes how Pinterest partnered with Intel to drive an open source first approach to developing profiling i.e. gProfiler and adopt, harden, extend it by contributing — (1) Dynamic Profiling, a heartbeat-based remote profiling layer that lets teams trigger profiling on-demand; (2) on-demand Intel PerfSpect collection for hardware telemetry like cache behavior, memory bandwidth, and CPU stalls; (3) wall time profiling to capture latency from threads blocked on I/O, locks, or RPCs that CPU flame graphs miss; (4) PMC flame graphs that tie specific functions to hardware events such as L3 misses and branch mispredictions; and (5) production-hardening fixes that keep the agent efficient at fleet scale.

The Profiling Challenge at Scale

Profiling at scale requires many features like continuous profiling, aggregation, multi-host profiling, Kubernetes integration, and industry-level support. Meanwhile, engineers were using disparate tools — perf, async-profiler, py-spy_ — each requiring manual setup and producing siloed results.

We needed a unified profiler that could:

• Support all our language runtimes under a single user interface
• Scale from a single instance to a fleet of hundreds of thousands of instances
• Provide both continuous and on-demand profiling
• Integrate hardware performance counters for deep bottleneck analysis
• Remain fully open source to avoid vendor lock-in and limit collaboration

Why gProfiler

gProfiler is a continuous profiler originally developed by Granulate (a former Intel subsidiary) and open sourced under Apache 2.0 as an outcome of collaboration between Pinterest and Intel. Unlike single-runtime profilers, gProfiler orchestrates multiple language-specific profilers and analyzer — perf, async-profiler, pyPerf, py-spy, rbspy, phpspy, and others — to produce unified Flamegraphs that capture CPU activity across all processes on a host, regardless of language or runtime. The table below shows the profilers gProfiler orchestrates for each supported runtime:

By merging runtime-specific stacks (e.g., Java method names from async-profiler) with system-wide native stacks from perf, gProfiler produces flame graphs that show the complete picture — from application code down through native libraries and into kernel routines.

Critically, gProfiler is fully open source, aligned with Pinterest’s strategy of building on and contributing to open source infrastructure. The entire pipeline — from agent to storage to visualization — is free of proprietary dependencies, and Pinterest can fork, patch, or extend any component.

The story of gProfiler open source

Pinterest originally tried to develop an internal profiling tool called ProfilerX to consolidate CPU profiling across our fleet. It quickly became clear that the engineering effort needed to build all of the capabilities we would want from scratch would be considerable. The first and foremost capability we were looking for was continuous / always on profiling. This capability was critical to our journey of multi-year infrastructure efficiency.

We had a clear decision to make Build or Buy or Partner. Trying to solve problems that are shared across multiple companies made it clearer that building one more closed source profiler project was not the answer. We should partner rather than pursue just a Build or Buy strategy. The level of profiling insights that a leading hardware vendor like Intel could provide was significantly ahead of what we could build on our own as was evident from the decades of capabilities Intel developed in Intel vTune™. Understanding and interpreting architecture specific counters anyways requires talking to the creators.

Beyond hardware Intel has demonstrated decades of clear leadership in software optimizations and open source contributions, therefore we decided to partner with Intel on this. This first and obvious choice was vTune™ but there were several gaps, first vTune was never developed for a cloud first environment and it had a significant customer base that Intel needed to continue to support therefore opensourcing / rebuilding it would been very disruptive to existing users and didn’t make sense for us to pursue.

Intel’s acquisition of Granulate presented a cleaner opportunity to start something more purpose built for the cloud. We really liked the ability for Granulate to continuously profile workloads. After months of discussion, in late 2023 Pinterest requested Intel to release gProfiler agent and gProfiler Performance Studio backend under a permissive Apache 2.0 license — enabling any organization, including Pinterest, to self-host and extend the full profiling stack. Intel generously agreed, recognizing the broader benefit to the customer base. The vision was to make this profiler tooling easily accessible so we get down to solving optimizations together. Intel with its ability to optimize vertically from silicon to application runtime allowed us to get one step closer to our optimization and efficiency goals.

Once Intel open sourced gProfiler and gProfiler Performance Studio on March 11, 2024, Pinterest began deploying gProfiler agents to our fleet and standing up the backend infrastructure internally.

To deploy gProfiler at scale we addressed critical reliability issues: Pinterest and Intel collaborated closely to harden gProfiler for production use at fleet scale. Together, we addressed critical reliability issues spanning agent resource consumption, disk usage, multi-terabyte ingestion, and build pipeline efficiency — and contributed the resulting fixes back upstream, improving gProfiler’s reliability for the broader community. We cover this reliability journey in detail later in this post.

Once the agent was running reliably across our fleet, Pinterest contributed the Dynamic Profiling feature to upstream gProfiler — system that allows developers to request on-demand profiles at fleet scale.

This capability also unlocked additional features — wall-clock time profiling, PMU hardware event-based flame graphs (intel/gprofiler#1014), and Intel PerfSpect integration (intel/gprofiler#959) — all of which can be dynamically configured to provide deeper profiling insights when needed.

gProfiler Open Source Architecture

To deep dive into Pinterest’s dynamic profiling contributions, let’s first review gProfiler’s architecture. The system consists of two components: the gProfiler agent, which runs on each node where profiling data is collected, and gProfiler Performance Studio, a backend that aggregates, stores, and visualizes profiling data across the fleet.

gProfiler Agent

The gProfiler agent is an orchestrator that coordinates multiple language-specific profilers on a single host. When it starts, it follows a well-defined profiling cycle:

Figure 1: gProfiler Agent profiling cycle — from process discovery through stack trace merge and upload.

1. Initiation: The gProfiler orchestrator initiates all configured profilers — perf for system-wide native profiling, plus runtime-specific profilers like async-profiler (Java), PyPerf (Python eBPF profiler), py-spy (Python fallback profiler), rbspy (Ruby), dotnet-trace (.NET), and phpspy (PHP).
2. Process discovery: Each profiler discovers eligible processes by scanning /proc/*/maps for language-specific library signatures (e.g., libjvm.so for Java, libpython*.so for Python) or by matching executable names (e.g., node for Node.js). This detection runs every profiling cycle and takes approximately 50ms to scan thousands of processes.
3. Profiling session: Each profiler attaches to its target processes using the appropriate mechanism — JVMTI for Java, ptrace for Python/Ruby/PHP, eBPF for PyPerf, EventPipe for .NET — and samples stack traces at the configured frequency (default: 11 Hz).
4. Stack trace output: Each profiler writes its collected stack traces to output files, with language-specific frame suffixes (_[j] for Java JIT, _[p] for Python, _[rb] for Ruby, _[k] for kernel, etc.).
5. Merge: The orchestrator parses all stack trace outputs and merges them. Runtime-specific stacks are scaled to match the perf sample counts for each process, then substituted in place of the native stacks that perf collected for those same processes. Processes without a runtime profiler (Go, C++, Rust) retain their perf stacks as-is.
6. Upload: The merged, collapsed stack trace file is uploaded to the backend for indexing and visualization.

This design means that a single gProfiler agent produces a unified view of CPU activity across all languages running on a host, without requiring any code changes to the profiled applications.

gProfiler Performance Studio (Backend)

The gProfiler Performance Studio is the backend infrastructure that receives, stores, and serves profiling data. It is structured as a set of microservices:

Figure 2: gProfiler Performance Studio backend — microservice architecture for profile ingestion, storage, and visualization.

Legend : Solid arrows are the continuous data plane (profile upload → AWS S3 → AWS SQS → indexer → ClickHouse → flame graphs). Dashed arrows are the dynamic control plane (UI creates a profiling request; agent fetches start / stop profiling requests via heartbeat).

• Backend (Webapp): The central web application that exposes all APIs to the frontend and API users. It receives profiling data uploads from agents, stores raw profile files in S3, enqueues indexing tasks in Amazon Simple Queue Service (SQS), and reads/writes service metadata in PostgreSQL.
• Frontend UI: A React-based interface for exploring flame graphs, filtering by service/host/container/time range, and comparing profiles across time periods.
• Indexer: A Go service that consumes events from SQS, downloads raw profiling data from S3, and bulk-inserts it into ClickHouse. It handles the transformation from collapsed stack format into the columnar schema optimized for flame graph queries.
• FlameDB REST: A Go query service that sits between the frontend and ClickHouse. It constructs flame graph data by querying the appropriate resolution tier (raw, 1-minute, 1-hour, or 1-day aggregates) based on the requested time range, and performs frame-level language identification using pattern matching on stack frame names.
• Logging Service: Collects and centralizes logs from agents for debugging and operational monitoring.

The backend uses S3 for durable profile storage, SQS for decoupling ingestion from indexing, PostgreSQL for service metadata and configuration, and ClickHouse as the columnar analytical store for fast flame graph queries across billions of stack trace samples.

This open source architecture gave Pinterest a strong foundation to build upon. Our contributions focused on extending it with dynamic profiling capabilities, hardening it for production scale, and integrating hardware-level insights — all contributed back upstream.

Pinterest’s Open Source Contributions with Intel

Dynamic Profiling: Single-Click Managed Profiling at Fleet Scale

The original gProfiler agent only supported a statically configured profiling mode, without the ability to dynamically reconfigure the agent in a given session.

Pinterest operates a large fleet of nodes across AWS EC2 and Kubernetes (Pincompute). At that scale, a static agent posed real challenges. There was no way to target profiling to specific services, no way to adjust the configuration on the fly, and no way to control profiling remotely without SSH access to each node.

The Dynamic Profiling feature we contributed to the open source gProfiler project solves exactly this problem. It offers zero-setup, single-click managed RBAC controlled profiling designed for scalability and ease of management:

• Service owners gain full control to start and stop profiling on demand, enabling targeted detection of service-specific bottlenecks, regressions, and optimization opportunities — all without shell access.
• Performance experts can fine-tune arguments to run various profilers, investigate common bottlenecks across services or processes, and dive deep into hardware-level insights using Intel PerfSpect — also without shell access.

This streamlined, accessible profiling capability has made efficiency and cost optimization a core part of Pinterest’s engineering culture.

Get Pinterest Engineering’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Under the hood, we needed the ability to:

• Install the agent on host in a dormant state
• Activate profiling remotely on specific hosts, services, or clusters
• Change profiling parameters (frequency, duration, profiler selection) without redeploying
• Stop profiling instantly if resource pressure is detected

Figure 3: dynamic profiling architecture — agents communicate via heartbeats, and profiling requests are delivered as heartbeat responses from the studio.

How it works:

1. gProfiler agents are installed on all hosts by default in dormant mode (less than 1% CPU, less than 100MB memory when idle). They send periodic heartbeats to the gProfiler Performance Studio every 60 seconds.
2. When an engineer submits a profiling request through the UI or API — specifying a service, host list, duration, frequency, and profiler configuration — the Profiling Service converts it into requests and queues them into the database
3. On the next heartbeat from a targeted host, the Heartbeat Handler responds with the pending profiling request. The agent validates, enqueues, and executes it.
4. Request execution is idempotent — agents track received and executed request IDs to prevent duplicate execution, and report completion status back through subsequent heartbeats.

Figure 4: The Dynamic Profiling UI — configure and control profiling on demand, no SSH.

he full dynamic profiling system — agent-side profiling request management, heartbeat client, and backend profiling-orchestration APIs — has been merged upstream into the Intel gProfiler repository and the Intel gProfiler Performance Studio repository, available for any organization to use.

PerfSpect: Deeper Insights for Infrastructure Decisions

Flamegraphs show where software spends time, but not how hardware is stalled or saturated. Intel PerfSpect adds CPU stalls, memory bandwidth, CPI, cache, and pipeline telemetry backed by hardware Performance Monitoring Counters (PMCs) — and Pinterest extended its gProfiler integration for on-demand collection of PerfSpect via the Dynamic Profiling UI, with metrics stored in the same backend for correlation with software hotspots. A standalone build pipeline bundles PerfSpect independently of Intel’s release cycle and runs on both x86 and ARM, covering the majority of Pinterest’s fleet including Graviton. The result: engineers pick optimal AWS instance types based on real workload behavior, not vendor specs.

Figure 5: PerfSpect TMAM dashboard — surfaces hardware-level bottlenecks like cache-miss-driven stalls directly in the gProfiler UI.

Wall Time Profiling

Default CPU flamegraphs miss a huge class of microservice latency — threads blocked on I/O, locks, DB queries, or RPC calls never appear. Pinterest contributed wall time profiling (intel/gprofiler#1012), which enables sampling stacks by elapsed wall-clock time to capture both running and waiting states. Engineers toggle CPU and wall time modes on the fly through the same UI — no agent redeploys — and compare them side-by-side in Performance Studio.

PMC Counter Flamegraphs

CPU profiling shows where time goes and PerfSpect shows system-wide hardware behavior, but neither pinpoints which functions cause cache misses or branch mispredictions. Intel and Pinterest contributed custom PMC flamegraphs (intel/gprofiler#1013) that sample on hardware events (L3 misses, TLB misses, branch mispredicts, memory stalls) so each frame’s width reflects event frequency, not CPU time. Agents advertise supported events, the backend validates requests, and results render alongside CPU and wall time profiles — letting engineers tie specific Java or C++ functions to hardware bottlenecks.

Figure 6: A PMU counter flame graph sampled on cache-miss events — each frame’s width reflects cache-miss frequency rather than CPU time, pinpointing the functions driving memory stalls.

Reliability Journey

Taking gProfiler from prototype to production-grade infrastructure at Pinterest scale required solving several reliability challenges, many of which resulted in fixes contributed back to the open source project(PRs #987, #988, #993, #995, #997, #998, #1002) :

Memory optimization: The original agent consumed 2.8 GB at steady state on typical hosts and spiked to 4–5 GB+ on high-process-count hosts (500+ processes) — frequently triggering OOM kills. The root cause was 3,000+ leaked pipe file descriptors from subprocesses, compounded by leaks in the grpcio library and the profile collection path. Through comprehensive subprocess cleanup (98% FD reduction), grpcio leak fixes, and collection-path optimizations, we brought the agent to under 400 MB stable across both scenarios — critical for running as a DaemonSet alongside production workloads.

Core dump elimination: The async-profiler component occasionally triggered core dumps consuming over 100GB/day of disk on storage-optimized instances. We disabled core dumps at the child process level and added zombie process checks, eliminating this entirely.

Staticx temp file accumulation: Each agent restart left behind extracted binary directories in /tmp, accumulating hundreds of gigabytes over time. We addressed this through proper cleanup lifecycle management.

ClickHouse “too many parts”: During aggressive backfill after an outage, high indexer concurrency with small batch sizes created more parts than the merge scheduler could compact, blocking inserts. We tuned batch sizes, added parts-limit monitoring, and implemented the Orion shard manager for automated cluster operations.

Build time reduction: gProfiler builds originally took over 30 minutes. Through build pipeline optimization — migrating from Jenkins to Buildkite, parallelizing architecture-specific builds, and caching intermediate artifacts — we achieved a 70% reduction in build time.

Surfacing recurring efficiency opportunities

Pinterest has started using gProfiler data to drive optimization across top CPU-bound services. The effort is still a prototype — engineers eyeball flamegraphs and track findings in a spreadsheet — but it has already paid off: we were able to identify common patterns across multiple services that lead to meaningful improvements e.g

1. Frequent Pattern.compile calls delivered 2–3% CPU savings in the affected sub-service.
2. compression tuning across Zstd/Zlibng/LZ4/Snappy
3. Serialization optimization with SIMD-accelerated alternatives like SimdJSON alongside Kryo, Jackson, Protobuf, and MessagePack.

We’re working to formalize this into a structured recommendations framework in alignment with our original vision of what we hope the service would deliver.

Results and Impact

Since deploying gProfiler across Pinterest’s infrastructure:

• Coverage: gProfiler agents are installed provide profiling observability across our fleet
• Productivity: Self-service dynamic profiling UI enables any engineer to start profiling their service in minutes, without needing to call the performance engineers or lookup documentation for profiling
• Engagement: Since its launch we see at least ~140 users across (engineers, efficiency leads, experts) using gProfiler on a monthly basis
• Efficiency: The profiling platform has detected several bottlenecks, regressions and optimizations with vetted recommendations translating to measurable savings
• Lightweight footprint: Agents stay nimble even at fleet scale — 0.15% CPU and ~200 MB memory in silent/dormant mode, and under 2% CPU and 2 GB memory while actively profiling
• Reliability: The infrastructure operates with 99.99% heartbeat success rate service level objective (SLO), sub-5-minute P99 indexing latency, and sub-second P95 flamegraph query latency
• Open source: Dynamic profiling, PerfSpect integration, memory optimizations, core dump fixes, and build improvements have all been contributed upstream

What’s Next

We’re continuing to invest in several areas:

• Latency profiling: Knowing what is blocking a service from running at higher utilizations
  Spark profiling: Extending gProfiler to support Spark job-level profiling, with optimization opportunities through JIT analysis and thread-name-aware flame graphs
• Workload-level profiling: Granular profiling by Kubernetes namespace, pod, and individual process — moving beyond host-level to workload-level visibility
• Structured optimization framework: Evolving today’s prototype recommendations effort into a formalized system with precision and accuracy KPIs, automated vetting, and hardware-informed instance selection using PerfSpect data (cache miss patterns, memory bandwidth saturation) to guide right-sizing decisions
• Regression detection: Automated detection of performance regressions across deployments using profiling data differentials
• GPU profiling: Extending the platform to cover GPU-bound workloads, integrating tools like NVIDIA Nsight Systems and Nsight Compute for kernel-level insights, alongside dynolog and DCGM for fleet-wide GPU telemetry
• LLM-powered analysis: Augmenting the optimization framework with large language models speed up RCA and analysis to provide fix suggestions with source pointers

Conclusion

Maintaining software performance at fleet scale is a fundamentally different challenge than occasionally profiling a single service manually. It requires a platform that spans languages, operates across deployment models, and empowers every engineer — not just performance specialists to make things efficient. By partnering with Intel on gProfiler, Pinterest helped build that platform on a fully open source foundation: dynamic profiling for remote, on-demand analysis across hundreds of thousands of hosts; PerfSpect integration for hardware-level insights; wall time profiling and PMU counter flame graphs to answer questions that CPU-only profiling can’t; and a reliability-hardened pipeline that ingests hundreds of terabytes of profiling data monthly. Every major capability we built has been contributed upstream, so any organization can deploy the same infrastructure. If you’re looking for a unified, vendor-free profiling platform that scales from a single host to an entire fleet, gProfiler is ready.

Resources to get started with gProfiler

gProfiler is open source and ready to deploy:

• gProfiler Agent: github.com/intel/gprofiler
• gProfiler Performance Studio: github.com/intel/gprofiler-performance-studio
• Intel PerfSpect: github.com/intel/PerfSpect

Whether you’re running a handful of services or a fleet spanning hundreds of thousands of hosts, gProfiler’s dynamic profiling infrastructure — contributed by Pinterest — gives you the control to profile what you need, when you need it, without the overhead of profiling everything all the time.

All performance metrics and infrastructure statistics cited in this post are based on Pinterest internal data, Q1 2026.

Acknowledgements

This work would not have been possible without the collaboration between Pinterest and Intel’s open source teams.

We’d like to thank the current Intel team for their continued partnership on gProfiler and PerfSpect Min Yeol Lim, Keith Sundberg, Rogerio Gelamo, and Kim McLaughlin through ongoing support, development and maintenance of gProfiler, and Jason Harper for perfspect. This wouldn’t have been possible without Intel business and engineering leadership by Kelly Hammond and Bill Pearson.

We also want to recognize leaders who were part of laying the original foundation of this open source project: Asaf Ezra, Brendan Greg, Dave Burgess, Greg Lavender, Greg Ernst, Harshad Sane, Peggy Irelan and Steve Beckis. Additionally all members of the original Granulate team supported the original project.

From Pinterest, we’d like to thank the following current and past members who contributed to and helped maintain gProfiler: Ambud Sharma for Pinterest Software Performance Vision, Prashant Patel for Technical design and Infrastructure scale out. We are especially grateful to engineering contractors who helped bring this work to life : Artur Sarlo, Ashok Chatharajupalli, Marc Antonio Vieira De Queiroz and Renato Gagliardi Sabo. We’d also like to recognize Eric Lopez, Vibhav Garg, and Vaibhav Shankar for their partnership, and Nasser Manesh, David Westbrook, and Molly Junck for their leadership support.

To learn more about engineering at Pinterest, check out the rest of our Engineering Blog and visit our Pinterest Labs site. To view and contribute to open source projects at Pinterest, visit our GitHub.