Stellar, Now Truly Observable

Stellar has established itself as a leading blockchain infrastructure, recognized for its speed, reliability, and suitability for payments and stablecoins. Until now, however, accessing and analyzing Stellar’s on-chain data has been complex and fragmented. With Agnostic, that’s changing: developers, analysts, institutions, and ecosystem stakeholders now have powerful tools to query, visualize, and share blockchain data, without the burden of managing technical infrastructure.

What’s Available Today

Agnostic’s release includes:

• ch-stellar: Custom functions that extend ClickHouse with Stellar-specific capabilities, including reading Galexie exports, decoding XDR, and making direct RPC requests to Soroban from SQL.
• Open Public Datasets: Continuously enriched and updated tables for accounts, transactions, contracts, liquidity pools, effects, operations, trades, and more, all in Apache Iceberg format, ready for the community.

Iceberg Endpoints

https://agnostic-stellar-iceberg-mainnet.agnostic.tech/accounts_v1/
https://agnostic-stellar-iceberg-mainnet.agnostic.tech/changes_v1/
https://stellar-iceberg-mainnet.agnostic.tech/contracts_data_v1/
https://stellar-iceberg-mainnet.agnostic.tech/events_v1/
https://stellar-iceberg-mainnet.agnostic.tech/ledgers_v1/
https://stellar-iceberg-mainnet.agnostic.tech/liquidity_pools_v1/
https://stellar-iceberg-mainnet.agnostic.tech/operations_v1/
https://stellar-iceberg-mainnet.agnostic.tech/trades_v1/
https://stellar-iceberg-mainnet.agnostic.tech/transactions_v1/
https://stellar-iceberg-mainnet.agnostic.tech/trustlines_v1/

• AGT ETL Pipelines: Open-source SQL pipelines for transforming and enriching Stellar data, making advanced research, institutional analysis, and DeFi monitoring straightforward.
• AGX: An interface for exploring, querying, and visualizing Stellar data, with AI-powered query generation, accessible to everyone from developers to business analysts.

• DGN: A collaborative hub for creating, publishing, and sharing dashboards and analytics notebooks focused on Stellar.

Our entire stack is open source (MIT), well-documented, and fully accessible. Dive into the code on GitHub or start exploring analytics at agnostic.dev.

Built for the Entire Stellar Community

Agnostic is designed for:

• Developers: Track contracts, monitor protocol activity, and accelerate application analytics.
• Stablecoin Issuers & Financial Institutions: Visualize flows, gain transparency, and ensure compliance with advanced reporting tools.
• DEXs, Market Makers, and Liquidity Providers: Analyze volumes, liquidity, trades, and market trends with flexible, SQL-native tools.
• Analysts & Researchers: Access clean, reproducible datasets and create public dashboards to share insights with the ecosystem.

Unlocking Stellar’s Full Potential, Together

With Agnostic, Stellar finally has a modern, open analytics layer. Whether you’re building the next DeFi application, tracking stablecoin flows, or just exploring blockchain data, Agnostic makes it accessible and actionable for everyone.

Start exploring at https://stellar.agnostic.tech, check our documentation, or join the community!

Agnostic Launches the First Open and Advanced Analytics Platform for Stellar was originally published in Agnostic on Medium, where people are continuing the conversation by highlighting and responding to this story.

The State of Blockchain Data

Blockchain analytics has come a long way, with powerful platforms like Dune Analytics, Nansen, and Glassnode shaping how we interact with on-chain data. These tools have brought valuable insights to the crypto space, each with its own strengths. However, they often come with inherent limitations — closed ecosystems, restricted transparency, and centralized data access. For those who value openness and decentralization, these constraints can hinder innovation and accessibility.

An Open and Decentralized Alternative

Enter Degen (dgn.space) — a groundbreaking, open-source, decentralized notebook platform designed to redefine how blockchain data is explored, shared, and understood.

Built on Agnostic’s open dataset and powered by ClickHouse infrastructure, Degen eliminates the barriers imposed by traditional platforms. Users can seamlessly query blockchain data and visualize insights with an intuitive experience. No gatekeeping, no hidden access controls — just raw, transparent, and community-driven analytics.

The Perfect Complement to AGX

Degen is the natural extension of AGX, offering rapid iterations, quick querying, and a social, gamified layer that enhances data exploration. It fosters collaboration, allowing users to combine Agnostic’s datasets with others to craft fully open-source insights. Whether you’re an analyst, a developer, or a crypto enthusiast, Degen turns blockchain data into a dynamic, interactive experience.

Why Does This Matter?

Transparency is at the core of blockchain, yet many analytics platforms remain centralized and closed. Degen changes this paradigm by ensuring complete control over your data and insights. It empowers users to explore, share, and collaboratewithout the restrictions of traditional platforms, fostering a more open, decentralized, and community-driven blockchain ecosystem.

Join the Future of Blockchain Analytics

The need for decentralized, transparent data tools has never been greater. Degen is here to unlock the full potential of blockchain analytics — making it more accessible, interactive, and engaging for everyone.

Ready to experience the next evolution of blockchain insights? Start your journey today at dgn.space 🚀

dgn.space, blockchain analytics made social was originally published in Agnostic on Medium, where people are continuing the conversation by highlighting and responding to this story.

In today’s data-driven world, the ability to access, analyze, and visualize data effectively is paramount. Whether you’re a data enthusiast, a researcher, or a business professional, having the right tools can make all the difference. Enter agx, a powerful desktop application designed to provide a seamless experience for exploring your own datasets and the multitude of open datasets available online.

Why agx Stands Out

agx combines the power of ClickHouse, a high-performance database engine, with a sleek and modern interface built using Tauri and SvelteKit. This combination ensures that users can enjoy native desktop application performance while leveraging the familiarity of web technologies. Here’s what sets agx apart:

1. Intuitive User Interface: The agx interface is designed with user experience in mind. Using Svelte for the front end ensures smooth and responsive interactions. The application provides an interactive SQL query editor complete with syntax highlighting, making it easy to write and test queries without hassle.

2. Comprehensive Schema Browser: Understanding the structure of your data is crucial for effective analysis. agx’s schema browser lets users delve into data structures effortlessly, enabling exploration with just a few clicks.

3. Dynamic Data Visualization: Results from your queries are displayed in a clear tabular format, and with agx’s integration of visualization tools, presenting your data in an easily digestible manner becomes a breeze. Whether you need to create quick charts or detailed reports, agx equips you with the necessary tools.

4. Cross-Platform Compatibility: AGX is expertly designed to function smoothly both as a native application and as a web app. This ensures that users from diverse backgrounds can easily access their data, regardless of their operating system.

5. Drag & Drop File Operations: AGX supports drag-and-drop file operations, simplifying data importation. This allows users to import datasets quickly without navigating convoluted menu options.

Getting Started with AGX

Setting up AGX is straightforward, whether opting for the live version or deploying it locally. Users can clone the repository and run it with Docker or build the native application directly by following easy installation steps. The detailed documentation provided ensures that even those new to data exploration can get started quickly.

The easiest way is via: https://agx.app

You can also run a local instance:

1. Clone the repository:

git clone https://github.com/agnosticeng/agx && cd agx

2. Run the application with Docker:

docker compose up

3. Access AGX via your web browser at;

http://localhost:8080

Or finally you can use the desktop app:

npm run tauri dev

Community and Contributions

One of the standout strengths of agx is its open-source nature, which is not just a feature but a driving force that empowers innovation. This project actively invites contributions from users and developers, fostering a vibrant and collaborative ecosystem that thrives on shared expertise and creativity. By participating—whether through submitting a Pull Request or sharing personal user experiences—you're not just joining a community; you’re becoming an integral part of a movement that is transforming the future of technology. Join the AGX community today and help shape what comes next!

Conclusion

agx is more than just a tool; it’s a gateway into the world of data exploration. Its superior performance, user-friendly interface, and robust feature set make it an indispensable resource for anyone looking to dive into datasets, whether they are personal or publicly available. By bridging the gap between complex data and clear insights, AGX empowers users to uncover hidden patterns and make informed decisions.

So, if you’re ready to elevate your data exploration experience, give AGX a try and unlock the potential of your data today!

Links:

• https://agx.app
• https://github.com/agnosticeng/agx
• http://producthunt.com/p/agx-2

agx, data analytics made easy was originally published in Agnostic on Medium, where people are continuing the conversation by highlighting and responding to this story.

Let’s dive in and show you how you can build something similar without needing to spend a fortune.

Step 1: Define Your Data Problem

Before jumping into development, it’s important to define what problem you want to solve with blockchain data. In my case, I wanted to track user growth for crypto projects over time. I noticed a strong correlation between the number of unique users and token price movements, which sparked the idea for Token Pulse.

Key Takeaway: Think about a specific problem or trend you want to analyze — whether it’s user growth, transaction volume, or on-chain liquidity.

Step 2: Use Pre-Indexed Blockchain Data

Instead of building your own node or indexing blockchain data yourself (which can be complex and costly), use pre-indexed data sources like Agnostic. This makes pulling information like unique user counts and transaction volumes across chains simple.

Key Takeaway: Explore available blockchain data APIs, such as Agnostic or Infura, which allow you to access on-chain data without handling the heavy infrastructure yourself.

Step 3: Set Up a FastAPI Backend

With the data problem defined and a data provider selected, it’s time to build the backend. I used Python’s FastAPI to create a lightweight server that fetches data from the Agnostic API through psycopg2 and natural Postgres SQL. This data can then be served to the frontend which we’ll look at in a later step.

Key Takeaway: Set up a simple server to act as the engine of your app, making API requests to gather blockchain data and process the results.

Step 4: Fetch, Transform, and Query Your Data

Once you have a connection to blockchain data through Agnostic, one of the key tasks is querying it in a way that surfaces meaningful insights. In Token Pulse, I needed to compare user growth and transaction volume week over week. While the SQL query might look complex at first, it’s actually a straightforward task if you’re familiar with SQL. Or let AI query tools like this one do the work for you after you decribe the problem.

Here’s the basic idea behind the query: for each token, we calculate the total unique users from the past week and compare it with the previous week’s data. The same goes for transaction volumes. In SQL, this requires window functions to roll up data over time, but the logic itself is quite simple — just comparing two sets of numbers.

The Query:

WITH token_activity AS (
    SELECT
        address AS token_address,
        DATE_TRUNC('day', timestamp) AS day,
        COUNT(DISTINCT input_0_value_address) AS unique_addresses,
        SUM(input_2_value_uint256) AS total_volume
    FROM evm_events_{chain}_mainnet_v1
    WHERE signature = 'Transfer(address,address,uint256)'
    AND timestamp >= NOW() - INTERVAL '14 days'
    GROUP BY token_address, DATE_TRUNC('day', timestamp)
    HAVING COUNT(DISTINCT input_0_value_address) > 100
),
rolling_7_day AS (
    SELECT
        token_address,
        day,
        SUM(unique_addresses) OVER (PARTITION BY token_address ORDER BY day ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS rolling_unique_addresses,
        SUM(total_volume) OVER (PARTITION BY token_address ORDER BY day ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS rolling_total_volume
    FROM token_activity
),
comparison AS (
    SELECT
        token_address,
        MAX(CASE WHEN day BETWEEN NOW() - INTERVAL '7 days' AND NOW() THEN rolling_unique_addresses END) AS current_week_unique_addresses,
        MAX(CASE WHEN day BETWEEN NOW() - INTERVAL '14 days' AND NOW() - INTERVAL '7 days' THEN rolling_unique_addresses END) AS previous_week_unique_addresses,
        MAX(CASE WHEN day BETWEEN NOW() - INTERVAL '7 days' AND NOW() THEN rolling_total_volume END) AS current_week_total_volume,
        MAX(CASE WHEN day BETWEEN NOW() - INTERVAL '14 days' AND NOW() - INTERVAL '7 days' THEN rolling_total_volume END) AS previous_week_total_volume
    FROM rolling_7_day
    GROUP BY token_address
)
SELECT
    token_address,
    COALESCE(current_week_unique_addresses, 0) - COALESCE(previous_week_unique_addresses, 0) AS unique_addresses_growth,
    COALESCE(current_week_total_volume, 0) - COALESCE(previous_week_total_volume, 0) AS total_volume_growth
FROM comparison
ORDER BY unique_addresses_growth DESC
LIMIT 50;

At first glance, it looks pretty detailed, but breaking it down, all it’s doing is:

1. Summing Unique Users: We group transactions by token and calculate how many unique users (input_0_value_address) interacted with the token over the past week and the week before.
2. Summing Volumes: The same process is applied to transaction volumes (input_2_value_uint256).
3. Comparing Weeks: Finally, we compare this week’s totals to the previous week’s to show growth or decline.

Why It’s Not So Bad: If you know SQL or use AI tools that can assist with query generation, this process becomes much easier. In fact, querying historical data is a common task in analytics, and tools like SQL make it possible to turn that into actionable insights quickly.

Key Takeaway: Don’t let SQL queries intimidate you! With a clear understanding of what you want to track, tools like SQL make it easy to pull the insights you need.

Step 5: Make It Work Across Multiple Chains

One of the key features of Token Pulse is its ability to pull data from multiple blockchains — Ethereum, Base, Polygon, and Arbitrum. Instead of building separate apps for each chain, we streamlined the process by applying dynamic variables based on the selected chain. This included customizing the styling, explorer links, and data-fetching logic, all of which can be extended as new chains are supported by the data tool.

How We Did It:

• Styling: For each chain, we applied different colors and visual styles to match its branding. For example, Ethereum uses its signature dark gray color, while Polygon uses purple. This subtle change enhances user experience and makes it clear which chain’s data is being displayed.
• Explorer Links: Each chain has its own block explorer. We dynamically created explorer links based on the selected chain, allowing users to click through to the relevant token or transaction on the correct blockchain explorer (e.g., Etherscan for Ethereum or Polygonscan for Polygon).
• Data Fetching: We applied different API endpoints for each chain so that the data fetch aligns with the user’s choice. This way, Token Pulse can easily pull data from any supported chain, and the same logic can be extended to additional chains in the future.

Here’s a snippet of the code that makes this possible:

const chainUrls = {
    ethereum: {
        explorer: "https://etherscan.io",
        barColor: "#3C3C3D",
    },
    polygon: {
        explorer: "https://polygonscan.com",
        barColor: "#8247E5",
    },
    base: {
        explorer: "https://basescan.org",
        barColor: "#0052FF",
    },
    arbitrum: {
        explorer: "https://arbiscan.io",
        barColor: "#1C39BB",
    }
};

// Fetch and apply different styles and links based on the chain selected
function loadCharts(chain) {
    const chainInfo = chainUrls[chain];
    document.body.style.backgroundColor = chainInfo.bgColor;  // Apply background color for the chain
    // Other logic for rendering charts and setting explorer links dynamically...
}

Why It’s Powerful: This approach means that Token Pulse can easily be extended to as many chains as the data provider supports. Instead of rebuilding everything for each blockchain, you just apply dynamic variables that adjust the style, data source, and links based on the user’s choice.

Key Takeaway: If you plan to support multiple chains in your app, set up dynamic variables for things like styles, links, and API endpoints. This keeps your codebase simple while offering cross-chain functionality.

Step 6: Build a Clean, User-Friendly Frontend

Now that the core functionality is in place, the next step is creating a simple frontend that users can interact with. In Token Pulse, we focused on a clean, intuitive design that lets users quickly switch between chains and view top tokens by user growth or transaction volume.

How We Did It:

• Charts and Data: We used JavaScript to create interactive charts that visualize token growth and transaction volume. Users can hover over bars to get more details or click on them to explore further on the respective blockchain explorer.
• Dynamic Content: The frontend adapts based on the selected chain (from Step 5), allowing users to switch between Ethereum, Polygon, Base, and Arbitrum data with ease.

Key Takeaway: When building your frontend, simplicity is key. Focus on clean visuals, clear information, and easy navigation. Don’t overcomplicate the interface — just let the data shine.

Step 7: Deploy and Share!

Once the app is built, it’s time to deploy. For Token Pulse, we chose Heroku for its simplicity and scalability. However, there are plenty of other options, such as Netlify or Vercel, that make deploying a web app easy and cost-effective.

How We Did It:

• Deployment: We deployed Token Pulse using Heroku, which allows for seamless integration with FastAPI and provides easy scalability as more users access the app.
• Sharing: After deployment, sharing the app with the crypto community was crucial for gathering feedback and driving improvements.

Key Takeaway: Deployment doesn’t have to be complicated. Choose a platform that fits your needs, deploy your app, and start gathering feedback. Continuous iteration is key to refining your tool.

Conclusion: Building a powerful crypto data app like Token Pulse doesn’t have to be daunting or expensive. By following these steps, you can create an app that provides real-time insights into the blockchain world. With the right tools and approach, blockchain data is not only accessible but also empowering.

I invite you to try out Token Pulse and explore how it tracks the top crypto projects by user growth and transaction volume. Or check out the code on GitHub. Whether you’re a crypto enthusiast, investor, or just curious, I’d love to hear your feedback! Let me know if you find it helpful and feel free to comment on any improvements or features you’d like to see.

7 Steps to Building a Powerful Crypto Data App (No Engineering Team Required) was originally published in Agnostic on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the first part of this series, we dove into how to gather on-chain data for top token holders and their transfer volumes using SQL queries powered by Agnostic’s API. We showcased how easily developers and analysts can unlock insights into wallet clusters and token movements.

Now, it’s time to take that data and bring it to life by converting our SQL queries into GraphQL APIs and integrating them into an interactive web app. We’ll walk through the GraphQL conversion process, and how we built our visualization using SvelteKit and D3.js.

Converting SQL Queries into Usable GraphQL APIs

In the first part of this series, we showed how to gather data for top token holders and their transfer volumes using SQL queries from Agnostic’s blockchain data collections. Now, let’s see how we can convert those SQL queries into usable GraphQL APIs so that we can pull this data dynamically into a web app.

At Agnostic, our platform includes a feature on the “Chart” interface where you can simply click the “Convert to GraphQL” button. This converts the SQL query into a GraphQL schema and endpoint. But what happens behind the scenes? Let’s break it down.

Using Aliases to Build GraphQL Queries

When converting SQL to GraphQL, we rely on aliases to structure the query. These aliases define which parts of the SQL query will be exposed as fields or arguments in the resulting GraphQL API.

• gql_field_: This prefix is used to define fields that will be returned in the GraphQL response.
• gql_arg_: This prefix is used to define dynamic parameters (arguments) that can be passed to the query.

Let’s take the two SQL queries from the first article and transform them into a GraphQL API.

Top Token Holders

The original SQL query used to retrieve the top token holders. To convert this query into GraphQL, we use aliases for both fields and arguments. Here’s how we adjust the query:

with
  'gql_arg_string_token' as token,
  (select argMax(value, block_number) from token_total_supplies_ethereum_mainnet_v1 where token_address = token) as total_supply

select
  wallet_address as gql_field_string_wallet,
  argMax(value, block_number) as gql_field_string_amount,
  gql_field_string_amount / total_supply as gql_field_string_percent
from
 token_balances_ethereum_mainnet_v1
where token_address = token
  and wallet_address != token
group by wallet_address
order by gql_field_string_percent desc
limit 150

Here, we replaced the token address with a GraphQL argument (gql_arg_string_token) so it can be dynamically passed when making API requests. The fields we want to return — wallet address, token amount, and percentage — are now exposed using gql_field_ aliases.

This creates the following GraphQL schema:

type token_holders {
  wallet: String
  amount: String
  percent: String
}

type Query {
  token_holders(token: String!): [token_holders]
}

Token Transfers

Let’s do the very same thing with the SQL query used to gather transfer volume data between wallets.

with
  'gql_arg_string_token' as token,
  (select argMax(value, block_number) from token_total_supplies_ethereum_mainnet_v1 where token_address = token) as total_supply,

  holders as (
    select
      wallet_address,
      argMax(value, block_number) / total_supply as percent
    from
      token_balances_ethereum_mainnet_v1
    where token_address = token
      and wallet_address != token
    group by wallet_address
    order by percent desc
    limit 150
  )

select
  input_0_value_address as gql_field_string_source,
  input_1_value_address as gql_field_string_target,
  token_unit_amount(chain_name, chain_network_name, any(address), sum(input_2_value_uint256)) as gql_field_string_amount
from
 evm_events_ethereum_mainnet_v1
where address = token
  and signature = 'Transfer(address,address,uint256)'
  and gql_field_string_source in (select wallet_address from holders)
  and gql_field_string_target in (select wallet_address from holders)
group by gql_field_string_source, gql_field_string_target
order by gql_field_string_amount desc

This creates the following GraphQL schema:

type transfers_between_holders {
  source: String
  target: String
  amount: String
}

type Query {
  transfers_between_holders(token: String!): [transfers_between_holders]
}

By converting these SQL queries into GraphQL, we make it easy to dynamically pull the latest token holder and transfer data into our app.

Integrating our GraphQL API with SvelteKit app

At Agnostic, we love using SvelteKit in our applications. Its simplicity, speed, and performance are key factors that make it ideal for building reactive web applications. Since SvelteKit compiles components into optimized vanilla JavaScript, we get smaller bundles and faster load times — perfect for handling large datasets like blockchain token balances and transfers.

Let’s integrate our GraphQL API into a SvelteKit app.

Fetching Data with GraphQL in SvelteKit

First, we need to fetch the data from our GraphQL API.

// /[token]/page.server.js
import { AGNOSTIC_TOKEN } from '$env/static/private';

const query = `
 query TokenHoldersWithTransfers($token: String!) {
  token_holders(token: $token) {
   wallet
   amount
   percent
  }

  transfers_between_holders(token: $token) {
   source
   target
   amount
  }
 }
`;

export const load = async (e) => {
  const { data } = await fetch('https://graphql.eu-west-1.agnostic.engineering/graphql', {
   method: 'POST',
   headers: { Authorization: AGNOSTIC_TOKEN },
   body: JSON.stringify({ query, variables: { token: e.params.token } })
  }).then((r) => r.json());
  
  return {
    holders: data.token_holders,
    transfers: data.transfers_between_holders
  };
};

Once the data is fetched, we pass it into our Svelte component for display.

Rendering the Graph with D3.js

Now that we have the data, let’s render the graph.

Here’s an example of how we can implement the rendering of top holders as nodes, with edges between the nodes representing token transfer volumes using D3.js and Svelte:

<script>
  import * as d3 from 'd3'
  
  export let width;
  export let height;
  export let holders = [];
  export let transfers = [];

  // Positionning for the first render and the radius of each circle
  const pack = d3.pack()
    .size([width, height])
    .padding((n) => n.r + 50);
  
  // Position the lines and circles
  const simulation = d3
    .forceSimulation()
    .force(
      'link',
      d3
        .forceLink()
        .id((d) => d.data.wallet)
        .distance((d) => d.source.r + d.target.r + 50)
    )
    .force('charge', d3.forceManyBody())
    .force('center', d3.forceCenter(width / 2, height / 2).strength(0.002))
    .on('tick', ticked);

  $: nodes = pack(
      d3.hierarchy({ children: holders }).sum((d) => d.percent)
    ).leaves();

  // small trick to keep record of the position of the nodes and edges 
  // when the holders and transfers are updated
  $: {
    const old = new Map(simulation.nodes().map((n) => [n.data.wallet, n]));
    simulation.nodes(nodes.map((n) => Object.assign(n, old.get(n.data.wallet))));
    simulation.force('link')!.links(transfers.map((t) => Object.assign({}, t)));
    simulation.alpha(1).restart();
  }
  
  // update circles and lines coordinates at each iteration of the simulation
  function ticked() {
    const nodes = simulation.nodes();
    for (let i = 0; i < nodes.length; i++) {
      const circle = document.querySelector(`circle[data-index="${i}"]`);
      if (circle) {
        const d = nodes[i];
        circle.setAttribute('cx', d.x);
        circle.setAttribute('cy', d.y);
      }
    }
    
    const links = simulation.force('link')!.links();
    for (let i = 0; i < links.length; i++) {
      const line = document.querySelector(`line[data-index="${i}"]`);
      if (line) {
        const d = links[i];
        line.setAttribute('x1', d.source.x);
        line.setAttribute('y1', d.source.y);
        line.setAttribute('x2', d.target.x);
        line.setAttribute('y2', d.target.y);
      }
    }
 }
</script>

<svg
 viewBox="0 0 {width} {height}"
 {width}
 {height}
>
  <defs>
    <marker
      id="arrow"
      markerWidth="8"
      markerHeight="6"
      refX="7"
      refY="3"
      orient="auto"
      markerUnits="userSpaceOnUse"
    >
      <polygon points="0 0, 8 3, 0 6" fill="#fff" />
    </marker>
  </defs>
  <g>
    <g class="circles" fill="#6536a3" stroke="#6536a3" fill-opacity="0.2">
      {#each nodes as d, i (d.data.wallet)}
        <circle
          data-index={i}
          r={d.r}
          cx={d.x}
          cy={d.y}
        />
      {/each}
    </g>
    <g class="lines" pointer-events="none">
      {#each transfers as _, i}
        <line
          data-index={i}
          stroke="#fff"
          stroke-width="1"
          stroke-opacity="0.6"
          marker-end="url(#arrow)"
        />
      {/each}
    </g>
  </g>
</svg>

You can also easily extend this to include features like zooming and dragging support, or even toggling wallet display based on user interactions.

Explore the Graph

We’ve now built an app that visualizes the top token holders and their transfer volumes as an interactive graph. Here’s a preview of the final graph:

This graph shows the top 150 Shiba Inu token holders as nodes, with the transfer volumes between them represented as edges. The size of each node reflects the amount of Shiba Inu each wallet holds, and each edge represents the volume of tokens transferred.

Want to see the live app we created throughout the article? Go check the app TokenFlow by Agnostic

Wrapping Up

With just a few SQL queries converted into GraphQL, we’ve turned raw blockchain data into a powerful, interactive visualization. At Agnostic, we believe that accessing and visualizing blockchain data should be easy, and our API makes it possible for anyone to build their own graph-based applications in no time.

We hope this two-part series has given you insight into how you can leverage Agnostic’s API, SvelteKit, and D3.js to unlock token secrets and visualize on-chain data.

Ready to build your own graph? Start by exploring our community page and dive into the world of on-chain data visualization!

Full code available here

Stay tuned for more updates and tutorials from Agnostic, and don’t forget to share your creations with us on social!

Unlocking Token Secrets: Mapping Out Top Holders and Transfers — Part 2 was originally published in Agnostic on Medium, where people are continuing the conversation by highlighting and responding to this story.

Blockchain data visualizations have become indispensable tools for analysts and developers alike, helping to uncover insights into token holders, transaction flows, and wallet clusters. Graph-based representations of wallets and token transfers have been highlighted by platforms like Arkham and BubbleMaps, showcasing how these visualizations can reveal hidden patterns in on-chain activity.

At Agnostic, we wanted to demonstrate how easy it is to build similar visualizations using our API. In this article, we’ll walk through how we collected and analyzed data about the top wallets and their transfer volumes using SQL queries from our API, allowing us to construct an interactive graph of token activity.

Ready? Let’s dive into the data-gathering process and show you just how easy it is to get these insights.

Data Gathering: Collecting the Top Wallets Holding a Specified Token

To construct our graph, the first step is to gather the top wallets holding the token of interest. These wallets will become the nodes of our graph. We need to retrieve the balance of each wallet at the latest block and determine what percentage of the total token supply each wallet holds.

Here’s the SQL query we used to gather the top 150 wallets for a specified token:

with
  '<token_address>' as token, 
  (select argMax(value, block_number) from token_total_supplies_ethereum_mainnet_v1 where token_address = token) as total_supply -- dynamically get the latest total supply

select
  wallet_address,
  argMax(value, block_number) as amount, -- get the balance of the token at the most recent block
  amount / total_supply as percent -- calculate what percentage of the total supply each wallet holds
from
  token_balances_ethereum_mainnet_v1
where
  token_address = token
  and wallet_address != token -- exclude the token contract itself from the list
group by
  wallet_address
order by
  amount desc -- rank wallets by the largest holders first
limit 150

This query extracts:

• The latest token balances for each wallet.
• The total token supply at the latest block.
• The percentage of the total supply that each wallet controls.

These wallets form the nodes of our graph, and the size of each node can be adjusted based on the percentage of tokens held.

Tracking Token Transfers Between Wallets

With the top wallets identified, the next step is to collect the volume of token transfers between these wallets. These transfer volumes will form the edges of our graph, connecting the nodes based on the amount of tokens moved between them over time.

Here’s the SQL query we used to collect the total volume of Shiba Inu tokens transferred between the top 150 wallets:

with
  '<token_address>' as token, -- specify the token contract

  holders as (
    select
      wallet_address,
      argMax(value, block_number) as balance -- same method to get the latest balances
    from
      token_balances_ethereum_mainnet_v1
    where
      token_address = token
    group by
      wallet_address
    order by
      balance desc
    limit 150 -- focus on the top 150 holders
  )

select
  input_0_value_address as "from", -- sender address
  input_1_value_address as "to", -- receiver address
  token_unit_amount(chain_name, chain_network_name, any(address), sum(input_2_value_uint256)) as value -- sum the total transfer value between wallets
from
  evm_events_ethereum_mainnet_v1 -- track all ERC-20 Transfer events
where
  address = token
  and signature = 'Transfer(address,address,uint256)' -- focus on the Transfer event signature
  and "from" in (select wallet_address from holders) -- include only transfers between the top holders
  and "to" in (select wallet_address from holders)
group by
  "from", "to" -- group transfers by sender and receiver
order by
  value desc -- order by the largest transfer values

This query:

• Defines the token address and collects transfer volumes between the top 150 wallets.
• Filters the Transfer(address,address,uint256) event, ensuring we capture token transfers between the key holders.
• Aggregates the total volume of tokens transferred between wallets and groups by the sender and receiver.

The result is a list of edges that connect the nodes in our graph, with the edge thickness representing the volume of tokens transferred between wallets.

To Go deeper: Explore the Results on Agnostic’s Marketplace

To give you a real-world example, we’ve published the results for the Shiba Inu token directly on Agnostic’s Marketplace. You can explore the live data, which includes:

• The top 150 wallets holding Shiba Inu, along with their balances and the percentage of the total supply they hold.
• The total transfer volumes between these wallets, showing how tokens flow between key players.

What’s Next: Visualizing the Data

With the data gathered, the next step is transforming it into a graph-based visualization. In the second part of this series, we’ll walk through how to integrate the data into a SvelteKit application, utilizing D3.js for the front-end visualization. We’ll demonstrate how the wallet nodes and transfer edges come to life in an interactive graph.

Stay tuned for Part 2, where we’ll turn this data into a fully interactive, real-time visualization.

Unlocking Token Secrets: Mapping Out Top Holders and Transfers — Part 1 was originally published in Agnostic on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

At Agnostic, our mission is to democratize access to well-structured blockchain data. We aim to provide a swift, user-friendly, and robust method for querying the vast volumes of data generated by smart contract blockchains. At the heart of our operations lies a fundamental process: aggregating extensive data from blockchain nodes, decoding it, and indexing it in diverse ways to facilitate rapid and flexible data queries. As we continually retrieve data from the blockchain, one of our primary prerequisites for delivering on our mission is ensuring fast and dependable access to blockchain nodes.

I. Blockchain Archive Nodes: A Vital Component

Blockchain archive nodes are the unsung heroes of the blockchain ecosystem. These specialized nodes play a pivotal role in Agnostic’s mission to provide comprehensive and reliable blockchain data.

Archive nodes are a distinct type of blockchain node that stores the complete history of a blockchain, including every transaction, smart contract interaction, and state change since its inception. They serve as the historical record-keepers of blockchain networks.

They allow Agnostic to access historical blockchain data, enabling users to trace the evolution of smart contracts, token transfers, and more over time.

II. The Cost Challenge: Why Self-Hosted Nodes?

In our pursuit of securing fast and dependable access to blockchain nodes for Agnostic indexers, we swiftly recognized that relying on third-party RPC providers would be financially unsustainable. These providers often impose charges based on request volume, and levy premiums for accessing historical data — a requirement crucial for indexers.

Drawing from our collective experience in managing extensive infrastructure, including bare-metal servers from previous roles, we opted for the self-hosted approach. However, achieving stability and robustness in our node fleet was no small feat and involved a significant trial-and-error process.

Our self-hosted node strategy was guided by several key technical decisions:

• We prioritized using Erigon-based clients whenever available, as they proved to be more cost-effective and easier to fine-tune.
• For reliable performance and satisfying availability, we employed fast NVMe disks configured in a RAID setup on bare-metal servers.
• We used the fastest network-attached disks in a given cloud environment, as local disks are a no-go for persistent data.
• We implemented ZFS, capitalizing on valuable features such as snapshots and transparent compression.

The result? We successfully established and maintained a robust fleet of archive nodes that deliver both speed and reliability while remaining cost-effective.

III. Balancing Act: Scaling our blockchain infrastructure while keeping costs under control

From the inception of Agnostic, we made a deliberate choice to address our infrastructure scaling needs by creating multiple, mostly independent Points of Presence (PoPs). This approach involves building relatively small, geographically distributed clusters that operate without shared infrastructure components like networking, storage, or control planes.

Within each PoP, we ensure the presence of at least one archive node for each supported blockchain, allowing indexers to access data locally. However, to fulfill our commitment as a reliable infrastructure provider, redundancy becomes imperative to withstand potential failures. This redundancy entails having a pair of archive nodes for each blockchain within each PoP. While this redundancy boosts resilience, it can also incur significant costs.

In our experience, most node failures have resulted from issues like failed software updates or temporary hardware or network glitches, leading to downtimes in the order of hours. This has prompted us to rethink our approach. Is there a more cost-effective way to address these transient failures without bearing the full expense of redundant nodes for each blockchain in each PoP, just in case?

Certainly, we believe there’s room for improvement.

IV. The Genesis of Agnostic’s Proxy: Cheap but effective High Availability strategy”

We embarked on a journey to devise a more cost-effective yet equally reliable solution. The vision was clear: to create a system that would allow PoPs to tap into the resources of other PoPs when their local infrastructure encountered unavailability. This ingenious approach aimed to achieve robust availability while optimizing operational costs — a fundamental principle guiding our actions at Agnostic.

This pivotal moment marked the birth of our in-house blockchain RPC proxy system — a cornerstone of Agnostic’s infrastructure.

V. Efficient Traffic Management with Node Proxy: Smart Routing for Blockchain RPC traffic

The core configuration of our proxy revolves around a fundamental concept: pools. Each pool comprises nodes organized into tiers (tier-1, tier-2, tier-3, and so on). Our proxy maintains ongoing, out-of-band communication with these nodes, gathering both general health data and blockchain-specific information, such as the height of the highest available block.

Additionally, within each pool, a designated probe node is configured. This probe node serves as a reference point for calculating a lag metric for every other node in the pool. This metric is derived from the difference between the highest block number reported by the probe node and the corresponding number reported by each individual node in the pool.

For every pool, we define a maximum allowable lag threshold, and nodes that exceed this threshold are marked as unhealthy. Incoming traffic to a pool is intelligently distributed among the healthy nodes belonging to the lowest, non-empty tier. For instance, as long as at least one tier-1 node remains healthy, all traffic destined for the pool is directed to this node.

Let’s consider an example of a pool configured for Ethereum Mainnet in PoP A:

• Probe Node: Alchemy RPC provider
• Tier-1: Ethereum archive nodes from the local Kubernetes (K8S) cluster
• Tier-2: Node Proxy of PoP B
• Tier-3: Node Proxy of PoP C
• Tier-4: Infura RPC provider

This setup ensures that traffic primarily flows to the local node. Only in the event of a failure or maintenance of the local node will traffic be directed to PoP B and subsequently to PoP C. In the rare scenario where all self-hosted nodes are unavailable, we resort to calling upon an RPC provider, albeit at a premium cost. This strategic approach buys us crucial time to address in-house node issues swiftly and efficiently.

VI. Simplified Application Configuration: How Agnostic’s Proxy Enhances Indexer Performance

When crafting high-performance indexers, we often employ various optimizations to harness the full potential of node RPC APIs. One of the most notable optimizations is the extensive use of request batching. However, this approach can become problematic when dealing with RPC APIs provided by a diverse range of nodes.

For instance, we used to set remarkably high batch size limits on our self-hosted nodes to maximize hardware utilization. In contrast, RPC providers tended to impose relatively smaller batch size limits. This discrepancy created a conundrum for us: either we could compromise on performance and use suboptimal settings to accommodate the more conservative nodes, or we risked lower availability by tailoring our application exclusively to the self-hosted nodes.

As part of our ongoing investment in the in-house proxy layer to enhance availability, we made a strategic decision to address these inconsistencies at the proxy level. This approach allowed us to fine-tune indexer configurations aggressively for ideal scenarios (when accessing self-hosted nodes), while simultaneously ensuring transparent and graceful handling of nodes with more conservative settings.

The proxy now excels in ensuring that we adhere to various limits on parameters such as maximum batch size and rate limits, regardless of the client’s settings. This proactive management guarantees uninterrupted operation, even under challenging conditions, offering both performance optimization and robustness.

VII. Securing Access and Monitoring Performance

Our proxy layer seamlessly integrates security and performance enhancement. Authentication serves as our security foundation, permitting access solely to authorized entities.

To provide controlled access for untrusted clients, we rely on JSON Web Tokens (JWTs) with encoded limitations, such as batch size, method restrictions, and rate limits. These safeguards prevent disruptions and misuse. Simultaneously, our proxy layer diligently monitors metrics in real time, offering insights into infrastructure performance, health, and usage patterns. This comprehensive approach guarantees a stable, secure, and transparent blockchain data access system.

Conclusion: A Glimpse into Agnostic’s Infrastructure Success

In our exploration of Agnostic’s blockchain infrastructure, we’ve unveiled its intricacies, challenges, and innovative solutions, showcasing the significance of our mission.

Our Commitment to Efficiency: At the core of Agnostic’s infrastructure is an unwavering dedication to efficiency. We self-host blockchain nodes, optimize costs, and establish a resilient architecture to deliver outstanding blockchain data services without imposing exorbitant expenses on users.

Resilience and Prudent Redundancy: We prioritize resilience by implementing multiple, mostly independent Points of Presence (PoPs). Geographically distributed resources and redundancy fortify our infrastructure against potential failures, ensuring uninterrupted service for our users.

Cost-Effective Redundancy: Our innovative approach to handling node failures via a proxy mesh network underscores our commitment to cost-effectiveness. By leveraging our distributed node fleet, we optimize costs while preserving availability.

This glimpse into Agnostic’s infrastructure success signifies more than technological achievements; it underscores our steadfast dedication to simplifying blockchain development and data access. Looking ahead, we remain devoted to innovation and excellence, poised to refine and expand our infrastructure to serve our growing user base, leading the charge in advancing the blockchain data landscape. Join us as we propel Agnostic into new frontiers of blockchain technology, where accessibility, efficiency, and reliability converge to redefine what’s possible in blockchain data services.

Building the Backbone of Agnostic: A Closer Look at Our Blockchain Infrastructure was originally published in Agnostic on Medium, where people are continuing the conversation by highlighting and responding to this story.

Join us as we navigate the labyrinth of blockchain data access and reveal how Agnostic is revolutionizing how builders and analysts harness the power of Web3.

Introduction

Blockchain technology has disrupted industries and promised a more transparent, decentralized, and secure future. Amid this revolution, our small but dedicated team at Agnostic, including myself, Arnaud Briche, and Yann Amsellem, embarked on a mission to unravel the complexities of accessing data from EVM-compatible blockchains like Ethereum, Arbitrum, and Polygon. As the CEO and co-founder, I am proud to be a part of this journey alongside my talented teammates. I invite you to discover how Agnostic has emerged as a game-changing solution for builders and analysts seeking to harness the power of blockchain data.

Chapter 1: The Genesis — Encountering the Beast

Our story began when we realized that despite the wealth of information contained within EVM-compatible blockchains, accessing, decoding, and utilizing this data was far from straightforward. The challenges we faced were numerous:

• The Node Barrier: We quickly discovered that accessing blockchain data requires relying on costly nodes provided by companies like Infura or Alchemy. This expense created a financial barrier that stood in the way of unlocking the full potential of blockchain data.
• Decoding the Puzzle: Each smart contract encoded data differently on the chain, making decoding a complex and time-consuming task requiring significant expertise and resources.
• Indexing Maze: We understood that properly indexing blockchain data was essential for efficient querying and analysis, but this process needed to be more manageable, resource-intensive, and challenging to implement effectively.
• A Dearth of Analytics Tools: We found that the unique nature of blockchain data demanded tailored analytics tools, but many existing solutions fell short, making the analysis process more difficult and time-consuming.

Chapter 2: The Quest — Building Agnostic

Undeterred by these challenges, we created a solution to overcome the complexities of accessing EVM-compatible blockchain data. Our unwavering commitment to this mission led us to develop Agnostic. This platform streamlines extracting and leveraging data from these blockchains by providing low-latency SQL access.

As we ventured deeper into the world of blockchain data, we continued to expand Agnostic’s capabilities:

• Decoding Mastery: We tirelessly worked to improve our decoding prowess, and today, Agnostic successfully decodes approximately 90% of the available data. This breakthrough allows users to access a wealth of previously obscured information, making more informed decisions and building more robust applications.
• Seamless Compatibility: We realized the importance of integrating with popular analytics and monitoring tools, so we ensured Agnostic’s compatibility with the PostgreSQL protocol. This feature allows Agnostic to effortlessly incorporate tools like Grafana and Superset, simplifying the analysis process and broadening the range of tools available to users.
• The Power of an Integrated SQL IDE: To make data extraction more intuitive and user-friendly, we developed an integrated SQL IDE that enables users to craft SQL queries easily, complete with autocomplete features. This design empowers users to extract the exact data they need while minimizing the learning curve typically associated with querying blockchain data.

• Unleashing the Potential of GraphQL API: We wanted builders to focus on creating cutting-edge applications without getting bogged down by data access challenges. So, we incorporated a feature that enables users to generate a GraphQL API directly from their SQL queries. This simplifies the process of building dApps and accessing live, low-latency data.

Chapter 3: The Reward — Agnostic’s Impact on the Blockchain Ecosystem

• Revolutionizing DevOps: Agnostic’s compatibility with Grafana allows developers to monitor their smart contract data like never before. This integration streamlines tracking and managing on-chain data, resulting in more efficient and proactive DevOps practices. Additionally, the alerting system ensures that potential issues are identified and addressed promptly, reducing the risks associated with operating in the Web3 environment.
• Empowering Analysts: As the leading data indexing platform for EVM-compatible blockchains, Agnostic has become an invaluable tool for analysts seeking insights from blockchain data. The platform’s ability to decode and index vast amounts of data, coupled with its compatibility with leading analytics tools, enables analysts to unlock the full potential of blockchain information, driving better decision-making and uncovering new opportunities.
• Building a Community: At the heart of Agnostic’s success is our commitment to fostering a vibrant and engaged community of users, partners, and enthusiasts. By providing comprehensive documentation, responsive support, and continuously improving our platform based on user feedback, we strive to create an environment that nurtures collaboration and innovation.

Chapter 4: The Road Ahead — Agnostic’s Vision for the Future

Our journey thus far has been challenging and rewarding, but we know much is still to be done. As we continue to refine and expand our platform, we remain focused on our core mission: to simplify and democratize access to EVM-compatible blockchain data.

Looking ahead, we are committed to:

• Expanding our coverage: As the blockchain landscape evolves, we will strive to support other EVM-compatible networks, ensuring that our users can access the most relevant and up-to-date data.
• Enhancing the developer experience: We will continue to invest in our integrated SQL IDE, autocomplete features, and GraphQL API generation capabilities, making it even easier for developers to build robust and innovative applications on top of EVM-compatible blockchains.
• Strengthening our community: We will deepen our engagement with users, partners, and the broader blockchain community, fostering collaboration and knowledge-sharing to drive the future of decentralized technology.

Conclusion

Our journey through the complexities of EVM-compatible blockchain data access has been one of discovery, innovation, and growth. As we continue to develop Agnostic, we remain dedicated to empowering builders and analysts with the tools they need to unlock the full potential of the Web3 world. Together, we will shape the future of decentralized technology and create a more open, transparent, and secure digital ecosystem.

Taming the Beast: A Journey Through Blockchain Data Access with Agnostic was originally published in Agnostic on Medium, where people are continuing the conversation by highlighting and responding to this story.