Navigating Web3 Data with Subsquid
Introduction
Subsquid is as an essential tool for parsing data from over 100 chains. It includes an open-source SDK, specialized data lakes for on-chain data (Archives) and a hosted service (Subsquid Cloud). I’ve spent a few months at Origin working with Subsquid to process OETH, OUSD, and OGV related data, and I’m impressed with its speed and agility. I hope this article will help you understand the benefits and capabilities of using Subsquid in your own projects.
Overview
Why did we choose Subsquid? It’s fast, easy to maintain, and uses TypeScript which we’re familiar with. We get an SDK that takes advantage of code generation, and a data archive that quickly looks up the data we care about. The SDK generates code from contract ABIs as well as a GraphQL schema which we define. The code generated from ABIs allows you to set filters for targeting relevant data and interact directly with blockchain contracts. The GraphQL schema is used to generate our TypeORM model and facilitates the creation of database migrations.
This code generation is great because we can spend most of our time writing our schema and mapping code instead of building infrastructure related code. If we decide later that our schema needs to change, the db migrations are easily regenerated and TypeScript helps find whatever we’ve broken. Once the data is mapped into our schema, we can access it through PostgreSQL or GraphQL. Subsquid Cloud allows for easy deployment of multiple versions of your squid, so you can safely transition from one deployment to the next.
Quick Example Project (repo)
I learn best by example, so here is a fairly easy example you can do yourself.
For this squid we’ll focus on tracking OETH balance increases for a couple of randomly chosen addresses.
Requirements
- Node.js
- Docker
- Mainnet RPC URL
Setup
First we’ll set up our project:
# Install Squid CLI
npm i -g @subsquid/cli@latest
# Scaffold a new squid from a template
sqd init my-awesome-squid --template abi
cd my-awesome-squid
# Install the dependencies
npm ciABIs
Next we’ll copy in our ABIs:
- OETH ABI to
abi/oeth.json - Chainlink Feed Registry to
abi/chainlink-feed-registry.json
Run sqd typegen to generate code using the ABI.
You should see the following output:
TYPEGEN
11:35:12 INFO sqd:evm-typegen saved src/abi/abi.support.ts
11:35:12 INFO sqd:evm-typegen saved src/abi/multicall.ts
11:35:12 INFO sqd:evm-typegen processing ./abi/chainlink-feed-registry.json
11:35:12 INFO sqd:evm-typegen saved src/abi/chainlink-feed-registry.abi.ts
11:35:12 INFO sqd:evm-typegen saved src/abi/chainlink-feed-registry.ts
11:35:12 INFO sqd:evm-typegen processing ./abi/oeth.json
11:35:12 INFO sqd:evm-typegen saved src/abi/oeth.abi.ts
11:35:12 INFO sqd:evm-typegen saved src/abi/oeth.tsSchema
Next we’ll create our schema file: schema.graphql
type BalanceIncrease @entity {
id: ID!
blockNumber: Int @index
blockTimestamp: DateTime! @index
address: String! @index
account: String! @index
amount: BigInt!
amountUsd: BigInt!
}Now generate our model files and db schema migration:
# Generate Model
sqd codegen
# Spin up postgres container
sqd up
# Generate db schema migration
sqd migration:generateProcessing
Next create the following files:
src/processor.ts
In this file we set up our processor and set our fields & filters.
import {
EvmBatchProcessor,
EvmBatchProcessorFields,
BlockHeader,
Log as _Log,
Transaction as _Transaction
} from '@subsquid/evm-processor'
import {lookupArchive} from '@subsquid/archive-registry'
import * as contractAbi from './abi/oeth'
export const processor = new EvmBatchProcessor()
.setDataSource({
archive: lookupArchive('eth-mainnet', {type: 'EVM'}),
chain: {
url: process.env.RPC_ENDPOINT!
}
})
.setFinalityConfirmation(10)
.setFields({
log: {
topics: true,
data: true,
}
})
.addLog({
address: ['0x856c4efb76c1d1ae02e20ceb03a2a6a08b0b8dc3'],
topic0: [
contractAbi.events.TotalSupplyUpdatedHighres.topic,
],
range: {
from: 16933090,
},
})
export type Fields = EvmBatchProcessorFields<typeof processor>
export type Block = BlockHeader<Fields>
export type Log = _Log<Fields>
export type Transaction = _Transaction<Fields>src/main.ts
In this file we parse the incoming data and map it into our database.
import {processor} from './processor'
import {TypeormDatabase} from '@subsquid/typeorm-store';
import * as oethAbi from './abi/oeth'
import * as chainlinkAbi from './abi/chainlink-feed-registry'
import {BalanceIncrease} from './model';
const oethAddress = '0x856c4efb76c1d1ae02e20ceb03a2a6a08b0b8dc3' // lowercase
const addresses = [ // randomly picked addresses
'0xbB0B2734e54f74AcE27645097bA33A4A19C56Be4',
'0xC7aD94376B7117DFd68e0E5c50Ce189DD3Fb55E9'
]
const chainlinkFeedRegistryAddress = '0x47fb2585d2c56fe188d0e6ec628a38b74fceeedf'
processor.run(new TypeormDatabase(), async (ctx) => {
const balanceIncreases: BalanceIncrease[] = []
for (const block of ctx.blocks) {
for (const log of block.logs) {
if (
log.address === oethAddress &&
log.topics[0] === oethAbi.events.TotalSupplyUpdatedHighres.topic
) {
// Any TotalSupplyUpdatedHighres event that comes in means all addresses have had their balance changed.
const previous = new oethAbi.Contract(ctx, {height: block.header.height - 1}, oethAddress)
const current = new oethAbi.Contract(ctx, block.header, oethAddress)
const chainlink = new chainlinkAbi.Contract(ctx, block.header, chainlinkFeedRegistryAddress)
const entities = await Promise.all(addresses.map(async address => {
const [previousBalance, currentBalance, ethUsdRate] = await Promise.all([
previous.balanceOf(address),
current.balanceOf(address),
chainlink.latestAnswer('0xEeeeeEeeeEeEeeEeEeEeeEEEeeeeEeeeeeeeEEeE', '0x0000000000000000000000000000000000000348')
])
const amount = currentBalance - previousBalance
return new BalanceIncrease({
id: `${log.id}:${address}`,
blockNumber: block.header.height,
blockTimestamp: new Date(block.header.timestamp),
address: oethAddress,
account: address.toLowerCase(),
amount,
amountUsd: amount * ethUsdRate
})
}))
balanceIncreases.push(...entities)
}
}
}
await ctx.store.insert(balanceIncreases.filter(bi => bi.amount > 0))
})Execution
Set your RPC_ENDPOINT url in the .env file.
Get Christopher Jacobs’s stories in your inbox
Join Medium for free to get updates from this writer.
Once we’ve done that, we can run our squid!
# Run the squid
sqd processIt may take 5 or 10 minutes to complete. We can fine-tune the performance by setting more proper from and to ranges.
If we make mistakes and want to reprocess the data, you can run the following:
# Spin down db, Spin up *fresh* db, Run the squid
sqd down && sqd up && sqd processResults
Now to see the results of the running processor:
# Start the GraphQL explorer
sqd serveNow visit the GraphQL explorer: http://localhost:4350/graphql
Use the following query to see some of the balance increase entries:
query MyQuery {
balanceIncreases(limit: 10) {
id
blockNumber
blockTimestamp
address
account
amount
amountUsd
}
}Output:
{
"data": {
"balanceIncreases": [
{
"id": "0017751416-000302-b1147:0xbB0B2734e54f74AcE27645097bA33A4A19C56Be4",
"blockNumber": 17751416,
"blockTimestamp": "2023-07-22T21:42:23.000000Z",
"address": "0x856c4efb76c1d1ae02e20ceb03a2a6a08b0b8dc3",
"account": "0xbb0b2734e54f74ace27645097ba33a4a19c56be4",
"amount": "15696759272278",
"amountUsd": "2964351"
},
...
]
}
}Other Thoughts
Handling Growing Complexity
If your squid grows rather large in size, it can get complicated understanding what processor filters apply to which mapping code. Later on you might discover some things need to change and unless you’ve documented things well, a fair amount of uncertainty can hit you.
So what I’ve chosen to do in the Origin Squid is segregate each logical processor into their own modules. We set the processor filters, the desired range, and do the mapping code all in the same module.
Each module follows the following interface:
interface Processor {
name?: string
from?: number
initialize?: (ctx: Context) => Promise<void> // To only be run once per `sqd process`.
setup?: (p: ReturnType<typeof createSquidProcessor>) => void
process: (ctx: Context) => Promise<void>
}We aggregate all these processors in main.ts and from there pass the filter and range into processor.ts. So long as we don’t have interdependencies between the various processors, they can each run asynchronously from each-other.
Helpers
You may find yourself making the same mistakes too many times, get fed up with it, and stop it from ever happening again. I’m guilty of sometimes neglecting address capitalization and padding.
Let’s say you’re looking to filter Transfer events of OETH on a specific address. You’ll have to make sure you pad your address for the topic1 and topic2 filters. It won’t tell you if you’ve made the obvious mistake of passing a Byte20 into a Byte32 because they’re both just string types.
To help prevent myself from making this mistake again in the future I created a function to create objects which can be used to set processor filters as well as later on match the appropriate log. This saves time coding, and ever so much more time in preventing the “why isn’t this working?” head scratchers.
Differential/Event Processing vs Contract Calls
You may find yourself tempted to recreate blockchain state by processing events and matching contract logic. For example, if we start at an ERC20’s deploy date and receive and process all of its transfer events, we should have accurate balances for all the ERC20 holders. This works fine, but you’ll always have to process from the beginning to have accurate data. Also, it won’t work with some special ERC20s that rebase without producing Transfer events, such as OETH and stETH. In some cases however, it is not possible to rely on events to give you the full picture of what’s going on internally. So we tend to make contract calls to get the data we care about instead. We use events and traces to help us determine when it is the right time to get new data. This isn’t always a great choice though as some contracts have an absurd number of events, which can hinder performance. For those, we rely on periodic updates instead.
Conclusion
Subsquid is quick to set up, easy to work with, and fast at processing. It’s absolutely worth trying out for your project.
You can take a look at the Origin Squid to see how we’ve been using Subsquid at Origin Protocol.
