Update (4/11/23): Cortina will activate on the Avalanche Mainnet at 11 AM ET on Tuesday, April 25th. Cortina includes protocol optimizations that are not compatible with versions < v1.10.0. If you do not upgrade to AvalancheGo@v1.10.0 before the activation time, your node will be unable to process new blocks and will be marked as offline by other nodes (potentially jeopardizing your staking rewards).

Update (4/6/23): Cortina activated successfully on the Fuji Testnet this morning at 11 AM ET!

On Monday, April 3rd, 2023, the pre-release code for the Avalanche Cortina Upgrade will be published. This upgrade will activate at 11 A.M. ET (3 P.M. UTC) on Thursday, April 6th, 2023. Note, this pre-release code will ONLY work on Fuji. If you run the code on Mainnet, it will exit on startup.

Pending a successful Cortina Upgrade on Fuji, the Avalanche Mainnet activation time will be announced and the official Cortina AvalancheGo release (v1.10.0) will be published.

The Cortina Upgrade includes protocol optimizations that are not compatible with AvalancheGo versions < v1.10.0. If you run a node on Fuji, you must upgrade your software to AvalancheGo >= v1.10.0 before the activation time on Fuji. If you are a Mainnet node operator, there is no action required until the official AvalancheGo@v1.10.0 code is published.

X-Chain Linearization

The X-Chain runs Avalanche Consensus, a leaderless, DAG-based protocol that allows for the concurrent processing of non-conflicting UTXOs at high throughput without establishing a total ordering of activity. The C-Chain, P-Chain, and all Avalanche Subnets, on the other hand, run Snowman++, a chain-based, totally-ordered protocol that sequences conflict-free block production without time-based slots over thousands of participants.

The existing semantics of the X-Chain prevent or significantly complicate the integration of Avalanche Warp Messaging (AWM), the addition of complex X-Chain transactions, the enablement of state syncing, and broad exchange support. AWM integration requires Snowman++ to verify the BLS Multi-Signature of incoming messages from other Avalanche Subnets. This limitation means the X-Chain, in its current form, cannot interact with Subnets and that the DAG-based consensus it runs cannot be broadly applied to Subnets, which overwhelmingly wish to communicate seamlessly with other Subnets. The partial-ordering on the X-Chain means there is no canonical state during vertex verification (a vertex is the container for batching transactions on the X-Chain, akin to a block in a blockchain) and that vertices are, by design, processed in a different order on different nodes. Without a canonical state, interacting with shared on-chain objects, such as exchanges, and state syncing to the tip of the network (to avoid re-processing all historical activity) becomes a non-trivial and error-prone problem that takes away from the time the community could spend further evolving Subnets. Finally, the non-deterministic ordering of on-chain activity greatly impedes the ability of many legacy exchanges to integrate with the X-Chain in its current form, as most legacy exchanges are designed for totally-ordered blockchains such as Bitcoin and Ethereum, and they have difficulty reconciling balances at different points in time on a partially-ordered blockchain

Cortina migrates the X-Chain to run Snowman++ consensus and operate as a totally-ordered blockchain in a process called “linearization”. When linearization begins, it will no longer be possible to add additional vertices to the X-Chain DAG. The terminal state of the DAG, now immutable, will then be used as the genesis state of the linearized X-Chain powered by Snowman++. The transaction format used on the X-Chain and the APIs to submit transactions, fetch transaction status, and fetch balance will not change during this process, so most wallets will not need to make any changes to support this linearization event. However, explorers that support the X-Chain will need to migrate to parsing X-Chain blocks instead of parsing X-Chain vertices, which look very similar to P-Chain blocks. Linearization is seamless and should not result in any downtime on the P-Chain, C-Chain, or any Subnets. The X-Chain will, however, be briefly inaccessible.

As alluded to above, this migration paves the way for Avalanche Warp Messaging integration, novel transaction types that modify shared X-Chain state, provides a straightforward path to enable state syncing, and enables exchanges to support the X-Chain, which will contain many of the tokens used on Elastic Subnets. While it is possible to introduce a total ordering over a DAG, doing so on the X-Chain would have required a rewrite of the existing Avalanche Consensus engine and would not have been useful for any Subnets. Migrating to a single consensus engine across the entire Avalanche Network, which reduces the size of the trusted computing base and increases the leverage of existing R&D efforts, will enable faster development and more broadly-applicable innovation.

We’ve prepared a migration guide for integrators here that highlights all changes to the AvalancheGo APIs required to support Cortina.

Batched Delegator Rewards

Since the launch of the Avalanche Network, validators have had the opportunity to charge a service fee to anyone that delegates to their node. If a validator is online for 80% of a delegation period, they receive a % of the reward (the fee) earned by the delegator. The P-Chain distributes this fee as a separate UTXO per delegation period.

As the number of delegators on the network increased substantially over the last few months (up to ~80k as of 3/20/23), the number of UTXOs that a validator may receive as fees also increased substantially. This often means that a validator will end up with thousands of small UTXOs that must be aggregated to be used for anything. Tracking thousands of UTXOs in explorers and wallets also makes providing a great UX more challenging than it needs to be.

Cortina modifies how these delegation fees are distributed for all validators that begin staking after the Cortina Activation (previously staked validators will see no change). Instead of sending a fee UTXO for each successful delegation period, fees are now batched during a node’s entire validation period and are distributed when it is unstaked.

Increased C-Chain Gas Limit

Since Apricot Phase 1, the C-Chain block gas limit has been set to 8M gas. Blocks on the C-Chain are produced every ~2s, so this setting limits the max amount of gas that can be consumed every 10s to ~40M gas. The gas target for every rolling 10s window, however, is set to 15M gas. This means that when more than 15M gas is used in a 10s window, the gas price will go up (and go down when less than 15M is used). You can read more about how dynamic fees work on the C-Chain here.

Outside of limiting the amount of gas that can be consumed in some window at any gas price, the block gas limit also limits the complexity of transactions that can be issued in a single block. As different developers on Avalanche began to deploy more complex dApps, they’ve expressed that 8M gas per block isn’t enough for their use case. Cortina increases the C-Chain block gas limit to 15M gas. To avoid increasing the amount of resources required to validate the Primary Network, the gas target will remain unchanged at 15M gas per 10s.

FAQ

How do I upgrade my node?

The process to upgrade to AvalancheGo v1.10.0 is the same as any other upgrade. If you build from source, run the build script as before. If you use the pre-compiled binaries, invoke them as before. If you use the installer script, use that as before.

Once you start AvalancheGo v1.10.0, you do not need to do anything else. More information on updating a node can be found here. As a reminder, it is best practice to have a backup of your staking key/certificate.

Do I have to upgrade my node?

If you don’t upgrade your validator to v1.10.0 before the Avalanche Mainnet activation date (which will be shared in coming days), your node will be marked as offline and other nodes will report your node as having lower uptime, which may jeopardize your staking rewards.

Is there any change in hardware requirements?

No.

Will updating decrease my validator’s uptime?

No. As a reminder, you can check your validator’s estimated uptime using the info.uptime API call:

I think something is wrong. What should I do?

First, make sure that you’ve read the documentation thoroughly and checked the FAQs. If you don’t see an answer to your question, go to our Discord server and search for your question. If it has not already been asked, please post it in the appropriate channel.

Cortina: X-Chain Linearization was originally published in Avalanche on Medium, where people are continuing the conversation by highlighting and responding to this story.

Update (10/18): 🎉🎉 Banff activated successfully on Avalanche Mainnet this morning at 12 PM EDT! 🎉🎉

Elastic Subnets

Banff unlocks the ability for Subnet creators to activate Proof-of-Stake validation and uptime-based rewards using their own token on their own Subnet. This means that for the first time anyone can become a validator of a Subnet by simply staking its token on the P-Chain. Subnets that choose to enable these new features must undergo a one-time transformation into a new type of Subnet called an Elastic Subnet.

When enabling Elastic Validation, the creator permanently locks the Subnet from future modification (they relinquish their control keys), specifies an Avalanche Native Token (ANT) that validators must use for staking and that will be distributed as staking rewards, and provides a set of parameters that govern how the Subnet’s staking mechanics will work (i.e. “what is the min stake amount”). Some of these configurable parameters are listed below:

• Asset ID (asset used for staking and rewards)
• Initial Token Supply (asset current supply after transformation)
• Maximum Supply (amount of asset that will exist after all rewards minted)
• Min Validator Stake (minimum amount of funds required to become a validator)
• Max Validator Stake (maximum amount of funds a single validator can be allocated, including delegated funds)
• Min Stake Duration (minimum number of seconds a staker can stake for)
• Max Stake Duration (maximum number of seconds a staker can stake for)
• Min Delegation Fee (minimum percentage a validator must charge a delegator for delegating)
• Min Delegator Stake (minimum amount of funds required to become a delegator)
• Max Validator Weight Factor (factor which calculates the maximum amount of delegation a validator can receive)
• Uptime Requirement (minimum percentage a validator must be online and responsive to receive a reward)

Enabling Elastic Validation on a Subnet is entirely optional and at the discretion of the creator(s). Those who prefer to have more control over a Subnet’s validator set will always have the option of retaining the default configuration of a new Subnet, which calls for specific nodes to be designated as validators by the creator.

At Banff activation, it will only be possible to use Avalanche Native Tokens from the X-Chain as stakeable tokens on Elastic Subnets. In a future release, support will be added for using ERC-20s as ANTs (which can be used as stakeable and rewardable assets on Elastic Subnets).

Early Subnet Validator Removal

Subnet creators must specify an explicit staking duration and stake weight for each validator they add to their Subnet’s validator set (unless the Subnet has already been transformed into an Elastic Subnet, in which case validators determine this when staking). Once set, neither a validator’s staking duration nor their stake weight can be modified.

While this isn’t a problem the vast majority of the time, a Subnet creator that adds a validator with the wrong stake weight or that is underperforming won’t be able to remove the troublesome validation until the predetermined staking duration concludes. Until now!

In Banff, Subnet creators gain the ability to remove a validator before the conclusion of its staking period from a Subnet. If a Subnet has already been made Elastic, a Subnet creator can also use this transaction to remove validators that were added before token-based staking was enabled.

P2P Protobuf Messaging

In all previous versions of AvalancheGo, all peer-to-peer (p2p) messages sent between nodes used a custom serializer to encode/decode structured data (see message/codec and wrappers). This implementation is highly optimized for AvalancheGo but is difficult to extend without invalidating existing messages/data because it isn’t backwards and forwards compatible. While this library is great for encoding data that requires a canonical format (e.g. blocks), it makes modifying AvalancheGo networking packages unnecessarily cumbersome.

In Banff, all p2p messages are serialized using Protocol Buffers (Protobuf). While there are hundreds of serialization approaches, Protobuf was the obvious choice for this enhancement because it is already used widely throughout AvalancheGo by the Custom VM handler. Outside of the wire format being backwards and forwards compatible, our new Protobuf-based serialization package uses less memory, uses less bandwidth, and is faster than our previous implementation. Not to mention, serialization code for other languages can be automatically generated. You can view all *.proto files used by AvalancheGo here.

Custom Delegation Fee Recipients

When designing Elastic Subnets, we created a new type of staking transaction to unify all token-based staking interactions (whether on the Primary Network or an Elastic Subnet). This new transaction type enables validators to specify a separate reward recipient for delegation fees, in addition to adding support for staking ANTs

This fee payment flexibility allows staking providers to directly compensate their partners for attracting delegations to their validators. As you might be guessing, this feature works equally well on the Primary Network and on Elastic Subnets.

FAQ

How do I upgrade my node?

The process to upgrade to AvalancheGo v1.9.0 is the same as any other upgrade. If you build from source, run the build script as before. If you use the pre-compiled binaries, invoke them as before. If you use the installer script, use that as before.

Once you start AvalancheGo v1.9.0, you do not need to do anything else. More information on updating a node can be found here. As a reminder, it is best practice to have a backup of your staking key/certificate.

Do I have to upgrade my node?

If you don’t upgrade your validator to v1.9.0 before the Avalanche Mainnet activation date (which will be shared in coming days), your node will be marked as offline and other nodes will report your node as having lower uptime, which may jeopardize your staking rewards.

Is there any change in hardware requirements?

No.

Will updating decrease my validator’s uptime?

No. As a reminder, you can check your validator’s estimated uptime using the `info.uptime` API call:

I think something is wrong. What should I do?

First, make sure that you’ve read the documentation thoroughly and checked the FAQs. If you don’t see an answer to your question, go to our Discord server and search for your question. If it has not already been asked, please post it in the appropriate channel.

About Avalanche

Avalanche is the fastest smart contracts platform in the blockchain industry, as measured by time-to-finality, and has the most validators securing its activity of any proof-of-stake protocol. Avalanche is blazingly fast, low cost, and green. Any smart contract-enabled application can outperform its competition by deploying on Avalanche. Don’t believe it? Try Avalanche today.

Website | Whitepapers | Twitter | Discord | GitHub | Documentation | Telegram | Facebook | LinkedIn | Reddit | YouTube

Banff: Elastic Subnets was originally published in Avalanche on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

TL;DR: Funds are safu. Multiple dev teams worked around the clock for the last few days to address a potential issue that is now fixed. This report explains what happened.

Last Sunday on September 4, 2022, we received a report of an issue from the Abracadabra and Sushi developer teams, who, in turn, were tipped off by Statemind on Immunefi. The Native Asset Call precompile, a special feature on the C-Chain that is used to interact with Avalanche Native Tokens, could be abused to trick certain contracts that performed blacklist-protected calls to user-supplied contracts to call undesirable targets. This feature does not exist on the subnet-evm and Subnets are thus unaffected.

We are incredibly happy and grateful to report that with the help and collaboration of the community all funds are safe, the reported issue has not been exploited, the code has been patched, and Avalanche remains stable. Out of an abundance of caution, the Abracadabra and Sushi teams temporarily transferred their users’ funds away from impacted contracts to secure them and will be returning them to their users. You can expect to hear more details from their team soon with how the users’ funds will be returned.

Ava Labs will be rewarding Statemind, Abracadabra, and Sushi for their responsible disclosure with a substantial bounty.

The Issue

Precompile contracts provide a standardized way to add custom functionality into the virtual machine. Such contracts can be called directly by specifying the precompile’s address and passing in the desired input. This feature is typically used to implement cryptographic functionality that would be difficult and expensive to implement in Solidity. Instead, common and well-tested libraries can be used directly to perform these operations efficiently.

The issue in this report comes from a precompile that is unique to the Avalanche C-Chain: Native Asset Call. This precompile is used to offer the same low-level call functionality for Avalanche Native Tokens as is available to the handling of AVAX.

The EVM provides the CALL opcode, which performs two actions:

1. Transfer the native coin from the caller to the callee
2. If the callee is a contract (or precompile contract), invoke it with specified calldata

Native Asset Call was added to provide the same functionality with the exception that it would perform a transfer of an Avalanche Native Token instead of the native coin of the EVM. It takes as input the following arguments:

1. Receiver address
2. AssetID of the Avalanche Native Token to transfer
3. Amount of the Avalanche Native Token to transfer
4. Calldata to pass in to a call to the receiver address

Native Asset Call performed two simple steps (similar to the EVM’s low-level CALL opcode):

1. Transfer the Avalanche Native Token to the receiver address
2. If the callee is a contract, invoke it with the specified calldata

Vital to the implementation of Native Asset Call is the address used as the calling context, when Native Asset Call invokes the receiver contract.

In order to provide the same functionality as the CALL opcode, the Call must be treated as if it comes from the original caller of Native Asset Call as opposed to Native Asset Call’s own address. Therefore, the caller address is set to the address that directly invoked Native Asset Call.

This makes an important assumption that the caller has explicitly chosen to call the precompile and therefore it should be safe to propagate the caller context. While this assumption holds for every invocation where the caller explicitly decides which address to call, it breaks down in the case of contracts that allow users to provide a callback address and do not filter out this precompile from their blacklist.

For example, this impacted the Abracadabra team’s Cauldron contracts because it provides the functionality to perform low-level calls on behalf of a user after checking that the recipient address of the call is not the BentoBox address or the Cauldron address:

https://github.com/Abracadabra-money/magic-internet-money/blob/23266d17969a95e69199670cba9d0060bff33340/contracts/CauldronV3_1.sol#L410

This low-level call, if directed to a Native Asset Call, led to a potential problem because the caller assumed that no callee would preserve the calling context, rendering the validation checks on input arguments ineffective.

Thanks to the Sushi and Abracadabra developers for their hard work to investigate and report this issue and to Statemind for tipping them off.

The Fix

As soon as the potential issue was reported, the Avalanche platform developers looked into the claims and verified the issue. The primary goal at this point was to protect any and all affected contracts on Avalanche as soon as possible without alerting any potential attacker of the issue.

Ava Labs promptly released a quick patch that prohibited the issuance of any future blocks that contained an interaction with the Native Asset Call as invalid. We then coordinated with node operators around the world to get as much stake on this version as possible. Thanks to the swift action of node operators, a majority of the stake upgraded within a few hours and prevented anyone from performing this interaction.

A new release was planned to provide a cleaner and more elegant fix by deprecating Native Asset Calls altogether. However, this upgrade was implemented incorrectly and failed to deprecate Native Asset Calls, while it disabled the previous patch. If you see a brief reappearance of Native Asset Calls right after the Apricot Phase 6 activation (typically used by auto-compounding bots), that is why. None of these transactions contain an exploit.

To re-address the issue, the developers released a final patch to again refuse future blocks that contained an interaction with the Native Asset Call. Thanks to the swift action of node operators, a majority of stake upgraded within a few hours to prevent anyone from performing this malicious interaction.

At this time, it is impossible to interact with the Native Asset Call precompile on the Fuji Testnet or on Mainnet. All contracts on the Avalanche platform that were open to this exploit are safe, with no need for interaction or modification required on their behalf. AvalancheGo@v1.9.0 will fully deprecate this precompile and introduce a safer replacement to restore full functionality for Avalanche Native Tokens on the C-Chain.

Conclusion

Many thanks to the Statemind, Abracadabra, and Sushi teams and to the entire Avalanche validator community for updating their nodes so promptly over a holiday weekend to prevent an attacker from exploiting this issue. No funds were lost and almost everyone continued to interact with Avalanche throughout the last few days without issue.

Responsible disclosure programs are a critical part of keeping ever-evolving blockchain platforms, like Avalanche, safe and we look forward to continuing to support security researchers through our Bug Bounty Program.

Apricot Phase 6: Native Asset Call Deprecation was originally published in Avalanche on Medium, where people are continuing the conversation by highlighting and responding to this story.

Apricot Phase Five will activate on the Avalanche Mainnet at 1 p.m. EST (6 p.m. UTC) on Thursday, December 2nd.

Update (11/24): Following a successful AP5 activation on the Fuji Testnet, Avalanche Mainnet AP5 activation has been scheduled for 1 p.m. EST (6 p.m. UTC) on Thursday, December 2nd. The code (AvalancheGo@v1.7.0) for Avalanche Mainnet activation can be found here.

Because this upgrade includes protocol optimizations that are not compatible with AvalancheGo versions < v1.7.0, your node will be marked as offline if you don’t upgrade by the activation time. If your node is a validator, that means other nodes will report your node as having lower uptime, which may jeopardize your staking rewards.

As a reminder, you can verify that your validator is online and healthy with the staking stats tool (https://stats.avax.network/dashboard/staking/) and register for upgrade notifications using the notify tool (https://notify.avax.network/).

This evening, the pre-release code was published for Phase Five of the Apricot Upgrade (“AP5”), which will activate at 10 a.m. EST (3 p.m. UTC) on Wednesday, November 24th on the Fuji Testnet. Note, this pre-release code only works on Fuji. If you run it on Mainnet, it will exit on startup.

Pending a successful AP5 activation on Fuji, the Avalanche Mainnet AP5 activation time will be announced and the official AP5 AvalancheGo release (v1.7.0) will be published.

This upgrade includes protocol optimizations that are not compatible with AvalancheGo versions < v1.7.0. If you run a node on Fuji Testnet, it is recommended that you update your software to AvalancheGo >= v1.7.0 before the activation time on Fuji. If you are a Mainnet node operator, there is no action required until the official AvalancheGo@v1.7.0 code is published.

P<>C Atomic Transfers

Avalanche uses an abstraction called Shared Memory to facilitate the movement of AVAX and ANTs (Avalanche Native Tokens) between the C-Chain, P-Chain, and X-Chain. In Shared Memory, all in-flight transfers are represented as UTXOs that can be consumed asynchronously by whichever chain the UTXO owner(s) calls `ImportTx` on. However there is one BIG exception, users can only go between X<>C and X<>P. This means someone trying to stake AVAX they received on the C-Chain has to travel through the X-Chain to get to the P-Chain. This extra step adds additional fees, complexity, and latency to the C<>P transfer process (one of the most common flows on the Primary Network).

Starting in AP5, users can consume UTXOs in Shared Memory on ANY Chain. This means you can now export funds from C and import them directly into P, or vice versa. This new functionality makes it easier to build interesting cross-chain mechanisms on the Primary Network and makes it easier for integrators that can’t index the X-Chain’s DAG to support staking.

Atomic Transaction Batching

When the Avalanche Network launched, the number of Atomic Transactions that could be processed in any block (on both the C-Chain and P-Chain) was capped at 1. When there is more than 1 pending transaction in the mempool, users can encounter uncharacteristic delays as they wait for their transaction to eventually be prioritized (sorted by AVAX burned) and included in a block. This throughput was sufficient for the first year of the network’s existence but has started to become a bottleneck. Since the start of Avalanche Rush, there has been a ~7–8x increase in the number of Atomic Transactions on the Primary Network each day (often coming in surges).

Starting in AP5, C-Chain and P-Chain blocks will contain multiple atomic transactions. For initial launch, this is limited to ~10 Atomic Transactions per block but could be increased in subsequent upgrades as usage increases. Paired with P<>C Atomic Transaction support, this upgrade will massively improve the cross-chain transfer UX.

C-Chain Fee Algorithm Optimizations

In Apricot Phase 3, dynamic fees were added to the C-Chain using an algorithm called “Moderato” (which targets a specific network utilization over time). This algorithm was parameterized to target 10M units of gas consumption every 10 seconds. Apricot Phase 4 introduced Snowman++ and Block-Based fees to reduce the amount of contention on the network and to encourage validators to batch transactions into fewer blocks, however, it left the target gas consumption and block production rate unmodified. We ran a large number of simulations to generate the parameter set used in AP4 and AP5, but there is nothing quite like the real world!

Although AP4 succeeded in greatly reducing contention on the C-Chain, we saw the minimum gas price oscillate at a higher rate of change than in our simulations. From one minute to the next, we observed changes of 20–50%. This directly impacted the experience of some users who saw their transactions “stall” as the minimum gas price jumped by the time their transaction made it into the mempool. Those users would then need to wait for the minimum gas price to go down before they would see their transaction included into a block (unless they “sped up” their transaction using their wallet).

Starting in AP5, Moderato is now parameterized to be ~66% more stable, target 15M units of gas consumption per 10 seconds, and better regulate block batching (less blocks that are more full). The additional gas price stability should make transaction construction more reliable, the increased target gas consumption should reduce the average gas price, and increased block batching should increase the number of transactions that are processed at a given gas price. Over the next few months, we will continue to pursue EVM optimizations that justify the further increase of target gas consumption.

Useful Links

• Apricot Phase Five Upgrade code can be found here.
• A tutorial for upgrading your node is available here.
• If you have any questions, please connect with the Ava Labs developer team here.

FAQ

How do I upgrade my node?

The process to upgrade to AvalancheGo v1.7.0 is the same as any other upgrade. If you build from source, run the build script as before. If you use the pre-compiled binaries, invoke them as before. If you use the installer script, use that as before.

Once you start AvalancheGo v1.7.0, you do not need to do anything else. More information on updating a node can be found here. As a reminder, it is best practice to have a backup of your staking key/certificate.

I use Metamask. Do I need to change anything?

No.

Do I have to upgrade my node?

If you don’t upgrade your validator to v1.7.0 before the Avalanche Mainnet activation date (which will be shared in coming days), your node will be marked as offline and other nodes will report your node as having lower uptime, which may jeopardize your staking rewards.

Is there any change in hardware requirements?

No.

Will updating decrease my validator’s uptime?

No. As a reminder, you can check your validator’s estimated uptime using the `info.uptime` API call:

I think something is wrong. What should I do?

First, make sure that you’ve read the documentation thoroughly and searched the FAQs. It might answer your question somewhere. If you don’t see the answer, go to our Discord server and search for your question. If it has not already been asked, please post it in the most appropriate channel.

Apricot Phase Five: P<>C Atomic Transfers, Atomic Transaction Batching, and C-Chain Fee Algorithm… was originally published in Avalanche on Medium, where people are continuing the conversation by highlighting and responding to this story.

Update (9/18): Following a successful AP4 activation on the Fuji Testnet, Avalanche Mainnet AP4 activation has been scheduled for 5 p.m. EDT (9 p.m. UTC) on Wednesday, September 22nd. The code (AvalancheGo@v1.6.0) for Avalanche Mainnet activation can be found here.

Because this upgrade includes protocol optimizations that are not compatible with AvalancheGo versions < v1.6.0, your node will be marked as offline if you don’t upgrade by the activation time. If your node is a validator, that means other nodes will report your node as having lower uptime, which may jeopardize your staking rewards.

As a reminder, you can verify that your validator is online and healthy with the new staking stats tool: https://stats.avax.network/dashboard/staking/

This morning, the pre-release code was published for Phase Four of the Apricot Upgrade (“AP4”), which will activate at 5 p.m. EDT (9 p.m. UTC) on Thursday, September 16th on the Fuji Testnet. Note, this pre-release code only works on Fuji. If you run it on Mainnet, it will exit on startup.

Pending a successful AP4 activation on Fuji, the Avalanche Mainnet AP4 activation time will be announced and the official AP4 AvalancheGo release (v1.6.0) will be published.

This upgrade includes protocol optimizations that are not compatible with AvalancheGo versions < v1.6.0. If you run a node on Fuji Testnet, it is recommended that you update your software to AvalancheGo >= v1.6.0 before the activation time on Fuji. If you are a Mainnet node operator, there is no action required until the official AvalancheGo@v1.6.0 code is published.

Optimization 1: Snowman++ (Reducing MEV & Contention)

In Avalanche, any node can produce a block at any time as long as that block contains some non-zero number of fee-paying transactions. After producing a valid block, it is up to the consensus engine, known as Snowman, to finalize one of the many potentially conflicting blocks at any given height. Regardless of the magnitude of the contention, Snowman eventually clears through all conflicts and accepts 1 block at each height (rejecting all others). As a general principle, however, the lower the contention, the faster the time to finality (less resources spent resolving conflicts).

Without the introduction of any optimizations, contention on the Avalanche network would increase commensurately with additional usage (as the competition for extractable value on-chain intensifies with additional TVL). This story has already started to play out as some network participants have begun to modify the node software to better take advantage of “Miner/Maximal Extractable Value” (MEV) by bypassing some of the throttling mechanisms meant to deter overly aggressive block production.

For those unfamiliar with MEV, it is generally considered “the total value that can be extracted permissionlessly (i.e. without any special rights) from the re-ordering, insertion or censorship of transactions within a block being produced” (from Flashbots). In the context of Snowman’s leaderless block production, a slightly broader definition is more applicable, one that also covers the value that can be extracted from modifying the time a block is produced and/or the parent referenced by the block being produced.

To preserve Avalanche’s fast time to finality and ensure node resource requirements don’t increase proportionally with (exponential) user growth, some low-hanging optimizations are warranted that reduce unnecessary contention and more accurately charge for expensive interactions with the network. At Ava Labs, we started a R&D effort ~6 months ago to research how to optimize the protocol to systemically limit the ability for block producers to put strain on the network (and on all the validators in the network) in their vicious competition with each other to beat each other out to perform particular on-chain interactions first. Today, that effort is unveiled: Snowman++.

Snowman++ is implemented as an overlay to Snowman (automatically applied to all user-created VMs based on Snowman) that restricts the amount of contention any given node can create by randomly enlarging the subset of validators that can produce a block at a given height over time. This subset of validators (and the priority for block production) is perturbed after the production of each block so the likelihood of producing the next block does not increase if a validator produced the previous block. Snowman++ does not make any user-facing modifications, so exchanges, wallets, etc. are not required to make any changes to their integrations (other than upgrading to AvalancheGo@v1.6.0, if they run a node). You can read the full Snowman++ technical specification here.

Engineers at Ava Labs found that Snowman++ reduced the amount of contention seen in highly adversarial environments by up to ~99.7%, compared to Snowman. This results in a direct reduction in validator bandwidth/CPU/disk IO usage in these scenarios.

Optimization 2: C-Chain Fee Reductions

Apricot Phase 3 migrated the C-Chain to use a new dynamic fee algorithm called “Moderato”. This algorithm targets a specific network utilization over time and adjusts the minimum fee any transaction must pay to be included in a block based on the amount of activity (“gas burned”) over the last 10 seconds. Since AP3 activation, gas price has averaged ~98.2 nAVAX, a ~56% reduction from AP2. The hourly average gas price over this time period is plotted below:

Generally speaking, it is responsible to only adjust the fee parameterization of a network when some innovation limits the net impact of the change to be somewhere close to neutral. For example, when DB IO goes down ~50%, fees can simultaneously be reduced by ~50%. In this vein, the gas price range on the C-Chain has been expanded to 25 nAVAX-1000 nAVAX from its previous range of 75 nAVAX-225 nAVAX, thanks to the resource reduction introduced by Snowman++, especially under load. Additionally, the “Moderato” smoothing parameters have been tuned such that the fee will increase/decrease much less rapidly, based on feedback from the community.

Optimization 3: Block-Based Fees

Last but certainly not least, Apricot Phase Four introduces the notion of a block fee that must be paid by transactions in a block if it is produced faster than the block target rate. Because any node can produce a block at any time in Snowman, it is important that blocks fairly compensate the network (i.e. burn sufficient AVAX) for the strain they generate, especially when produced in very quick succession.

The block fee is a continuously evaluated function (like “Moderato”) that increases/decreases in value based on how many blocks are being produced faster than the block target rate. The block fee is paid by the “effective tip” of all the transactions in a block, where “effective tip” is the gas price paid over the minimum required gas price multiplied by the gas used by the transaction (i.e. the unnecessary burn). This looks something like: (“gasPrice” — “minGasPrice”) * “gasUsed”.

Most people (~99%) that interact with Avalanche, will never even notice that block-based fees exist because they don’t go out of their way to expedite the inclusion of their transactions into blocks (typically there is no need to do so as the average time-to-finality is 1–2s). However, the people that do attempt to produce blocks at an aggressive clip will be charged for that ability.

Useful Links

• Apricot Phase Four Upgrade code can be found here.
• A tutorial for upgrading your node is available here.
• If you have any questions, please connect with the Ava Labs developer team here.
• For a full list of all the items that are targeted in upcoming upgrades, check out the roadmap.

FAQ

How do I upgrade my node?

The process to upgrade to AvalancheGo v1.6.0 is the same as any other upgrade. If you build from source, run the build script as before. If you use the pre-compiled binaries, invoke them as before. If you use the installer script, use that as before.

Once you start AvalancheGo v1.6.0, you do not need to do anything else. More information on updating a node can be found here. As a reminder, it is best practice to have a backup of your staking key/certificate.

What are the minimum/maximum gas prices?

25 nAVAX (66% reduction from AP3) and 1000 nAVAX, respectively.

Please note that the maximum gas price is not capped by the static gas price of AP2 (225 nAVAX). If you don’t migrate to use the built-in dynamic fee endpoints prior to AP4 activation, it is possible that the processing of your transactions will be delayed (if the minimum network gas price rises above 225 nAVAX).

You can get the current suggested gas price by calling the eth_gasPrice API method. For example:

I use Metamask. Do I need to change anything?

No. Metamask will automatically account for the new minimum/maximum gas price once they go into effect (just like it did for the introduction of dynamic fees in AP3).

Do I have to upgrade my node?

If you don’t upgrade your validator to v1.6.0 before the Avalanche Mainnet activation date (September 22nd at 5 p.m. EDT), your node will be marked as offline and other nodes will report your node as having lower uptime, which may jeopardize your staking rewards.

Is there any change in hardware requirements?

No.

Will updating decrease my validator’s uptime?

No.

I think something is wrong. What should I do?

First, make sure that you’ve read the documentation thoroughly and searched the FAQs. It might answer your question somewhere. If you don’t see the answer, go to our Discord server and search for your question. If it has not already been asked, please post it in the most appropriate channel.

Apricot Phase Four: Snowman++ and Reduced C-Chain Transaction Fees was originally published in Avalanche on Medium, where people are continuing the conversation by highlighting and responding to this story.

Update (8/17): A previous version of this post stated that Avalanche Mainnet activation would occur at 7 a.m. EDT (11 a.m. UTC) on Tuesday, August 24th, however, this time has been updated to 10 a.m. EDT (2 p.m. UTC) on Tuesday, August 24th based on community feedback.

The code (AvalancheGo@v1.5.0) for Avalanche Mainnet activation can be found here. This upgrade includes breaking changes and we recommend that all validators upgrade their nodes to AvalancheGo >= v1.5.0.

Today, we are publishing the pre-release code for Phase Three of the Apricot Upgrade (“AP3”), which will activate at 3 p.m. EDT (7 p.m. UTC) on Monday, August 16th on the Fuji Testnet. Note, the pre-release code referenced above only works on Fuji. If you run it on Avalanche Mainnet, it will exit on startup.

Pending a successful AP3 activation on Fuji, we will publish the official release that will be used on Avalanche Mainnet for AP3. AP3 will activate at 10 a.m. EDT (2 p.m. UTC) on Tuesday, August 24th on Avalanche Mainnet.

This upgrade includes breaking changes and once we publish the official Avalanche Mainnet release we recommend that all validators upgrade their nodes to AvalancheGo >= v1.5.0.

Moderato: Dynamic Fees Overview

AP3 introduces dynamic fees on the C-Chain using a new algorithm (“Moderato”) that targets a specific network utilization over time. When the network operates over the target utilization, the minimum fee a transaction must pay to be included in a block increases. When the network operates under the target utilization, the minimum fee a transaction must pay to be included in a block decreases. As the network gains capacity to handle additional load (through the activation of future protocol enhancements), this algorithm can be tuned to increase this targeted utilization (increasing the number of transactions that can be processed at a specific fee rate).

Because block production is asynchronous in Snowman, Moderato cannot depend on a height-based mechanism to determine if utilization is above or below the target (many blocks could be produced in a short window to game the fee mechanism) nor can it rely on the content of blocks alone (there is additional overhead to splitting the same content in 20 blocks instead of 1). To satisfy these constraints, Moderato forms a view of the network using a rolling, time-based window of activity, where block production is reduced into the same measure of activity as the block content itself.

When applied to the Coreth (the VM that powers the C-Chain), the result is a mechanism that computes the minimum gas price a transaction must pay as a function of the gas usage over a recent window where each block produced in the window contributes some synthetic gas overhead. When the gas usage of the window is above the gas target for the parameterized window, the gas price will increase. When the gas usage of the window is below the gas target for the parameterized window, the gas price will decrease. To ease the transition for ecosystem participants to this new mechanism, the gas price computed using this mechanism is upper-bounded by the current static gas price of 225 nAVAX (225 gwei) that was introduced in AP1.

In summary, Moderato has the following characteristics:

• Time-Based, Rolling Window Fee Calculation: The minimum gas price any transaction must pay is calculated based on the amount of gas consumed in the last 10 seconds (including the synthetic gas usage of any blocks produced during that same window). If that amount of gas used is below the target usage, the minimum gas price decreases. If it is above the target gas usage, it increases.
• Capped Fee Range (75–225 nAVAX): The dynamic fee algorithm caps the gas price at 225 nAVAX (static price introduced in AP1). This means that if you don’t change how you create transactions, your transactions will still be valid and included in a block. If you decide to take advantage of the new dynamic fee mechanism, however, you could pay a gas price as low as 75 gwei (a 66% fee reduction!). In future upgrades, these bounds may be relaxed so that the minimum gas price can drift even lower when there is low network usage.
• Fee Burning: The entire fee paid by each transaction is burned (as has been the case since the launch of the Avalanche Network).
• Block Gas Limit (8 Million Gas): The block gas limit on the C-Chain does not expand or contract depending on congestion. Instead, it is fixed at its current value (introduced in AP1): 8 million gas.

To maintain compatibility with EVM tooling that has/is upgrading to utilize the recently activated EIP-1559 (Fee market change for ETH 1.0 chain), we mapped “BaseFeePerGas” and the new “Transaction1559Payload” to the new algorithm’s minimum gas price mechanism. When broadcasting a transaction using this new format, you are charged the minimum gas price allowed in the block your transaction is included in (instead of the gas price you specify). Using the “TransactionLegacy” format, however, will still result in the gas price specified being paid. Because the “BaseFeePerGas” field is populated after AP3 activates, common tooling (like MetaMask) will automatically take advantage of flexibility provided by this new transaction type.

London EIPs Support

Additionally, AP3 activates 2 London EIPs in an unmodified form (which activated on Ethereum in early-August). These are:

• EIP-3198 (BASEFEE opcode): Access the current value of the BASEFEE
• EIP-3541 (Reject new contracts starting with the 0xEF byte): Disallow new code starting with the 0xEF byte to be deployed

EIP-1559 (Fee market change for ETH 1.0 chain) was applied in the modified form described above. EIP-3529 (Reduction in refunds) was not applied because gas refunds were removed entirely in Apricot Phase 1. EIP-3554 (Difficulty bomb delay to December 1st, 2021) was not applied because the C-Chain does not use Proof-of-Work.

Useful Links

• Apricot Phase Three Upgrade code and full details can be found here.
• A tutorial for upgrading your node is available here.
• If you have any questions, please connect with the Ava Labs developer team here.
• For a full list of all the items we are targeting in the upcoming Apricot upgrades, check out the roadmap.

How do I upgrade my node?

The process to upgrade to AvalancheGo v1.5.0 is the same as any other upgrade. If you build from source, run the build script as before. If you use the pre-compiled binaries, invoke them as before. If you use the installer script, use that as before.

Once you start AvalancheGo v1.5.0, you do not need to do anything else. More information on updating a node can be found here. As a reminder, it is best practice to have a backup of your staking key/certificate.

Please note that AvalancheGo >= v1.5.0 no longer places binaries in the “build/avalanchego-latest” and “build/avalanchego-preupgrade” directories because the node wrapper is no longer required (used to support the v1.4.5 database migration). Binaries are now placed in their pre-v1.4.5 locations, “build/avalanchego” and “build/plugins/evm”.

I use MetaMask. Do I need to change anything?

No. MetaMask will automatically account for dynamic fees once they go into effect.

How can I know the current gas price?

You can call the eth_gasPrice API method. For example:

What are the minimum/maximum gas prices?

75 nAVAX and 225nAVAX, respectively. This means that the gas price will always be less than or equal to the old, static gas price.

What if the price of AVAX increases, causing transactions to become more expensive?

The minimum/maximum gas prices may be updated in future network upgrades.

Do I have to upgrade my node?

Yes, you must upgrade your node to AvalancheGo >= v1.5.0 by 10 a.m. EDT (2 p.m. UTC) on Tuesday, August 24th. If you don’t upgrade to v1.5.0 by this time, your node will be marked as offline and other nodes will report your node as having lower uptime, which may jeopardize your staking rewards.

Is there any change in hardware requirements?

No.

Will updating decrease my validator’s uptime?

No.

I think something is wrong. What should I do?

First, make sure that you’ve read the documentation thoroughly. It might answer your question somewhere. If you don’t see the answer, go to our Discord server and search for your question. If it has not already been asked, please post it in the most appropriate channel.

Apricot Phase Three: C-Chain Dynamic Fees was originally published in Avalanche on Medium, where people are continuing the conversation by highlighting and responding to this story.

Development across the Avalanche ecosystem is growing rapidly, with new teams, applications, and assets being created every day. To keep the community up-to-speed on the work Ava Labs is doing to support these amazing efforts, we publish this weekly blog to recap our contributions to key technical areas.

This blog also provides information on how developers can get more involved in the Avalanche ecosystem, through programs like bug bounties, developer office hours, and career opportunities. Without further ado, here is this week’s engineering update:

Database Migration Recap

Last week, the code for the database migration was released as AvalancheGo@v1.4.5. Because that release was far and away the biggest item we worked on last week, I opted to summarize things in a short Twitter thread instead of on Medium. I think the folks on Discord summed up our expectation of a > 90% reduction in read IO best:

This week, we introduced a number of improvements and optimizations based on community feedback and recently released AvalancheGo@v1.4.7. If you haven’t migrated yet (you are still on version < v1.4.5), it is now easier than ever to do so with our new step-by-step instruction guide. You DO NOT need to upgrade from v1.4.4 to v1.4.5 before upgrading to v1.4.7 (the automated migration manager can go from v1.4.0-v1.4.4 directly to v1.4.7). For all the details around this upgrade and for answers to FAQs, you can check out the full upgrade guide.

Since the release of this code, many people reached out to me asking how we were able to achieve these large gains in DB efficiency. I worked with the Platform team to compile a summary of the three biggest changes:

Removed All Database Iteration from Block Processing

Previously, the current validator set was stored as a list sorted by the end times of the stakers and the pending validator set was stored as a list sorted by the start times of the stakers. When AddDelegatorTxs and RewardDelegatorTxs were verified, it required a complex analysis of the validator set. This caused AddDelegatorTxs and RewardDelegatorTxs to perform iterations over each of these databases, which resulted in a significant number of database reads.

Now, the validator set is cached in its object format, which avoids database reads while also avoiding repeated parsing of the data structure. To maintain the validator set over multiple processing blocks without a blowup in memory, the validator set is now managed as a Copy on Write data structure that is passed between blocks.

Added Additional Platformvm Caching

Previously, the platformvm had little to no caching implemented. The platformvm used to track block diffs using an implementation of the database interface. The database interface intentionally didn’t contain caching in order to avoid a large increase in memory utilization. This increase would have been non-trivial because the caching would have to be done on a per-layer basis, keeping duplicate values scattered throughout memory.

Now, there are two different implementations that manage the chain state. This first implementation lives on top of the database and caches parsed values. The second implementation tracks the diff between two chain states values. When a block is accepted, the diff is applied to the base chain state manager. This enables the efficient caching of parsed values while minimizing the memory footprint.

Grouped Related Values into Single Key-Value Pairs

Previously, related key-value pairs were stored separately. For example, a single validator would cause a transaction, a transaction status, an entry in the sorted validator set, and the node’s uptime to be tracked in the database.

By combining these values, we now store far fewer key-value pairs in the database and perform less database reads. Increasing the size of a key-value pair does increase the amount of time to read and write the value, however, this increase is significantly less than the improvement from reducing the number of reads and writes.

API Re-Architecture Progress

Yesterday, we rolled out our first round of API performance improvements and are now serving cached responses from a CDN 30% of the time:

Why should this matter to you? Well, the more responses we are able to cache the more the CDN can serve you directly without needing to relay your request to a node. Responses returned by the CDN tend to arrive in ~80% less time than responses served directly by an API node. So, anything that uses api.avax.network will appear dramatically faster without any changes on the client side.

For example, you can now fetch a block in ~50–60 ms from anywhere in the world:

curl --location \
  --request POST 'https://api.avax.network/ext/bc/C/rpc' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "jsonrpc": "2.0",
    "method": "eth_getBlockByHash",
    "params": [
        "0x73e8bc7dc66c50714f5b5240a4d478db411355dac5e3ecf69335ccf3aa8a284b",
        true
    ],
    "id": 1
  }'

Documentation and Tutorial Blast

One of the great new features enabled by the database migration was better tracking of rewarded UTXOs. We added a new documentation page that shows how these can be accessed platform.getRewardUTXOs. For all historical validation periods, it is now possible to see exactly how much was rewarded.

Additionally, we completed multiple new tutorials for the C-Chain. We now have docs and tutorials for:

• Transferring an ANT on the C-Chain
• Getting the balance of an ANT on the C-Chain
• Forking Mainnet/Fuji state on the C-Chain for local smart contract development
• Verifying smart contracts on the C-Chain Explorer
• Creating ARC-20s

Everyday Updates

Each week there are many small but meaningful improvements made to the repos we maintain (usually based on feedback from the Avalanche community). Oftentimes they don’t warrant their own section but are still worth calling attention to. Here are this week’s “everyday” improvements:

• Wallet staking rewards now accessible in an address’s transaction history (using the new platform.getRewardUTXOs endpoint):

• AvalancheJS v3.5.0 was released (includes support for the new websocket endpoint for the X-Chain)
• Ortelius updated to use new Index API (all releases >= v1.4.5)

Office Hours

Over the last few weeks, the Ava Labs engineering team has been hosting focused office hours on Discord from 2–4 PM ET each Wednesday. Last week the Client Apps (Wallet, Ledger, Ortelius) team hosted office hours. This Wednesday, office hours will be hosted by the Platform (avalanchego, coreth) team.

If you can’t make the office hours, you can always reach out with questions on Discord or on Twitter.

Join Ava Labs

Ava Labs was founded by Cornell computer scientists who brought on talent from Wall Street to execute their vision. The company has received funding from Andreessen Horowitz, Initialized Capital, and Polychain Capital, with angel investments from Balaji Srinivasan and Naval Ravikant.

We are actively hiring for a number of key technical roles. To ensure we attract the best and brightest to join our team, we support hiring remote candidates from anywhere in the world. If the work we are doing interests you, we’d love to chat! You can apply here.

If you don’t know what role to apply for or want to bump your application to the top of the stack, you can take a short 25 minute technical assessment to demonstrate your skills. We’ll reach out promptly if you demonstrate the talent that would make you a great fit at Ava Labs.

Weekly Update (5/28/2021): Ava Labs Engineering was originally published in Avalanche on Medium, where people are continuing the conversation by highlighting and responding to this story.

Development across the Avalanche ecosystem is growing rapidly, with new teams, applications, and assets being created every day. To keep the community up-to-speed on the work Ava Labs is doing to support these amazing efforts, we publish this weekly blog to recap our contributions to key technical areas.

This blog also provides information on how developers can get more involved in the Avalanche ecosystem, through programs like bug bounties, developer office hours, and career opportunities. Without further ado, here is this week’s engineering update:

Bootstrapping and Memory Optimizations: AvalancheGo@v1.4.3

Last week’s engineering update mentioned that we would be releasing the code for the upcoming Database Migration as AvalancheGo@v1.4.3, however, some node operators mentioned in Discord that they were running into issues bootstrapping their node on v1.4.2. We decided to dig into these concerns and to shift the focus of v1.4.3 to adding bootstrapping optimizations to improve startup reliability. Additionally, AvalancheGo@v1.4.3 includes the following changes:

• Networking optimizations that bring the node’s bandwidth usage to <= that of AvalancheGo@v1.3.2
• P-Chain improvements to make managing the validator set more efficient
• Reductions to the size of unnecessarily large caches on the C-Chain
• Fix for a non-deterministic bug that could cause nodes that receive a lot of transaction broadcast requests to temporarily stop producing blocks on the C-Chain until they process a block produced by another node
• Changes to how validators evaluate the uptime of unresponsive peers

To elaborate on the uptime monitoring change mentioned above, AvalancheGo@v1.4.3 is the first release that penalizes validators for being unresponsive for some period of time (known as being “benched”). Specifically, another validator will mark your validator offline if it fails to respond to at least 10 messages in a row over the course of at least 5 minutes. Your validator will be marked by that validator as online again when it becomes more responsive.

For the vast majority of validators, this change will have no effect. However, If your validator is running on a computer with insufficient CPU or a poor internet connection, other validators may consider it as having lower uptime, which could cause it to not receive a staking reward. Each node has a different perspective as to which nodes are unresponsive, so it is not possible to tell exactly how your validator’s uptime or responsiveness is perceived by other validators. However, you can use explorers like Avascan or Vscout to get a few data points:

Database Migration: AvalancheGo@v1.4.4

We are planning on including the previously discussed database migration code in AvalancheGo@v1.4.4. You can read about the performance improvements (90% reduction in DB reads) and mechanisms that enable this upgrade in last week’s engineering update. We are actively testing a pre-release of v1.4.4 on a small collection of nodes run by Ava Labs before a planned public release early next week.

At the time of release, we will distribute a detailed upgrade guide that explains what validators and exchanges should expect when running the migration. To make the rollout of this upgrade as smooth as possible, AvalancheGo@v1.4.4 includes a daemon that automatically orchestrates the entire upgrade if a series of sanity checks are passed (i.e. that there is enough free space to complete the migration).

Wallet SDK

As part of a recently kicked off refactor of the wallet (wallet.avax.network), it became apparent that its core functionality (mnemonic wallet utilities, address derivation, Ledger communication, UTXO management, etc) could be abstracted into a standalone SDK that could be used by any wallet. But why just tell you about it, when I could show you?

AVAX Send (4 lines of code):

AVAX Delegation (6 lines of code):

Over the last few weeks, we’ve been formalizing the structure of this new SDK and plan to make it publicly available in the coming weeks. We can’t wait to check out the new wallets and tools that the community creates on top!

graph-node.avax.network Deprecation

On May 28th (2 weeks from now), we will be shutting down the graph-node we are hosting at graph-node.avax.network. Anyone relying on this service should migrate to the graph-node service provided by TheGraph. The only two subgraphs deployed on graph-node.avax.network can be accessed on TheGraph at the following URLs:

• https://api.thegraph.com/subgraphs/name/dasconnor/avalanche-blocks
• https://api.thegraph.com/subgraphs/name/dasconnor/pangolin-dex

Everyday Updates

Each week there are many small but meaningful improvements made to the repos we maintain (usually based on feedback from the Avalanche community). Oftentimes they don’t warrant their own section but are still worth calling attention to. Here are this week’s “everyday” improvements:

• Apricot Phase 2 support added to AvalancheJS
• AvalancheJS hit a new weekly high in downloads:

• Major new features from wallet-beta.avax.network (ERC-20 and ERC-721 support, new login flow supporting multiple accounts, show C-Chain private key, and fixes for C -> X MAX amount send) were rolled out to wallet.avax.network

• C-Chain block explorer now includes the error that caused a transaction to revert

• Docs now include a tutorial for porting your dApp from Ethereum

Office Hours

Over the last few weeks, the Ava Labs engineering team has been hosting focused office hours on Discord from 2–4 PM ET each Wednesday. Last week the dApps team hosted office hours. This Wednesday, office hours will be hosted by the Developer Services (AvalancheJS, Avash, Ledger) team.

If you can’t make the office hours, you can always reach out with questions on Discord or on Twitter.

Join Ava Labs

Ava Labs was founded by Cornell computer scientists who brought on talent from Wall Street to execute their vision. The company has received funding from Andreessen Horowitz, Initialized Capital, and Polychain Capital, with angel investments from Balaji Srinivasan and Naval Ravikant.

We are actively hiring for a number of key technical roles. To ensure we attract the best and brightest to join our team, we support hiring remote candidates from anywhere in the world. If the work we are doing interests you, we’d love to chat! You can apply here.

If you don’t know what role to apply for or want to bump your application to the top of the stack, you can take a short 25 minute technical assessment to demonstrate your skills. We’ll reach out promptly if you demonstrate the talent that would make you a great fit at Ava Labs.

Weekly Update (5/14/2021): Ava Labs Engineering was originally published in Avalanche on Medium, where people are continuing the conversation by highlighting and responding to this story.

Development across the Avalanche ecosystem is growing rapidly, with new teams, applications, and assets being created every day. To keep the community up-to-speed on the work Ava Labs is doing to support these amazing efforts, we publish this weekly blog to recap our contributions to key technical areas.

This blog also provides information on how developers can get more involved in the Avalanche ecosystem, through programs like bug bounties, developer office hours, and career opportunities. Without further ado, here is this week’s engineering update:

Apricot Phase Two (AP2) Upgrade Reminder

As alluded to in last week’s engineering update, the code for “Apricot Phase Two: Berlin EIPs & Enhanced Avalanche Native Token (ANT) Support” was released earlier this week. You can read about all the improvements in AP2 here.

AP2 activated at 10 a.m. EDT (2 p.m. UTC) on Wednesday, May 5th on the Fuji Testnet and will activate at 7 a.m. EDT (11 a.m. UTC) on Monday, May 10th on the Avalanche Mainnet. This upgrade includes breaking changes to consensus rules and validators that do not upgrade their node by this time will become unhealthy (as they won’t be able to process blocks produced with AP2 consensus rules) and marked offline by other nodes. Make sure to upgrade!

Networking Optimizations (AvalancheGo v1.4.1 and v1.4.2)

In AvalancheGo@v1.4.0, we added some new message types to support authenticated peer gossiping. Simulations we ran on our internal test networks indicated these new message types wouldn’t meaningfully impact the bandwidth usage of the node. However, nodes that updated to v1.4.0 found that this wasn’t the case. Some node operators reported as high as a 10x increase in upload bandwidth usage.

We ran some additional analysis on our internal networks with data we collected from Avalanche Mainnet and were able to reduce the bandwidth usage of the node to within 10% of v1.3.2 in the v1.4.2 release. We’ve identified a few more optimizations that will roll out in subsequent releases that will bring the bandwidth usage down below that of v1.3.2.

As part of this exercise, we also identified some opportunities to improve stability on low bandwidth devices by reducing bursty network layer usage. In short, nodes with limited bandwidth can, in some cases, saturate their bandwidth by trying to do too much at once, such that all activities time out and nothing gets done (looping endlessly in this state). These changes will roll out with the additional bandwidth optimizations mentioned above.

Preparing for AvalancheGo@v1.4.3: Database Migration

Next week, we plan to release some significant database optimizations for AvalancheGo that will require a database migration. To prevent a drop in connected stake in the network while nodes perform this migration, we’ve developed an automatic upgrade daemon that performs this migration while keeping your existing node running. This daemon will run your validator on the previous version, bootstrap the new database using the v1.4.3 code, and then automatically cut over when complete. The binary orchestration logic developed to implement this daemon will be reused in future database migrations and will be a useful tool for orchestrating custom VMs in the additional subnets a node chooses to participate in.

To minimize network usage, the new database will bootstrap from the locally running node (on the previous version) instead of connecting to the rest of the Avalanche network over the internet. During this process, you will notice an increase in CPU and RAM usage as multiple instances of the node software will be running on your machine. We recommend preparing a bootstrapped volume and moving it onto your node if your machine will not be able to support this increase in CPU and RAM usage. Instructions will be provided on how to do this when the v1.4.3 code is released.

To ensure that validators can roll back in the case of any issue, this migration does not modify any legacy data. Rather, an entirely new database is created that uses an optimized format (this also prevents any race issues during the initial bootstrap process because the v1.4.2 node will still be running). A side effect of this approach is that your disk usage will temporarily double during this process (your machine will have both the old database and the newly formatted database). Once the new database is bootstrapped, you are then free to delete the old database.

In our testing on mainnet, we observed a ~90% reduction in read IO for a validating node. To give you an idea of what this looks like, you can check out the diagram below:

Degraded API Performance + Wallet Downtime

Early this week, the Avalanche non-custodial wallet (wallet.avax.network) encountered some downtime due to API degradation. We posted frequent updates to the new status page (status.avax.network) during the downtime. To stay up-to-date on status updates, make sure to subscribe!

In short, we encountered a ~5x increase in peak API requests in 7 days. Let’s just say this rapid growth pushed the limits of our legacy API architecture (which was still fully deployed at that time on Avalanche Mainnet).

At no time, however, did the Avalanche network stop processing blocks and/or transactions. In fact, there was a pretty large cohort of users interacting with the C-Chain thorough MetaMask throughout the downtime that did not encounter meaningful issues. Some people expressed frustration that the wallet was not available during this downtime. Just a reminder: the wallet source code is open-source and can be run by anyone with their own AvalancheGo if the hosted wallet encounters downtime.

On a different note, I’m proud to share that we finished the rollout of the new API architecture today. This new architecture should eliminate any rate limiting issues users have encountered over the last few weeks. By the end of next week, we also plan to have the performance optimizations we are working on deployed to provide better performance than direct node access for the majority of users. Imagine getting an eth_call, eth_getBlockByNumber, or eth_getTransactionReceipt response in 20ms no matter where in the world you are! If you notice any issues as this rolls out over the next few days, please let us know!

C-Chain Exchange Integration Guide

To better support exchanges integrating with the Avalanche network, we added a new C-Chain exchange integration tutorial to the docs website. We’ve heard stories of people integrating with the C-Chain in 1 day by reusing their previous ETH infrastructure. This will be integrated more deeply into the docs website early next week. If you notice any issues or have any suggestions of how to improve this tutorial, please let us know!

New Websites

This week we launched an updated website for Ava Labs. This new site includes an updated list of the members of our awesome engineering team:

We also created a website specifically for the Avalanche platform:

You can read a blog about this website redesign effort here.

“Behind-the-Scenes” Improvements

Each week many small but meaningful improvements are made to the repos we maintain (usually based on feedback from the Avalanche community. Oftentimes they don’t warrant their own section but are still worth calling attention to. Here are this week’s “behind-the-scenes” improvements:

• Mac binaries for AvalancheGo are now notarized
• Bug fixed when signing large smart contract interactions (like DEX swaps) on Ledger
• New version of Avash released to support new AvalancheGo flags (added in AvalancheGo@v1.4.2)
• Added support for ANT “atomic transfers” in AvalancheJS

Office Hours

Over the last few weeks, the Ava Labs engineering team has been hosting focused office hours on Discord from 2–4 PM ET each Wednesday. Last week the Platform team (AvalancheGo / Coreth) held we hosted office hours for Platform (AvalancheGo / Coreth). This Wednesday, office hours will focus on C-Chain dApps.

If you can’t make the office hours, you can always reach out with questions on Discord or on Twitter.

Join Ava Labs

Ava Labs was founded by Cornell computer scientists who brought on talent from Wall Street to execute their vision. The company has received funding from Andreessen Horowitz, Initialized Capital, and Polychain Capital, with angel investments from Balaji Srinivasan and Naval Ravikant.

We are actively hiring for a number of key technical roles. To ensure we attract the best and brightest to join our team, we support hiring remote candidates from anywhere in the world. If the work we are doing interests you, we’d love to chat! You can apply here.

If you don’t know what role to apply for or want to bump your application to the top of the stack, you can take a short 25 minute technical assessment to demonstrate your skills. We’ll reach out promptly if you demonstrate the talent that would make you a great fit at Ava Labs.

Weekly Update (5/7/2021): Ava Labs Engineering was originally published in Avalanche on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today, we’re releasing the code for Phase Two of the Apricot upgrade, which will activate at 10 a.m. EDT (2 p.m. UTC) on Wednesday, May 5th on the Fuji Testnet and at 7 a.m. EDT (11 a.m. UTC) on Monday, May 10th on the Avalanche Mainnet. This upgrade includes breaking changes, and we recommend that all validators upgrade their nodes before then.

Apricot Phase Two contains new C-Chain precompiles (you can find a good explanation of what a “precompile” is in Appendix E of the Ethereum Yellow Paper) that makes Avalanche Native Tokens (ANTs) more powerful. After this upgrade, you’ll be able to import custom tokens from the X-Chain and wrap them as ARC-20s on the C-Chain for use in the 50+ live dApps deployed there.

Additionally, Apricot Phase Two activates the Berlin EIPs (which activated on Ethereum in mid-April). These include:

• EIP-2565 (ModExp Gas Cost): Defines the gas cost of the ModExp (0x00..05) precompile.
• EIP-2929 (Gas cost increases for state access opcodes): Increases gas cost for SLOAD, *CALL, BALANCE, EXT* and SELFDESTRUCT when used for the first time in a transaction.
• EIP-2718 (Typed Transaction Envelope): Defines a new transaction type that is an envelope for future transaction types.
• EIP-2930 (Optional access lists): Adds a transaction type which contains an access list, a list of addresses and storage keys that the transaction plans to access. Accesses outside the list are possible, but become more expensive.

The Ethereum Cat Herders provided a great overview of each EIP and why it is important in this blog post.

For a full list of all the items we are targeting in the upcoming Apricot upgrades, check out the Apricot Phase 1 post.

Node Operators can review the below resources to prepare for the upgrade:

• Apricot Phase Two Upgrade code and full details can be found here.
• A tutorial for upgrading your node is available here.
• If you have any questions, please connect with the Ava Labs developer team here.

Apricot Phase Two: Berlin EIPs & Enhanced Avalanche Native Token (ANT) Support was originally published in Avalanche on Medium, where people are continuing the conversation by highlighting and responding to this story.