Euler Labs - Medium

Securing Euler

Miguel Cruz — Tue, 13 Aug 2024 13:13:29 GMT

Security came first in every step of building Euler v2.

The vision for transitioning Euler into a modular protocol dates back to 2022. Since then, various code designs have been explored as proof of concepts long before work officially began on what will be launched in the very near future.

As we head toward that milestone, we’re proud to share today that Euler v2 has undergone one of the most expensive and comprehensive security processes for any DeFi protocol to-date. Roughly $4m was allocated by Euler DAO to security — an incredible budget for an app-layer protocol, with many protocols budgeting closer to low six figures. Those costs covered the hiring of a dedicated Senior Cybersecurity Engineer to oversee development processes, engagements with world-class security researchers early in protocol development, the creation of bespoke fuzz testing and formal verification testing suites, 29 audits from 12 different firms, all culminating in the largest ever ($1.25m) code audit competition.

We draw inspiration from Morpho’s excellent framework on the same topic. Note that we have limited the content to only the key processes in the development of the smart contracts, but this is only one part of a bigger picture covering the security of Euler.

Reinforcing Security Through Modularity

The primary goal of Euler v2’s modular design was to improve the security and risk management of the protocol. We believe modularity is the way forward for protocol development because it can lead to more secure software that isolates risk, is easier to test, analyze, and audit, and reduces single points of failure. The core modules in Euler v2 are designed to do a small number of things in a very well-specified way. These modules include:

Ethereum Vault Connector (EVC)
Euler Vault Kit (EVK)
Euler Price Oracle (EPO)

The audit process detailed later in this blog post highlights the key benefits of Euler v2’s modular design, which enabled internal and external teams to develop and test modules both in isolation and in composition with other modules.

Security-first processes

While it is common to develop a protocol and later bring in auditors to consider its security after it has been developed, Euler v2 development turned this process upside down.

With the goal of implementing the “Swiss Cheese” model of security for v2, Euler DAO allocated roughly $4m to security — an incredible budget for a DeFi protocol, with many protocols budgeting low six figures. This capital allowed us to bring in world-class protocol engineers and security researchers early on during the design and proof-of-concept stages of Euler’s development.

To oversee processes and operations for security, Erik Arfvidson was hired by Euler Labs as Head of Security. Erik has over 16 years of experience in security, spanning web2 and web3. In addition to managing day-to-day security operations and working closely with external security partners, Erik has been working to ensure that all development practices at Euler are compliant with ISO27001 standards, which is a certification standard used by banks and defense companies.

At the onset, we engaged:

Certora to bring two security engineers in-house for six months to help guide development and formally prove assumptions about code operations;
Alberto Cuesta Cañada, ERC-4626 pioneer and developer of Yield protocol, to challenge the structure behind v2 and helped dive deep into thorny attack vectors for lending protocols. This work led to our publishing of an article on exchange rate manipulation of lending protocols; and
Cmichel and StErMi, leading security engineers from Spearbit and experts in lending protocol attack vectors, to come in-house to challenge the designs behind the protocol’s architecture, carry out threat modeling, and help conduct line-by-line analysis of key parts of the early codebase.

Testing, fuzz testing and formal verification

As the protocol began to take shape, a comprehensive suite of unit and invariant tests were also developed alongside Euler v2. Enigma Dark’s Victor Martinez was hired to build a dedicated fuzz testing suite using Medusa and Echidna, while Certora helped build a formal verification testing suite.

These two methodologies are important complementary tools for testing protocol invariants and finding edge cases in which, despite best efforts, unexpected things happen in a protocol’s code. Fuzz testing bombards a protocol’s code with randomness to find when key assumptions might be broken. Formal verification turns the codebase into a system of equations and helps formally prove that certain invariants cannot ever be broken. Fuzz testing is generally more flexible to apply but cannot prove things, while formal verification is harder to use in all cases but provides formal proofs when it works.

One of the most important invariants a lending protocol needs to test is that a user’s health should not change unless their balances or the prices of the assets they hold do. Breaking this invariant, or proving it couldn’t be broken, was set as a key challenge for both invariant testing suites. Formally proving that it couldn’t be broken was considered a ‘Holy grail’ for formal verification, which is something Certora’s team, with significant contributions from Andrew Ferraiuolo, duly delivered on (see below).

Internal and external audits

Once each module of Euler v2 was completed, it was manually audited both internally (within Euler Labs) and externally (by leading web3 security firms). Euler Labs is fortunate to house multiple smart contract teams, each of which can be used to audit another team’s code once it is completed prior to it being sent out to an external firm. To date, 29 external audits by 12 different firms have been carried out on the Euler v2 codebase.

The audit process for Euler v2 highlights a key benefit of its modular design. Each module in Euler is designed to do a few things in a very well-specified way, which allows for better documentation of assumptions about what the code should do and ultimately makes things easier for auditors. In particular, it allows for independent teams to develop and test modules both in isolation and in composition with other modules.

Utilising multiple audit firms allowed world-class security researchers with diverse backgrounds to bring their domain expertise to designing and battle testing the codebase. This helped to break ground on new areas of research, especially technical issues, such as those surrounding Dutch-auction-style liquidations.

Euler was audited by the world’s best web3 security firms.

Coordinating the Largest Code Audit Competition Ever

Code audit competitions have emerged as an essential part of the auditing pipeline, which go beyond manual reviews and formal audit engagements by experts to crowdsource security to the general public. This style of auditing is highly complementary to formal auditing processes as it invites a far more diverse cross-section of candidates to challenge a codebase using whatever experience and techniques they have at their disposal.

Cantina was therefore engaged to carry out the world’s largest ever* code audit competitions on the Euler v2 codebase with $1.25m allocated to security researchers who could find any additional post-audit bugs. The largest prize was for issues with High severity. The contest went viral, with more than 600 participants worldwide participating and, in some cases, live blogging about their attempts to claim the top prize.

Ultimately, no high or medium severity issues were found in the post-audit code (see Appendix for severity definitions), which is almost unheard of in contests of this scale. Despite this, we awarded researchers by allocating $200k for low issues found, as a way of thanking the security community for all of their hard work and contribution.

*Other projects have since followed Euler’s trend and hosted even larger competitions.

Conclusion

To summarise, Euler v2 has been secured by experienced security professionals who have monitored and shaped its development from start to finish, from its modular design to line-by-line code reviews.

It is crucial to note, however, that security does not stop with audits and is not limited to securing a protocol’s smart contracts only. The work highlighted here is only part of a much bigger security picture for the protocol. This article has not touched on countless other security initiatives that help secure the protocol, including bug bounties, monitoring, penetration tests for the user interface, plans for critical incidents, and much else.

When it comes to security, there is always more to be done.

Securing Euler was originally published in Euler Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Euler v2 Code Reveal

Miguel Cruz — Wed, 24 Apr 2024 15:09:15 GMT

TL;DR:

Euler Vault Kit (EVK) and Euler Price Oracle code is made public today.
Auditors encouraged to begin familiarising themselves with the code ahead of $1.25m record-breaking Cantina code audit contest begins on May 20th.
Builders and integrators can begin learning how to create and connect together their own custom lending and borrowing vaults, or their own synthetic assets.
For further questions or inquiries about integrating with Euler, reach out to us on Discord or Telegram

Building in public

We are delighted to unveil today the codebase for the Euler Vault Kit (EVK), one of the main modular components of Euler v2 alongside the Ethereum Vault Connector (EVC). This follows from the publication of the EVC code several months ago.

Additionally, we’re also unveiling the Euler Price Oracle, a library of immutable integrations for engaging with pricing systems which benefits not only the EVK but also the broader DeFi ecosystem.

Whilst the EVK + Price Oracle repos are still undergoing audits and further minor developments, the code is being made available publicly now as part of our commitment to build in public wherever possible and ensure that best security practices are being followed at all times.

Although it is important to note that the code and documentation that ultimately ships at launch could differ from the preliminary codebase and documents being shared today, early publication of the code will ensure that security professionals and future vault creators have sufficient time to familiarise themselves with how things work and scrutinise the code thoroughly ahead of future code audit competitions and deployments.

Cantina Code Audit Competition

A record-breaking $1.25m Cantina code audit competition for Euler v2 begins on May 20th. The EVK, Euler Price Oracle and EVC codebases are all in scope for that competition and now publicly available for security engineers to scrutinise.

Whilst there will no doubt be improvements and alterations to code and its documentation ahead of the start of the competition, security engineers who begin reviewing the code early will likely give themselves the best chance possible of uncovering valuable bugs.

Time to get building

Euler v2 works quite differently from Euler v1, with users able to create their own customised lending and borrowing vaults in a permissionless fashion using the EVK and then chain them together using the EVC. Today’s publication of the EVK code allows builders and integrators to start better understanding how to get started using Euler v2.

Advanced builders can create their own vaults from scratch and build on top of the open-source EVC, or use the EVK’s tailor-made, productised vaults, which come equipped with popular features out of the box and can typically be deployed in just a few clicks.

You can read more about the philosophy of how lending protocols work in this blog post.

Resources for getting started

For those wanting to dig in straight away, the main resources are available here:

Repos

https://github.com/euler-xyz/euler-vault-kit

https://github.com/euler-xyz/euler-price-oracle/

https://github.com/euler-xyz/ethereum-vault-connector

Docs

https://evc.wtf/docs/whitepaper

https://docs.euler.finance/euler-vault-kit-white-paper/

Join the Community

Follow us on Twitter. Join the Discord. Keep in touch on Telegram (community, announcements). Check out the Euler Finance website.

Disclaimer:

This piece is provided by Euler Labs Ltd. for informational purposes only and should not be interpreted as investment, tax, legal, insurance, or business advice. Euler Labs Ltd. and The Euler Foundation are independent entities.

Neither Euler Labs Ltd., The Euler Foundation, nor any of their owners, members, directors, officers, employees, agents, independent contractors, or affiliates are registered as an investment advisor, broker-dealer, futures commission merchant, or commodity trading advisor or are members of any self-regulatory organization.

The information provided herein is not intended to be, and should not be construed in any manner whatsoever, as personalized advice or advice tailored to the needs of any specific person. Nothing on the Website should be construed as an offer to sell, a solicitation of an offer to buy, or a recommendation for any asset or transaction.

This post reflects the current opinions of the authors and is not made on behalf of Euler Labs, The Euler Foundation, or their affiliates and does not necessarily reflect the opinions of Euler Labs, The Euler Foundation, their affiliates, or individuals associated with Euler Labs or The Euler Foundation.

Euler Labs Ltd. and The Euler Foundation do not represent or speak for or on behalf of the users of Euler Finance. The commentary and opinions provided by Euler Labs Ltd. or The Euler Foundation are for general informational purposes only, are provided “AS IS,” and without any warranty of any kind. To the best of our knowledge and belief, all information contained herein is accurate and reliable and has been obtained from public sources believed to be accurate and reliable at the time of publication.

The information provided is presented only as of the date published or indicated and may be superseded by subsequent events or for other reasons. As events and markets change continuously, previously published information and data may not be current and should not be relied upon.

The opinions reflected herein are subject to change without being updated.

Euler v2 Code Reveal was originally published in Euler Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Euler v2: The Modular Lending Platform

Michael Bentley — Thu, 22 Feb 2024 14:56:28 GMT

It’s official. Euler is relaunching! Today, we’re excited to unveil Euler v2— The Modular Lending Platform. Check out the lite paper or read on below to find out more.

Introduction

Euler v2 is a modular lending platform that will have two main components at launch: 1) the Euler Vault Kit (EVK), which empowers builders to deploy and chain together their own customised lending vaults in a permissionless manner; and 2) the Ethereum Vault Connector (EVC), a powerful, immutable, primitive which give vaults superpowers by allowing their use as collateral for other vaults. Together, the EVK and EVC provide the flexibility to build or recreate any type of pre-existing or future-state lending product inside the Euler ecosystem.

Euler v2 provides a best-in-class experience for lenders and traders alike, by providing unparalleled access to diverse risk/reward opportunities, new collateral options, lower net borrowing costs, advanced risk management tools such as sub-accounts and profit and loss simulators, custom-built limit order types (including stop-loss and take-profit), and greatly reduced liquidation costs.

Features will include:

Customizable Vaults: Euler v2 introduces the Euler Vault Kit (EVK), allowing developers to build ERC-4626 vaults with complete freedom. No constraints on governance, oracles, or upgradability.
Core Vault Class: Create governed lending products for risk-managed, capital-efficient lending experiences. Euler v2 enhances the user journey for borrowers and traders.
Edge Vault Class: Develop ungoverned, modular vaults for establishing permissionless lending markets. Euler Edge forms the base layer for a free-market approach to risk management.
Escrow Vault Class: Enable any ERC20 token as collateral without earning yield. Escrow vaults ensure collateral accessibility for liquidations and offer protection to borrowers.

New Features of EVK

Synthetic Assets: Euler v2 enables the creation of collateralized debt positions and synthetic assets. Explore new synthetic assets within Euler with a permissionless approach to product creation.
Nested Vaults: Tokenize vault shares and create nested vaults for innovative lending and borrowing opportunities. Opt-in to riskier collateral types without affecting other lenders.
Permissionless Rewards: Leverage RewardStreams, an open-source module, for seamless distribution of rewards without staking. Foster dynamic and efficient user engagement.

Ethereum Vault Connector (EVC)

Interoperability Layer: The Ethereum Vault Connector (EVC) serves as an open-source interoperability layer, allowing easy connection of vaults within the Euler ecosystem.
Advanced Trading and Risk Management: For developers, the EVC provides multicall batching, flash liquidity, gasless transactions, and an “operator concept” for advanced trading and risk management strategies.

FeeFlow: Empowering the Euler DAO

Fee Auction Mechanism: FeeFlow, an open-source module, introduces a reverse Dutch auction mechanism, empowering the Euler DAO to control fees on Euler markets and convert them into other types of assets.
Passive Income for Vault Creators: Vault creators can set fees, creating a passive income stream while contributing to the Euler DAO’s financial flexibility.

We’re Just Getting Started

Euler v2 addresses the limitations of monolithic and isolated lending protocols, aiming to become the primary liquidity layer for DeFi. The modular architecture resolves liquidity fragmentation, allowing permissionless creation of vaults that seamlessly connect across the ecosystem.

In the decentralised finance (DeFi) landscape, the foundational practice of lending and borrowing digital assets is crucial for the emergence of capital markets. Traditional monolithic lending protocols often impose restrictions on borrowing, limiting asset selections and enforcing conservative loan-to-value (LTV) requirements. Additionally, heavy fines for liquidation further hinder the trading experience for borrowers and traders. On the other hand, isolated lending markets offer greater flexibility, but come with the downside of fragmenting collateral and inhibiting rehypothecation, resulting in lower capital efficiency.

Euler v2 will be a first-of-its kind solution to these challenges, with the goal of becoming the primary liquidity layer for DeFi. Its modular architecture addresses the liquidity fragmentation problem by enabling the permissionless creation of Edge vaults, allowing seamless collateral usage across the broader ecosystem. This innovative approach connects different vaults from various product lines via the Ethereum Vault Connector (EVC), ensuring unparalleled flexibility for lenders, borrowers, builders, and traders without compromising on security or risk management.

But Euler’s vision extends beyond existing possibilities. We see a world that includes real-world assets, non-fungible tokens, IOUs for uncollateralized lending, peer-to-peer lending, and oracle-free lending. The modular design and forthcoming developments aim to establish Euler as a global liquidity layer and a comprehensive hub for lending, borrowing, and trading on EVM-based networks.

To join us on this journey, please see our lite paper here, and more technical white paper here. Happy building.

Join the Community

Follow us on Twitter. Join the Discord. Keep in touch on Telegram (community, announcements). Check out the Euler Finance website.

Euler v2: The Modular Lending Platform was originally published in Euler Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Paternalism Versus The Invisible Hand in the Risk Management of Lending Protocols

Michael Bentley — Wed, 21 Feb 2024 16:29:21 GMT

Introduction

Suppose a lending protocol allows borrowers to use USDC as collateral to take out loans of ETH. What is the maximum amount of ETH that can be borrowed with the USDC collateral?

Asked in terms of protocol risk management, what is the correct loan-to-value (LTV) ratio the protocol should adopt for such a trade?

A straightforward answer is that it is the LTV that maximises the risk/reward payoff for lenders. Unfortunately, knowing what this is at any given point in time is practically impossible to calculate, especially on-chain. Not only does the optimal LTV for a protocol depend on the expected price volatility of underlying assets, but also on liquidity, the exposure of the protocol to other assets, the number of open positions on other protocols, the rate of market arbitrage, and many other factors. The true optimal LTV therefore not only changes from moment to moment, but it needs extensive calculations to be able to verify its value at each of those moments.

Given these challenges, lending protocol design always requires heuristics and pragmatic choices. These choices mean that there is no one-size-fits-all solution when it comes to lending protocols. Significant trade-offs exist between different designs, especially when it comes to the degree to which a lending protocol is supposed to be able to adapt to changing external conditions. In this article, I divide the major design patterns into three classifications (Fig. 1) and highlight some of their trade-offs.

Figure 1 | Liquidity and risk management models. In A, under global paternalism, lenders all deposit to a single governed pool. The collateral factor setting the required loan-to-value (LTV) ratio for borrowers is determined globally in a paternalistic manner. If the risk environment deteriorates, then governance may opt to lower the LTV for the entire protocol from a higher value (Before) to something lower (After). This model allows lenders, and to some degree borrowers, to hold open positions in a relatively passive manner. In B, under the invisible hand, lenders deposit directly into a range of pools with different fixed risk parameters. In this model, if the risk environment deteriorates, lenders must actively pull deposits from higher risk pools themselves (Before) and manually move them to lower risk pools (After). This model allows lenders maximum flexibility into where they put their deposits, and helps them avoid fees and the risks of bad governance, but still requires borrowers to actively manage opening and closing loans across different pools. In C, under local paternalism, lenders deposit into pools with different fixed risk parameters via an aggregator. The aggregator may opt to put a small percentage of their liquidity into higher risk pools and some in lower risk pools. Aggregators compete against each other for deposit customers based on their ability to manage risk and deliver a reward. If the risk environment deteriorates, they will pull deposits from higher risk pools (Before) and move them to lower risk pools (After). This model allows lenders to deposit passively into the aggregator of their choosing and have risk/reward managed for them, but still requires borrowers to actively manage opening and closing loans across different pools.

Global Paternalism via DAO Governance

One of the most popular solutions for risk management of lending protocols has been a mechanism of governance over key risk parameters by decentralised autonomous organisations (DAOs) and their risk management associates (specialist risk manager organisations like Gauntlet, Chaos and Warden). I call this the ‘paternalistic’ model of risk management (Fig 1). It is ‘global’ as opposed to ‘local’ paternalism (see below), because risk is managed globally for the entire protocol and its users, not for a subset of pools or users.

This model has been used by so-called ‘monolithic’ lending protocols, like Euler v1, Compound v2, Aave v2/v3 and Spark, where LTV ratios are generally set conservatively, but not too conservatively, at some value at which every user deposit goes into. If the environment becomes less favourable for lending, then it is the responsibility of governance or governance-appointed risk manager to move the protocol-wide LTV ratios for all users from risk-on (Fig 1A — Before) to a more risk-off setting (Fig 1A — After).

Governance of risk parameters allows risk specialists to monitor on-chain activity and risk factors and adjust governed pools in a paternalistic fashion as the environment changes. It also helps prevent a fragmentation of liquidity because liquidity converges inside a single governed pool. Importantly, monolithic lending protocols are also capital efficient for borrowers, because they allow them to earn yield on their collateral, reducing their net borrowing costs. These kinds of protocols were the first to find product-market fit in DeFi and have helped brands like Compound and Aave establish themselves as early market leaders, serving as useful tools for 80% of users even if not providing a perfect risk/reward match for any given user.

However, governance-based risk management also has a number of drawbacks. DAOs are often composed of people with varied skill sets, many of whom may not be qualified to vote directly on risk parameters themselves. Delegation of voting power can help put control in more qualified DAO member hands, but this only helps to centralise decision making over a large number of parameters in the hands of a few individuals, who often end up wielding considerable power. Even when these specialists make ‘good’ decisions, DAO governance takes time and decisions might not be implemented sufficiently quickly if the environment changes rapidly. Governance also forces protocol users to accept or reject a single risk/reward outcome, when in reality users have very different tolerances. It also arguably trains users to expect that risk will be managed for them paternalistically, so that they are ill-equipped to take responsibility for managing their own risk/reward decisions.

The Invisible Hand via (Immutable) Isolated Pools

The other major class of lending protocol relies on more free market principles to allow lenders to choose a risk/reward suitable for themselves and live or die by their decisions. The philosophy behind these models is that the ‘invisible hand’ will manage risk for the market. Lenders have more freedom with respect to risk/reward under this model, but they must be prepared to take more of an active role in the management of their deposits. Free market models generally have lenders depositing into a range of different, typically ungoverned, isolated pools. They can do this either directly themselves (Fig 1B) or sometimes through an aggregator (Fig 1C, see below).

Variants of this model have been used by protocols like Kashi, Silo, Compound v3, Morpho Blue, Ajna and FraxLend. With many pools to choose from, users are free to lend across a wide range of possible LTV ratios (and other risk parameters). Some might take a cautious approach, lending at low LTV ratios, but attracting fewer borrowers, whilst others might be more open to risk, lending at higher LTV ratios, attracting more borrowers and people taking on leveraged positions. As users express their own preferences, a distribution of risk/reward tolerance emerges, which might have a central tendency, but has no single value to suit everyone (Fig 1B). If the environment becomes less favourable for lending, then it is the responsibility of individuals to move their liquidity from more risk-on pools (Figs 1B — Before) to more risk-off pools (Figs 1B — After).

Free market risk management allows lenders and borrowers maximum flexibility with respect to managing their own risk/reward tolerance. This, in turn, allows different use-cases for lending and borrowing to emerge. Borrowers taking short-term trades may be happy to pay a very high interest rate for a few days, whilst borrowers looking to collateralise their crypto assets might seek lower cost, longer-term borrowing opportunities. These different types of users can seek out similarly-minded risk/reward lenders. At the protocol layer things are often somewhat simpler with free market models too. The absence of governance allows immutable primitives to be constructed that can be used by anyone. Complexity and product-specifics can be pushed to an aggregation-layer or user-interface layer (see below). Whilst this does not necessarily reduce the complexity of the system overall, it does simplify the complexity of the trusted codebase for the subset of users who are happy to manage their own risks. Ultimately, for most users, system complexity is like a bubble under the carpet: it can be minimised and moved, but never entirely removed.

There are a number of drawbacks with free-market systems too, however. Perhaps the most important of these is the impact on capital efficiency that liquidity fragmentation brings. New isolated markets often struggle to compete with paternalistic lending protocols due to the costs of use for lenders and borrowers alike. Isolated pools not only make it harder for lenders and borrowers to find one another, but they often also make borrowing more expensive (even when users are able to find a match). That’s because in most isolated lending market protocols borrowers must use collateral which earns them no yield (see Morpho Blue, Compound v3, FraxLend). In contrast, in monolithic lending protocols borrowers can simultaneously use an asset as collateral and lend it out at the same time. This can substantially reduce the costs of borrowing, and even make borrowing profitable, enabling interest-rate arbitrage (via “carry trades”). And with more borrowing, comes more yield for lenders. Note that there is no free lunch here. Lenders are exposed to rehypothecation risks on monolithic lending protocols in a way that they are not on isolated lending protocols.

Local Paternalism via Aggregators

The major drawback of isolated lending markets is that they may require lenders to take their life into their own hands and manage their own risks. This will simply not be practical for many lenders. This is where aggregators can be useful. Aggregator protocols allow users to deposit assets into a single managed pool, where risk management is delegated to a local risk manager (which may be an organisation or an automated smart contract) who further deposits assets into isolated downstream lending markets (Fig 1C). Note that aggregators are therefore a layer above the lending protocol layer and can be used in conjunction with monolithic lending protocols and isolated lending markets simultaneously.

Aggregators today come in several flavours. There are free market aggregators, like Yearn and Idle, which are generally agnostic about the downstream lending markets they deposit into. They simply try to maximise the risk/reward payoff for their users, regardless of how rewards are achieved. And there are aggregators, like MetaMorpho, that are more opinionated about where the yield comes from. They generally try to preserve capital inside their own ecosystem products. In either case, the goal of aggregators is for a defined risk manager (which may be an organisation or an automated smart contract) to construct a portfolio of lender deposits in pools with different risk properties. If the environment becomes less favourable for lending, then it is the responsibility of the risk manager to construct a new portfolio for the lenders it serves by moving liquidity from more risk-on pools (Figs 1C — Before) to more risk-off pools (Figs 1C — After).

Aggregators are especially useful for lenders because they help them to access a wide range of risk/reward opportunities across a variety of different pools or protocols in a relatively passive manner. They can also limit the gas costs of constant risk/reward optimisation, especially for smaller depositors. On the downside, aggregators will often charge additional fees for their services. Being paternalistic in nature, they also suffer many of the same drawbacks as paternalistic governance too.

It is also often claimed that aggregators help to solve the liquidity fragmentation problem associated with isolated pools, since the isolation is largely abstracted away for lenders. However, lenders are only half the equation here. Even when lenders use aggregators, the picture for borrowers is still fragmented. Borrowers will generally still need aggregator systems of their own to manage their risks, and may therefore continue to prefer using monolithic lending protocols whose liquidity is typically larger and less volatile (especially when borrowers can earn yield on their collateral). Otherwise borrowing from illiquid pools can make for a particularly poor experience for borrowers, whose cost of borrowing can become unpredictable during supply side shocks, and even drive them towards liquidation.

Scaling lending protocols

In order for DeFi lending to compete with traditional finance, we should accept that there is no one-size-fits-all solution to building a lending protocol. DeFi needs a lending ecosystem with modularity at its core, in which different protocol designs serve different user needs.

Governed monolithic lending protocols provide DeFi users today with unparalleled capital efficiency, but have significant drawbacks when it comes to providing diverse risk/reward opportunities and exposing users to the many risks of governance. Isolated lending markets relying on the flow of the free market help provide a wide range of choice for users, and may form primitives on top of which custom products can be built, but often lack capital efficiency and expose users to risks that they may not be able to manage sufficiently well or in a cost-effective manner. They may also make life more difficult for borrowers.

Conclusion

To truly scale lending in DeFi requires development of an agnostic ecosystem in which different types of lending protocols can flourish alongside one another. This requires increased composability and connectedness of diverse collateral types from different protocols. True freedom is not about choosing between paternalism vs the invisible hand; it is about being able to switch seamlessly between whichever kind of model you prefer at any time you want.

This is why we’re building Euler v2: a modular platform allowing users to create and use a suite of highly customizable lending vaults tailored to their needs. Built on top of the Ethereum Vault Connector (EVC), Euler will bridge the gap between monolithic lending protocols and isolated pools. It will recognise that different users have different risk/reward appetites and enable users to deploy and link together their own customised lending vaults in a permissionless manner. With this flexibility, any type of pre-existing or future lending protocol — monolithic or otherwise — can be recreated inside the Euler ecosystem and connected to vaults from elsewhere in the ecosystem.

The EVC and Euler v2 will be a 0 to 1 moment for the composability and connectedness of collateral in DeFi. Together, they will provide vault creators with unparalleled opportunities for innovation and growth via network effects as more and more diverse vault types are deployed within the Euler ecosystem.

Join the Community

Disclaimer:

The opinions reflected herein are subject to change without being updated.

Paternalism Versus The Invisible Hand in the Risk Management of Lending Protocols was originally published in Euler Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introducing the Ethereum Vault Connector — A New Era Of Lending Is Here

Kasper — Thu, 11 Jan 2024 13:09:18 GMT

Introducing the Ethereum Vault Connector — A New Era Of Lending Is Here

Today marks a significant day for Euler Labs, as we proudly introduce The Ethereum Vault Connector, a public good, open-source primitive designed to make lending markets more customizable and interconnected, powering the creation of novel use cases.

DeFi is Dead, Long live DeFi.

DeFi emerged as the answer to traditional finance’s shortcomings — limited choices, opacity, and centralised control. For DeFi to continue fulfilling its mission, we believe our efforts must extend beyond just decentralising applications and focus on making it easier to develop new foundational structures. In lending, this means empowering users and developers to go beyond the current status quo and giving them an expanded range of customizable options to create tailored lending markets, products, and protocols.

In this spirit, we’re excited to introduce the Ethereum Vault Connector (EVC) — A powerful primitive that enables everyone and anyone to build protocols and new products on top.

The EVC code has been subjected to three smart contract audits by yAudit, Certora, and Hunter Security, along with a comprehensive review by Spearbit. A fourth audit by Trail of Bits is currently underway. While additional audits are already scheduled we welcome feedback and discussions from the community on the current version of the code and the white paper. We also encourage everyone to participate in our ongoing bug bounty.

The Ethereum Vault Connector — What does it do exactly?

The Ethereum Vault Connector (EVC) is a primitive that facilitates seamless communication between ERC-4626 vaults, amongst other smart contracts.

It supercharges vaults with new capabilities and allows them to accept collateral from one another, in a network composed of other vaults.

The EVC empowers developers with absolute freedom to innovate by streamlining the intricacies within core lending and borrowing contracts, lowering barriers to development.

With the EVC, developers no longer need to start from scratch, when building lending protocols, stablecoins, yield aggregators, margin trading apps, or others.

Immutable, unopinionated, without governance , zero fees — it’s intended to be a true public good for DeFi.

Why build on the Ethereum Vault Connector?

1. Unified interoperability layer

The EVC creates network effects by facilitating access to unified liquidity. It enables protocols to accept deposits as collateral from the vaults in the network, offering convenience to users by eliminating the need to transfer their collateral assets from one protocol to another.

Additionally, this integration allows vault creators to easily bootstrap lending products by connecting with vaults and utilising existing deposits within the network as recognized collateral.

2. Underlying asset flexibility

Its unopinionated nature extends to assets, giving you full control over the assets that can be used as collateral or liabilities.

This flexibility enables the creation of lending products backed not only by ERC-20 tokens but also by irregular asset classes, such as RWAs, NFTs, uncollateralized IOUs, or synthetics.

3. Advanced position management

The EVC also excels in the domain of automations as it allows external contracts to act on behalf of your account so the following can be implemented:

Position Managers
Conditional Orders
Intent-Based Systems
Custom Liquidation Flows

4. Generalised flash loans and batching

Should you choose, you can rely on the EVC to perform multiple operations affecting multiple vaults and external smart contracts within a single batch transaction.

This is more convenient for UI users, more gas efficient, and allows deferring account liquidity and vault constraints checks until the end of the batch.

Infinite use cases, infinite possibilities

EVC allows for a modular design that opens up up a limitless number of use cases for products that can be built on top of it. These include, but are not limited to:

Traditional, overcollateralized lending products
Uncollateralized lending products
Real World Assets (RWA) lending products
NFT lending products
P2P lending products
Oracle-free lending products
Automations (conditional orders, custom liquidation flows, strategies, position managers, optimizers, guardians etc.)
Smart contract tooling (i.e. swap hubs using new dexes or new dex aggregators)

Build (and audit) with us

When finally deployed, the EVC will be one of the most audited contracts in DeFi. To date, it has undergone three smart contract audits (yAudit, Certora, Hunter Security) and a detailed code review conducted by Spearbit. A fourth audit with Trail of Bits is currently in progress. We are also proud to have Alberto Cuesta Cañada, co-founder of Yield and ERC-4626 pioneer, contributing to the EVC and assisting us with architecture and security design.

We intend to continue to review the EVC, as further audits are already scheduled. As for bug bounties, the first one is now open. Similar to our audit approach, you can expect more bounties to be announced soon.

Additionally, it’s important to add that the EVC is set to be included in the Euler v2’s final audit, contributing to the overall scope of the security review.

To learn more about the EVC and its full possibilities we invite you to join our workshops with Encode starting today at 11 AM ET, which will also have a bug bounty up to $100,000.

The Future of Euler v2

The EVC will be the foundation on which Euler v2 will be built, enabling it to be more flexible, with a stronger emphasis on risk management and yield optimization.

Euler v2’s modular design will allow users incredible freedom to create and deploy lending markets tailored to their needs. This includes choosing oracles, setting governance structures, selecting asset types, and defining risk management mechanisms and interest rate models.

Whether users are activating a market with our ready-to-use components, customising them to fit their unique requirements, or building completely new ones from scratch, the possibilities to create truly personalised markets will be endless.

Euler v2 holds many more surprises, but this is all we’re revealing for now. Worry not, though; the full details of Euler v2 will soon be announced.

Disclaimer:

The opinions reflected herein are subject to change without being updated.

Introducing the Ethereum Vault Connector — A New Era Of Lending Is Here was originally published in Euler Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hello 2024, Hello new Euler look.

Matija Skrnjug — Wed, 10 Jan 2024 15:25:10 GMT

“Tell me I can’t, and then watch me work twice as hard to prove you wrong” — Unknown

This feeling has resonated with us and with everyone in crypto for the last two years. As we now see, crypto is coming back, stronger than before. And it’s not just crypto making a comeback — Euler is too.

Behind the scenes, we’ve been diligently building. Over the past several months, our team has been in stealth mode, developing not one, but three projects. The moment to draw back the curtain and unveil to the world what we have built is now closer than ever.

The same Euler, but wiser and stronger

As we enter 2024, we are excited to present a refined Euler brand that reflects our vision for this year: The same Euler, but wiser and stronger. While the Euler brand still represents our foundational values and who we are, it now reflects our resilient journey and emboldened future.

The newly refined Euler brand includes a new logo, typography, color palette, and website.

Logo Design

The Euler logo has three parallel lines that come together in the shape of an “E”, symbolizing forward movement. The clean, PP Mori font used alongside the logo reflects our brand’s focus on clarity and precision.

Symbol

Our symbol is a timeless representation of Euler, it serves as an alternative to the full logo when space is limited, ensuring our brand’s identity remains distinct and recognizable at any size.

Color Palette

Our brand palette showcases a vibrant, bright green — embodying vitality and growth — complemented by sophisticated shades of navy. These colours, mirroring our product, evoke innovation and trust, fostering a harmonious brand-product relationship.

Typography

Our brand’s choice of serif headlines embodies experience and wisdom, exuding elegance and authority. Paired with sans-serif body text, this deliberate contrast signifies modernity and readability, ensuring impactful, harmonious communication.

Looking Forward

The rebrand unveiled today is just the beginning. We are committed to making 2024, Euler Labs’ most impactful year. The transformative projects that we will launch this year will bring us closer to our mission of creating a truly modular DeFi ecosystem. Stay tuned and keep an eye on our socials, who knows we might announce one of those projects, sooner than you think!

Disclaimer:

The opinions reflected herein are subject to change without being updated.

Hello 2024, Hello new Euler look. was originally published in Euler Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

War & Peace

Michael Bentley — Wed, 10 Jan 2024 13:33:26 GMT

War & Peace: Behind the Scenes of Euler’s $240M Exploit Recovery

TL;DR:

In March 2023, the Euler Protocol was exploited for ~$197 million in assets.
Over the course of the following three weeks, the Euler Labs team engaged in a highly strategic recovery effort that led to the return of ~$240 million in the stolen assets.
During the operation, multiple twists and turns threatened to throw things off course. These included the intervention of the notorious Lazarus Group, multiple red herrings, an unpredictable attacker and frontrunning bots.
The team that pulled off this exceptional feat is the same Euler Labs team that has been hard at work over the past year building a stronger, more powerful and modular V2 of the Euler platform which we’re excited to help rollout over the coming weeks and months.

Introduction

As we enter 2024, I want to give you an update on what the Euler Labs team has been up to. While 2023 was not the year we had hoped for, we are more bullish than ever that 2024 will be a huge year for DeFi, and Euler especially. The brand refresh being unveiled today is just the beginning of a series of announcements we have planned over the coming months. The Euler team has been working on some transformative projects that will bring us ever closer to our mission of creating a truly modular DeFi trading ecosystem. But before we talk about the future let’s reflect on the series of events that took place this past year.

2023 — A look back

There’s no other way to put this: 2023 was a bit of a shit year.

I can’t think of a better way to say goodbye to it than by telling the story of how it played out. The tale starts very positively, for me at least, with the birth of my son, but then very quickly takes a dark turn with a devastating $200M attack on the Euler protocol and its users, before picking back up with an unprecedented $240M recovery story. So strap in for a tale of babies, bank runs, exploits, MEV bots, sleuthing, trolling, counter hacks, North Korea, negotiations, heart attacks, Laurence Day, recoveries, and much more.

It is my hope that by telling the story of the Euler exploit we can set the record straight and move onto the next chapter of the Euler story. As with many big news cycles, there was a lot of misinformation published and an endless amount of innuendo and rumours on social media. We enforced a communication blackout as part of our recovery strategy, which turned out to be essential for the recovery itself, but allowed external observers to have a field day speculating on how the recovery came about. Right now whenever we talk about Euler, people always want to look backwards — I want to put an end to that this year and get people excited about Euler’s future.

The most important part of this story is that Euler users’ assets were fully recovered and that the Euler community ultimately came out on top. The how and why will be interesting to some, perhaps insightful to others. Considering the complexities of crypto, the fact that we were dealing with what turned out to be an irrational bad actor(s), and interference from one of the world’s most ruthless hacking teams, this is a remarkable outcome.

I also want to make sure the people on our team and the broader investigation get credit for the monumental effort they put into the recovery. On one of our company all-hands calls shortly after the attack I told everyone I would help them get new jobs, to which one person replied: “I’m not going anywhere until we find him”. That attitude really sums up how our entire team responded to this incident. Our team really gave everything to the recovery effort, and we simply wouldn’t have seen the outcome we achieved without this extraordinary commitment. We’re a stronger, more resilient team today than we were in March 2023. Not only have we retained almost everyone that was with us a year ago, we’ve added top talent that have been drawn to Euler Labs because they believe in our potential to build great software.

Finally, it’s also just an objectively interesting story that I think deserves to be told. I can’t tell the story in its entirety for a variety of reasons. Not least because I have no intention of revealing the techniques we used to bring about the recovery. But the twists and turns and human elements are far more interesting than the technical details anyway.

I also want to emphasise that a lot of people were impacted by the events of 2023 and their stories are just as important as mine, but ultimately not mine to tell. This is my story as I experienced it. I’m going to tell it from a first-person perspective in chronological order because I can’t think of a better way to do it. I’ve generally avoided mentioning anyone else by name to protect people’s identities, but those people who helped in this case know who they are and have my eternal gratitude.

Ok, let’s dig in.

Euler in the beginning

I am one of three co-founders of a project called Euler, which started in the midst of the pandemic back in 2020. Back then, my co-founders and I started Euler with the goal to develop a decentralised finance (DeFi) protocol that would allow people to lend/borrow a wide variety of crypto assets directly from/to one another in an open, non-custodial manner.

Before starting the project I worked as an interdisciplinary research scientist at the University of Oxford, where I used to build game-theoretic mathematical and computational models of populations of organisms cooperating or competing with one another. I worked on topics as wide ranging as the evolution of cancer, antibiotic resistant evolution in bacteria, and the spread of invasive Hornet species. This might sound like a world away from DeFi, but the underlying game-theoretic principles are really not that dissimilar. That’s ultimately what attracted me to the DeFi movement that was starting to really take off in late 2019. I realised that many of the tried and tested mathematical techniques I was using in my day job could be applied to a brand new discipline.

My co-founders and I initially tried to differentiate Euler from market leaders like Compound and Aave with novel mechanisms around how interest rates could be set in a decentralised manner. We also designed the protocol to compose tightly with decentralised exchanges like Uniswap in order to offer users access to more lending markets than ever before. Over time, however, we ended up introducing a range of new innovations to the DeFi lending space, many of which focused on new ways of quantifying and managing risk. This helped the project to find admirers, especially among the DeFi developer community.

As the team and project grew, we also gained a reputation for taking security seriously. We weren’t building fast and breaking things as some others were boasting about doing at the time. We didn’t want the project to be a flash in the pan. We developed the protocol slowly and methodically, even as the wider crypto market was taking off and many other projects rushed to launch.

To reinforce our security-first approach, we had the protocol audited at great expense many more times than is the industry standard. Our audits invariably came back with excellent feedback from auditors. We had the UI audited. We put in place a $1M bug bounty for anyone who found critical bugs in Euler, higher than anything offered by other leaders in the space. We secured $10M insurance cover for the protocol by forging a partnership with a DeFi insurance firm (extremely rare in DeFi). Overall, we tried to go above and beyond the industry norms when it came to security wherever possible.

Following deployment in Dec 2021, the protocol continued to grow in popularity. Even as the broader market was cooling due to the collapse of Luna, Celsius, FTX and others, Euler continued to find new users and grow its market share. And its enhanced liquidation module had been proven a success, lowering the cost of liquidations for borrowers more than 10x in some cases. Owing to its gas-efficient design and risk management features, it was also particularly popular with other decentralised organisations (DAOs) seeking to deposit treasury assets. Some of its biggest users resulted from integrations with other protocols like Angle, Balancer, and Idle.

Going into 2023 things looked very positive for Euler. It had been live on chain for over a year and had survived unprecedented and hostile market conditions for much of its existence, growing where others were wavering, pushing on where others were slowing down. That is until March 13th 2023, when things changed dramatically.

March 9th

Babies and bank runs

In the first few months of 2023 there was a sense of cautious optimism in the crypto industry. Most of the bad actors had been washed out the year prior. Green shoots of recovery were seemingly on the horizon. At Euler, we had been hard at work developing the core of what will soon become the next iteration of the Euler platform, what we’re currently calling “Euler V2.”

I was also waiting for the birth of my second child, who was due in early March. On Thursday March 9th, close to midnight, he arrived. My wife and I spent the next day with him and I told my team to expect me to be off on paternity leave for a couple of weeks. Everything was seemingly perfect, but as we all know, this didn’t last.

Outside of crypto, the wider global economy had been faltering and inflation was soaring. Interest rates had risen faster than at any time in history and many banks in the US were caught short. A lot of those struggling were holding US treasuries that were rapidly decreasing in value and lenders started to worry they might not be able to satisfy short-term requests for withdrawals. Silicon Valley Bank was the first domino to fall.

One might think that a bank run on a fairly small bank in the US would have had a limited impact on crypto markets, but that wasn’t the case. One of the largest tokens by market capitalisation is a USD denominated stablecoin called USDC, which gives users digital dollars to use on chain. USDC should ordinarily be redeemable for exactly $1, because it is matched 1:1 with USD bank deposits, but hours after my son was born the bank run started to threaten its backing and it began to de-peg from $1 and trade lower as redemptions stalled.

The problem for Euler was twofold: First, the Euler DAO had a treasury holding more than $25m in USDC. Second, the Euler protocol had many millions in USDC used as collateral for loans. If USDC went to zero, it could send the treasury to zero and cause problems for protocol users along with it.

With my son less than 24 hours old, I had to get to work. I spent the whole weekend making calls to other teams, banks, investors, and others, trying to map out what, if anything, could be done. It felt like Euler could be in real trouble.

Fortunately, as quickly as the drama started, it went away. By late Sunday night, U.S. banking regulators had taken extraordinary measures to shore up deposits in the impacted banks. Likewise, over the next days and weeks international regulators took steps to provide additional liquidity and oversight to their own shaky depository institutions. All that worry for nothing it seemed. I went to bed exhausted, but looking forward to spending more time with my son on Monday.

March 13th

Euler exploited

The next morning, on March 13th, during our usual breakfast routine, I could hear something ringing upstairs in my office. I rushed up to see what all the noise was about, expecting it to be something related to the bank run, but it was much much worse. Messages were coming from all directions telling me something unusual was happening with the Euler protocol. An attack was ongoing and tens of millions of dollars appeared to have already been drained. I jumped on a call with other members of our team as we tried to figure out what was actually happening.

Of all the times for this to happen as well, it seemed so cruel. The ups and downs of the previous days gushed over me and I knew that things would likely get worse before they got better. Unfortunately there was no time to grieve for the situation though. As the adrenalin kicked in I got to work with my team. There was so much to do.

WTF is the donateToReserves function?

One of the first tasks was figuring out exactly how the attack was carried out. Fortunately this was done exceptionally quickly because the entirety of the Ethereum security community had been digging through the rubble of the exploit within minutes of it being detected. The cause was identified as a single missing line of code in an obscure code path involving the now famous ‘donateToReserves’ function inside of Euler.

To understand how the attack worked, it’s important to understand the purpose of the donateToReserves function in the first place. People looking into the details of this function noticed that it was not present in the initial Euler code base when it was deployed back in 2021.

So where did it come from? Why would you ever want to donate your own assets to the protocol’s reserves? Unfortunately some people rushed to speculate about this on social media. Rumours were soon spread that donateToReserves was unaudited code introduced for vanity purposes. This was simply not true at all. The code was introduced for an important reason, and most definitely audited. Some people were even saying that the Euler developers had introduced the function deliberately as a back door to drain the protocol themselves — insinuating that we were about to go on the run! There was a lot of noise to filter out in those first few hours.

The truth is that donateToReserves was introduced in order to fix a much smaller ‘first depositor’ bug in the protocol that had been missed by all previous auditors of Euler. That bug was eventually reported by a white hat hacker (ethical hacker) as part of the Euler bug bounty program via Immunefi almost a year prior to the 2023 attack. The white hat had identified in their submission that the original version of the Euler protocol was vulnerable to allowing first depositors in new, uninitialised pools to have their deposits stolen by a front-runner. The reason they highlighted was that an internal exchange rate in the protocol was not properly initialised when there were no pre-existing depositors, allowing would-be front-runners to set it arbitrarily.

At the time the bug bounty submission came in for the first depositor bug it had not been spotted by prior auditors or been exploited. Had it been exploited, it would have likely led to minimal losses and would very likely have been covered by the $10M insurance programme Euler had in place. Nevertheless, bugs that permit theft of assets, no matter how small, need to be mitigated. So a plan was put in place to make sure it was patched. That patch created the vector for the March 2023 exploit.

The proposed solution was developed and sent to the auditors for review. That solution would work for all new markets activated after the patch, but for a subset of pre-existing it would not. So a new function — donateToReserves — was included to make sure that the internal exchange rate of those pre-existing markets could be addressed too.

The solution was sent to audit by the most recent auditor of Euler. That auditor was also the protocol’s insurer, with a $10M insurance payout potentially on the line if anything went wrong. So they were strongly incentivised to get it right. On completion of the audit, an upgrade to the smart contracts was proposed to the Euler DAO community, and by July 2022 the first depositor bug was patched.

Unfortunately there was a critical issue with the introduction of donateToReserves that no one spotted at the time (or for a prolonged period after). Specifically, the function lacked a health check to make sure that an existing user wouldn’t be able to put themselves in an unhealthy state just by donating to the reserves. Clearly this is something that an ordinary user should never want to do, so in retrospect it’s not hard to see how it was missed at the time.

Ultimately, the attacker exploited the missing health check in this function to instead donate a large amount of their own collateral to the reserves in order to make themselves eligible for liquidation. They then self-liquidated themselves to claim a liquidation bonus that was greater than the losses caused by the donation.

First clues left by the attacker

When something like this happens in the crypto industry, the worst case scenario is that it is perpetrated by organised crime or a state-sponsored actor, because then there’s very little hope of recovery, regardless of how good the investigation into what happened is. But if the attack is carried out by an individual or is opportunistic in nature, there is a much better chance of finding out who did it and getting some or all of the stolen assets back.

Fortunately for Euler users, there were early signs that this attack was not carried out by an experienced black hat hacker. There was a sliver of hope. There is a small sub-group of security specialists, sleuths, and white hat hackers within the Ethereum community that will often offer to help projects in such circumstances, and they were the first to notice something rather strange about the way the exploit had been carried out. We were hugely appreciative for their support in the immediate aftermath of the exploit and for their continued and dedicated support in the weeks that followed.

Most exploits begin with the withdrawal of funds to a privacy-preserving protocol like Tornado Cash (a protocol which provides users with a greater degree of anonymity than they would normally expect to have from transacting on Ethereum, which is ultimately a public ledger). The withdrawn funds, whose origins are then hard to trace, are then used to pay for deployment of exploit code and exploit transactions. Most exploits will take place in a single clean transaction very soon after the code encoding the exploit logic has been deployed on chain. However, in the case of the Euler exploit, the first few transactions were anything but clean.

The exploit in real-time

Note that hereafter sections involving communications and transfers are often told in present tense rather than past for clarity and ease of understanding.

At 8:50:23 am UTC the attacker uses an address labelled as Euler Finance Exploiter 1 to deploy the exploit smart contract (labelled as Euler Exploit Contract 2) to Ethereum. This contains the logic for the theft of assets from Euler, but does not actually steal anything itself unless the attacker carries out further transactions using it. Note that the labelling of the exploit smart contract by Etherscan is a little confusing here (why 2 and not 1?). This will become clear in a moment.
We would normally expect an attacker to start using their exploit smart contract very soon after they have deployed it to carry out the attack itself, but that is not what happens here. Instead, at 8:50:35 am UTC, just seconds after the first exploit smart contract is deployed, a second copy of the exact same smart contract is deployed (labelled as Euler Exploit Contract 1) by another address labelled as Euler Finance Exploiter 3.
Shortly after, at 8:50:59 am UTC, Euler Finance Exploiter 3 carries out the first exploit: a single transaction which steals around ~$8.8M worth of assets from the protocol. Then they halt their attack and go strangely quiet.
At 8:56:35 am UTC Euler Finance Exploiter 1 picks up where Euler Finance Exploiter 3 left off, triggering several more exploit transactions over a period of around 15 minutes. They then convert most of their stolen assets to ETH and DAI and withdraw the majority to a second attacker-controlled address, labelled as Euler Finance Exploiter 2.
At 10:33 am UTC, Euler Finance Exploiter 2 then sends 100 ETH to yet another attacker-controlled address, labelled as Euler Finance Exploiter 4.
At 10:38 am UTC Euler Finance Exploiter 1 then sends 100 ETH onwards to Tornado Cash. This was considered to be bad news, because anyone sending funds to a privacy-preserving protocol probably does not have plans to return them.

Immediately after the attack, the balances controlled by the four main addresses involved are:

Main attacker address Euler Finance Exploiter 1 which holds ~$13.6M worth of ETH (~8,080 ETH)
Main attacker address Euler Finance Exploiter 2 which holds ~$34M worth of DAI and ~$149.2M worth of ETH (~88,752 ETH)
Main attacker address Euler Finance Exploiter 4 which holds only 1 ETH after the attack, but which becomes important in the story later on
Secondary attacker address Euler Finance Exploiter 3 which holds ~$8.8M worth of DAI

The main attacker has also transferred ~$168k worth of ETH (100 ETH) from Euler Finance Exploiter 4 to Tornado Cash.

In total, stolen assets are worth ~$197M at the time of the attack, with several thousand affected users. Among the largest impacted users are other DAOs. Angle protocol, in particular, is severely hit, with ~$17M in losses. Those losses lead its over-collateralized stable coin to de-peg, threatening the viability of the entire protocol. Also impacted are protocols like Balancer, Idle, Yield, and many more. The number of other protocols with deposits in Euler really highlights its composability and just how much it was trusted by the wider developer community at the time.

Over the next few weeks, with the price of ETH rising during the negotiation period, the attacker eventually returns ~$240M worth of assets to Euler users. Some affected users even joke on social media that the attacker is a better trader than they are. It’s hard to overstate just how special an outcome this is for the Euler community. For users to come out ahead after an attack like this would be an outcome very few would expect — but they did.

Before we get there though, there’s a lot to cover in between.

Attacker gets front-run by a bot

Overall, the attack is a bit of a mess. What on earth happened here? Euler Finance Exploiter 1 is the first to deploy the exploit smart contract, but second to steal the funds. Euler Finance Exploiter 3 is second to deploy the exploit smart contract, but the first to steal funds. Digging into the details, we see that Euler Finance Exploiter 1 and Euler Finance Exploiter 3 both paid for the exploit to be carried out using different sources of funds.

Euler Finance Exploiter 1 is funded by Tornado Cash. They are clearly the developer of the exploit smart contract, but it is difficult to trace the source of their funds back any further than that because of the privacy-preserving nature of Tornado Cash.

Euler Finance Exploiter 3 is funded by a mixture of two other protocols, called FixedFloat and Multichain, respectively. Interestingly, the source of their funds can be traced back a little further, all the way back to an exploit of another protocol just 30 days prior.

So, is there a single exploiter or two? If there is a single exploiter, did they make a mistake and accidentally deploy their exploit smart contract twice, leaving clues as to their identity?

At 11:38:11 am UTC the Euler Finance Exploiter 3 withdraws ~$8.8m worth of DAI back to Euler Finance Exploiter 1. This might make it seem like Euler Finance Exploiter 1 and Euler Finance Exploiter 3 are one and the same, which could be good news for identifying the attacker because of the traceability of the source of funds for Euler Finance Exploiter 3.

Unfortunately, however, at 12:08:35 pm UTC a message from Euler Finance Exploiter 3 is sent on chain which gives further details about what happened. They claim that the two attackers are unrelated parties after all. It reads:

“Hi, I am the owner of a MEV bot that has accidentally frontrun the first tx of the hacker. It tried to frontrun the second tx but failed and only the contract has been created. I tried to return the money but the contract can only send to an address already in bytecode. Unfotunately it sent the money to the hacker’s address. I tried my best and I am very sorry for anyone who lost money.”

The claim here is that Euler Finance Exploiter 3 did not mean to steal the first batch of funds. How is this possible? If not, why did they send funds to the hacker and not back to the protocol?

For those that aren’t aware, transactions on Ethereum are often made semi-public before they are processed by network validators for inclusion on the blockchain. This gives rise to opportunities for other users to extract value from them (so called ‘maximum/miner extractable value’ or MEV for short). Most forms of MEV involve some kind of front-running. A classic example is a sandwich attack, in which an MEV searcher places an order right before and after a trade on a decentralised exchange protocol takes place. This allows them to front and back-run the transaction to make profit. A related type of profiteering happens in traditional finance via high-frequency trading (HFT).

Front-running is not just possible for exchange transactions though. Sophisticated MEV searchers will take any transaction they get advance notice of, rapidly simulate it, and if it makes a profit, try to copy it for themselves. In the case of the Euler exploit, the claim by Euler Finance Exploiter 3 is that their bot saw a very profitable transaction ahead of time and simply copied it, front-running the attack from Euler Finance Exploiter 1 on Euler.

Why then did Euler Finance Exploiter 3 send the stolen assets back to Euler Finance Exploiter 2 if they had no intention to steal anything? Decompilation of the exploit smart contract from bytecode back to human-readable code showed that the code they copied had a withdraw function that pointed back to Euler Finance Exploiter 2. So it was simply not possible for Euler Finance Exploiter 3 to do anything but send the assets back to the main exploiter, even if a smart contract deployed by their own address held them for a short period of time.

Did the MEV front-runner help? In many ways, clearly not. Their intervention led to a lot of wasted time by investigators trying to figure out the relationship between all the addresses involved that ultimately could have been better spent focussing on the main exploiter.

However, the front-running did help reveal some information about the main exploiter. In particular, it confirmed to us that this was unlikely to be a sophisticated state-sponsored actor or organised crime group. Why? Because, if it had been, the attack would have been orchestrated in a much cleaner way. There are techniques more sophisticated users can use to prevent front-running.

Information gathering

In the aftermath of an attack there is an enormous amount of information that needs to be gathered. Our team quickly organised around this task and started the painstaking process of identifying potential leads. To use a cliché, no stone was left unturned.

The amount of data we had to work with would take weeks to sift through and it was a largely manual task. The search would throw up countless leads and red herrings. The world of crypto is still small, and everyone is connected to everyone else if you look hard enough. And believe me, our team couldn’t have looked harder.

In conjunction with the decompiled smart contracts, analysis of the content and style of the exploit code logic, and some other tricks we had up our sleeves from information gathering, we were able to formulate a list of leads early on and figure out what kind of attacker we were likely working with.

Communications begin

As is normal after any exploit, we tried to reach out to the attacker the very same day with an on chain message. Our opening gambit to the attacker was simple and neutral in tone:

“We understand that you are responsible for this morning’s attack on the Euler platform. We are writing to see whether you would be open to speaking with us about any potential next steps.”

Ultimately there is a low probability that an attacker will respond to any message, regardless of what it says. However, it is obviously important to try to say something as a way to break the ice. Until sufficient evidence has been gathered to the contrary, it is always important to leave the door open to the (very) remote possibility that an attacker might be able to provide an explanation for what has happened. They might be open to exiting the situation as a white hat.

We were not particularly hopeful though. There were already several strong indications that this attacker was not likely to be open to negotiating in good faith. First, they had already converted most of the stolen assets into ETH and DAI. This incurred huge losses in slippage costs and is not something someone would normally do if they intended to exit as a white hat. Second, the attacker had already transferred 100 ETH to Tornado Cash within an hour or so of the attack. Given that they were already laundering stolen assets through a US-sanctioned protocol, it seemed unlikely that they would be open to negotiating in good-faith.

Normally the people handling exploit communications will start with a more deferential tone than the one we took. Something that tries to paint the attacker in a positive light. It might even say ‘congratulations’ to them for finding the vulnerability or highlight their skill. I don’t know why that approach has become the industry norm, but, for good reasons, including those stated above, it was not the approach taken here.

Ultimately, there is very little evidence that taking a deferential tone in negotiations actually works. If an attacker is not already minded to return assets voluntarily, they are unlikely to be persuaded of this by an overly-friendly message. In general, pretending to be nice to someone who has clearly harmed you is more likely to make you look weak and perhaps even comes across as patronising/infantilising.

In the meantime, we also enforced a company-wide blackout on social media communications except for a handful of carefully crafted and very deliberate Twitter updates. This went against industry trends, where the people dealing with the fallout of an exploit will often spend time on social media answering questions and trying to bring clarity to the situation. This is a completely reasonable and appropriate approach in many circumstances.

However, it wasn’t appropriate in our case because we had already gathered enough information about the attacker to know that there was a good chance that they might be anxious about what they had done. Anxiety is generally a feeling that you reduce by getting information that helps you collapse the vast state of uncertainty in front of you to something more contained and manageable. Even something as simple as a reply to an unrelated social media message can help collapse uncertainty and leak information. Limiting communication helped maintain maximum uncertainty in the attacker’s mind by starving them of information.

Our communications approach was quite different to how people have tended to handle exploit communications in the past and was initially criticised by a lot of people. Since we could not discuss our communications strategy at the time, it is my hope that this post goes some way to making it clear that the communications choices we made throughout the whole process were extremely strategic, almost obsessively so.

Laurence Day shows up

Late into the evening my wife and I had a visit.

Laurence Day is a crypto Twitter micro celebrity who gets accused of being behind every and all hacks in crypto as part of a longstanding joke. The reason for this is because he was a core contributor to another protocol that was exploited, and when the international media first wrote the story up about it they accidentally put a picture of his face next to the sub-heading ‘exploiter’, which then inevitably got copied far and wide. And so he became ‘the exploiter’ of his own protocol and pretty much everything else after that.

I didn’t know him all too well before this event, but as luck would have it he lives quite close to me and he had a newborn himself arrive just a few weeks before my son. I was on a lengthy call discussing the case when Laurence arrived and so I left him downstairs talking to my wife. After everything that had happened that day I can only imagine what she made of his appearance.

Laurence wasn’t the only person to show such kindness in the aftermath. Ethereum is an amazing community and we really felt that in the days and weeks that followed. Many people reached out to offer their support. Another person I’d never met, but who’d been through something similar, reached out just to talk about how I was feeling and see if they could offer some emotional support. That really helped me process things mentally.

March 14th

Reward for information launched

Unsurprisingly, no response was received to our first message. So the next day, on March 14th, we decided to take a more direct approach. We made a recommendation to the Euler Foundation to urgently launch a reward for information that might lead to an arrest and a return of all funds. Shortly after a new message was sent on chain:

“Following up on our message from yesterday. If 90% of the funds are not returned within 24 hours, tomorrow we will launch a $1M reward for information that leads to your arrest and the return of all funds.”

Beyond actually getting information from a whistleblower or motivated sleuth, the goal here was to increase pressure on the attacker, to maintain their state of anxiety, and perhaps provoke a response from their side in order to gain further information about who they were.

Whilst a reward for information is not common in the industry, and some people felt that it was too soon and/or too aggressive from our side to be taking an approach like this, it was a strategic move that we felt we needed to make early to keep the attacker on edge.

Later we will learn from private communications with the attacker that this approach had worked better than we had ever imagined possible, but given what happened next, we wouldn’t know it at the time.

March 15th

A final message

Before the 24-hour deadline was up, we followed up with what we expected would be a final message that gave the attacker a final chance to work towards a more positive resolution:

“The simplest way to move forward today is to return 90% of both the DAI and ETH under your control to the EulerDAO treasury address: 0xcAD001c30E96765aC90307669d578219D4fb1DCe. Then investigations can be halted, and the focus here can turn to distribution of that back to protocol users, without needing to go the legal route.”

Here we wanted to give just enough of a hint that we had already gathered information to go the legal route whilst still offering an olive branch to achieve a swift resolution.

Community messages

We were not the only people sending messages to the attacker in the early days after the attack. Many other affected Euler users had been sending their own messages on chain pleading with the attacker to reverse their actions. One message that plays an important role in the story read:

“Please consider returning 90%/80%. I’m just a user that only had 78 wstETH as my life savings deposited into Euler, I’m not whale or millionaire. You can’t imagine the mess I’m into right now, completely destroyed. I’m pretty sure 20M is already life changing for you and you’ll bring back joy to a lot of affected people.”

Among the sad messages like this one there were also people messaging the attacker offering them tips on how to launder the proceeds of the attack (pretty common after exploits), as well as plenty of people just trying to bring a bit of humour to the situation (also pretty common).

Whilst probably quite hard to understand for people outside the industry, the gallows humour that follows these kinds of bad events in crypto is a key part of what binds the broader community together. It’s a nascent industry, bad things happen and so people build resilience through humour. It’s one of my favourite aspects of the culture. Anyway, one message that was sent to the attacker that got a few laughs on social media purported to be from the aforementioned Laurence Day:

March 16th

More Tornado Cash transfers

At 1:19 am UTC on March 16th we finally seem to get a response to our messages, and it is not good. The attacker transfers another 2500 ETH to Euler Finance Exploiter 4, and then transfers another 1000 ETH of this into Tornado cash. This doesn’t look like behaviour consistent with someone planning to imminently negotiate a return of funds. Although it isn’t all the ETH, which gives us some hope.

There’s something odd about the way the attacker is working here though — the transfers are all in batches of 100 ETH. There is no reason to send ETH around in batches like this. Sleuths assisting the investigation and a number of people on social media picked up on this unusual behaviour too and started to consider its implications.

We saw it as more evidence that the attacker was either inexperienced at using Tornado Cash, or that they were on edge and acting irrationally. Either way, we felt like it might be good news for our chances of identifying and catching them.

A victim is refunded — who are they, and why?

Moments after the Tornado Cash transfers something even more remarkable happens. Out of nowhere the attacker sends 100 ETH to the victim who sent the message quoted above asking for the attacker to take a deal. The refund is even more than the victim lost in the attack (closer to 88 ETH). Why did the attacker do this? Social media lights up. Theories start flying.

Perhaps the attacker just craves attention or enjoys the power dynamic. They know the impact of this refund will be to provoke many more people to start sending messages asking for refunds too. Why would they want to encourage that kind of behaviour?

Or perhaps the attacker felt genuine sympathy for the victim after their message. If so, why send assets back to only one person? Many other people sent similar heartfelt messages of loss. What is special about this user?

Or perhaps the refund victim knows the attacker. Even wilder, maybe they are the attacker.

Or perhaps the attacker is just trolling, responding negatively to the messages we have sent them with a ‘fuck you’ signal. This is how many users interpret the situation. Frustration boils over in some cases as people start to express their concern about our approach to communication. One user gave particularly stinging feedback in a message on chain:

“This dude is clowning on you all, are you not embarrassed? Making threats against him with no leverage. Escalating then practically begging for him to return our money when you had NOTHING to offer. Now we re almost certain it isn t NK and what, still nothing? Why would your first move be to threaten him legally, as if he wasn t aware the FBI was looking into it already? Your only shot was to appeal to his evidently existing good nature and offer up the 10%, but instead you re out here sheepishly asking about next steps ? What a joke you ve made yourselves out to be.”

Messages like these were difficult for our team to process, but also understandable considering the situation Euler’s users were in. People had been hard hit by the attacker’s actions and wanted to know that everything possible was being done to get a positive outcome. Whilst our team was working extremely hard towards that goal in the background, and nothing was done without a reason, ultimately we had nothing to show for our hard work at this point. And we didn’t want to jeopardise our strategy by commenting publicly. So the only thing we could do was keep pushing with the investigation. It seemed clear that we were going to need to make further progress towards getting some leverage on the attacker.

Refund victim red herring

We began to investigate the refund victim’s address, and what we found made our eyes light up. Unlike the attacker’s address, which was harder to connect to any real-world identities directly, the address belonging to the refund victim participated in many transactions that meant it was not too difficult to track it back to a likely owner.

That person turned out to be a developer who had contributed to building another lending protocol. Which seemed very coincidental indeed.

At this point we have identified that the refund victim has means to commit the attack. We also find evidence of their familiarity with the Euler code base and there are other significant links too. No smoking guns, but a lot of coincidences. Little did we know that we weren’t the only ones looking into the identity of the refund victim though.

Later that day a crypto media outlet tracked the refund victim down independently and put it to them that they might be the hacker (see the article here). To which the refund victim replied: “Seriously? No, I’m not the hacker.” Shortly after the user returned the excess 12 ETH sent by the attacker to the EulerDAO treasury with the message:

“I was affected by the recent Euler Finance hack and fortunately, I received back 100 ETH from the hacker, which is 12 ETH more than my original deposit of ~78 wstETH. With this transaction, I’m returning the extra 12 ETH that doesn’t belong to me to the Euler Finance Deployer.”

Could this all be a coincidence? Ultimately it turned out to be just that. For a short period of time though it appeared that it could be a major breakthrough. But it was just one of several red herrings that turned up during the investigation.

March 17th

Attacker sends funds to North Korea

At 3:48 am UTC on March 17th the attacker makes a new transaction. This time it is not to a victim, himself or to Tornado Cash, but, astonishingly, to an address labelled as Ronin Bridge Exploiter. This is the address of an exploiter that carried out an $600M+ attack on a bridge protocol a year earlier. What is astonishing about the address is that it was attributed by the US Treasury’s Office of Foreign Assets Control as belonging to Lazarus, a group of state-sponsored professional hackers that do work for North Korea.

This is a major escalation and marks a major shift in the nature of the investigation. On the one hand, this could be a major blow as it could be a signal that the attacker is Lazarus themselves or is so confident in their ability to not get caught that they can comfortably troll Euler users by transferring funds to a sanctioned entity.

On the other hand, we had already gathered a lot of information about the attacker at this point and were confident that they were not associated with Lazarus. So we felt like this was either a poor attempt at misdirection or someone spiralling and out of their depth. Both of which were good news for our chances of catching them.

March 18th

Attacker sends a small percentage of funds back to Euler

Given what happened the day before, we were not expecting the attacker to send assets back to Euler users anytime soon, but at 6:20 am UTC on March 18th, that’s exactly what they started doing.

Beginning with a transaction for 1000 ETH, a total of 3000 ETH is sent back (worth ~$5.3M). Overall, this amounts to less than 3% of the total stolen funds, so is a drop in the ocean in the scale of things. Whilst the initial transaction gets everyone’s hopes up, it’s hard to take this as a positive signal when the assets stop being sent back. Given what happened the day before, it feels like the actions of someone trolling or at best unsure of what to do next.

The attacker has now sent stolen funds to a second attacker-controlled address, Tornado Cash, a seemingly random victim, Lazarus, and Euler. There seems to be no real rational explanation for this behaviour, unless they are just trolling, which is how most people interpret it at the time.

A few hours later, at 10:20 am UTC, another transaction is made. This time it is an empty transaction, with no message or ETH sent. This could be seen as yet more trolling, and we think that is the most likely reason for it, and yet, there’s an outside chance it is a prompt to communicate. We need to respond. Despite it feeling like the attacker is still just trolling, at 12:00 pm UTC we send a message that attempts to encourage a continuation of funds returned:

“Thank you for returning a portion of the assets. The original offer still stands if you would like to continue by returning the funds. The reward for information will be removed immediately and all our investigations will be dropped.”

In the meantime, we have now gathered a lot of information about the leads we have been following. We have been working non-stop on the investigation since the day of the exploit.

We ponder whether or not to get on the front-foot a bit more, reaching out via back channels to leads. Doing this could be risky though, because we could alert the attacker themselves, causing them to flee, or disturb someone entirely unrelated, causing them to protest their innocence on social media. Ultimately we decide to bide our time and keep working.

Team safety

Meanwhile, one of the key team members helping on the investigation called me in the early hours of their day. They wanted to discuss their personal situation because, for a variety of reasons I can’t disclose, they weren’t feeling safe. We explored options to help increase the safety of their situation and even talked about getting them on a plane that very day and putting them up in alternative accommodation abroad. It was traumatic in the extreme for the person involved. They could have been forgiven for walking away at that point, but if anything it only redoubled their resolve in the coming days. Words can’t express my appreciation for the dedication and resolve of our team.

March 19th

The team continues to do intensive research potential leads, but it’s Sunday and the attacker appears to be taking a day off. Not so for the Euler team — much was happening behind the scenes. Although there were some days where little was publicly disclosed or there was little visible communication, there was hard work around the clock — running down each and every lead, helping to coordinate multiple parallel and tangential investigations, and other things we cannot disclose.

March 20th

A breakthrough?

On March 20th at 4:10 pm UTC, a new mysterious message comes through from Euler Finance Exploiter 1. This one simply reads:

“T_27”

There is lots of speculation among our investigation about what this might mean. It shows up as a tank on google search. Perhaps it is something encoded as bytes. 2 and 7 are the first few digits of Euler’s number ‘e’. Is this just more trolling?

As the debate rages, a breakthrough moment happens. At 4:51 pm UTC, a new message is sent from Euler Finance Exploiter 2:

“We want to make this easy on all those affected. No intention of keeping what is not ours. Setting up secure communication. Let us come to an agreement.”

At 7:14 pm UTC we respond:

“Message received. Let’s talk in private on blockscan via the Euler Deployer address and one of your EOAs, via signed messages over email at contact@euler.foundation, or any other channel of your choice. Reply with your preference.”

We eagerly await a response, but none is forthcoming. Was this just yet more trolling? The use of the plural ‘we’ is a point of interest. We continue to keep digging and uncover more information about the leads we have been working on. The team has been working around the clock on this, with many of us getting 3 hours a night sleep.

March 21st

The return of Lazarus

It’s now March 21st, and we are still hard at work, but the day is going slowly. The risk feels like it is growing that the attacker is buying time in preparation for something unpleasant, perhaps to flee or strike a deal with another black hat. It’s been 8 days since the exploit and less than 3% of the funds have been returned.

All of a sudden, at 5:02 pm a new message is sent. This time it is not from the Euler attacker though. It is from the Ronin Bridge Exploiter to the Euler Finance Exploiter 2 address:

“Decrypt with the private key of 0xb66cd966670d962c227b3eaba30a872dbfb995db (https://github.com/LimelabsTech/eth-ecies).

bKsKXCYxBlQjY2opFlBVGQQrnjzy67s6xFc/+GIVX9YeGakmzz/EXsNZlAUhF37Q8RjVBn3DRVJP94ncxGS+j1wu5dLo4RRXrKSIrZzceoqVZ2pvsx02Pyl3K5SW3Yf/nzt96To3KOxN5sQqJZkfH6+RWwc+KoTAmomW1FNVWhlwV9UqKuuLwEo5heFmSFgBSJYNztvAEzZ/8Ra1BWu9P1OzBbmx67W0/2DdDMAK31tX”

This is extremely alarming. It’s clear immediately what is happening. Lazarus is trying to counter-exploit the Euler exploiter. If the Euler exploiter loses the funds to Lazarus they are gone forever. We respond as quickly as we can with a flurry of messages. The first message we send out straight away reads:

“Be very careful using that decryption tool. The simplest way out here is to return funds.”

Shortly after we follow up with another message:

“Do not try to view that message under any circumstance. Do not enter your private key anywhere. Reminder that your machine may also be compromised.”

And finally, after digging deeper, with yet another message:

“Do NOT use the suggested decryption tool. It has an old version of ellyptic, which has a vulnerability:

https://security.snyk.io/package/npm/elliptic/6.4.0

“There is no check to confirm that the public key point passed into the derive function actually exists on the secp256k1 curve. This results in the potential for the private key used in this implementation to be revealed after a number of ECDH operations are performed.””

The drama of this is not lost on people watching events unfold on social media:

Moments later, a new message is sent from Euler Finance Exploiter 2 to us:

“We still want to do the right thing returning funds to the Euler team. Will communicate shortly.”

Breakthrough! Or so we think.

March 22nd

Private communication attempt number 1 begins

Hours go by without any communication. It feels like another dead end. I could not be more tired having stayed up all night discussing the previous day’s events. Then finally, at 2:24 pm UTC on March 22nd, a new message comes in from Euler Finance Exploiter 2:

“Euler: from INTACH3ZZ@PROTONMAIL.COM”

After 9 days since the exploit, we appear to have a line of private communication. The attacker’s opening message is:

“Dear Dr. Michael,

From this email address we should talk privately with you and the Euler Team. It is in our best interest to step out of this unfortunate situation as soon as possible. I hope that we can define a course of action in a few correspondences.

Thanks,

Jacob”

Is this someone acting in good faith or just more games though? It’s interesting that the email refers to ‘us’ and ‘we’ and is signed from ‘Jacob’. Is the plural a bluff, or are we really dealing with a team here? Why did they address me specifically, and not the project more generally?

And what is the significance of the name Jacob? Clearly that is not the attacker’s real name. One possibility we explore is that it is a biblical reference. Jacob is a character in Genesis sometimes referred to as the ‘righteous deceiver’. In some bibles the name also apparently means ‘he cheats’.

Then we discover there is potentially a link back to the refund victim. Someone from the investigation points out:

“fyi the santichez name on twitter is Santiago. which is Jacob. is a male Spanish name that derives from the Hebrew name Jacob. https://en.wikipedia.org/wiki/Santiago_(name)“

We also explore if there is any further information to be gained from the email address itself. One of the sleuths comes back with an answer:

“errrrrrrrrrrrrrr what if he’s fucking with us, inta chez would translate to “you are gay””

Overall it still feels like this is still someone not taking things seriously. Nevertheless, we write a short email response and wait.

March 23rd

An eventful hospital trip

Meanwhile, at home, my eldest child is ill (later, we discover, with covid). With my wife looking after the newborn, I have been looking after my eldest as best I can during the night time on top of everything else. In the first few days after the exploit I had pretty much no sleep. That increased to 3–4 hours a night for a few days, but it dropped back to more or less zero with the new health issues at home. Obviously that isn’t a sustainable amount of sleep to be having, especially given everything else going on.

After a stressful start to the morning, sometime after lunch on March 23rd I suddenly start having pains in my chest and I’m struggling to breathe. My wife brings me to the hospital, where it could not be busier. As we enter the waiting room, at 1:45 pm UTC, a new email comes in from the attacker. I sit waiting for quite a while to be seen, responding to messages from our team as we explore options for replies.

Eventually I’m met by a doctor and told that I need an electrocardiogram to check my heart, which they are supposed to do very quickly for someone with my symptoms. But as anyone that has dealt with the NHS recently can attest, things rarely go smoothly. There are simply none of the machines available in the Accident and Emergency part of the hospital. So one of the doctors offers to walk me down to the nearby Resus (i.e. ‘resuscitation’) department where there is likely to be one free.

I kid you not, as we walk into the room, I hear the characteristic long beep as someone in a nearby bed goes into cardiac arrest. Alarms suddenly started going off everywhere and I was pushed out into the hallway. I could hear family members crying and doctors shouting. My anxiety hits a new high, but the experience helps keep things in perspective.

Over the course of these hours, I was conscious that we’d not replied to the attacker. That amount of delay is simply not ok when so much is on the line. So I sat among the other patients and crafted a response with our team while having blood tests drawn. After this, I promptly passed out from exhaustion.

After waking up and settling again in the waiting area with a dead phone and dying laptop, a few hours later I was told my symptoms were more likely the result of anxiety and a panic attack than a heart attack, and was sent home where I finally had a few hours sleep.

March 24th

A lot of investigative forensic work continued, much of which we can’t discuss in detail. The weekend is bearing down on us and we don’t want the attacker to have time off to think, but do we push them again? We decide against it, but continue our internal investigations.

March 25th

Almost 50% of stolen assets are returned

Then, on March 25th, at 3:08 pm UTC, 51,000 ETH (worth ~$90m) is suddenly returned by Euler Finance Exploiter 2 to Euler in a new transaction. That’s almost half of the assets returned now. Are they planning on returning all of them? Either way it’s a huge moment for everyone involved, especially Euler users. Even a 50% recovery from an attack like this is almost unheard of. Moments later a new message is sent on chain though:

“sheeps4music at xyzmailhub.com”

It seems like maybe the attacker wants to open a new line of private communication. What is wrong with INTACH3ZZ@PROTONMAIL.COM though? We quickly check and the latter email is longer operational. Then things go crazy again, as they always seem to do. Two steps forward one step back.

Prisoner’s dilemma or a bluff?

The attacker sends from Euler Finance Exploiter 2 4 more transactions of ~7700 ETH each (worth ~$13.5M each), this time not back to Euler, but to four new attacker-controlled addresses: 0xa1b44d…F4D8e676, 0x46e0Be…8c50Cf55, 0xC4e04A…3b7b208E, and 0x8765A3…D8E38A4c. These addresses also subsequently each receive ~10.7M DAI from Euler Finance Exploiter 2 as well. This means each of the four addresses controls around $24M worth of the stolen assets (~$13.5M ETH + ~$10.7M DAI).

The last of these transactions is at 3:32 pm UTC. At this point, it looks like the exploiter has decided to keep more than 50% of the stolen funds. Why have they distributed the stolen assets among four addresses though? Are there four attackers?

A few minutes later though, at 3:38 pm UTC, address 0x46e0Be…8c50Cf55, sends a transaction of all their ~7700 ETH back to Euler, along with a cryptic message seemingly pointing to an email address for further communication:

“x @ proton.me”

Around 30 minutes later, at 4:08 pm UTC, they then send their DAI back to Euler too. Well, sort of. Of the 10.7M DAI they hold, they send ~1.23M DAI back to Euler in this transaction. Then a flurry of peculiar messages follows. First, at 4:09 pm UTC, they message:

“email me ASAP will give up every fucking thing about hacker for 15%”

Then, at 4:15 pm UTC, they bizarrely lower their offer with another message:

“email me ASAP will give up every fucking thing about the hacker for 10% like offered.”

And then at 5:07 pm UTC, they appear to drop negotiations entirely with yet another message:

“Euler exploiter 3 here.. please just email xxxyyy990@umail.edu.pl .. will reply with info ASAP.. dont care about buonty”

We send emails to the two email addresses x@proton.me and xxxyyy990@umail.edu.pl but both seem to be dead ends. What on earth is happening in all this?

One interpretation is that the attacker is actually a group of four individuals, and they have decided to send ~50% of the funds back to Euler and then split the rest equally among themselves. But then, in a classic prisoner’s dilemma style standoff, one of the individuals has decided to break rank. Whoever controls 0x46e0Be…8c50Cf55 seems worried about being caught and is trying to lessen their future sentence by trying to return stolen assets voluntarily in a show of good faith.

At this point, however, the story is not compelling. We send an email to sheeps4music@xyzmailhub.com calling the attacker’s bluff, making it clear that the time for games is up. Minutes later they give up the charade, transferring the ~9.5M DAI remaining in 0x46e0Be…8c50Cf55 to one of the other four attacker-controlled addresses in this transaction.

At 7:43 pm UTC the sleuths assisting the investigation notice a new message come through on chain from the attacker offering a new email address sheeps4music@mail2tor.com to talk over. They ask what it is about (they were not privy to private communication over email) and I tell them we had received an email from that email address and had asked for confirmation on chain that it was genuine. The message they were seeing was simply a confirmation message. Meanwhile, over email, a new round of discussions are underway with the attacker, this time with a very different tone. At 8:45 pm UTC I message the sleuths:

The conversation with the attacker continues into the early hours of the next day. Our approach appears to be working.

March 26th

It’s a frustratingly quiet day on the communication front, but the team continues the forensic work at all hours. It is unclear if the response from the attacker was merely more trolling or an attempt to buy more time.

March 27th

On the following day, the pressure campaign begins to pay off and the attacker starts sending more of the stolen assets back.

First, at 6:13 pm UTC, a transaction from 0xa1b44d…F4D8e676 sending ~7700 ETH (~$13M) back to Euler.

Quickly followed at 6:21 pm UTC by another transaction from 0x8765A3…D8E38A4c sending its ~7700 ETH back.

And then another at 6:29 pm UTC from 0xC4e04A…3b7b208E sending its ~7700 ETH back.

A few minutes later at 6:40 pm UTC the attacker sends a further ~10.7m DAI back to Euler from 0x8765A3…D8E38A4c.

At this point it looks like they might be about to return all of the stolen assets, but as usual, they take pause.

At 6:58 pm UTC they send a transaction for 20M DAI from 0xa1b44d…F4D8e676 to a brand new attacker-controlled address 0x0d1B0c…70DA9843. What are they planning to do with this? Are they going to try to keep 10% of the total stolen? It seems possible. That offer is no longer on the table though. It had long been withdrawn.

At 7:24 pm UTC they then send another transaction for 3M DAI from this new address back to Euler, leaving 17M outstanding in the new wallet.

New emails are then exchanged over the course of the next few hours. The attacker mentions retaining hold of assets as a ‘safeguard’. We are pushing hard at this point.

March 28th

A few hours later, in the early hours of the 28th, they ask for forgiveness for the harm done to Euler’s reputation and the time taken away from me with my son. At 1:23 am UTC we reply:

“I’m sitting with my wife and son right now typing this message. You’re right that you can’t change those first few days with my newborn. But you can change tomorrow for me. He’s two weeks old now. I would love to spend the day with him. I really do not care about winning or losing or the reputation of the protocol. None of that matters in the end. The same is true for you. The money, which you can never spend, is completely useless to you if you can’t live a full life after this.”

A little while later several new messages are sent on chain.

First message from the attacker at 1:56 am UTC:

“The rest of the money will be returned ASAP. I only look after my safety, and that is the reason for the delay. I’m sorry for any misunderstanding. Please read my next message.”

Second message from the attacker at 1:57 am UTC:

“Jacob here. I don’t think what I say will help me in any way but I still want to say it. I fucked up. I didn’t want to, but I messed with others’ money, others’ jobs, others’ lives. I really fucked up. I’m sorry. I didn’t mean all that. I really didn’t fucking mean all that. Forgive me.”

Back on email the attacker requests to talk to me personally over a call or through voice messages to get reassurances. It’s 4:14 am UTC, and I’m beyond tired, but I agree to do it on the proviso that the remaining funds are returned. Sometime after, more funds are returned to Euler.

A transaction at 4:51 am UTC sends a further 5M DAI back to Euler from 0x0d1B0c…70DA9843.

A few more hours go by, and then at 8:28 am UTC I receive my first direct message over a social media platform. This was clearly essential to engage with to continue to build trust and encourage the return of funds, but also extremely high risk. I knew that if I were to say the wrong thing the remaining assets might never be returned. I was very conscious that it could also have been a trap.

In the conversation I continue to press for the return of the remaining assets whilst also trying to build trust and keep things moving. I asked at one point whether he considered the bug bounty too small and why he didn’t consider taking that route instead of carrying out the exploit. He said it wasn’t something he really considered. His answer was that he wasn’t experienced enough to know about all the options he had available to him once he knew about how to carry out the exploit; the attack was apparently hypothetical for him until the last minute, and he wasn’t even sure it would work.

We talk for some time until the midwife arrives to see my wife and son. It feels like a natural point to end the conversation so I urge him to return the funds whilst our appointment is underway.

Then it all goes quiet again. Recovery stands at around 84% at this point. The pressure to say something publicly is enormous by this point, but we know that if we do and get it wrong, any progress we’ve made could quickly unravel.

March 29th

Following the private conversation our team has a lot to work on.

March 30th

We sense the attacker’s anxiety and fear they may have changed their mind. I replay the conversation we had over and over again in my head. That night I can barely sleep again. I send my wife this message at 1:01 am UTC:

I started experiencing this exploding head syndrome phenomenon most nights before I went to bed after this point. It’s apparently not that uncommon, but I had never experienced it before and it really shook me. Fortunately I haven’t had it in a long while.

March 31st

The next day starts slowly, though quickly picks up. In the early afternoon, a number of transactions come through. Are we done? Of course not.

Unfortunately the transactions all involve sending small amounts of dust back to Euler. There are still millions of dollars worth of assets unreturned. Are these dust transactions just more trolling? Or are they some kind of signal?

A transaction at 2:41 pm UTC sends dust ETH back to Euler from 0xC4e04A…3b7b208E.

A transaction at 2:42 pm UTC sends dust ETH back to Euler from 0x46e0Be…8c50Cf55.

A transaction at 2:43 pm UTC sends dust ETH back to Euler from 0x8765A3…D8E38A4c.

Shortly after, at 3:12 pm UTC a new cryptic message is sent from the attacker to a seemingly brand new address. It simply reads:

“Was asked for this”

It looks like a confirmation message for someone the attacker is in private conversation with. We know it isn’t us who asked for it though. Who is it for? There is a lingering concern that he is in the process of making some kind of deal with another black hat to launder the remaining stolen assets.

A little while later we start to better understand what’s happening though. I received a message from none other than Laurence Day saying that someone has been trying to contact me. The message to him was from a member of the Euler community who claims to be the intended recipient of the ‘was asked for this’ message.

It turns out that starting March 29th, during the period of silence we have had with the attacker, they have forged direct contact with two Euler community members on social media. The motivation for doing this isn’t entirely clear. It’s hard to understand why the attacker hasn’t just returned the remaining assets. Is this more misdirection? Is it delaying tactics? Is it trolling? Can the community members be trusted? Clearly we need to continue to tread carefully.

From talking to one of the community members it seems the motivation of the attacker for reaching out to other people from the community is to get advice on reputation management and a second opinion on the best course of action next in terms of returning the remaining assets. They are also talking about taking out loans, which sounds concerning. The attacker also tells one of the community members they are waiting to meet someone later that evening to discuss their plans and help them manage their safety. Again, this sounds more than a little ominous.

April 1st

I ask one of the sleuths from the war room to reach out independently to the attacker over social media to bring a fresh impetus to things and reiterate the message that the best way out is to return the stolen assets as soon as possible. As with all communications that went out, it’s a carefully crafted message, deliberate and strategic.

April 2nd

By the evening of April 2nd things have once more become intolerable. The promise to return all assets has not been met and it seems like the attacker is once again not operating in good faith. We push more, taking a direct approach and sending them an email reminding them of what is at stake for them.

I’m starting to feel incredibly unwell by this point. Something about the symptoms reminded me of how I felt when I first caught covid, so I decided to take a test. Sure enough it was positive. When I spoke to my wife she too was feeling unwell and tested positive. It seems likely our eldest was patient 0. We assumed our newborn son would be fine since covid rarely affects babies.

Meanwhile our latest email has had the desired effect, and contact is reestablished just a few hours later. This time the attacker messages me again from a different social media account. I urge them to end things immediately. They request that things can be ended without resentment. At 11:49 pm UTC we offer reassurance:

“Not that kind of team. Everyone just wants to move on. Be great if it ends with a positive story.”

It seems like their conversation with the war room sleuth has also picked up, but as usual, things then go quiet. It’s 5 am local time by the time I can finally sleep.

April 3nd

Safe to say we’re all ready for this to be over at this point. At 8:02 am UTC the attacker sends me a new message saying funds will be back within the hour. I can’t allow myself to believe it. Sure enough, at 9:36 am UTC they say it is taking longer than expected. We ask them if it is guaranteed to be done by 2:00 pm UTC. They say it is.

At 12:56 pm UTC they ask where the remaining assets should be transferred to. We tell them to the DAO treasury address. This feels like it could be it. I message the Euler team a couple of minutes later:

Does it happen? No.

Minutes go by. And keep going by. Each one feels like an hour. At 1:50 pm UTC we nudge the attacker. They respond at 2:07 pm UTC saying there’s not much left to do. We demand an update. We then let the Euler team know things are still very uncertain:

At 3:35 pm UTC we press the attacker again. A minute later they respond, making a request. We reject it and remind them it is important to return the funds before people think they have changed their mind.

At 4:13 pm UTC we ask what the delay is about.

At 5:52 pm UTC they message saying 30 more minutes.

At 7:03 pm UTC they message saying they are ready to go.

At 7:45 pm UTC they message saying they have emailed a list of requests. One of them asks me to:

“organize a party for you and your team, with nice food and drinks, and beautiful girls, somewhere in the future”

Are they trying to actually be nice or is this a joke? It certainly seems at this point like this could be just another game. I’m at the end of my tether. I would love nothing more than to scream back at this point. But there’s obviously little sense doing that. So we get to work crafting a response to the requests. We opt to hit each one with a straight bat.

At 9:14 pm UTC we reply to a direct message saying the email response is nearly ready. We end by saying:

“Has to be now though. Further delay is a red line.”

At 10:01 pm UTC the email response is sent.

Every question answered. Including the party one, to which we say simply:

“Respectfully, this is a time for humility and reflection and does not seem like the right time to think about a party.”

At 10:16 pm UTC they respond on direct message:

“Going for it”

There is another frustrating wait, and I’m trying not to lose my patience with them. Then, finally, a transaction at 10:49 pm UTC sends a further 12M DAI back to Euler from 0x0d1B0c…70DA9843.

At 10:51 pm UTC they send a message saying that they know I hate them. We reply:

“I don’t hate you. Really wish you hadn’t done it obviously, but everyone makes mistakes. Important thing is you’re making it right so that everyone can focus now on the future.”

Minutes later the remaining funds are returned. First a transaction at 10:54 pm UTC sends all the remaining assets from Euler Finance Exploiter 1, ~8080 ETH (~$14.6M) back to Euler. Then a transaction at 10:59 pm UTC sends all remaining assets from Euler Finance Exploiter 4, ~2500 ETH (~$4.5M) back to Euler.

It’s over. In the end, the strategy worked. There is much that cannot be disclosed, but the attacker was compelled for a number of reasons to return the funds.

At 11:07 am UTC we announce the news with a single tweet:

At 00:15 am UTC we write to the sleuths in the war room to thank them for their support.

April 4th

Covid setback creates communications challenges

The next morning there were lots of messages of congratulations, but I was feeling so rough that I could honestly barely feel anything about the outcome. Our team was exhausted, but were already hard at work handling the enormous challenges still ahead of us.

Meanwhile, my son was also getting quite ill. We ended up having to call an ambulance. Both him and my wife suffered complications from covid and spent several days in hospital.

During those days there were a lot of questions about what happened from journalists and questions about what the next steps would be from affected users. Given the health issues in my family it was hard to contribute to communications though and whilst many people were thankfully sympathetic to the situation, some grew a bit frustrated.

The challenge was that very few people actually knew what had happened or had information to share because we generally kept things incredibly tight-lipped during the recovery process. People in the broader Euler team and many in the war room had only a little more information than the general public.

Unfortunately, the information vacuum led people to speculate on what might have happened, and some on social media got quite creative with their ideas and came to the conclusion that it must have been us all along:

You got us, Chris!

The media also started speculating about what had happened. One article claimed to have interviewed a key Euler negotiator. Others speculated that Euler had been bailed out by influential partners. Seeing people who had played relatively minor roles — or no role at all — in the recovery story receive public adulation was quite disheartening for our team who had really given everything.

There was plenty of other misleading information floating around too, including about the attacker. So much so that the attacker even messaged us about some of the stories at one point, seemingly annoyed that our team had perhaps been briefing against him to the media. The reality was that we had not spoken to the media about the attacker. The only time we had done so was to seek corrections for misleading stories.

Rest of the year

At some point during this process my wife and I began to worry more about our home security. There were stories being published around the time of the exploit about people being tortured in their homes over their crypto holdings. And here we had hostile messages, an active attacker, Lazarus involvement, potential organised crime links, a social media firestorm fueled by misinformation, among other things. Some of the hostility has been pretty relentless — the joys of being a public builder in crypto.

At home, we had additional security systems fitted and I began exploring better ways to secure my private keys. I wanted them sharded and records proving that they weren’t kept inside the house. I eventually found a solution and everything seemed to be fine. Then in November one of my hardware wallets developed a fault. Not a problem I thought, I’ve got the private key. Ultimately it turned out I didn’t though.

Somewhere in between the birth of my son, our exploit recovery efforts, personal and family illnesses, I made an error and it turns out that one of the private keys is no longer recoverable. This means that I’ve now lost a substantial percentage of the crypto assets I held in cold storage, accumulated over more than seven years, including the majority of the EUL allocated to me for participating in Euler governance (~4.4% of the total supply is now effectively removed from circulation).

In terms of my Euler allocation, I am currently considering my options and will likely make some form of proposal to the DAO at some point. My focus right now is entirely on the relaunch though. Ultimately I take responsibility for whatever happens next. I would have once found this outcome absolutely devastating, but the earlier events of the year really helped put the important things in life into perspective for me.

January 2024 Update

Good news! I have recovered my wallet. Thanks to everyone who supported me. Fingers crossed 2024 continues as it started.

Conclusion

So there it is. A turbulent tale, but one with a (mostly) happy ending. While many might have lost their resolve following an ordeal such as this, I couldn’t be prouder of the way the Euler Labs team came together to solve an almost insurmountable challenge. Even in the best of circumstances, a 100% recovery of assets is a rare outcome. With crypto, it’s almost unheard of.

We’re hugely grateful for all the support and warm wishes we received along the way. While the words above share my perspective, they represent the work of dozens of people on the Euler team and beyond, from all around the world. That same Euler team has been hard at work over the past year building a stronger, more powerful and modular V2 of the Euler platform which we’re excited to help rollout over the coming weeks and months.

The story doesn’t end here — the only way is forward.

Disclaimer:

The opinions reflected herein are subject to change without being updated.

War & Peace was originally published in Euler Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Modularity, The Way Forward To Building DeFi Systems

Doug Hoyte — Tue, 12 Dec 2023 14:48:56 GMT

Containing Complexity

Engineering is largely about containing complexity. By creating well-defined interfaces, and enforcing that modules interact through these interfaces, each module can ignore the inner workings of the other modules and focus on its own implementation.

Consider engineers building a bridge. They don’t need to work out a new chemical composition of metal specific for their project. Instead, they will take an existing steel “off the shelf”. The steel is a module, and the interface is the desired properties, such as tensile strength, weight etc. If some new material came along with these same properties, it could be used for future bridges without requiring any new bridge-level redesigns.

In electronics, products are built out of reusable components such as microchips. These components were often designed without any knowledge of the final product they will be used in. In fact, they are usually designed to first be plugged into a testing fixture, during which they are called the “device under test”. Later on, when the end product is assembled, it too will be a device under test before it is finally shipped to customers. Treating components as modules that conform to interfaces tames the complexity.

Leaky Abstractions

That is the theory. As components themselves become more complicated, using them effectively requires understanding their inner design. In software engineering this is common enough to be given a name: Leaky abstraction.

Consider the simple case of numbers. Programmers can’t even treat these as pure modules. For example, you might assume that adding 1 to a number will always give you a new number that is greater than the original. In many programming systems, including Solidity, this is not so: a fixed amount of memory is allocated to store a number, and if you exceed this it will either throw an error or wrap back around to a lower number, as in a car odometer.

Modularity in Finance

Financial systems are also designed to be modular and standardised.

When you trade stocks, you will typically see them listed according to their ticker symbols, such as GOOG for Google. But underneath this, securities also have special numbers called CUSIPs. For instance, Google’s class A shares have code 38259P508. In a sense, you can think of these as equivalent to Ethereum addresses: They are unique identifiers that exchanges, brokers, etc., can use to ensure they (or their computers) are all talking about the same thing.

How do their computers talk to each other? Most often, it is over a protocol called FIX. This protocol specifies various fields that are common to different message types and gives them numeric tags. For example, CUSIP codes are stored in the SecurityID field, which is given tag 48. Since every company is speaking FIX, it is relatively easy to plug in new connections to existing systems.

That is the theory. In practice, the numbering system is a mess. Before using the SecurityID field, you need to also check IDSource (22). As well as CUSIPs, many other numbering standards are in common use: RIC, ISIN, ISO currency codes, exchange-specific numbers, and more. And this is just talking about the US. Most other countries have their own numbering standards, such as SEDOL in the UK.

The FIX protocol itself is only superficially compatible. Every different provider has their own custom variations of which FIX version they support, which fields they use, and the meanings of those fields. Each entity has its own bespoke interface documentation (often hundreds of pages long), and it is a significant engineering effort to integrate anything at all.

In my opinion, this is the true opportunity of DeFi: finally making good on the largely unfulfilled promise of FIX and the numbering authorities and creating a seamlessly interoperable international financial system.

Smart Contracts

On the blockchain, the primary unit of modularity is the smart contract. Smart contracts have their own internal storage, and other smart contracts cannot directly read this storage. Why is this? It is not so that contracts can store secret values: Everything on the blockchain is public information and can be read off-chain by anyone with access to a node.

This restriction is there to enforce modularity. Contracts have no way to interact with other contracts except through the interfaces that these other contracts choose to provide.

In the context of DeFi, sophisticated systems will typically be constructed as a constellation of contracts that communicate via well-defined interfaces. For example, rather than having a single lending contract, Compound uses special-purpose cToken contracts to hold each asset.

Building systems as intercommunicating contracts has the benefit that individual components can be accessed by external systems as well, as long as they conform to the exposed interfaces. Additionally, separation into contracts can be used to form trust boundaries between components to increase system security.

Defect Containment

In the original version of the Euler contracts, there was a module called the Swap module. This allowed users to swap their eTokens for other eTokens by effectively withdrawing them, swapping on a DEX like Uniswap, and then re-depositing them. Because all user funds were retained in a “singleton” contract, tokens never needed to transit any other wallet for a swap, and this operation was maximally gas efficient.

However, the Swap module had to be very carefully coded and audited because it ran in the context of the main contract. In other words, it was part of the trusted code base (TCB).

To add an additional layer of defense, we created a new module called SwapHub that invoked “swap handlers” outside the TCB. Swap handlers are isolated contracts and are treated as though they could themselves be malicious actors. They are given the funds the customer wants to swap, and then are invoked. If they do not send back a sufficient amount of the desired token to meet the customer’s slippage checks, then the transaction is aborted. In this way, any defects in the swap handler — malicious or not — are fully isolated.

Based on decomposing how DeFi systems work today, our own experience, and some of the current thinking around intents-based systems, we have decided to take this concept much further. In the new system we’re designing at Euler Labs, we have created a flexible but minimal TCB, and have moved as much of the possible system outside of this trust boundary.

Money Lego

As the number of modules in a system increases, the number of possible interconnection pairs grows quadratically. Therefore, coding each contract to specifically interact with all other possible contracts is infeasible.

The way we solve this is with standardised interfaces: Every token-like contract exposes a balanceOf method, no matter what type of asset it is storing, or how it is implemented internally. ERC-20 is the basis for standard tokens on Ethereum. Even though it has several well-known shortcomings, all tokens are written to conform to it. This itself is proof of how powerful standards are: Any possible gain from improving the interface would be much smaller than the loss of breaking compatibility with the rest of the network.

Another standard is ERC-4626, the Tokenised Vault Standard. This standard is wisely backward-compatible with ERC-20, but adds new interfaces suitable for yield-bearing tokens.

At Euler Labs, we believe ERC-4626 will be a fundamental standard. In our interpretation, lending activity will involve multiple vault contracts working together: at least one as collateral and one as a liability. To make this work, we are building a meta-layer for vaults to interact with each other and generate interconnected liquidity, which will also benefit Euler v2. Powered by a modular design, these vaults will also be highly customisable: creators will be able to choose oracles, set LTV, risk managers, and take advantage of additional functionality such as batching, gasless transactions, simulations, and more. Ultimately, standardising modularity at this level will create an ecosystem that will eclipse isolated and bespoke lending markets.

Disclaimer:

The opinions reflected herein are subject to change without being updated.

Modularity, The Way Forward To Building DeFi Systems was originally published in Euler Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Protocols vs Products

Michael Bentley — Fri, 10 Nov 2023 17:56:14 GMT

Which is better?

I recently saw this claim: “If a smart contract is feature-oriented, then it is a product, not a protocol.” I thought this was an interesting distinction worth exploring in more detail, especially given the current trend of people advocating for the development of more primitive, base layer-type protocols.

https://x.com/PaulFrambot/status/1696900204727791999?s=20

Protocols and products

The traditional definition of a protocol is a set of simple rules determining how information is transmitted between different parties or devices. The Internet (i.e. TCP/IP) is perhaps the best known set of protocols in existence today. Protocols can be extraordinarily valuable, but they seldom extract value. The Internet would not be what it is today, for example, if it had an in-built fee switch or tried to extract value from those who built on top of it. Protocols are usually unopinionated and minimalistic, targeting a broad user base, and designed to last for a long time.

The Internet stood the test of time because it didn’t try to build in complex features. If it had, most of those features would have quickly become out-of-date technical debt and it would have become obsolete. Protocols usually form the base layer of a stack on top of which more complex operations can be performed and more complex *products* can be built. The Internet became valuable because it targeted a broad user base and millions of people felt comfortable staking their future on it by building products on top of it.

In contrast to protocols, the best products are usually extremely opinionated and feature rich, have a very specific target audience in mind, and are not designed to last forever. They always seek to create and extract value from users too. Steve Jobs is the canonical master of building great products.

The rise of product-protocols

So, what are DeFi smart contracts? Protocols or products? I think there’s an argument DeFi has led to the creation of a new class entirely: the product-protocol.

DeFi smart contracts are definitely protocols, in the sense that they provide a strict set of rules determining how information is transmitted between different parties, but they are often feature-rich products too. Some protocols are closer to the original meaning. Others are more product-like. Is this good or bad?

Recently, a narrative has formed around the idea that feature-rich products are somehow inherently bad. I can’t agree. To be bearish on products is to be bearish on composability of protocols, and the entire premise of DeFi, for there’s not much point in building a protocol if it isn’t used to build great products on.

At the core of this narrative seems to be the view that simple protocols are more secure than products. At some level, this is trivially true. It’s easier to build something secure in itself if it only does a couple of small tasks. When it comes to writing secure code, the less complexity the better is a good rule of thumb. So, let’s all build simple protocols and stay away from complex products?

Well, no. Protocols are great for developers to build on, but ordinary end users typically don’t just want great protocols, they want great products. And if products are the desired end state, then you get them one way or another in a system over time, regardless of who builds in that complexity.

Protocol developers might be able to sleep better at night knowing that their protocol is so simple it can’t possibly fail. But as they push the risk and requisite skill required to design and bring to market products further up the stack, they should also be prepared to relinquish the rewards that development of those products brings. The Internet is incredibly valuable, but it was famously not that valuable to its developers.

Product-protocols

Modular product development

The security argument for protocols is really an argument for the benefits of modularity. Modules are simple and do a few things well, and nothing else. They are easy to understand and easy to test. In this regard, protocols can be thought of as base layer modules. Good products usually build on base layer modules and add further features and complexity in a modular fashion.

Whilst modularity may help with security, there are very different flavours of modularity. Consider the different ways that Mac and PC were originally brought to market. Both systems used modular design principles. However, whereas Mac famously opted to build all its modules in-house, PC famously opted to let anyone build products on top of it.

There was a stark difference in pace of growth, but also in the final product. PC grew rapidly, but ended up feeling like a jumbled mess, plagued by a proliferation of rent-seeking apps, inefficiencies and security issues. Mac grew much slower, but ended up feeling like a cohesive product, largely free from value extracting apps and security issues.

There’s an important lesson here for smart contract developers. Modularity per se is not enough to bring about great products in the long-term. You want *cohesive* modularity, not chaotic modularity. There’s a risk that the proliferation of DeFi primitive protocols, each with their own governance system and their own fee extraction mechanisms, is simply replacing middle-men with middle-protocols.

Example

Let’s consider an example. One of the most popular use-cases for lending protocols is to compose them with a DEX in order to build a leveraged long or short position. A user will borrow assets and swap them on the DEX, redeposit the swapped assets in the lending protocol, borrow more, swap again, and so on, up to their desired risk/reward tolerance.

There are many advantages to having the lending protocol and DEX built in a modular fashion by entirely different specialised teams. Indeed, the fact that two different protocols built by different developers from different parts of the world can be composed together inside a single transaction is one of the major benefits of DeFi over traditional finance. However, there are important downsides to this way of building things too.

If both lending protocol and DEX extract fees and demand governance by entirely different communities, each with their own interests, the end user experience could be suboptimal. The modularity might not be cohesive if the use-case didn’t have a tailor made solution. Furthermore, if the end user needs to pay UI fees and lending protocol fees and DEX fees and hook fees, and so on, is DeFi really living up to its disintermediated claims?

Conclusions

Overall, I broadly agree with claims that many DeFi projects today look more like products than protocols. I’m not sure this is a bad thing. There’s clearly some truth in the idea that simpler, more primitive, protocols are inherently more secure than complex products. But I’m sceptical of the idea that simply building lots of primitive protocols will result in a more secure decentralised finance ecosystem and better user experience in the long-run. I expressed a similar view in a thread about oracle-free lending protocols a little while ago. Users want products, and this requires complexity at some level. If you’re only building primitives, don’t expect to be able to extract value from users at that level.

Stay tuned to see what we’re building at Euler Labs. Spoiler alert: both primitive and more feature-rich developments are on the roadmap. Cohesive modularity is the design goal.

References

https://x.com/euler_mab/status/1667190907161182210?s=20

https://x.com/delitzer/status/1661755779240841222?s=20

https://twitter.com/PaulFrambot/status/1696900204727791999?s=20

Morpho Labs 🦋 on Twitter: "1/ A new era for risk management?Historically, in DeFi, risk has been managed by a DAO, but the landscape is changing.Let's dive into the current state of risk management collateralized lending and what this change might look like 👀 / Twitter"

1/ A new era for risk management?Historically, in DeFi, risk has been managed by a DAO, but the landscape is changing.Let's dive into the current state of risk management collateralized lending and what this change might look like 👀

Disclaimer:

The opinions reflected herein are subject to change without being updated.

Protocols vs Products was originally published in Euler Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Have you heard about this SSTORE quirk, anon?

Kasper — Tue, 19 Sep 2023 20:41:59 GMT

tl;dr SSTORE is one of the most expensive EVM opcodes. A savvy developer always optimizes their smart contract to use SSTORE in the most efficient manner. However, there’s a catch buried in EIPs that nobody talks about. Or at least, I haven’t encountered any information about it until today.

tl;dr, Read the Conclusion.

Background

There are a ton of resources out there that can teach you how to reduce and optimize the gas consumption of your smart contract. Recently, I went through some of them to improve the smart contract I am working on. Like everyone, I was especially interested in storage optimization which can have a substantial impact on gas costs. Minimizing unnecessary storage modifications and utilizing packed data structures that optimize storage usage can help reduce gas consumption significantly. How surprised I was when I looked at the hardhat-tracer dump of SLOADs and SSTOREs to find out that sometimes their gas cost might not be as one would expect after going through the resources. A direct deep dive into relevant EIPs was needed.

EIP-2929

source: https://hackmd.io/@fvictorio/gas-costs-after-berlin

From the gas optimization point of view, the most important seems to be the fact that if the value was previously modified during the same transaction, all the subsequent SSTOREs cost 100. It means that if I have a storage variable and I want to change its value multiple times in one transaction, the first write should be the most expensive and all the subsequent ones should cost pennies.

Unintuitive behaviuor

Let’s use this simple example to explain the problem. Say you need to use a reentrancy guard in your smart contract. For that, you inherit from ReentrancyGuard from Open-Zeppelin. The contract is as follows:

// SPDX-License-Identifier: MIT
// OpenZeppelin Contracts (last updated v4.9.0) (security/ReentrancyGuard.sol)

pragma solidity ^0.8.20;
/**
 * @dev Contract module that helps prevent reentrant calls to a function.
 *
 * Inheriting from `ReentrancyGuard` will make the {nonReentrant} modifier
 * available, which can be applied to functions to make sure there are no nested
 * (reentrant) calls to them.
 *
 * Note that because there is a single `nonReentrant` guard, functions marked as
 * `nonReentrant` may not call one another. This can be worked around by making
 * those functions `private`, and then adding `external` `nonReentrant` entry
 * points to them.
 *
 * TIP: If you would like to learn more about reentrancy and alternative ways
 * to protect against it, check out our blog post
 * https://blog.openzeppelin.com/reentrancy-after-istanbul/[Reentrancy After Istanbul].
 */
abstract contract ReentrancyGuard {
    // Booleans are more expensive than uint256 or any type that takes up a full
    // word because each write operation emits an extra SLOAD to first read the
    // slot's contents, replace the bits taken up by the boolean, and then write
    // back. This is the compiler's defense against contract upgrades and
    // pointer aliasing, and it cannot be disabled.
    // The values being non-zero value makes deployment a bit more expensive,
    // but in exchange the refund on every call to nonReentrant will be lower in
    // amount. Since refunds are capped to a percentage of the total
    // transaction's gas, it is best to keep them low in cases like this one, to
    // increase the likelihood of the full refund coming into effect.
    uint256 private constant NOT_ENTERED = 1;
    uint256 private constant ENTERED = 2;
    uint256 private _status;
    /**
     * @dev Unauthorized reentrant call.
     */
    error ReentrancyGuardReentrantCall();
    constructor() {
        _status = NOT_ENTERED;
    }
    /**
     * @dev Prevents a contract from calling itself, directly or indirectly.
     * Calling a `nonReentrant` function from another `nonReentrant`
     * function is not supported. It is possible to prevent this from happening
     * by making the `nonReentrant` function external, and making it call a
     * `private` function that does the actual work.
     */
    modifier nonReentrant() {
        _nonReentrantBefore();
        _;
        _nonReentrantAfter();
    }
    function _nonReentrantBefore() private {
        // On the first call to nonReentrant, _status will be NOT_ENTERED
        if (_status == ENTERED) {
            revert ReentrancyGuardReentrantCall();
        }
        // Any calls to nonReentrant after this point will fail
        _status = ENTERED;
    }
    function _nonReentrantAfter() private {
        // By storing the original value once again, a refund is triggered (see
        // https://eips.ethereum.org/EIPS/eip-2200)
        _status = NOT_ENTERED;
    }
    /**
     * @dev Returns true if the reentrancy guard is currently set to "entered", which indicates there is a
     * `nonReentrant` function in the call stack.
     */
    function _reentrancyGuardEntered() internal view returns (bool) {
        return _status == ENTERED;
    }
}

First of all, the contract doesn’t use a boolean, but uint; it’s cheaper as per the comment in the code. Second, we set the _status to 1 in the constructor (one-time gas cost of 20,000) so that later on it’s cheaper to make the 1->2 transition in the nonReentrant() modifier. Finally, the _status may be considered a transient variable as it should always be brought back to the initial value within the transaction (the reentrancy lock should always be released). Moreover, because of that final 2->1 transition, we should expect a gas refund which, due to EIP-3529, is capped at 20% of the total transaction’s gas. Hence, it’s best to keep the gas refunds low, otherwise, we may not get the full refund we would be entitled to.

I’ve just said that thanks to setting the _status to 1 in the constructor, it’s cheaper to use the nonReentrant() modifier, but how much gas exactly do we expect to spend using nonReentrant()? Taking into account what’s been considered so far, let’s write some comments in the code as we go:

modifier nonReentrant() {
    // the first SLOAD should cost us 2100.
    // if we were to call this modifier again, we should pay 100 here
    if (_status == 2) {
        revert ReentrancyGuardReentrantCall();
    }

    // considering we already accessed this storage slot above,
    // we should pay 2900 here if it's the first call of the modifier.
    // if we were to call this modifier again, it would mean it's a
    // subsequent SSTORE hence we should pay 100 here
    _status = 2;

    _;

    // considering we already accessed this storage slot above 
    // and the fact that it's always a subsequent SSTORE, we 
    // should pay 100 here
    _status = 1;
}

To sum up, if it’s the first call of the nonReentrant(), we expect to pay 5100 gas. If it’s a subsequent call in the same transaction, we expect to pay 300 gas. Sounds good, doesn’t it?

If there’s a smart contract out there that integrates with ours and in one transaction it calls a couple of non-reentrant functions from our contract, each subsequent call should be cheaper than the first one. Thanks to that behaviour, our reentrancy protection should not be that costly!

Well, that’s theory. In practice, things appear vastly different. To show that, let’s write two simple contracts and look at the hardhat-tracer dump.

contract MyContract is ReentrancyGuard {
    // some non-reentrant function that does nothing but is protected
    function nonReentrantFunction() public nonReentrant {}
}

contract IntegrationContract {
    MyContract internal immutable myContract;

    constructor(MyContract _myContract) {
        myContract = _myContract;
    }
    
    // a function that calls the protected function twice.
    // we expect that the second call should be cheaper
    function test() external {
        myContract.nonReentrantFunction();
        myContract.nonReentrantFunction();
    }
}

We have the MyContract with a simple reentrancy-protected function and IntegrationContract that simply calls the MyContract.nonReentrantFunction() twice in its test() function. Do you see anything odd? We expected to pay 100 for the 1->2 transition in the second call, instead, we paid 2900. At the same time, we paid expectedly less for the second SLOAD. Why is that?

EIP-2200

When I shared my findings with the team members, some of them said it was illogical, some of them said it was inconsistency, and others claimed it must be a bug. I dived deeper and apparently, everything became clear.

EIP-2200 is heavily referred to by EIP-2929 and pertains to the so-called Wei Tang’s approach to net gas metering. Its implementation predates EIP-2929 and was introduced in the Istanbul hard fork. The idea behind it was to provide a structured definition of net gas metering changes for SSTORE opcode, enabling new usages for contract storage, and reducing excessive gas costs where it doesn’t match how most implementation works. It says: “Usages that benefit from this EIP’s gas reduction scheme include: Subsequent storage write operations within the same call frame. This includes reentry locks, same-contract multi-send, etc.”. As you can see, it explicitly mentions the reentrancy locks gas cost reduction, but has it done the job?

The gas calculation logic proposed by EIP-2200 is as follows:

source: https://eips.ethereum.org/EIPS/eip-1283

source: https://eips.ethereum.org/EIPS/eip-2200

As you can see, the algorithm does not recognize the case in which the original value equals the current value now, but it had been previously set to some other value within the same call frame. It does not care whether the storage slot key was previously accessed. Therefore, when we set our _status variable from 1 to 2, no matter if it’s the same or a subsequent call, we are exercising the same path and pay SSTORE_RESET_GAS for that, which happens to be 2900 after EIP-2929. Bye bye gas savings!

Questionable rationale

EIP-2200 offers a rationale for the above behaviour. It claims that the approach mostly achieves what transient storage tries to do (EIP-1087 and EIP-1153) but without the complexity of introducing the concept of “dirty maps”, or an extra storage struct. Personally, I find this questionable. Controversial EIP-1153 that introduces new opcodes for transient storage — TLOAD and TSTORE — would work far better here and in other scenarios. In cases like the above, each use of nonReentrant() modifier would cost 300 vs. 5100 (or 3100 in the subsequent calls). Moreover, although simplicity shall be praised and “dirty maps” implementation might be indeed tricky, it might sometimes lead to the implementation of ugly design patterns like the one I had to come up with myself. On the other hand, if nobody knows about the SSTORE quirk, nobody except me will implement any ugliness :)

Ugly gas optimizations

Imagine you have a function where reentrancy is allowed. Every time it happens you just increase the callDepth counter to be able to revert at some point. At the same time, you have a critical section that you want to protect from reentrancy, you use the criticalLock for that. The critical code should only be executed once when the call stack unwinds back to the top-level function call. You implement the smart contract as follows:

// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.19;

contract UglyContract {
    error CriticalLockViolation();
    error CallDepthViolation();

    struct ContextStruct {
        bool criticalLock;
        uint8 callDepth;
        uint8 stamp;
        uint8 dummy;
    }

    ContextStruct internal context;
    
    constructor() {
        // set the storage slot to non-zero so that it's cheaper to alter it later
        context.dummy = 1;
    }

    // this modifier is used to protect critical sections of the code
    modifier critical() {
        if (context.criticalLock) {
            revert CriticalLockViolation();
        }
        _;
    }

    function prettyFunction(address target, address criticalTarget) external critical {
        uint8 callDepth = context.callDepth;

        if (callDepth >= 5) {
            revert CallDepthViolation();
        }

        unchecked {
            context.callDepth = callDepth + 1;
        }

        // do something here: make external calls, allow reentrancy back into this function.
        // low-level call used to avoid compiler optimization and show the issue
        target.call("");

        context.callDepth = callDepth;

        if (callDepth == 0) {
            context.criticalLock = true;
            
            // do some critical stuff here: make external calls, but disallow reentrant callback into this function.
            // low-level call used to avoid compiler optimization and show the issue
            criticalTarget.call("");
            
            context.criticalLock = false;
        }
    }

    function uglyFunction(address target, address criticalTarget) external critical {
        uint8 callDepth = context.callDepth;

        if (callDepth >= 5) {
            revert CallDepthViolation();
        }

        unchecked {
            context.callDepth = callDepth + 1;

            // gas optimization to keep the slot in altered state until the end of the uglyFunction
            context.stamp = 1;
        }

        // do something here: make external calls, allow reentrancy back into this function.
        // low-level call used to avoid compiler optimization and show the issue
        target.call("");

        context.callDepth = callDepth;

        if (callDepth == 0) {
            context.criticalLock = true;
            
            // do some critical stuff here: make external calls, but disallow reentrant callback into this function.
            // low-level call used to avoid compiler optimization and show the issue
            criticalTarget.call("");
            
            context.criticalLock = false;

            // bring the slot back to the original state
            context.stamp = 0;
        }
    }
}

As you can see, we have two functions that do exactly the same thing. Gas optimized uglyFunction() and not optimized prettyFunction(). To avoid the user being charged additional gas, in the uglyFunction() the stamp field is set to 1 at the top and reset at the end of the transaction. Savings? 2800 for each call. Of course one can argue here that the final storage slot value will be equal to the original value hence gas refund is expected. In this case, you would be right, but it’s only because the context is transient in nature which does not have to be the case. Also, you should not rely on the gas refunds as they are capped at 20% of the total transaction’s gas as I already mentioned before.

Conclusion

It appears that due to EIP-2200, when it comes to the gas cost for the SSTORE opcode, within a transaction, the EVM only cares about the current vs. final value of the storage slot and not intermediate values. In scenarios when your smart contract sets the storage slot from X to Y, then back to X, and from X to Z you will not pay the reduced gas cost of 100 for changing the storage slot from X to Z, even if you’re within the same call frame. You will pay a full cost of 2900 or 20,000 if X is 0. Amongst others, it has implications for reentrancy locks, toggled boolean flags, counters incremented and decremented within the same call frame, etc. Maybe after all controversial EIP-1153 would be a good idea?

Disclaimer:

The opinions reflected herein are subject to change without being updated.

Have you heard about this SSTORE quirk, anon? was originally published in Euler Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.