<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Miguel Hervas on Medium]]></title>
        <description><![CDATA[Stories by Miguel Hervas on Medium]]></description>
        <link>https://medium.com/@miguel_64041?source=rss-138d26687430------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*0RN93S0ejpxBZh8a</url>
            <title>Stories by Miguel Hervas on Medium</title>
            <link>https://medium.com/@miguel_64041?source=rss-138d26687430------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 15:24:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@miguel_64041/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Altair Runtime Upgrade 2, A Post-Mortem]]></title>
            <link>https://medium.com/altair-network/altair-runtime-upgrade-2-a-post-mortem-618d8c3b12d8?source=rss-138d26687430------2</link>
            <guid isPermaLink="false">https://medium.com/p/618d8c3b12d8</guid>
            <category><![CDATA[defi]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[substrate]]></category>
            <category><![CDATA[polkadot-network]]></category>
            <dc:creator><![CDATA[Miguel Hervas]]></dc:creator>
            <pubDate>Fri, 19 Nov 2021 14:51:04 GMT</pubDate>
            <atom:updated>2021-11-19T14:51:04.853Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Qm37LFnfOVRPjJI2hzHlw.jpeg" /><figcaption><a href="https://pixabay.com/photos/trees-wilderness-nature-woods-3822149/">source</a></figcaption></figure><blockquote>This post will go into more detail about the event that caused Altair to halt block production for over a day and try to share the thought process that helped identify and fix the issue in addition to highlighting the learnings.</blockquote><p>First some current <em>Altair</em> relevant context:</p><ul><li>Altair Polkadot/Substrate/Cumulus at v0.9.10</li><li>Altair number of vesting schedules at 4k</li></ul><p>Content of the Altair Runtime Upgrade 2:</p><ul><li>Bring node and runtime to Polkadot/Substrate/Cumulus v0.9.12</li><li>Enabling Crowdloan Claiming functionality</li><li>Support for Compressed Runtime WASM</li></ul><p>More information about the release: <a href="https://github.com/centrifuge/centrifuge-chain/releases/tag/v3.0.0-alpha4">https://github.com/centrifuge/centrifuge-chain/releases/tag/v3.0.0-alpha4</a></p><p>Runtime Upgrade sequence of events:</p><ol><li>Executed Upgrade in Centrifuge’s <em>Charcoal</em> Testnet (connected to <a href="https://medium.com/centrifuge/launching-chachacha-rococos-wilder-cousin-13dcb3820db2"><em>Chachacha</em></a><em> </em>Relay Chain). At this stage, <em>Charcoal</em> had same data set as <em>Altair</em> in terms of content, not data hierarchy (will explain why this was important later on)</li><li>Verified Claiming functionality successfully</li><li>Executed Upgrade in Altair. <strong>Block</strong> <strong>production</strong> <strong>stopped</strong> trying to propose the next candidate.</li></ol><p>Time to put on the troubleshooting hat:</p><p>First we started looking at the logs and noticed that the collators were not able to build the block in the first place:</p><pre>2021-11-09 15:33:43.004  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ⚠️ Vesting declares internal migrations (which *might* execute). On-chain `StorageVersion(0)` vs current storage version `StorageVersion(0)`</pre><pre>2021-11-09 15:33:43.733  INFO tokio-runtime-worker aura: [Parachain] ⌛️ Discarding proposal for slot 136372668; block production took too long</pre><p>In the first line, we can see a migration in the Vesting Pallet. <a href="https://github.com/paritytech/substrate/pull/9202">This</a> was introduced as part of Substrate v0.9.11 to support multiple vesting schedules and we were inheriting it on our move from v0.9.10 to v0.9.12.</p><p><em>To clarify, we already knew that and we tested it with same data content beforehand on Charcoal, but we couldn’t trigger the same behavior.</em></p><p>In the second log, we can se that a timeout for block proposal was triggered at aroung ~700ms. Now we were wondering what was the maximum block proposal time allowed on the collator side for our Aura implementation:</p><pre>// We got around 500ms for proposing    <br>block_proposal_slot_portion: SlotProportion::new(1f32 / 24f32),    // And a maximum of 750ms if slots are skipped</pre><p>(from <a href="https://github.com/centrifuge/centrifuge-chain/blob/a988e21fed2d971265cae3150116f519781c9603/src/service.rs#L523-L524">source</a>)</p><p>We had 0.5 seconds (500ms) to propose the block before proposal slot is discarded. Since the block slot is discarded after 0.5s we didn’t know for sure how much longer did the collator need to build it even if there was no time constraint, so we rebuilt the node with a unrealistically high block proposal timeout of 12s so we could know how far we were from the max target:</p><pre>2021-11-10 11:13:06.600  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ⚠️ Vesting declares internal migrations (which *might* execute). On-chain `StorageVersion(0)` vs current storage version `StorageVersion(0)`</pre><pre>2021-11-10 11:13:12.096  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Identity</pre><p>Well that was not very encouraging. It took roughly 5.5s to wrap up the vesting migration, which is 5s above target. So why we couldn’t identify this behavior during our testing phase? Why did it take 0.3s on <em>Charcoal </em>and 5.5s in <em>Altair</em>?</p><p>We started looking at the infrastructure where our collators were running in and couldn’t find an obvious bottleneck neither on Memory, CPU or I/O. Therefore, if the code is the same, the physical resources are the same, the data content is the same, then it had to be something around the data morphology.</p><p>Since <em>Altair</em>-<em>Kusama</em> storage trie is significantly bigger than <em>Charcoal</em>-<em>Chachacha</em> then the hierarchical path where the vesting storage is located can be significantly different from one network to another, then the time it can take to iterate over all the vesting schedules can as well increase depending on the location of the vesting schedules storage node. (<em>Thanks to @basti &amp; @rphmeier from Parity for pointing that out</em>)</p><p>At this point, we were talking about how to fit 100 shoes in a box of 10, essentially trying to execute 4k vesting schedule migrations in a limited time of 0.5s, we were running out of options to get this through without runtime changes (through <em>Kusama</em> Governance).</p><p>So we needed to go from 5.5s to 0.5s, that is an improvement of order 11. We know that running some process in memory vs on SSD can be around 10–100 times faster, so in theory if we run our chain storage in memory we could improve the computation time by that much and <em>potentially</em> hit the 0.5s mark. And that is what we did, we provisioned an instance of 16 cores and 64GB of memory and modified the node code to use an in-memory DB (thanks @<em>rphmeier </em>for the suggestion and @<em>mustermeiszer </em>for the fast plan execution):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H6TPAq8RvfX2kCQ0cuv0Gg.png" /><figcaption><a href="https://github.com/centrifuge/centrifuge-chain/pull/520/files#diff-de6b9cfd1a256da7900d35e26aa7279bd713110e68e91b52616befb7397848dfR355">source</a></figcaption></figure><p>Then when running the new node, had to sync up from scratch but since we were using in-memory db it was very fast (800 bps). Slowly the free memory of the host was going down (expected since it had to host the whole DB in it), once the <em>parachain</em> DB was synced up, the node attempted to produce the <strong>block</strong> and it <strong>went through</strong>! We were able to decrease the block time from 5.5s to 0.06s (60ms), that is 91x faster in memory than against an SSD.</p><h4>Summary of Learnings:</h4><ul><li>Be aware of every migration that can be inherited from Substrate and assess the impact upfront</li><li>Try to build a similar data storage hierarchy on your test environments before pushing the upgrade on your production ones (not trivial, and we will try to work on that soon)</li><li>Ensure that you follow the suggested host specifications for both running validators and collators</li><li>Good logging around your services to kick off the troubleshooting process</li><li>Even though increasing vertically our host fixed the issue, once the network is big enough this might not be an option. Therefore paying attention and being proactive on the above will save many headaches.</li></ul><h4>Last Note:</h4><p>Many thanks to @basti, @rphmeier &amp; @alejandro @ <a href="https://medium.com/u/433b0bc6fa9e">Parity Technologies</a> for being responsive and helpful on this matter.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=618d8c3b12d8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/altair-network/altair-runtime-upgrade-2-a-post-mortem-618d8c3b12d8">Altair Runtime Upgrade 2, A Post-Mortem</a> was originally published in <a href="https://medium.com/altair-network">Altair Network</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Altair’s Winning Parachain Bid: Making Sense of the Kusama Parachain Candle Auctions]]></title>
            <link>https://medium.com/altair-network/altairs-winning-parachain-bid-making-sense-of-the-kusama-parachain-candle-auctions-7c2e8e52df2?source=rss-138d26687430------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c2e8e52df2</guid>
            <category><![CDATA[kusama]]></category>
            <category><![CDATA[altair]]></category>
            <category><![CDATA[parachain-auction]]></category>
            <dc:creator><![CDATA[Miguel Hervas]]></dc:creator>
            <pubDate>Sat, 02 Oct 2021 12:44:49 GMT</pubDate>
            <atom:updated>2021-10-05T14:38:36.239Z</atom:updated>
            <content:encoded><![CDATA[<h4>May the VRF odds be forever in your favor</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*tBC5JcM2dADLMxJW" /><figcaption>From <a href="https://media1.giphy.com/media/JRPobx8QnK5nOtgNxb/giphy.gif?cid=ecf05e47wuapogq4nm648ruldi4u3hj35tefrkzmde3let73&amp;rid=giphy.gif&amp;ct=g">giphy</a></figcaption></figure><p>First of all — a huge THANK YOU to the entire Altair community for your support in the Altair crowdloan! Altair won the 9th parachain slot auction in the most dramatic fashion — so we wanted to lay out the details for you on how that happened!</p><h3>How does the Kusama Candle Auction Work?</h3><p>Candle auctions are a variant of open auctions that were originally employed in 16th century for the sale of ships and get their name from the “inch of a candle” 🕯 that determined the open period of the auction. When the flame extinguished and the candle went out, the auction would suddenly terminate and the standing bid at that point would win.</p><p>Parachain slot auctions follow a modified <a href="https://wiki.polkadot.network/docs/en/learn-auction#mechanics-of-a-candle-auction">candle auction</a> format, with each auction lasting approximately 1 week. To prevent auction sniping (the strategy of bidding at the last moment) and promote more accurate price discovery, the exact endpoint of a particular auction remains unknown until after the auction’s close. Kusama parachain slot auctions have a known open phase of 1 day and 21 hours, and at the normal close of the bidding period (1 week) it will be retroactively determined to have ended at some point in the past during the ending phase of 5 days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n0sAO1KI79JNLvWj" /></figure><p><a href="https://polkadot.network/blog/the-first-parachains-launch-on-kusama/">Kusama’s modified candle auction format</a></p><p>So during the ending phase, bids will continue to be accepted, but later bids have higher probability of losing since the retroactively determined close moment may be found to have preceded the time that a bid was submitted.</p><h3>Why all this complexity?</h3><p>Fair auctions are hard. Especially when you involve blockchains. How blockchains execute their code is generally public and can be seen and predicted by everyone. This means automated bidding by a onchain logic (i.e. a smart contract) can’t be private and thus its strategy would always be at a disadvantage to an off chain entity doing bidding as they could look at all the automated bidding and find a way to exploit it. Ultimately the goal for parachains is that they can bid for their own slots so the auction mechanism wants to incentivize participants to give their highest bid as soon as possible but still allow some price discovery.</p><h3>The VRF: deciding when the candle goes out</h3><p>The auction’s end point can be any time within the ending period. The end point needs to be uniformly random over the ending phase (i.e. the likelihood of any block in the period chosen is the same). It also needs to be impossible to manipulate. Kusama has a built in Verifiable Random Function (VRF). It is a cryptographic function that produces this randomness that is then used to automatically choose the ending block at the close of the 1-week bidding period.</p><h3>The Endpoint of Auction #9</h3><p>Altair was leading the bids for auction 9 for 35% of the ending period. That means that Altair had a 35% probability of winning the candle auction.</p><p>In the end, the retroactively determined ending time for auction #9 on Kusama by the VRF was September 25th at 19:16:48 UTC. So in other words, that’s when the candle blew out. At this block — Altair was the highest bidder!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*amU4O6wGxi2k3Ai6" /><figcaption>From <a href="https://parachains.info/auctions#timeline">Parachains.info</a></figcaption></figure><h4>What’s Next?</h4><p>We will be sharing more details in the next days on the detailed launch process, timeline, and what to expect from Altair (and for future AIR token holders!) so make sure to stay tuned to our channels for updates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c2e8e52df2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/altair-network/altairs-winning-parachain-bid-making-sense-of-the-kusama-parachain-candle-auctions-7c2e8e52df2">Altair’s Winning Parachain Bid: Making Sense of the Kusama Parachain Candle Auctions</a> was originally published in <a href="https://medium.com/altair-network">Altair Network</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Launching Chachacha: Rococo’s wilder cousin]]></title>
            <link>https://medium.com/centrifuge/launching-chachacha-rococos-wilder-cousin-13dcb3820db2?source=rss-138d26687430------2</link>
            <guid isPermaLink="false">https://medium.com/p/13dcb3820db2</guid>
            <category><![CDATA[polkadot]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[altair]]></category>
            <dc:creator><![CDATA[Miguel Hervas]]></dc:creator>
            <pubDate>Thu, 25 Feb 2021 20:22:24 GMT</pubDate>
            <atom:updated>2021-10-13T15:06:24.171Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Chachacha: Rococo’s wilder Cousin</h3><p><em>Chachacha</em> is a <em>Rococo</em> based relay chain testnet configured and supported by <a href="https://centrifuge.io/">Centrifuge</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2g0y4jyne2lybvriKB-qBA.png" /></figure><p>Centrifuge is excited to contribute this testing environment for Polkadot’s parachain ecosystem! We started building <em>Chachacha</em> to prepare Centrifuge Chain for <em>Rococo</em>, Polkadot’s Parachain testnet but have since heard feedback from several teams that this would be a useful tool to prepare for Rococo.</p><p>We hope that teams waiting to on-board onto <em>Rococo</em> will find <em>Chachacha</em> helpful to ease this process, and that teams which have already gone live on <em>Rococo</em> can use <em>Chachacha</em> as a support network to test future updates.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3o6Mbr4ICoxdkaZNWU%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fthesimpsons-3o6Mbr4ICoxdkaZNWU&amp;image=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2F3o6Mbr4ICoxdkaZNWU%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb5ca55109e8cf9b6df9f3c5a19fd698/href">https://medium.com/media/eb5ca55109e8cf9b6df9f3c5a19fd698/href</a></iframe><h4>Purpose</h4><p>The main purpose of <em>Chachacha</em> is to serve as a support network to ease and speed up the on-boarding of new parachains in <em>Rococo</em>.</p><p>It helps:</p><ul><li>To get the parachains in the waiting list to a state that is ready to be added to Rococo for further performance and stability testing</li><li>Parachains to find issues beforehand, by integrating earlier.</li></ul><h4>Main Characteristics</h4><ul><li><em>Chachacha</em> will be at par with the latest <em>Rococo</em> polkadot/cumulus/substrate version</li><li>The network will be refreshed and restarted frequently</li><li>Inclusion process is analogous to <em>Rococo’s</em></li><li>Notifications and Support will be given in the <a href="https://matrix.to/#/!bNsgeAIUuMfcyVXKAu:matrix.org?via=matrix.org&amp;via=matrix.parity.io"><em>Chachacha</em></a> Element channel</li></ul><h4>How to include your parachain in Chachacha</h4><p>Maintain at least <strong>two</strong> validators (full block authoring node) for <a href="https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Ffullnode-relay.chachacha.centrifuge.io#/explorer"><em>Chachacha</em></a>:</p><p>Treat this like a production grade Polkadot node — see <a href="https://wiki.polkadot.network/docs/en/maintain-guides-how-to-validate-polkadot#docsNav">Run a Validator (Polkadot)</a></p><p>Node Setup — use one of the options below</p><ol><li>Chachacha Bootnodes</li></ol><pre>/ip4/34.89.248.129/tcp/30333/p2p/12D3KooWD8CAZBgpeZiSVVbaj8mijR6mfgUsHNAmCKwsRoRnFod4</pre><pre>/ip4/35.242.217.240/tcp/30333/p2p/12D3KooWBthdCz4JshkMb4GxJXVwrHPv9GpWAgfh2hAdkyXQDKyN</pre><p>2. Build from source:</p><pre>git clone <a href="https://github.com/centrifuge/polkadot">https://github.com/centrifuge/polkadot</a></pre><pre>cd polkadot</pre><pre>git checkout chachacha</pre><pre>cargo build --release </pre><pre>./target/release/polkadot --validator --chain rococo --name &lt;your_chachacha_validator_name&gt; </pre><p>3. Use Docker:</p><pre>docker run -d centrifugeio/polkadot:chachacha --validator --chain rococo --name &lt;your_chachacha_validator_name&gt;</pre><p>Check your node on the<a href="https://telemetry.polkadot.io/#list/0x5719ed0c1a5f4a11ef4d206f3a35d350365adc35943f2834b8d5bc7ff6e1a1d2"> Chachacha Telemetry</a></p><p>Generate your Chachacha ValidatorId Addresses</p><p>Follow <a href="https://matrix.to/#/!bNsgeAIUuMfcyVXKAu:matrix.org?via=matrix.org&amp;via=matrix.parity.io">Chachacha</a> announcements for Chachacha validator updates, which can require one of the following scenarios:</p><ol><li>Update client</li><li>Update client and purge-chain</li></ol><p>Maintain at least <strong>one</strong> collator (full block authoring node) for your team’s parachain.</p><pre>cd &lt;root_cumulus_based_parachain_code&gt;</pre><pre>cargo build --release</pre><pre>./target/release/&lt;parachain_collator_name&gt; --version</pre><pre>./target/release/&lt;parachain_collator_name&gt; export-genesis-state --<br>parachain-id &lt;your_registered_parachain_id&gt; &gt; genesis-state</pre><pre>./target/release/&lt;parachain_collator_name&gt; export-genesis-wasm &gt; genesis-wasm</pre><pre>wget -O rococo-chachacha.json https://raw.githubusercontent.com/centrifuge/polkadot/f9ed42d49ee825784ba8d55a82fc2742b88766af/node/service/res/rococo-chachacha.json</pre><pre>./target/release/&lt;parachain_collator_name&gt; --collator --parachain-id &lt;your_registered_parachain_id&gt; -- --execution wasm --chain rococo-chachacha.json</pre><ol><li>Reach in the <a href="https://matrix.to/#/!bNsgeAIUuMfcyVXKAu:matrix.org?via=matrix.org&amp;via=matrix.parity.io">matrix</a> channel and provide validator keys above generated.</li><li>After receiving CHA’s to one of the ValidatorId Addresses initiate the Submitting the setKeys Transaction by executing `session-&gt;set_keys` in <a href="https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Ffullnode-relay.chachacha.centrifuge.io#/extrinsics">Chachacha Extrinsics</a> for each validator with its correspondent output of the rotate keys call</li><li>Follow the <a href="https://docs.substrate.io/tutorials/v3/cumulus/connect-parachain">registration process</a></li><li>You are free to do runtime upgrades after the parachain is connected, so you can still iterate on features later on</li></ol><p><em>Until we update PolkadotJs types, please use the following ones:</em></p><pre>{<br>  &quot;AccountInfo&quot;: &quot;AccountInfoWithProviders&quot;,<br>  &quot;Address&quot;: &quot;MultiAddress&quot;,<br>  &quot;FullIdentification&quot;: &quot;()&quot;,<br>  &quot;LookupSource&quot;: &quot;MultiAddress&quot;,<br>  &quot;Keys&quot;: &quot;SessionKeys6&quot;<br>}</pre><p><em>Technical Questions? Head to our </em><a href="https://matrix.to/#/!bNsgeAIUuMfcyVXKAu:matrix.org?via=matrix.org&amp;via=matrix.parity.io"><em>Element</em></a><em>.</em></p><p><em>For more, jump on our </em><a href="http://gov.centrifuge.io/"><em>Forum</em></a><em>, join our </em><a href="https://centrifuge.io/newsletter/"><em>newsletter</em></a><em>, or join our </em><a href="https://t.me/centrifuge_chat"><em>telegram</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13dcb3820db2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/centrifuge/launching-chachacha-rococos-wilder-cousin-13dcb3820db2">Launching Chachacha: Rococo’s wilder cousin</a> was originally published in <a href="https://medium.com/centrifuge">Centrifuge</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>