<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mark Andreev on Medium]]></title>
        <description><![CDATA[Stories by Mark Andreev on Medium]]></description>
        <link>https://medium.com/@mark.andreev?source=rss-5001da3331c8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*pQNyjnh4uzSobfquPFpMgA.jpeg</url>
            <title>Stories by Mark Andreev on Medium</title>
            <link>https://medium.com/@mark.andreev?source=rss-5001da3331c8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:06:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mark.andreev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Loading Two Versions of the Same Library in One JVM with JPMS]]></title>
            <link>https://medium.com/@mark.andreev/loading-two-versions-of-the-same-library-in-one-jvm-with-jpms-cf52abe9ac46?source=rss-5001da3331c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf52abe9ac46</guid>
            <category><![CDATA[jpm]]></category>
            <category><![CDATA[jvm]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Mark Andreev]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 17:33:35 GMT</pubDate>
            <atom:updated>2026-11-10T17:33:35.010Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_gRmw6rDe075cgF10kj4w.png" /></figure><p>Java applications can have dependency conflicts when two components require different versions of the same library. Can we load both versions into one JVM?</p><p><strong>Yes. The Java Platform Module System (JPMS) provides the necessary APIs.</strong></p><h3>What is JPMS?</h3><p>JPMS was introduced in <strong>Java 9 (September 2017)</strong> as part of Project Jigsaw (<a href="https://openjdk.org/jeps/261">JEP 261</a>).</p><p>It addresses several problems in Java:</p><ul><li><strong>Dependency management:</strong> Modules declare their dependencies explicitly. Java checks the module graph during resolution.</li><li><strong>Encapsulation:</strong> Modules control which packages other modules can access.</li><li><strong>JDK modularity:</strong> Developers can create smaller runtime images with jlink.</li><li><strong>Class isolation:</strong> Separate module layers can use separate class loaders.</li></ul><p>Before Java 9, applications primarily used the classpath. When two JARs contained the same class, a single class loader typically loaded the first matching definition. This could cause dependency conflicts and runtime errors.</p><h3>Loading multiple library versions</h3><p>JPMS provides ModuleLayer, which lets applications create separate module configurations at runtime.</p><p>I created a small demonstration using <strong>Guava 10.0 and Guava 33.1.0</strong>.</p><p>Each version runs in a separate module layer with its own class loader:</p><pre>JVM<br>           |<br>      Application<br>       /       \<br> ModuleLayer A  ModuleLayer B<br>      |              |<br>  Guava 10.0     Guava 33.1.0<br>      |              |<br> ClassLoader A   ClassLoader B</pre><p>Both versions contain com.google.common.collect.ImmutableList. The demo confirms that each version has a different class identity and that both can run in one JVM.</p><p>The implementation uses ModuleFinder, Configuration.resolve() and ModuleLayer.defineModulesWithOneLoader().</p><h3>Limitations</h3><p>The example demonstrates dependency isolation. The two Guava JARs have different JPMS module names.</p><p>JPMS does not provide a direct API to unload a module layer. Class unloading depends on garbage collection. Applications must also manage plugin lifecycle and release references when dynamic unloading is required.</p><p>For applications that do not need dynamic unloading, this is usually not a problem.</p><h3>Try it</h3><p><strong>Source code:</strong> <a href="https://github.com/mrk-andreev/demo_jpms-multiversion">github.com/mrk-andreev/demo_jpms-multiversion</a></p><p>Requires JDK 21 or later.</p><pre>git clone https://github.com/mrk-andreev/demo_jpms-multiversion.git<br>cd demo_jpms-multiversion<br>./run.sh</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf52abe9ac46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FIX Protocol Debugging: A Practical Guide to Common Issues and Best Practices]]></title>
            <link>https://medium.com/@mark.andreev/fix-protocol-debugging-a-practical-guide-to-common-issues-and-best-practices-19133ed8e427?source=rss-5001da3331c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/19133ed8e427</guid>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[fix-protocol]]></category>
            <dc:creator><![CDATA[Mark Andreev]]></dc:creator>
            <pubDate>Tue, 14 Oct 2025 20:39:40 GMT</pubDate>
            <atom:updated>2025-10-14T20:39:40.887Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’ve worked with FIX protocol for any length of time, you know that debugging sessions can feel like detective work. Messages get rejected, sequences break, and connections drop at the worst possible moments. After spending countless hours troubleshooting FIX issues across different trading systems, I’ve learned that most problems fall into predictable patterns. This guide covers the issues you’ll actually encounter and the approaches that work when things go wrong.</p><h3>Understanding the Debugging Landscape</h3><p>FIX debugging splits into two distinct layers. The session layer handles connection management, message sequencing, and reliable delivery. The application layer deals with business logic — order management, execution reports, and market data. When something breaks, your first task is figuring out which layer is causing the problem.</p><p>Session-level issues typically manifest as logon failures, sequence number mismatches, or garbled messages. Application-level problems show up as rejected orders, unexpected execution reports, or business rule violations. The debugging approach differs significantly between these two categories.</p><h4>Session Layer Issues: Where Most Problems Start</h4><p><strong>Logon Failures</strong></p><p>The most common problem developers face is failed session establishment. When your FIX engine refuses to connect, start with the basics. Check your log files for the logon message (MsgType=A or 35=A). If you don’t see it, the problem is at the TCP level — your application never established a socket connection.</p><p>Use `<em>netstat -a | grep port</em>` to verify whether a TCP connection exists. If the connection shows as ESTABLISHED but you still see logon failures, examine the rejection reason in the logout message (MsgType=5 or 35=5). The Text field (tag 58) contains the explanation.</p><p>Here’s a typical logon message that succeeds:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*pKS1QD3JDDd9Od5dYr2T1w.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNoNzbsKAjEQRuF3Sa1LZjLRJPAXC0ax0GLdQhGLRVYRvBRBEBzf3VSHrzpfc7l9NmMpw3UsJpmA5XrfSCMaEYM6j1YlYpe3i9ypn6Fvu1Xu1QlIPYMte7IkU4rJzhM7jQFWyQa4GiEcKsCO1EzM%2BXV%2FP571czz9%2FlFgIg4%3D">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=98|35=A|49=SENDER|56=TARGET|34=1|52=20251014–19:07:23|98=0|108=30|141=Y|10=231|</blockquote><p>Compare this to a rejected logon where wrong credentials were provided:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-jytRE-3tDmzhxuWPG3RTA.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNqrVkrLrPBNLS5OTE8tVrJSsrB184zQM9EzqbG0NTQ0qjE2tTWtMbG0DXEMcncNqTE1sw129XNxDaoxNrE1rDE1sjUyMDI1NDA0edQw2dDSysDcysiixtTC1jOvLDEnM0UhODUvJbXIOT%2B3wNOlxtDA1tDCvEZJRyk5P6c0Nw9oYXRsLQALyifP">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=112|35=5|49=TARGET|56=SENDER|34=1|52=20251014–19:07:28|58=Invalid SenderCompID|10=187|</blockquote><p><strong>Sequence Number Mismatches</strong></p><p>Sequence number problems terrify people unnecessarily. The logic is straightforward: each side maintains two counters — one for outgoing messages and one for incoming messages. When your engine receives a sequence number higher than expected, it sends a Resend Request (MsgType=2). When it receives a sequence number lower than expected without PossDupFlag(43) set to Y, it logs out.</p><p>Here’s what a sequence mismatch looks like in practice. Your engine expects message 145 but receives:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jHQZpZuS7Re3wc-_FAnU5Q.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNqrVkrLrPBNLS5OTE8tVrJSsrB184zQM9EzqbG0NTQ1qzE2tXWpMbG0DXb1c3ENqjE1sw1xDHJ3DakxNrE1NDGvMTWyNTIwMjU0MDTRNbS0MjSwMjSt0dPTqzE0sDWwNKlR0lFKzs8pzc0DGh4dWwsA6k8e%2BA%3D%3D">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=156|35=D|49=SENDER|56=TARGET|34=147|52=20251014–19:10:15|…|10=094|</blockquote><p>The incoming MsgSeqNum(34) is 147, but your NextNumIn is 145. Your engine responds with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HnSQdt1gHYRvDb10sqlJmQ.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNoNzb0KAjEMAOB3yaxHkyatLWQQrOKgw3mDIA4ipwj%2BDEUQjO9up2%2F8vnC5fTZjrafrWCHDTJfrfccdW9JI5kXJOOkw71dlMAm6K9tF6c2zkvMm1CBBhzzFlNFlDBYVWQxDIxo6RQkGEzi%2F7u%2FHsyWH4%2B8P3iMg7A%3D%3D">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=72|35=2|49=TARGET|56=SENDER|34=203|52=20251014–19:10:16|7=145|16=147|10=156|</blockquote><p>This Resend Request asks for messages 145 through 147. The counterparty should retransmit with PossDupFlag(43)=Y.</p><p>The most common cause of sequence breaks is someone manually resetting sequence numbers on one side without coordinating with the counterparty. Less common but equally frustrating: your FIX engine crashed and forgot to persist the sequence numbers, or someone deployed a new build that wiped the session state.</p><p><strong>Version Mismatches</strong></p><p>BeginString(8) must be the first tag in every message. When counterparties use different FIX versions, you’ll see immediate rejection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sr1rUD02tshlOIbYvB4Ftw.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNqrVkrLrPBNLS5OTE8tVrJSsrB184zQM9EzqrG0tbSoMTa1dazR09OrMTSwNTI2rFHSUUrOzynNzQOqjY6tBQBTGxKy">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.2|9=98|35=A|…|10=231|</blockquote><p>If the acceptor expects FIX.4.4 but receives FIX.4.2, the logout will specify:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RN3cKff3NOqjH7QTQTj6_w.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNoNzbEKwjAQANBfkcxaemlOksANilE66NB2EMShhFgCmkJTQfD8d7s%2FeF%2FxiJ9zyLkfQhZWaDrW10IVig1pwxUSsjLU7ZqT6xi31LrLwTVcKQJGSbKUCCWoDRgLaJVk1FQnP05T8PNqH4aY2nmKaWAoFw0s1sKPz%2FcrLd3t%2FvsDlu0mkw%3D%3D">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=89|35=5|49=TARGET|56=SENDER|34=1|52=20251014–19:15:42|58=Incorrect BeginString|10=201|</blockquote><p>Always verify that both sides agree on BeginString, and for FIXT.1.1 sessions, ensure DefaultApplVerID matches.</p><h4>Garbled Messages and Checksum Errors</h4><p>A message is considered garbled when fundamental structure problems occur:</p><p>- BeginString(8) isn’t the first tag<br>- BodyLength(9) isn’t the second tag or contains incorrect byte count<br>- MsgType(35) isn’t the third tag <br>- CheckSum(10) isn’t the last tag or contains incorrect value</p><p>The FIX engine should disregard garbled messages without incrementing sequence numbers. The next valid message triggers sequence gap detection, initiating recovery.</p><p>Checksum calculation sums every byte up to but excluding the checksum field, then applies modulo 256. A sample message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hmxr43IO8WUdFowdPh7Mxw.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNpFTssKwkAQ%2B5c9a5nZnanbQg7WBxYUhPYgiAeRKoKPQxEEx393exJCHgRCPu58fW%2B6vj9eut6VLmJZ7zLJxAqwBguKuUmBar01zdHMVpUFgRcy9fDklYll7KlMUDJmFETERNEYSUJKwTzDmyraRdNOTQVsIUJSZ0JDVSA5QUg8QW05%2Fbc5DtvBGxMoZ3Mjd3reXvdH%2Brs%2FfH%2BQnTKy">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=153|35=D|49=BLP|56=SCHB|34=240|52=20251014–20:20:50|11=90001008|1=10030003|21=2|55=TESTA|54=1|38=4000|40=2|59=0|44=30|47=I|60=20251014–18:20:32|10=061|</blockquote><p>If you calculate checksum incorrectly, you’ll see rejections referencing tag 10. The fix: verify your SOH delimiter is actually 0x01, not the literal string “\\001”.</p><p><strong>SOH Delimiter Issues</strong></p><p>The SOH (Start of Header) character is 0x01 in hexadecimal. One of the most common bugs I’ve seen in custom implementations is using the wrong representation. This won’t work:</p><blockquote>byte SOH = “\\001”.getBytes()[0]; // Wrong!</blockquote><p>This will:</p><blockquote>char SOH = (char) 0x01; // Correct</blockquote><p>If you’re seeing every message rejected as garbled, verify your SOH delimiter first. Use Wireshark with a display filter on the counterparty’s IP address to compare your output against a known working client.</p><h3>Application Layer Debugging</h3><p><strong>Required Tag Missing</strong></p><p>The session accepted your message structurally, but validation failed. You’ll receive a Reject (MsgType=3):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8bAQOubGJpAYhc_F6-ZXpw.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNoNzrEKwjAQANBfKZm15JI7TQs3CEZx0KF2EMSh1FgCtqKxIHj%2Bu13e%2FL7qFj%2F7kFLThaRK5XizO%2BWYoxQMBsUSW8GC61W19bXQgo%2F%2BsPaVWGQgIcNGGwINODe6NFSCEyQGFHJchecYX%2BGavZsu62NKcejELi3DJDCRgGZwhaiZah%2F3sR%2Bmwvny%2BwOIHitV">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=124|35=3|49=TARGET|56=SENDER|34=15|52=20251014–20:25:18|45=14|58=Required tag missing|373=1|371=55|10=189|</blockquote><p>RefTagID(371) tells you which tag caused the rejection — in this case, tag 55 (Symbol). SessionRejectReason(373)=1 confirms it’s a required field violation. Always check your data dictionary for required vs optional fields.</p><p><strong>Incorrect Value Format</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jyy7zXpfawMS86Ck1iEG9A.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNpl0MFKw0AQBuBXCTlrmNndSTeBOQQSJVCrxFIE9bDEVIQ2gSQVwRF8B9%2FQJ3EXclA8DfyH%2F%2F%2BY93j%2F8nbVTZN77qY4jy1f1HeJSYxkjIZEE5diMr6tNmXVCKW8LZrLaivasBVSrEARAppzBbmyudaCyNdNCYBCxEVxsxYyjKIt15tdsa59H7AS40OCBEBS%2BNeiBMHvK3no%2F4gwDSIdRIvDixabF2W%2FRd%2BfX0ubEUNB6wV9O4xj187Rk5tdtB%2FGo5vDiV7d4dSJXmn2EytkbQNBgZb4LG6Hw%2BnY%2B%2F%2FcP378APSKVc8%3D">View in FixParser</a></figcaption></figure><p>When you send the right tag with the wrong data type:</p><blockquote>8=FIX.4.4|9=145|35=D|49=SENDER|56=TARGET|34=8|52=20251014–20:28:33|11=ORD001|55=AAPL|54=1|38=INVALID|40=2|44=150.00|60=20251014–20:28:32|10=142|</blockquote><p>OrderQty(38) expects a number but received “INVALID”. The reject explains:</p><blockquote>8=FIX.4.4|9=116|35=3|49=TARGET|56=SENDER|34=9|52=20251014–20:28:34|45=8|58=Incorrect data format for value|373=6|371=38|10=203|</blockquote><p>SessionRejectReason(373)=6 means incorrect data format, and RefTagID(371)=38 points to OrderQty.</p><p><strong>Business Message Rejections</strong></p><p>Application-level business logic violations use BusinessMessageReject (MsgType=j). For example, attempting to cancel an already-filled order:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fdysHWRIjOQv3aznzs8eWA.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNoNzsEKgkAQgOFXiT2X7MzOlApzCNLoUIF5CKKD5BrGVqAERdO7t5f%2F%2BPF%2FTde%2Ft34cm6sfTW5SKTfHhBLSTMCROpabUib1sloXtfJcDsVuVVTqSBCVUdAigwWaoc0d5sBKLAjqFihlbHTQEatLrYByKvuh9cOkCYNv2s%2Bk60PwrYKNHKiZmsszvO6P%2BHI6%2F%2F7ecS2E">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=134|35=j|49=TARGET|56=SENDER|34=22|52=20251014–20:32:15|45=21|372=F|379=12345|380=1|58=Order already filled|10=221|</blockquote><p>RefMsgType(372)=F indicates the Order Cancel Request triggered the rejection. BusinessRejectReason(380)=1 means “Unknown ID” — the order couldn’t be found or is in a state that doesn’t permit cancellation.</p><h4>Repeating Groups: A Common Source of Confusion</h4><p>Repeating groups allow multiple instances of the same field set within a message. The structure requires:</p><p>1. A leading field (NoXXX) specifying the count<br>2. Each entry starting with a designated first field</p><p>Consider a Security Definition with multiple legs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wblW6OUpbmv9XGyr3-BmBQ.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNpNzrsKwkAQheF32VrDzpmdJRuYwkIlhQqaIiAWolEEL8UiCI7vbhQLu8P%2FNefpDqfHrMt5e%2Byyq1ypk7otQhEsKREbi%2B4tJF3M63ZlErUZLafjxjiomEDhIeQpDOErlgowEVFY9NBETP1gLS0ifCJY6UvgEn9EPyKvxDA3cLvb%2BX659o%2FWm9cbiCEsXg%3D%3D">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=113|35=d|49=ONIXS|56=TARGET|34=5|52=20251014–20:35:22|555=2|602=9131|603=8|624=2|623=1|602=2382|603=8|624=1|623=1|10=132|</blockquote><p>NoLegs(555)=2 declares two entries. Each entry starts with LegSecurityID(602), serving as the delimiter between entries. The count in NoLegs must match the actual number of groups, or the message will be rejected as malformed.</p><p>A parsing error occurs when tags appear out of sequence within the repeating group. FIX requires fields to follow the order specified in the message definition. If you send:</p><blockquote>555=2|603=8|602=9131|624=2|623=1|…</blockquote><p>The parser can’t identify where one entry ends and the next begins because LegSecurityIDSource(603) appears before LegSecurityID(602).</p><h3>Best Practices for Effective Debugging</h3><p><strong>Log Everything, But Organize It</strong></p><p>Your FIX engine should log every incoming and outgoing message. When analyzing logs, you need to quickly identify patterns across multiple messages. Raw FIX messages aren’t human-friendly — having the ability to parse and visualize them significantly reduces debugging time.</p><p>Modern approaches involve tools that parse the raw FIX stream and present it in readable format. Some advanced implementations support custom data dictionaries, which becomes essential when counterparties use proprietary tags. Being able to quickly cross-reference tag 5000 in one message with tag 5000 in another helps spot inconsistencies immediately.</p><p><strong>Maintain a Data Dictionary</strong></p><p>Every counterparty interprets FIX slightly differently. Document the custom tags, valid values, and required fields for each connection. When debugging, refer to this dictionary rather than relying on memory. Custom tags (typically 5000–9999) won’t appear in standard FIX documentation.</p><p>Example data dictionary entry for a custom tag:</p><blockquote>&lt;field number=”5001&quot; name=”ClientStrategy” type=”STRING”&gt;<br> &lt;value enum=”VWAP”&gt;Volume Weighted Average Price&lt;/value&gt;<br> &lt;value enum=”TWAP”&gt;Time Weighted Average Price&lt;/value&gt;<br> &lt;value enum=”POV”&gt;Percentage of Volume&lt;/value&gt;<br>&lt;/field&gt;</blockquote><p><strong>Validate Before Sending</strong></p><p>Most rejection issues stem from malformed messages leaving your system. Implement validation that checks:</p><p>- All required tags are present<br>- Data types match expectations <br>- Values fall within acceptable ranges<br>- Repeating group counts match actual entries<br>- Custom tags comply with bilateral agreements</p><p><strong>Automate Regression Testing</strong></p><p>Once you fix a bug, create a test case that reproduces it. Use test frameworks that can simulate both FIX clients and servers, sending specific message sequences to verify your application handles edge cases correctly.</p><p>A regression test for sequence number recovery:</p><blockquote>def test_sequence_gap_recovery():<br> # Send messages 1, 2, 3<br> # Skip message 4 <br> # Send message 5<br> # Verify ResendRequest for message 4<br> # Resend message 4 with PossDupFlag=Y<br> # Verify session continues normally</blockquote><p><strong>Use Heartbeats Strategically</strong></p><p>Heartbeat(MsgType=0) messages maintain connection integrity during idle periods. HeartBtInt field in the logon message defines the interval. If you don’t receive any message within HeartBtInt seconds, send a TestRequest(MsgType=1). If the counterparty doesn’t respond within a reasonable time, consider the connection dead.</p><p><strong>Handle Resend Requests Carefully</strong></p><p>When you receive ResendRequest(MsgType=2), decide whether to retransmit each message. Market data from hours ago is meaningless. Orders might no longer be relevant given changed market conditions. You can skip messages using SequenceReset-GapFill(MsgType=4) with GapFillFlag(123)=Y.</p><p>Example: counterparty requests messages 100–150. You decide messages 100–120 are stale:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KjSK4tCwMEGjMsK7SAqigA.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNoNzT0KwkAQBtC7bK1hZnYmJgtTCK5ioUVMoYiFSBTBn2IRBD%2Fvru8C7xPO1%2FdqKOV4GUpIofH5cltppWi9EURzhba%2ByetZ7mC199NukXtEdSaCiQuJMbGOhZJaYgFL9B1i7SwMJueJIozC6Xl73R%2F%2FZH%2F4%2FgDgSiD%2F">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=82|35=4|49=SENDER|56=TARGET|34=100|52=20251014–20:45:12|123=Y|36=121|10=174|</blockquote><p>This tells the counterparty to reset their expected sequence number to 121, skipping 100–120. Then transmit messages 121–150 normally with PossDupFlag(43)=Y.</p><p><strong>Monitor SendingTime Accuracy</strong></p><p>SendingTime(52) must be accurate. Some counterparties reject messages when SendingTime deviates too far from actual time. If your system clock drifts, you’ll see SessionRejectReason(373)=10 (SendingTime accuracy problem).</p><p><strong>Cross-Reference Messages</strong></p><p>When debugging complex scenarios, you need to trace the lifecycle of an order across multiple messages. An order starts with NewOrderSingle(MsgType=D), receives ExecutionReport(MsgType=8) acknowledgments, might get modified via OrderCancelReplaceRequest(MsgType=G), and eventually fills with more ExecutionReports.</p><p>Each message contains ClOrdID(11) linking them together. Being able to filter your logs by ClOrdID and see the complete order history in chronological order is invaluable. Visual tools that highlight the same tag value across different messages make this analysis much faster than grepping through raw logs.</p><h3>Recovery Strategies</h3><p><strong>Infinite Resend Loops<br></strong>A particularly nasty scenario occurs when a garbled message keeps getting retransmitted. Counterparty A can’t parse message 100 and ignores it without incrementing sequence numbers. When it receives message 101, it detects a gap and requests resend of message 100. Counterparty B retransmits the same garbled message. The loop continues indefinitely.</p><p>The FIX specification recommends implementing logic to detect this pattern. After several failed attempts to process the same message, terminate the session and investigate manually.</p><p><strong>State Mismatch After Recovery<br></strong>After a network disruption and session recovery, your application’s internal state might not match the counterparty’s view. Perhaps an ExecutionReport confirming a fill was sent but never acknowledged. Your system thinks the order filled; the counterparty thinks it’s still working.</p><p>Implement state reconciliation mechanisms. Some venues provide mass status requests or snapshot messages that let you synchronize state after recovery. Use these actively rather than assuming your cached state is correct.</p><h3>Conclusion</h3><p>FIX protocol debugging rewards systematic thinking. When issues arise, start at the session layer — verify connectivity, check sequence numbers, validate message structure. Only then move to application-level business logic. Keep detailed logs, maintain accurate data dictionaries, and invest in tools that help you visualize message flows.</p><p>The patterns repeat. Sequence number mismatches, garbled messages due to SOH handling, required field violations, and incorrect data types account for the vast majority of issues you’ll encounter. Master these fundamentals, and you’ll spend less time fighting fires and more time building robust trading systems.</p><p>Remember that FIX is forgiving in structure but unforgiving in sequencing. A single missed message can cascade into session failures if not handled properly. Test your error handling as thoroughly as your happy paths, because production environments will find every edge case you missed in development.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19133ed8e427" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FIX Protocol for Market Data Feeds: A Comprehensive Developer’s Guide]]></title>
            <link>https://medium.com/@mark.andreev/fix-protocol-for-market-data-feeds-a-comprehensive-developers-guide-b3ffc7a82d25?source=rss-5001da3331c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3ffc7a82d25</guid>
            <category><![CDATA[fix-protocol]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Mark Andreev]]></dc:creator>
            <pubDate>Mon, 13 Oct 2025 19:51:47 GMT</pubDate>
            <atom:updated>2025-10-13T19:51:47.779Z</atom:updated>
            <content:encoded><![CDATA[<p>The Financial Information eXchange (FIX) protocol has become the industry standard for electronic trading communication since its inception in 1992. Originally developed to replace phone-based communications between Fidelity Investments and Salomon Brothers, FIX now facilitates millions of transactions daily across global financial markets.</p><p>This comprehensive guide explores how FIX protocol enables efficient market data feeds. Market data distribution is one of the most commonly implemented applications of FIX, allowing trading systems to receive real-time price updates, order book snapshots, and trade information from exchanges and market data providers.</p><h3>Understanding FIX Protocol Structure</h3><p>FIX messages follow a standardized tag-value format where each field is identified by a numeric tag, followed by an equals sign and the field value, terminated by the ASCII Start of Header (SOH) character (0x01). This structure ensures reliable parsing and processing across different systems and platforms.</p><h4>Standard Message Components</h4><p>Every FIX message consists of three essential parts:</p><p><strong>Standard Header</strong>: Contains message identification, routing, and control information<br><strong>Message Body</strong>: Includes application-specific data fields<br><strong>Standard Trailer</strong>: Provides message integrity validation</p><p>The fundamental structure follows this pattern:</p><blockquote>8=FIX.4.4|9=XXX|35=Y|…message body…|10=XXX|</blockquote><p>Where vertical bars (|) represent the non-printable SOH delimiter in actual messages.</p><h4>Critical Header Fields</h4><p>- <strong>BeginString</strong> (8): Protocol version identifier (e.g., “FIX.4.4”)<br>- <strong>BodyLength</strong> (9): Message length in bytes, excluding header and trailer<br>- <strong>MsgType</strong> (35): Defines the message type and purpose<br>- <strong>SenderCompID</strong> (49): Identifies the message originator<br>- <strong>TargetCompID</strong> (56): Specifies the intended recipient<br>- <strong>MsgSeqNum</strong> (34): Sequential message numbering for session management<br>- <strong>SendingTime</strong> (52): Message transmission timestamp</p><h3>Market Data Message Types</h3><p>FIX protocol defines several message types specifically for market data distribution:</p><h4>Market Data Request (MsgType=V)</h4><p>The Market Data Request message initiates data subscriptions and specifies what market information to receive. This is a general request for market data on specific securities, and a successful request returns one or more Market Data messages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IB6cDumq6pkVjIWnH6uIfA.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNodzksKwkAQBNC7zFrCdE%2F3mAR6ETRCIBGJEpTgQiSK4GcRBMEWvIM39CSO2TyK2lQ9zeH0qLq%2B3x273qQmllmxjigiTQRiVsfSKCUyKYt8vlL2sszrJq%2FVkRAro6BFAgv4fX%2BAUmdT5AjQKXqUalpbCyE6%2BUtigzxkL5vgWDCYDH0YVCAfZJYsW5QKVtCRmpHZ3873yzUcbLevH7TLMdE%3D">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=185|35=V|49=CLIENT|56=SERVER|34=45|52=20241012–14:30:25.123|262=MDR001|263=1|264=0|265=1|266=Y|267=2|269=0|269=1|146=1|55=AAPL|10=234|</blockquote><p><strong>Key fields breakdown</strong>:<br>- <strong>MDReqID</strong> (262): “MDR001” — Unique request identifier<br>- <strong>SubscriptionRequestType</strong> (263): “1” — Snapshot + Updates subscription<br>- <strong>MarketDepth</strong> (264): “0” — Full depth of book<br>- <strong>MDUpdateType</strong> (265): “1” — Incremental refresh updates<br>- <strong>AggregatedBook</strong> (266): “Y” — Level 2 aggregated book data<br>- <strong>NoMDEntryTypes</strong> (267): “2” — Number of entry types requested<br>- <strong>MDEntryType</strong> (269): “0”, “1” — Bid and Offer entries<br>- <strong>NoRelatedSym</strong> (146): “1” — Number of symbols<br>- <strong>Symbol</strong> (55): “AAPL” — Apple Inc. stock symbol</p><h4>Market Data Snapshot Full Refresh (MsgType=W)</h4><p>This message provides complete order book snapshots in response to subscription requests. The Market Data Snapshot message is used as the response to a Market Data Request, and in all cases, one message refers only to one request:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwQcisMqsjfrcefT-QOqTQ.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNqV0M1qAkEMAOB3mXNZkkwy%2BwM5LO0WBBVZpRXEg8i2CFYPS6HQCL6Db%2BiTOLqUhR5avIQk5Esg3%2B5t8zVq2nb13rSucJk%2BD%2BYJJ2y5eiTzoq%2FGuU6r%2BqWqTYI%2BDgfVeGaeNaQmpATECEjn4wm58FCQJCzBKJCOnmoANBEty8kwtjLlGHMFoxQUBRKSmKIiwLXXr4uF15%2BFvw13Rv412JtwM9k9JO3O0J8Gr8PB3INb77efH7v4xcXycAEtRlaw">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=312|35=W|49=SERVER|56=CLIENT|34=67|52=20241012–14:30:25.456|262=MDR001|55=AAPL|268=4|269=0|270=150.25|271=1000|272=20241012|273=14:30:25|269=0|270=150.24|271=1500|272=20241012|273=14:30:25|269=1|270=150.26|271=800|272=20241012|273=14:30:25|269=1|270=150.27|271=1200|272=20241012|273=14:30:25|10=156|</blockquote><p><strong>Field explanations</strong>:<br>- <strong>MDReqID</strong> (262): “MDR001” — References original request<br>- <strong>Symbol</strong> (55): “AAPL” — Security identifier<br>- <strong>NoMDEntries</strong> (268): “4” — Four market data entries<br>- <strong>MDEntryType</strong> (269): “0” (Bid), “1” (Offer) — Entry type indicators<br>- <strong>MDEntryPx</strong> (270): “150.25”, “150.24”, “150.26”, “150.27” — Price levels<br>- <strong>MDEntrySize</strong> (271): Quantities at each price level<br>- <strong>MDEntryDate</strong> (272): “20241012” — Trade date<br>- <strong>MDEntryTime</strong> (273): “14:30:25” — Entry timestamp</p><p>The snapshot message can transmit a 2-sided book of orders or list of quotes, a list of trades, index values, opening, closing, settlement, high, low, or VWAP prices, or any combination of these.</p><h4>Market Data Incremental Refresh (MsgType=X)</h4><p>Incremental updates deliver real-time changes to previously sent market data. These messages are sent after a snapshot to provide updates as they occur:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fhIWmPLHGePYUmpGC3cCyw.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNp90M0KwjAMB%2FB36VlGkrZbO8hBdIKgIlNEEA8iKoIfhyEIRvAdfEOfxDhQD4KX8E8TfoFezHp77q%2BqarFZVSY3gTvdaeISJ5HJebGep%2BIij4pyUpTiU271usVgLNZxGsQTE5BDQHrc7uhyCzmlSRaiUErcb5cAqDEwCWWRQXNds6DD18x7bjaHPX0BRg8JWY2oKry2vrw2lt8HagtrCz8W%2FVqhtuJfCnU3BjENszzuTvuD%2FsJsfn0COA9JKw%3D%3D">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=245|35=X|49=SERVER|56=CLIENT|34=68|52=20241012–14:30:26.789|262=MDR001|268=2|279=0|269=0|278=MD001|55=AAPL|270=150.23|271=2000|272=20241012|273=14:30:26|279=1|269=1|278=MD002|55=AAPL|270=150.28|271=900|272=20241012|273=14:30:26|10=198|</blockquote><p><strong>Update actions</strong>:<br>- <strong>MDUpdateAction</strong> (279): “0” (New), “1” (Change), “2” (Delete)<br>- <strong>MDEntryID</strong> (278): “MD001”, “MD002” — Unique entry identifiers<br>- First entry: New bid at $150.23 for 2000 shares<br>- Second entry: Changed offer to $150.28 for 900 shares</p><h4>Market Data Request Reject (MsgType=Y)</h4><p>This message is sent to reject a Market Data Request or indicate that market data will no longer be available for a subscription:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lb1sfkr_opdb30l2x-H3Ag.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNoNzksKwkAMANCryKylTNKkdgpZaYWCdVE%2FKOJCpYrYTheDqBjBO3hDT6LvBO9pjud7WYewO9XBZCaVcbGKKCJ1AsQas6yVnMzyaplXyokMJ0U%2BnWtMMrDKKGiRwAJ%2B3x%2BgLLYZcuScU0xQylFlLSmmIKCcysJffHfzvfBo912jYAU4UdM3h665tv4f2GxfP2WBK5Q%3D">View in FixPaser</a></figcaption></figure><blockquote>8=FIX.4.4|9=145|35=Y|49=SERVER|56=CLIENT|34=70|52=2024101214:30:25.999|262=MDR004|281=1|58=Unknown symbol|10=156|</blockquote><p><strong>MDReqRejReason</strong> (281) values:<br>- “0”: Unknown symbol<br>- “1”: Duplicate MDReqID<br>- “4”: Unsupported SubscriptionRequestType<br>- “5”: Unsupported MarketDepth<br>- “6”: Unsupported MDUpdateType</p><h3>Advanced Market Data Features</h3><h4>Multi-Symbol Subscriptions</h4><p>FIX supports subscribing to multiple instruments in a single request:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zDWrfkgR2RqJ23vTdVooig.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNodzksKwkAQBNC7zFrCdE%2F35AO9CBpDIFGJIoq4EFER%2FCyCINiCd%2FCGnsQkm1cFtamXOZ6f1aFpdqdDYxITybhYBRSQxoJhpI5lqRTLsCyyyULZyzyrl1mtjoS8MgpaJLCAv88XKHGcAAfoSNGjVKPaWmyrE2ilXu71sm4NpVtjsb2gQF6cMkuazsouq%2Fl40WU%2BnealghVAp2Zg9vfL43prH2%2B27z8vvzbV">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=278|35=V|49=CLIENT|56=SERVER|34=46|52=20241012–14:35:15.234|262=MDR002|263=1|264=1|265=1|266=Y|267=2|269=0|269=1|146=3|55=AAPL|55=MSFT|55=GOOGL|10=123|</blockquote><p>This request subscribes to top-of-book data for Apple, Microsoft, and Google simultaneously.</p><h4>Trade Data Subscription</h4><p>Subscribe to executed trades alongside order book data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rq_pyTH8wXYsZX2LqQjEBQ.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNodjk0KwjAQRu%2BStZSZZJI0hVmIVii0LtpSFHEhUkXwZ1EEwRG8gzf0JMZuHo%2F3bb6nOpweVT8Mu2M%2FqEylvChWCSUkgTGkYix3QoFnZZEvW7GOm7zu8loMMXmxmjVoQkD9fX%2BQMoIMIbHOi3aaq3kNYKIaxkhiiLSjO15Hev6vYexh7IG1ILno1nLblFNBYEiDqIna3873yzXe3GxfP%2BcgM4s%3D">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=198|35=V|49=CLIENT|56=SERVER|34=47|52=20241012–14:40:10.567|262=MDR003|263=1|264=0|265=1|266=Y|267=3|269=0|269=1|2692|146=1|55=TSLA|10=089|</blockquote><p><strong>MDEntryType values</strong>:<br>- “0”: Bid<br>- “1”: Offer <br>- “2”: Trade</p><h4>Trade Execution Updates</h4><p>Trade data appears in incremental refresh messages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kal3eO0Ph7uBYcz7FLLewg.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNpFjs0KwkAMhN9lz1KS7G7%2FIAexFQqth20RQTyIVBH8ORRBMILv4Bv6JKYH8TTfhGEmD7M%2F3pt%2BGLaHfjC5SXlerSIXOcmYAMV6XonLuC3DsgziY57VVbnoxDqOM%2FGkKXIISJ%2FXG13uIEeMkKxQTNwUAWDElFEoyRiUtVg55S4UoAvec9fWUz0Bk%2FOR10yC7GHUf70ay78BQWCMEzETs7uebueLvr7ePL%2FJCDc0">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=201|35=X|49=SERVER|56=CLIENT|34=69|52=20241012–14:40:11.123|262=MDR003|268=1|279=0|269=2|278=TRD001|55=TSLA|270=245.50|271=500|272=20241012|273=14:40:11|10=167|</blockquote><p>This indicates a trade execution at $245.50 for 500 shares of Tesla.</p><h3>Aggregated vs Non-Aggregated Books</h3><p>Market data can be provided in two forms:</p><p><strong>Aggregated Book</strong>: Price levels show combined volume from multiple orders at the same price<br><strong>Non-Aggregated Book</strong>: Individual orders are shown separately with unique identifiers</p><p>The AggregatedBook field (266) in the Market Data Request specifies which format to receive.</p><h3>Subscription Management</h3><h4>Snapshot vs Snapshot + Updates</h4><p>FIX supports different subscription models:</p><p><strong>Snapshot Only</strong>: Receive current market state once without ongoing updates<br><strong>Snapshot + Updates</strong>: Receive current state followed by real-time incremental updates</p><p>When Snapshot + Updates is requested, the server first sends a full refresh, then continues sending incremental updates until the subscription is disabled.</p><h3>Unsubscribing from Market Data</h3><p>To stop receiving updates, send a request with SubscriptionRequestType=2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-m9U7c_cgMUnYkND4yen9w.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNoNzbEKwjAQANB%2FySzh7nIJbeAmjVCwDlGKIA4iVYRWhyAInuA%2F%2BId%2Bid3e9l7mfH22fSnHS19MNJUsm51ly1oL%2BqDOS6dcy3zVpPVWfZBNyl3K6li4Uk9CQIyA9Pt8kSP7CGABQCmQtIsMgBOdkCIIOVYzM6f78BhvU7c%2FvP9yUySr">View in FixParser</a></figcaption></figure><blockquote>8=FIX.4.4|9=156|35=V|49=CLIENT|56=SERVER|34=48|52=20241012–14:45:00.000|262=MDR001|263=2|10=234|</blockquote><p>This unsubscribes from the previous Snapshot + Updates request identified by MDReqID.</p><h3>Market Data Implementation Best Practices</h3><h4>Connection Isolation</h4><p>Always use separate connections for market data versus other trading activities. This isolation prevents data surges from impacting time-sensitive operations and ensures stable system performance.</p><h4>Subscription Optimization</h4><p><strong>Batch Subscriptions</strong>: Group multiple symbol subscriptions in one request (up to 500 channels where supported) for efficiency[19].<br><strong>Filter Aggressively</strong>: Subscribe only to the data needed, avoiding wildcards that deliver unnecessary information.<br><strong>Use Aggregated Data</strong>: When full order details aren’t required, opt for aggregated book data to reduce message volume.</p><h3>Message Sequencing</h3><p>FIX market data implementations must handle message sequencing properly:</p><p><strong>Sequence Numbers</strong>: Every message includes MsgSeqNum (34) for ordering<br><strong>Gap Detection</strong>: Missing sequence numbers indicate potential data loss<br><strong>Duplicate Handling</strong>: PossDupFlag (43) indicates potential duplicate messages.</p><h3>Performance Optimization</h3><p><strong>Efficient Parsing</strong>: Pre-compile field dictionaries for faster message processing<br><strong>Memory Management</strong>: Reuse message objects to minimize garbage collection overhead<br><strong>Raw vs Aggregated Updates</strong>: Subscribe to raw feeds only when every tick is needed; otherwise use aggregated updates</p><h3>Real-World Market Data Flow</h3><p>A typical market data session follows this pattern:</p><p>1. <strong>Session Establishment</strong>: Logon message exchange with authentication<br>2. <strong>Market Data Request</strong>: Subscribe to required instruments and data types<br>3. <strong>Initial Snapshot</strong>: Receive current market state via Snapshot Full Refresh<br>4. <strong>Incremental Updates</strong>: Process real-time changes via Incremental Refresh<br>5. <strong>Heartbeats</strong>: Maintain session connectivity with regular heartbeat messages<br>6. <strong>Subscription Changes</strong>: Add/remove instruments or modify data types<br>7. <strong>Unsubscribe</strong>: Disable subscriptions when no longer needed<br>8. <strong>Session Termination</strong>: Clean logout with proper sequence number handling</p><h3>Testing and Validation</h3><h4>Testing Scenarios</h4><p><strong>Tailor Testing to Market Conditions.</strong> Simulate high-volume trading days, news events, and market openings to understand system response[23].<br><strong>Automate Testing</strong>: Leverage automated tools to increase testing scope and frequency.<br><strong>Stress Testing</strong>: Evaluate performance under extreme message traffic conditions.<br><strong>Session Layer Testing</strong>: Ensure seamless connection handling and efficient message traffic management.</p><h4>Performance Testing</h4><p><strong>Latency Measurement</strong>: Track end-to-end message processing times from receipt to application<br><strong>Throughput Testing</strong>: Validate handling of peak message volumes during active trading<br><strong>Memory Profiling</strong>: Monitor memory usage under sustained market data loads<br><strong>Connection Stability</strong>: Test recovery mechanisms with simulated connection failures</p><h4><strong>Practical Example: Complete Market Data Session</strong></h4><p>Here’s a complete example showing a typical market data session from connection to data reception:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0dZuW6bMh4CSU3tN-FtM6w.png" /><figcaption><a href="https://fixparser.mrkandreev.name/#/?state=eNqVk91Kw0AQhd8l1xJmZn%2ByG5gbbYSACqa2tqgXIlUErRdBEFzBd%2FANfRIn24Q2obUawjAJnD3fnEzek%2FvHt9NFXd8%2BLOokTxwfl7NUpzp4RmeCMjwN2vPRSVmcXQRjeVxU06IKSrM2wRATkEZA9f35RZCjyRWkSCqQJa6KcwCUVnFTNYNUE3vLc6kZk1Qf34thQG2lGsPFpJqMRwGBSelwvdzkUkgN12XD1dIIV0soXDbbxaWN3eRa%2B5B1rDuSDBhTcKZpkRHiJf36THlQ3J06kGm%2Fkpm%2FyHDthlHm%2FqmilRntk0mQKMP3gyTJQIKc7QzSbQ8S08z5%2Fgd2ESqitWk4PixHg5T7s9KeZDFSezegbub4dS23UMsNkEaz%2FlpSt2LJQXL38vT6vJS%2F4Orm4wdIrsv8">View in FixParser</a></figcaption></figure><blockquote><strong>Step 1: Subscribe to Market Data</strong><br>8=FIX.4.4|9=185|35=V|49=CLIENT|56=SERVER|34=45|52=20241013–20:15:30.123|262=REQ001|263=1|264=0|265=1|266=Y|267=2|269=0|269=1|146=1|55=EURUSD|10=234|</blockquote><blockquote><strong>Step 2: Receive Initial Snapshot<br></strong>8=FIX.4.4|9=312|35=W|49=SERVER|56=CLIENT|34=67|52=20241013–20:15:30.456|<br>262=REQ001|55=EURUSD|268=4|269=0|270=1.0850|271=1000000|<br>272=20241013|273=20:15:30|269=0|270=1.0849|271=1500000|<br>272=20241013|273=20:15:30|269=1|270=1.0851|271=800000|<br>272=20241013|273=20:15:30|269=1|270=1.0852|271=1200000|<br>272=20241013|273=20:15:30|10=156|</blockquote><blockquote><strong>Step 3: Receive Incremental Updates</strong><br>8=FIX.4.4|9=201|35=X|49=SERVER|56=CLIENT|34=68|52=20241013–20:15:31.789|<br>262=REQ001|268=1|279=1|269=0|278=BID001|55=EURUSD|270=1.0851|<br>271=2000000|272=20241013|273=20:15:31|10=198|</blockquote><blockquote><strong>Step 4: Unsubscribe When Done</strong><br>8=FIX.4.4|9=156|35=V|49=CLIENT|56=SERVER|34=48|52=20241013–20:20:00.000|<br>262=REQ001|263=2|10=234|</blockquote><h3>Conclusion</h3><p>FIX protocol provides a robust foundation for market data distribution in electronic trading systems. Its standardized message structure, comprehensive error handling, and flexible subscription model make it ideal for receiving real-time market data from exchanges and data providers.</p><p>Understanding market data-specific message types — from Market Data Requests to Incremental Refresh updates — enables developers to build reliable, efficient market data consumption systems. The examples provided in this guide demonstrate real-world usage patterns for subscribing to market data, processing snapshots and updates, and managing subscriptions effectively.</p><p>For testing and validation of your FIX market data implementations, tools like <a href="https://fixparser.mrkandreev.name/">fixparser.mrkandreev.name</a> provide essential capabilities for message parsing, validation, and debugging, ensuring your systems comply with FIX protocol specifications and perform reliably in production environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3ffc7a82d25" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What the difference between currentTimeMillis and nanoTime in Java?]]></title>
            <link>https://medium.com/@mark.andreev/what-the-difference-between-currenttimemillis-and-nanotime-in-java-f9658968b21a?source=rss-5001da3331c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9658968b21a</guid>
            <category><![CDATA[time]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[cpp]]></category>
            <dc:creator><![CDATA[Mark Andreev]]></dc:creator>
            <pubDate>Wed, 01 May 2024 15:45:43 GMT</pubDate>
            <atom:updated>2024-05-01T15:45:43.249Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*aIH9O_4f0ChavcYR3pQ0VA.png" /></figure><p>There are two similar methods in Java: System.currentTimeMillis() and System.nanoTime() that relate to time measurement. CurrentTimeMillis returns the current time in milliseconds from the Epoch (January 1, 1970, 00:00:00 GMT) and nanoTime returns a nanosecond-precision time related to some elapsed time. So you can measure time intervals using both methods but estimate offset related to common begin only with System.currentTimeMillis() . Let’s go deeper..</p><p>Lets specify levels of difference:</p><ul><li><strong>Estimated value</strong> | Epoch time vs Escaped time</li><li><strong>Estimation unit</strong> | Millis vs Nanos</li><li><strong>Clock type</strong> | Realtime vs Monotonic</li></ul><p>Under the hood HotSpot JVM (for POSIX) use clock_gettime (<a href="https://linux.die.net/man/3/clock_gettime">https://linux.die.net/man/3/clock_gettime</a>) with two different arguments:</p><ul><li><strong>CLOCK_REALTIME </strong>for System.currentTimeMillis()</li><li><strong>CLOCK_MONOTONIC</strong> for System.nanoTime()</li></ul><pre>jlong os::javaTimeMillis() {<br>  struct timespec ts;<br>  int status = clock_gettime(CLOCK_REALTIME, &amp;ts);<br>  assert(status == 0, &quot;clock_gettime error: %s&quot;, os::strerror(errno));<br>  return jlong(ts.tv_sec) * MILLIUNITS +<br>    jlong(ts.tv_nsec) / NANOUNITS_PER_MILLIUNIT;<br>}<br><br>jlong os::javaTimeNanos() {<br>  struct timespec tp;<br>  int status = clock_gettime(CLOCK_MONOTONIC, &amp;tp);<br>  assert(status == 0, &quot;clock_gettime error: %s&quot;, os::strerror(errno));<br>  jlong result = jlong(tp.tv_sec) * NANOSECS_PER_SEC + jlong(tp.tv_nsec);<br>  return result;<br>}</pre><p>You can find this in <a href="https://github.com/openjdk/jdk/blob/4f529f8c232b4082aa4aa39766bcf42b09885ee4/src/hotspot/os/posix/os_posix.cpp#L1412">https://github.com/openjdk/jdk/blob/4f529f8c232b4082aa4aa39766bcf42b09885ee4/src/hotspot/os/posix/os_posix.cpp#L1412</a></p><p>As we know the difference between CLOCK_REALTIME and CLOCK_MONOTONIC:</p><blockquote>CLOCK_REALTIME<br>System-wide realtime clock. Setting this clock requires appropriate privileges.</blockquote><blockquote>CLOCK_MONOTONIC<br>Clock that cannot be set and represents monotonic time since some unspecified starting point.<br>[<a href="https://linux.die.net/man/3/clock_gettime">Linux Die Net</a>]</blockquote><p>And also:</p><blockquote>Another difference is CLOCK_REALTIME can jump forward or backward according to NTP. By default, NTP allows the clock rate to be speeded up or slowed down by up to 0.05%, but NTP cannot cause the monotonic clock to jump forward or backward.<br>[<a href="https://stackoverflow.com/a/65875696/1966494">StackOverflow</a>]</blockquote><p>What is more important is that CLOCK_MONOTONIC are guaranteed to be monotonic only on the same core.</p><p>Mark Andreev, SWE @ Conundrum AI</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9658968b21a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to write Dockerfile for Rust project]]></title>
            <link>https://medium.com/@mark.andreev/how-to-write-dockerfile-for-rust-project-f5446f4b2bcb?source=rss-5001da3331c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5446f4b2bcb</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[dockerfiles]]></category>
            <dc:creator><![CDATA[Mark Andreev]]></dc:creator>
            <pubDate>Fri, 26 Jan 2024 14:35:52 GMT</pubDate>
            <atom:updated>2024-01-26T14:35:52.652Z</atom:updated>
            <content:encoded><![CDATA[<p>Rust is a powerful language that allows you to build native binary. Docker multi-stage builds separate compile and runtime stages. In the compile stage you can install all dependencies that you need for compilation. In the runtime stage you need only executable binary with dependent libs without a compiler.</p><p>As result you can use Alpine Linux as base image for runtime. It is allowed to minimise image size that increases microservice time to start because image pull operation goes faster for smaller images.</p><p>For alpine we have to use Rust Cross compile build with target x86_64-unknown-linux-musl and install musl-tools to build the binary.</p><pre>FROM rust:1.75.0-buster as builder<br>WORKDIR /opt/app<br><br>RUN apt-get update -qq &amp;&amp; apt-get install -yqq musl-tools<br>COPY src/ src/<br>COPY Cargo.toml .<br>COPY Cargo.lock .<br>RUN rustup target add x86_64-unknown-linux-musl &amp;&amp; \<br>  cargo build --target x86_64-unknown-linux-musl --release<br><br>FROM alpine:3.19.0<br><br>COPY --from=builder /opt/app/target/x86_64-unknown-linux-musl/release/medium_f5446f4b2bcb /usr/local/bin<br><br>ENTRYPOINT [&quot;/usr/local/bin/medium_f5446f4b2bcb&quot;]</pre><p>Result docker images size = 21.1MB</p><p>Repository: <a href="https://github.com/mrk-andreev/medium_f5446f4b2bcb">https://github.com/mrk-andreev/medium_f5446f4b2bcb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5446f4b2bcb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to fix “The default package ‘.’ is not permitted by the Import-Package syntax” in OSGI]]></title>
            <link>https://medium.com/@mark.andreev/how-to-fix-the-default-package-is-not-permitted-by-the-import-package-syntax-in-osgi-3b59a6c18e71?source=rss-5001da3331c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b59a6c18e71</guid>
            <category><![CDATA[maven]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[osgi]]></category>
            <dc:creator><![CDATA[Mark Andreev]]></dc:creator>
            <pubDate>Tue, 16 Jan 2024 10:09:10 GMT</pubDate>
            <atom:updated>2024-01-16T10:09:10.529Z</atom:updated>
            <content:encoded><![CDATA[<p>When you build your OSGI project with “org.apache.felix:maven-bundle-plugin” you can found error like this:</p><blockquote>[ERROR] Bundle {groupId}:{artifactId}:bundle:{version} : The default package ‘.’ is not permitted by the Import-Package syntax.<br> This can be caused by compile errors in Eclipse because Eclipse creates<br>valid class files regardless of compile errors.<br>The following package(s) import from the default package null<br>[ERROR] Error(s) found in bundle configuration</blockquote><p>That means that one of your dependent packages places their class in root (default package). Unfortunately the error message describes that issue exists but doesn’t tell the source of the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/333/1*JZk23NdvMghWP8cdISUIPg.png" /><figcaption>Example Jar with classes in default package</figcaption></figure><p>You can debug this on one’s own. Add “org.apache.felix:maven-bundle-plugin” as a dependency with scope=test. That will allow you to create the break point.</p><p>Add a conditional breakpoint into “aQute.bnd.osgi.Packages” class in all places where values inserts into “Map&lt;PackageRef, Attrs&gt; map” with condition that key contains “Descriptors.DEFAULT_PACKAGE”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*EPlW9lc9p2gvnihi3iTbNw.png" /><figcaption>How to create conditional breakpoint</figcaption></figure><p>When this breakpoint is triggered go to frame “analyzeJar” and look at the resource that represents the jar with the default package.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vd8bbeRKj0zkpMm-vxVtwQ.png" /><figcaption>How to find resource with class in default package</figcaption></figure><p>You have to replace dependency with another version that doesn’t contain a class inside the default package. In my case updating the library “dnsjava:dnsjava” to the latest version fixed the problem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b59a6c18e71" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Find the Number of Occurrences of a Substring in a String (JMH based Comparison)]]></title>
            <link>https://medium.com/@mark.andreev/find-the-number-of-occurrences-of-a-substring-in-a-string-d924cf933fc3?source=rss-5001da3331c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d924cf933fc3</guid>
            <category><![CDATA[jmh]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Mark Andreev]]></dc:creator>
            <pubDate>Wed, 17 May 2023 07:31:40 GMT</pubDate>
            <atom:updated>2023-05-17T17:05:40.886Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ir3RUZHtWZBk4j6KyGWLog.jpeg" /></figure><p>In this post we will solve the problem “<a href="https://stackoverflow.com/questions/767759/find-the-number-of-occurrences-of-a-substring-in-a-string">Find the Number of Occurrences of a Substring in a String</a>” using different approaches. More importantly we will measure solution performance using <a href="https://github.com/openjdk/jmh">JMH</a>. So you can consider this post as a JMH example project.</p><p>We will start with interface declaration and unit tests that will describe our algorithm cases:</p><pre>public interface SubstringCounter {<br>    int count(String string, String substring);<br>}<br><br><br>class SubstringCounterTest {<br>    @ParameterizedTest<br>    @MethodSource(&quot;getImpls&quot;)<br>    void shouldReturnZeroIfNoSubstringPresent(SubstringCounter counter) {<br>        assertEquals(0, counter.count(&quot;abcd&quot;, &quot;x&quot;));<br>    }<br><br>    @ParameterizedTest<br>    @MethodSource(&quot;getImpls&quot;)<br>    void shouldReturnOneIfStringEqualsToString(SubstringCounter counter) {<br>        assertEquals(1, counter.count(&quot;abcd&quot;, &quot;abcd&quot;));<br>    }<br><br>    @ParameterizedTest<br>    @MethodSource(&quot;getImpls&quot;)<br>    void shouldReturnOneIfStringContainsSingleSubstring(SubstringCounter counter) {<br>        assertEquals(1, counter.count(&quot;abcd&quot;, &quot;bc&quot;));<br>    }<br><br>    @ParameterizedTest<br>    @MethodSource(&quot;getImpls&quot;)<br>    void shouldReturnMultipleSubstring(SubstringCounter counter) {<br>        assertEquals(3, counter.count(&quot;abxxabxab&quot;, &quot;ab&quot;));<br>    }<br><br>    static Stream&lt;Arguments&gt; getImpls() {<br>        return Stream.of(<br>                Arguments.of(new IndexOfSubstringCounterImpl())<br>                // ...<br>        );<br>    }<br>}</pre><p>As you can see we use @ParameterizedTest for inject custom implementation. That allows us to add more implementation without copy pasting test cases.</p><h3>Benchmark</h3><p>We will use <a href="https://github.com/openjdk/jmh">JMH</a> to write new test cases. You can explore samples in the official repository (<a href="https://github.com/openjdk/jmh/tree/master/jmh-samples">https://github.com/openjdk/jmh/tree/master/jmh-samples</a>). Theoretical understanding of microbenchmarks is well described in Aleksey Shipilëv’s blog (<a href="https://shipilev.net/">https://shipilev.net/</a>).</p><p>In SubstringCounterBenchmark we use @Param annotation that contains all implementations of SubstringCounter that we will benchmark. For setup benchmark we use @Setup annotation that is equivalent of @BeforeEach in JUnit5 framework. We mark the measured method using @Benchmark annotation. To prevent dead code elimination we use Blackhole that consume output of the counter and processes it in a specific manner. @BenchmarkMode specify benchmark measurement type: Throughput (ops/time), AverageTime (time/op), SampleTime (Sampling time), SingleShotTime (Single shot invocation time).</p><p>We also specify @Warmup for warmup JVM that causes JVM <a href="https://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/underst_jit.html">runtime optimizations</a> and @Measurement that specify measurable iterations.</p><pre>@BenchmarkMode(Mode.Throughput)<br>@OutputTimeUnit(TimeUnit.MILLISECONDS)<br>@State(Scope.Benchmark)<br>@Warmup(iterations = 5, time = 10, timeUnit = SECONDS)<br>@Measurement(iterations = 10, time = 60, timeUnit = SECONDS)<br>public class SubstringCounterBenchmark {<br>    public SubstringCounter counter;<br><br><br>    @Param({<br>        &quot;IndexOfSubstringCounterImpl&quot;<br>        // ...<br>    })<br>    public String impl;<br><br>    @Setup<br>    public void prepare() throws Exception {<br>        counter = (SubstringCounter) getClass().getClassLoader().loadClass(&quot;name.mrkandreev.code.&quot; + impl).getDeclaredConstructor().newInstance();<br>    }<br><br>    @Benchmark<br>    public void bench(Blackhole blackhole) {<br>        blackhole.consume(counter.count(&quot;abxxabxab&quot;, &quot;ab&quot;));<br>    }<br><br>    public static void main(String[] args) throws RunnerException {<br>        Options opt = new OptionsBuilder().include(SubstringCounterBenchmark.class.getSimpleName()).forks(1).build();<br><br>        new Runner(opt).run();<br>    }<br>}</pre><h3>Implementations</h3><p>In real world implementation we have to check input: nullable or empty. But in this implementation it is out of scope.</p><pre>public int count(String string, String substring) {<br>    if (string == null || string.length() == 0 <br>        || substring == null || substring.length() == 0) {<br>    }<br>    // ...<br>}</pre><p>As the worst implementation we can use regular expression that compile in each call.</p><pre>public class RegExSubstringCounterImpl implements SubstringCounter {<br>    @Override<br>    public int count(String string, String substring) {<br>        return (int) Pattern.compile(Pattern.quote(substring))<br>                .matcher(string)<br>                .results()<br>                .count();<br>    }<br>}</pre><p>Also we can use one-liner replacements that will produce solutions without loops.</p><pre>public class ReplaceBasedSubstringCounterImpl implements SubstringCounter {<br>    @Override<br>    public int count(String string, String substring) {<br>        return (string.length() - string.replace(substring, &quot;&quot;).length()) / substring.length();<br>    }<br>}</pre><p>And we can use loop over indexOf calls with or without post = idx + substring.length() optimization.</p><pre>public class IndexOfSubstringCounterImpl implements SubstringCounter {<br>    @Override<br>    public int count(String string, String substring) {<br>        int pos = -1;<br>        int count = 0;<br><br>        while ((pos = string.indexOf(substring, pos+1)) != -1) {<br>            count++;<br>        }<br><br>        return count;<br>    }<br>}</pre><pre>public class IndexOfWithJumpSubstringCounterImpl implements SubstringCounter {<br>    @Override<br>    public int count(String string, String substring) {<br>        int count = 0;<br><br>        int idx;<br>        for(int pos = 0; (idx = string.indexOf(substring, pos)) != -1; pos = idx + substring.length()) {<br>            ++count;<br>        }<br><br>        return count;<br>    }<br>}</pre><h3>Benchmark results</h3><pre># JMH version: 1.36<br># VM version: JDK 17.0.7, OpenJDK 64-Bit Server VM, 17.0.7+7-LTS<br># VM options: -javaagent:/opt/jetbrains/idea-IU-213.6777.52/lib/idea_rt.jar=46463:/opt/jetbrains/idea-IU-213.6777.52/bin -Dfile.encoding=UTF-8<br># Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)<br># Warmup: 5 iterations, 10 s each<br># Measurement: 5 iterations, 10 s each<br># Timeout: 10 min per iteration<br># Threads: 1 thread, will synchronize iterations<br># Benchmark mode: Throughput, ops/time</pre><pre>(impl)                                Mode  Cnt      Score     Error   Units<br>IndexOfWithJumpSubstringCounterImpl  thrpt   10  86171.752 ± 225.064  ops/ms<br>IndexOfSubstringCounterImpl          thrpt   10  77560.418 ± 154.745  ops/ms<br>ReplaceBasedSubstringCounterImpl     thrpt   10  29758.761 ±  35.899  ops/ms<br>RegExSubstringCounterImpl            thrpt   10   5121.197 ±  10.030  ops/ms</pre><p>As we can see the best score (in our case it is throughput) is IndexOfWithJumpSubstringCounterImpl:</p><pre>public class IndexOfWithJumpSubstringCounterImpl implements SubstringCounter {<br>    @Override<br>    public int count(String string, String substring) {<br>        int count = 0;<br><br>        int idx;<br>        for(int pos = 0; (idx = string.indexOf(substring, pos)) != -1; pos = idx + substring.length()) {<br>            ++count;<br>        }<br><br>        return count;<br>    }<br>}</pre><h3>References</h3><ul><li><a href="https://stackoverflow.com/questions/767759/find-the-number-of-occurrences-of-a-substring-in-a-string">https://stackoverflow.com/questions/767759/find-the-number-of-occurrences-of-a-substring-in-a-string</a></li><li><a href="https://github.com/mrk-andreev/medium_d924cf933fc3">https://github.com/mrk-andreev/medium_d924cf933fc3</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d924cf933fc3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to write Comparable in Modern Java (Java 8+)]]></title>
            <link>https://medium.com/@mark.andreev/how-to-write-comparable-in-modern-java-java-8-85f8ee25374e?source=rss-5001da3331c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/85f8ee25374e</guid>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Mark Andreev]]></dc:creator>
            <pubDate>Tue, 09 May 2023 14:12:34 GMT</pubDate>
            <atom:updated>2023-05-09T14:12:34.984Z</atom:updated>
            <content:encoded><![CDATA[<p>In Java when we want to add order for objects of our class we have to implement interface Comparable&lt;T&gt; . Typically, in the old days we had to write complex code and remember when answers had to be equals 0, when less than zero and when bigger then zero. But in Java 8+ we can use Comparator.comapringInt/Double/Long for this needs.</p><pre>public record Point(int x, int y) implements Comparable&lt;Point&gt; {<br>    @Override<br>    public int compareTo(Point o) {<br>        if (o == null) {<br>            throw new NullPointerException();<br>        }<br><br>        return Comparator<br>                .comparingInt(Point::x)<br>                .thenComparingInt(Point::y)<br>                .compare(this, o);<br>    }<br>}</pre><p>In this example we use Comparator that compares x and y fields of Point . We can optimise our code using the “Introduce constant” refactoring technique.</p><pre>public record Point(int x, int y) implements Comparable&lt;Point&gt; {<br><br>    public static final Comparator&lt;Point&gt; POINT_COMPARATOR = Comparator<br>            .comparingInt(Point::x)<br>            .thenComparingInt(Point::y);<br><br>    @Override<br>    public int compareTo(Point o) {<br>        if (o == null) {<br>            throw new NullPointerException();<br>        }<br><br>        return POINT_COMPARATOR<br>                .compare(this, o);<br>    }<br>}</pre><p>As well as compareInt we can compare fields with reference typed field:</p><pre>public record ExtendedPoint&lt;T extends Comparable&lt;T&gt;&gt;(T payload, int x, int y)<br>        implements Comparable&lt;ExtendedPoint&lt;T&gt;&gt; {<br>    @Override<br>    public int compareTo(ExtendedPoint&lt;T&gt; o) {<br>        if (o == null) {<br>            throw new NullPointerException();<br>        }<br><br>        return Comparator<br>                .comparingInt(ExtendedPoint&lt;T&gt;::x)<br>                .thenComparingInt(ExtendedPoint::y)<br>                .thenComparing(obj -&gt; payload.compareTo(obj.payload))<br>                .compare(this, o);<br>    }<br>}</pre><p><strong>Summary</strong>. Since Java 1.8 we can use Comparator ‘s methods compare and thenComparing if we would like to simplify compareTo method.</p><p>This example will help you to remember value that compareTo should return:</p><pre>public static int compare(int x, int y) {<br>   return (x &lt; y) ? -1 : ((x == y) ? 0 : 1);<br>}</pre><p>Less -1 | Equals 0 | Bigger 1</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85f8ee25374e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to write tests for PySpark]]></title>
            <link>https://medium.com/@mark.andreev/how-to-write-tests-for-pyspark-38f923dd659c?source=rss-5001da3331c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/38f923dd659c</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Mark Andreev]]></dc:creator>
            <pubDate>Tue, 07 Feb 2023 17:39:06 GMT</pubDate>
            <atom:updated>2023-02-07T17:39:06.188Z</atom:updated>
            <content:encoded><![CDATA[<p>You have to write tests in 2023. This technique lets you check code correctness at an early stage and reduce risk of regress issues. This statement is well known for classic development but what about data processing pipelines?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*STlcHdGUBfsyYKHE.jpg" /></figure><p>We can do the same. At least we can run Spark in local mode without a cluster. Just define a fixture with session scope for performance sense because creation of new context is time expensive.</p><pre>import pytest<br>from pyspark.sql import SparkSession<br><br>@pytest.fixture(scope=&#39;session&#39;)<br>def spark():<br>    return SparkSession.builder.getOrCreate()</pre><p>For example we have functionevaluate_feature_avg_price_per_metchant that evaluates average price for each merchant that is written in pySpark.</p><pre>from pyspark.sql.dataframe import DataFrame<br>from pyspark.sql.functions import avg<br><br><br>def evaluate_feature_avg_price_per_merchant(df: DataFrame) -&gt; DataFrame:<br>    return df.groupBy(&#39;merchant&#39;).agg(avg(&#39;price&#39;).alias(&#39;avg_price&#39;))</pre><p>Using spark fixtures which you can find in previous code snippets we can write a test case that compares two data frames. Remember about order in data frames. You have to specify order in your tests because Spark ignores global order. In assert we call collect that convert data frame to List[Row] this move allows pytest compare dataframe line by line.</p><pre><br>from pyspark.sql import Row<br>from pyspark.sql import SparkSession<br><br><br>def test_evaluate_feature_avg_price_per_merchant(spark: SparkSession):<br>    df = spark.createDataFrame([<br>        Row(merchant=1, price=1.0),<br>        Row(merchant=1, price=2.0),<br>        Row(merchant=2, price=3.0),<br>    ]).orderBy(&#39;merchant&#39;)<br>    df_expected = spark.createDataFrame([<br>        Row(merchant=1, price=1.5),<br>        Row(merchant=2, price=3.0),<br>    ]).orderBy(&#39;merchant&#39;)<br><br>    df_out = evaluate_feature_avg_price_per_merchant(df)<br><br>    assert df_out.collect() == df_expected.collect()<br></pre><p><strong>Note</strong>. Remember that for local pySpark execution you need Java. Set the <a href="https://en.wikipedia.org/wiki/Environment_variable">Environment variable</a> JAVA_HOME before starting your tests.</p><h3>Conclusion</h3><p>You have to write tests for your pySpark application. It is so easy and valuable for your project. I recommend you “<a href="https://stackoverflow.com/questions/67299/is-unit-testing-worth-the-effort">Is Unit Testing worth the effort?</a>” from Stackoverflow that describes the value of tests.</p><p>Github repository with code: <a href="https://github.com/mrk-andreev/medium_38f923dd659c">https://github.com/mrk-andreev/medium_38f923dd659c</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38f923dd659c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to migrate your Python library to R programming language]]></title>
            <link>https://medium.com/@mark.andreev/how-to-migrate-your-python-library-to-r-programming-language-a8a166e564b3?source=rss-5001da3331c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8a166e564b3</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Mark Andreev]]></dc:creator>
            <pubDate>Tue, 07 Feb 2023 07:02:29 GMT</pubDate>
            <atom:updated>2023-02-07T07:02:29.140Z</atom:updated>
            <content:encoded><![CDATA[<p>Migrating a library from one language to another can be a challenge but you can skip a lot of issues when you choose to migrate just bindings and reuse previous codebase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/0*LtYTAkS56UzBlqp8.png" /><figcaption>Reticulate Logo</figcaption></figure><p><a href="https://neptune.ai/">NeptuneAI</a> (ML meta store) decided to migrate their Python SDK to R language in this way. They use <a href="https://rstudio.github.io/reticulate/">Reticulate</a> that provides Python API in your R scripts.</p><p>Lets look at simple reticulate’s code sample that allows you to use functions defined in py-file:</p><pre>## install<br>install.packages(&quot;reticulate&quot;)<br><br>library(reticulate)<br>use_virtualenv(&quot;myenv&quot;)<br><br>## python script (flights.py)<br>import pandas<br>def read_flights(file):<br>  flights = pandas.read_csv(file)<br>  flights = flights[flights[&#39;dest&#39;] == &quot;ORD&quot;]<br>  flights = flights[[&#39;carrier&#39;, &#39;dep_delay&#39;, &#39;arr_delay&#39;]]<br>  flights = flights.dropna()<br>  return flights<br><br>## r script<br>source_python(&quot;flights.py&quot;)<br>flights &lt;- read_flights(&quot;flights.csv&quot;)<br><br>library(ggplot2)<br>ggplot(flights, aes(carrier, arr_delay)) + geom_point() + geom_jitter()</pre><h3>Example migration</h3><p><a href="https://scikit-learn.org/stable/">Scikit-learn</a> is one of the most famous machine library in Python. With Reticulate you can call already written scripts in the R environment. In the next demo we will use a pipeline (preprocessing &amp; estimator) trained on <a href="https://www.kaggle.com/c/titanic">Titanic dataset</a> . Fitted models can be used in the R environment the same way as R other functions.</p><pre>## train.py<br>from sklearn.compose import ColumnTransformer<br>from sklearn.datasets import fetch_openml<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.impute import MissingIndicator<br>from sklearn.pipeline import Pipeline<br>from sklearn.preprocessing import OrdinalEncoder<br><br><br>def train():<br>    X, y = fetch_openml(&quot;titanic&quot;, version=1, as_frame=True, return_X_y=True, parser=&#39;liac-arff&#39;)<br>    X.drop([&#39;boat&#39;, &#39;body&#39;, &#39;home.dest&#39;], axis=1, inplace=True)<br><br>    pipeline = Pipeline([<br>        (&#39;encode_sex&#39;,<br>         ColumnTransformer(<br>             [<br>                 (&quot;age&quot;, MissingIndicator(), [&quot;age&quot;]),<br>                 (&quot;fare&quot;, MissingIndicator(), [&quot;fare&quot;]),<br>                 (&quot;sex&quot;, OrdinalEncoder(), [&quot;sex&quot;]),<br>                 (&quot;pclass&quot;, OrdinalEncoder(), [&quot;pclass&quot;]),<br>             ],<br>             remainder=&quot;passthrough&quot;<br>         )<br>         ),<br>        (&#39;model&#39;, RandomForestClassifier()),<br>    ])<br>    pipeline.fit(X[[&#39;pclass&#39;, &#39;sex&#39;, &#39;age&#39;, &#39;fare&#39;]], y)<br>    return pipeline<br><br>## predict_pipeline<br>import timeit<br><br>import pandas as pd<br>from sklearn.datasets import fetch_openml<br><br>from train import train<br><br><br>def load_dataset():<br>    X, y = fetch_openml(&quot;titanic&quot;, version=1, as_frame=True, return_X_y=True, parser=&#39;liac-arff&#39;)<br>    return X<br><br><br>def do_predict(pipeline, X):<br>    return pipeline.predict_proba(X[[&#39;pclass&#39;, &#39;sex&#39;, &#39;age&#39;, &#39;fare&#39;]])[:, 0].astype(float).tolist()</pre><p>In R we will define a path to python and scripts.</p><pre>library(reticulate)<br>use_python(&quot;/usr/bin/python3&quot;)<br>source_python(&quot;train.py&quot;)<br><br>pipeline &lt;- train()<br>source_python(&quot;predict_pipeline.py&quot;)<br><br>X &lt;- load_dataset()<br><br>do_predict(pipeline, X)</pre><p>As we can see it is not so hard to reuse models in R scripts. But what about the performance impact of this communication?</p><h3>Perfomance impact</h3><p>As you know, there is <a href="https://en.wikipedia.org/wiki/No_free_lunch_theorem">no free lunch</a>. Every communication between R and Python using <a href="https://rstudio.github.io/reticulate/">Reticulate</a> will cost you. We can use <a href="https://cran.r-project.org/web/packages/microbenchmark/index.html">microbenchmark</a> to measure the difference.</p><p>As a sample project we will use demo project with <a href="https://www.kaggle.com/c/titanic">Titanic — Machine Learning from Disaster</a>. At the preparation step we will train models in python and compare model prediction time in Python and in R with reticulate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*j5iGQyskpv-sZnSpmGlUFQ.png" /><figcaption>Experiments design</figcaption></figure><pre>library(&quot;microbenchmark&quot;)<br>microbench_out &lt;- microbenchmark(do_predict(pipeline, X))<br>microbench_out<br><br># Unit: milliseconds<br>#                    expr      min       lq     mean   median       uq      max<br># do_predict(pipeline, X) 55.42288 66.04502 85.75898 81.54774 98.76258 137.4046<br># neval<br>#   100</pre><pre>import timeit<br><br><br>def py_benchmark():<br>    pipeline = train()<br>    X, y = fetch_openml(&#39;titanic&#39;, version=1, as_frame=True, return_X_y=True, parser=&#39;liac-arff&#39;)<br><br>    def evaluate_predict():<br>        do_predict(pipeline, X)<br><br>    t = timeit.Timer(lambda: evaluate_predict())<br>    out = t.repeat(repeat=100, number=1)<br>    print((pd.DataFrame({&#39;t, milliseconds&#39;: out}) * 1000).describe())<br><br>if __name__ == &#39;__main__&#39;:<br>    py_benchmark()<br><br>#       t, milliseconds<br># mean         50.587171<br># std          15.673094<br># min          30.254079<br># 25%          39.684958<br># 50%          45.808600<br># 75%          58.351191<br># max         118.226873</pre><p>Python: median=46ms, R: median=86ms (+87%)</p><p>But when we think about numbers we have to split evaluation in Python evaluation phase and R communication overhead. In this case we have a complex computation that happens in the Python side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*-p6jb4V9V-oJA9YBF0yyhQ.png" /><figcaption>How many time we spend in each phase</figcaption></figure><p>Now we will create simple benchmark based on numpy evaluation of expression max(np.ones(1000))</p><pre>def next_seq():<br>    return np.ones(1000)<br><br>def py_benchmark():<br>    t = timeit.Timer(lambda: max(next_seq()))<br>    out = t.repeat(repeat=100, number=1)<br>    print((pd.DataFrame({&#39;t, milliseconds&#39;: out}) * 1000).describe())</pre><pre>library(reticulate)<br>use_python(&quot;/usr/bin/python3&quot;)<br><br>source_python(&quot;numpy_functions.py&quot;)<br><br>library(&quot;microbenchmark&quot;)<br>microbench_out &lt;- microbenchmark(max(next_seq()))<br>microbench_out</pre><p>And numbers of this experiment will be stunning:<br>Python: median=0.2ms, R: median=298ms</p><h3>Conclution</h3><p><a href="https://rstudio.github.io/reticulate/">Reticulate</a> is a simple way to use Python written functions in an R enviroment but you have to minimise communication between Python and R. If you would like to use heavy functions like Machine learning models it may be your choice but when you want reuse pandas or numpy in R — think twice about that.</p><p>Demo project available in github: <a href="https://github.com/mrk-andreev/medium_a8a166e564b3">https://github.com/mrk-andreev/medium_a8a166e564b3</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8a166e564b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>