<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Data Analysis Center - Medium]]></title>
        <description><![CDATA[Machine learning for Oil and Gas Exploration. Advanced methods to locate oil deposits - Medium]]></description>
        <link>https://medium.com/data-analysis-center?source=rss----9faef2f3af0f---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Data Analysis Center - Medium</title>
            <link>https://medium.com/data-analysis-center?source=rss----9faef2f3af0f---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 13:34:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/data-analysis-center" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[segair: A Rust-Powered Engine for Accelerated SEG‑Y Slicing]]></title>
            <link>https://medium.com/data-analysis-center/segair-a-rust-powered-engine-for-accelerated-seg-y-slicing-a78a6bd119ad?source=rss----9faef2f3af0f---4</link>
            <guid isPermaLink="false">https://medium.com/p/a78a6bd119ad</guid>
            <category><![CDATA[rust-programming-language]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[oil-and-gas]]></category>
            <category><![CDATA[seismic]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Razim Garipov]]></dc:creator>
            <pubDate>Fri, 21 Mar 2025 11:01:15 GMT</pubDate>
            <atom:updated>2025-03-24T07:12:52.099Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_f1r2pY1P5dVDWzoWbgvmA.png" /></figure><p>Seismic data processing and interpretation are at the heart of modern exploration workflows, and the ability to efficiently parse and visualize massive <a href="https://medium.com/data-analysis-center/the-splendors-and-miseries-of-seg-y-an-introduction-for-software-developers-ad3b3999c0ed">SEG-Y files</a> is more critical than ever. In this context, inline and crossline slicing — methods for extracting 2D sections from a 3D seismic post-stack cube — plays a pivotal role in revealing subtle subsurface features such as faults, horizons, facies, and hydrocarbon traps.</p><p>They are even more important when it comes to applying AI: modern GPUs perform efficiently only when fed data at very high speeds. The sheer volume and complexity of SEG-Y data have long posed performance challenges to existing processing solutions.</p><p>Our team has consistently focused on enhancing both computational performance and data throughput. Previously, we developed libraries such as <a href="https://medium.com/data-analysis-center/making-seismic-interpretation-easy-with-seismiqb-ac62d01a477">seismiqb</a> and <a href="https://medium.com/data-analysis-center/making-seg-y-fast-fda702894907">segfast</a> to accelerate seismic interpretation tasks, yet increasing demands continue to push the limits of current methods.</p><p>This article introduces <strong>segair</strong>, an innovative in-house library developed in Rust, specifically engineered to tackle these challenges. Leveraging Rust’s renowned performance, memory safety, and concurrency features, segair has been designed to dramatically accelerate the slicing process.</p><p>To establish its performance potential, we benchmarked segair’s inline and crossline slicing capabilities against other open-source SEG-Y solutions, with a focus on processing speed. The benchmark results highlight segair’s potential to streamline seismic data workflows, offering a robust, high-performance alternative for rapid data interpretation and real-time decision making.</p><h4>TLDR</h4><ul><li>Segair is primarely the fastest. Usually 2x compared to segyio</li><li>Random crossline of a 7.8 GB cube is obtained in 0.1 s on average on a 16&#39;000 iops SSD.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*Was4fqAaFAIPcVeq6hFiHw.png" /><figcaption>Relative slowdown for 5 crossline random slices on B-26–92-TX</figcaption></figure><ul><li>OpenVDS is strangely extremely slow (might be 10x, 100x, …) everywhere except sequential reading from cache</li><li>With caching in play, the performance gap across engines narrows as files reside in RAM. OpenVDS can sometimes edge segair in sequential slicing tests under these conditions</li></ul><h3>Benchmark setup</h3><p>When benchmarking SEG-Y file libraries, several challenges arise. First, caching can significantly skew results; both operating system and hardware caching may artificially boost performance. To measure true performance, it is crucial to flush caches and isolate the environment before each run. Memory layout and access patterns are also critical factors — contiguous versus random access can markedly affect performance, particularly for slicing operations that depend on the file layout.</p><p>To address these challenges, we conducted multiple benchmark runs, flushing the OS cache between each run. Additional details are provided in subsequent sections.</p><h4>Inline vs crossline slicing</h4><p>Inline slicing extracts 2D sections along the direction in which the seismic data is stored — typically, this means the data is laid out contiguously in the file. Because the needed traces are stored one after another, inline slicing benefits from sequential (contiguous) disk access, which is generally faster.</p><p>In contrast, crossline slicing involves extracting slices perpendicular to that primary storage order. The data for a crossline slice is scattered throughout the file, meaning that the reading process must jump around to gather the necessary traces. This pattern results in random access, which is slower due to additional overhead in seeking and retrieving non-sequential data.</p><p>These differences in access patterns are key factors in performance benchmarks, where contiguous (inline) reads are typically much more efficient than random (crossline) reads.</p><p>For more details on the file layouts, see <a href="https://pkirkham.github.io/pyrus/efficient-reading-for-seg-y-format/">Benchmarking Inefficiencies in the SEG-Y Format</a>.</p><h4>Competitors</h4><p>We compare segair to the following instruments (also referred to as “engines”):</p><ul><li><a href="https://github.com/analysiscenter/segfast">segfast</a> with <a href="https://github.com/equinor/segyio">segyio</a> engine (denoted as <em>segyio</em>),</li><li>segfast with memmap engine (denoted as <em>memmap</em>),</li><li><a href="https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds">OpenVDS</a>. Requires scan and import of a SEG-Y file to a volume data store (VDS). May take a considerable amount of time.</li></ul><p><a href="https://medium.com/data-analysis-center/making-seismic-interpretation-easy-with-seismiqb-ac62d01a477">Seismiqb</a> serves as a uniform interface to call the instruments above.</p><h4>Compute environment</h4><p>The benchmarks were conducted on Amazon’s <a href="https://aws.amazon.com/ec2/instance-types/c6a/">EC2 c6a.xlarge</a> instance, equipped with 4 vCPUs and 8 GiB of memory, running Ubuntu 24.04.1 LTS and Python 3.11.</p><h4>Storage</h4><p>Benchmarking on different drives helps determine whether performance limitations are due to the libraries’ I/O efficiency or the hardware’s capabilities. For instance, while different SEG‑Y readers may report similar processing times, one may be more efficient if it reaches the drive’s IOPS limit more quickly. For this benchmark, files were stored on Amazon Elastic Block Store using gp3 SSDs (3&#39;000 IOPS) and io1 SSDs (16&#39;000 IOPS).</p><p>For this benchmark, files were stored on <a href="https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volume-types.html">Amazon Elastic Block Store</a> using gp3 SSD (3&#39;000 IOPS) and io1 SSD (16&#39;000 IOPS).</p><h4>SEG-Y files</h4><p>The benchmark was performed on the following post-stack SEG‑Y files:</p><ul><li><strong>Vincent.</strong> 1.2 GB, 309 inlines, 1126 crosslines, 1150 samples</li><li><strong>B-26-92-TX.</strong> 7.8 GB, 418 inlines, 2033 crosslines, 2250 samples</li></ul><p>Each volume arranges its traces by inline, and the data is encoded using the big-endian IEEE float format.</p><h3>Results and discussion</h3><p>The fundamental unit of our benchmark is the time taken to acquire a slice. We define a “scenario” comprised of</p><ul><li>number of slices,</li><li>inline or crossline direction,</li><li>sequential (1, 2, 3, …) or random (5, 78, 16, …) indices of the slices.</li></ul><p>To make a huge number of measurements comprehensible, we run each scenario 30 times, take the mean of each run, and then take the mean of means, thus getting the <a href="https://en.wikipedia.org/wiki/Grand_mean">grand mean</a>. This is done for all the tables in this article.</p><h4>Edge slices</h4><p><strong>First inline</strong></p><p>Let’s start with the simplest test possible — just reading the very first slice of a volume.</p><p>All tables display time rounded to milliseconds, except when rounding would eliminate the difference between engines.</p><p>Also notice that horizontal axis in barplots may be linear or logarithmic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rFk_WPJlSfRJroFYGHEs2Q.png" /><figcaption>Grand mean of getting the first inline slice. Green highlight shows the quickest engine for a file-storage combination. The last row shows how much quicker <em>segair</em> is relative to <em>segyio.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4MxSPsvFOo_wwtjAeoUCMA.png" /><figcaption>Relative slowdown for the first inline slice on Vincent. Slowdown is obtained by dividing an engine’s time by segair’s time.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g68c1DY8PTFH9aGCi_Aauw.png" /><figcaption>Relative slowdown for the first inline slice on B-26-92-TX.</figcaption></figure><p>Right away we see that segair is at least 1.6–2 times faster than the closest competitor in the face of segyio.</p><p>Notice that memmap on Vincent is about 10 times slower than on B-26-92-TX. Why is that? Occasionally, the memmap engine on Vincent exhibits an anomalously high acquisition time of the first slice, sometimes exceeding 1 second. After the initial slice, memmap gives virtually the same timings and slowdowns as segyio. This is not the only oddity we encountered in this benchmark.</p><p>OpenVDS obtains the first slice orders of magnitude slower (20–130x) than the others. At the same time it benefits the most from the faster storage. We will see that it still doesn’t help reaching the competitors’ times in many cases.</p><p>Storage speedup on Vincent is either missing or much lower compared to B-26-92-TX. Seems like Vincent’s volume is so small that even the slower drive is good enough for quick reads.</p><p>For the <strong>last inline</strong> on both files and the <strong>first crossline</strong> on Vincent, segair is about twice faster than segyio and memmap, and tens and hundreds times than OpenVDS. However, OpenVDS becomes the second fastest option for the <strong>first crossline</strong> on B-26-92-TX on the 16&#39;000 IOPS drive with a 2.23x slowdown compared to segair, while segyio and memmap have it at 4x.</p><p><strong>Last crossline</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Venn15eYmy7h_xxQ1e-OHg.png" /><figcaption>Segair stays on top for the last crossline</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAX1UVCexN-3MmzPeb1LjA.png" /><figcaption>Relative slowdown for the last crossline on Vincent. Difference between segair and segyio on a small volume shrinks</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wqY24QmPwHGHvTJiiuD1pg.png" /><figcaption>Relative slowdown for the last crossline on B-26-92-TX. OpenVDS overtakes segyio and memmap becoming the second best after segair</figcaption></figure><p>The importance of benchmarking on multiple files is pronounced even more here. On Vincent, segyio approaches segair. On B-26-92-TX, segair achieves a 4x acceleration compared to segyio and memmap. OpenVDS comes much closer but is still 1.38 times slower.</p><h4>Multiple slices</h4><p>We carry on with a more loaded use case: you’re trying to track a subtle, laterally continuous stratigraphic feature — such as a thin sand body or a channel system — that may only appear intermittently in a single slice. If you suspect that a channel is present in a particular area but its amplitude or continuity is not obvious on one slice, you could extract 5 sequential inline or crossline slices.</p><p>Another common scenario is exploration of the seismic volume. A user may look for significant geological features or analysing suspicious places from the horizon map view. To do that, they would frequently jump across the file here and there, changing axes too. This is why we measure how long it takes to get slices of random indices.</p><p>Random slicing also emulates a multi-user environment: when several people work on a single file at once, they are not likely to request slices right next to the ones others have loaded.</p><p>You probably use machine learning models too, so you need many sequential and random slices to form training and validation batches.</p><p><strong>5 inline</strong></p><p>The overall picture for 5 inline sequential and random slices stays about the same:</p><ul><li>Segair’s time reduction over segyio and memmap is in 1.6–2.3x range</li><li>OpenVDS is still all over the place but comes close with a 5x slowdown for sequential on both files on the faster drive</li><li>Times are just a little lower than for edge slices. That tells us that it doesn’t matter where the inline slice is spatially located — it takes the same time if it hasn’t been read before (uncached).</li></ul><p><strong>5 crossline sequential</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LeEKUm3gCjPNEMoeiCPGfA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_w4mjIbhJRjKZmEBR9Bppw.png" /><figcaption>Relative slowdown for 5 crossline sequential slices on Vincent</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T_oAJom7PyJkkSqeNaYZFw.png" /><figcaption>Relative slowdown for 5 crossline sequential slices on B-26-92-TX</figcaption></figure><p>Segyio comes close to segair on Vincent and OpenVDS is now within an order of magnitude with the others.</p><p><strong>5 crossline random</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wZnyoVFsLa6oGu3AXxRudA.png" /></figure><p>Times have increased substantially compared to the sequential slicing. That probably indicates that some bufferization is performed by the OS and/or hardware in the former scenario but not here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ehNR70QA4DA5MJDxhNln4A.png" /><figcaption>Relative slowdown for 5 crossline random slices on Vincent</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Es5O0fpJX6Fn0fPmTrVK9A.png" /><figcaption>Relative slowdown for 5 crossline random slices on B-26-92-TX</figcaption></figure><p>There’s a pattern emerging: OpenVDS comes to shine if you have a faster storage and throw larger data at it. Still can’t beat segair though!</p><h4>Cached slicing</h4><p>So far we have reviewed scenarios in which slices weren’t cached. Clearly, after we do some work with the file, caching comes into play. We need to see how everything works with it.</p><p>To guarantee caching, we first generate the indices of the slices we plan to request, then preload them. Only after this do we measure the time required to load the same slices again.</p><p><strong>10 inline sequential cached</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aUqfelUKx-RhY71ru_dLjg.png" /></figure><p>Faster drive doesn’t offer any speedup anymore. When files are cached, they’re typically stored in RAM, which is much faster than any drive. This means that once a file is cached, subsequent accesses pull the data from RAM instead of the drive, rendering any differences in drive speed negligible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*avtGOSuyk3hhVoOUS287NQ.png" /><figcaption>Relative slowdown for 10 inline sequential cached slices on Vincent. OpenVDS comes forward.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GMtFnPhutrFrF7JlbGydaA.png" /><figcaption>Relative slowdown for 10 inline sequential cached slices on B-26-92-TX. The engines are close to parity.</figcaption></figure><p>This is the first time OpenVDS is faster than segair. Seems like VDS as a <a href="https://bluware.com/blog/understanding-the-capabilities-of-bluware-vds-and-openvds/">brick-based format</a> fits cached sequential slicing very well.</p><p><strong>10 inline random cached</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VOkzQDWALHUUO1S8SvnCSg.png" /></figure><p>The retrieval times are comparable to those observed with sequential slicing, and the faster drive still does not provide any additional speedup. Not for OpenVDS though, which is massively slower in random and quicker on the faster storage.</p><p>In <a href="https://bluware.com/blog/understanding-the-capabilities-of-bluware-vds-and-openvds/">Understanding the Capabilities of Bluware VDS and OpenVDS</a> under <em>Random Access Reading</em> section we read:</p><blockquote>Unlike the SEG-Y format, which stores data sequentially, VDS is a brick-based format. This allows for data to be read in any direction without performance degradation. Whether reading an inline, crossline, time-slice, arbitrary line, or a cube, VDS delivers exceptional performance.</blockquote><p>We have seen that OpenVDS’s sequential slicing is indeed the fastest when the files are cached. It is strange that we don’t see the same picture in our random test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_gc9e7p_K15C_DZ_KhJDag.png" /><figcaption>Relative slowdown for 10 inline random cached slices on Vincent</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k4gJfH_sNg59vvg973thfQ.png" /><figcaption>Relative slowdown for 10 inline random cached slices on B-26-92-TX. Segyio and memmap are almost equal to segair</figcaption></figure><p><strong>10 crossline sequential and random cached</strong></p><p>The overall pattern is similar to <strong>10 inline sequential cached</strong> we reviewed above. However, there are some differences:</p><ul><li>segair, segyio, memmap, OpenVDS order from fastest to slowest is preserved, except for <strong>crossline sequential</strong>, where OpenVDS stands first just like for <strong>inline sequential cached</strong></li><li>No performance gain with the faster drive for any of the engines</li></ul><h3>Conclusion</h3><p>Our comprehensive benchmark tests demonstrate that segair delivers superior performance across a wide range of slicing scenarios making it an excellent choice for rapid SEG‑Y file processing. In many cases, segair achieves acquisition times that are 2x faster than that of the traditional solution like segyio, and it outperforms OpenVDS by orders of magnitude in raw, uncached tests.</p><p>It is important to note, however, that under certain conditions — particularly in cached slicing scenarios and when utilizing high-performance storage (e.g., on faster SSDs) — OpenVDS can sometimes outperform segair. For instance, in some cached inline slicing tests, OpenVDS even exceeded segair’s performance. Despite these isolated cases, the overall performance, consistency, and robustness of segair across the majority of scenarios underscore its advantages.</p><p>Leveraging Rust’s state-of-the-art performance, memory safety, and concurrency features, segair not only streamlines seismic data processing but also sets a new standard for real‑time data analysis in exploration workflows. Segair is proving itself as the superior option for most practical applications in seismic interpretation.</p><p>Stay tuned for our upcoming post, where we’ll dive into benchmarking more challenging operations — like depth slice loading and geometry building for larger seismic cubes — to uncover segair’s deeper potential.</p><p>If you have any questions about our methodology or potential collaborations, please feel free to reach out!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a78a6bd119ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analysis-center/segair-a-rust-powered-engine-for-accelerated-seg-y-slicing-a78a6bd119ad">segair: A Rust-Powered Engine for Accelerated SEG‑Y Slicing</a> was originally published in <a href="https://medium.com/data-analysis-center">Data Analysis Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Splendors and Miseries of SEG-Y: an introduction for software developers]]></title>
            <link>https://medium.com/data-analysis-center/the-splendors-and-miseries-of-seg-y-an-introduction-for-software-developers-ad3b3999c0ed?source=rss----9faef2f3af0f---4</link>
            <guid isPermaLink="false">https://medium.com/p/ad3b3999c0ed</guid>
            <category><![CDATA[seismic]]></category>
            <category><![CDATA[oil-exploration]]></category>
            <category><![CDATA[oil-and-gas]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Alexey Kozhevin]]></dc:creator>
            <pubDate>Wed, 17 Apr 2024 11:19:46 GMT</pubDate>
            <atom:updated>2024-04-17T11:19:46.105Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UZJY3nYQ_8s1NBN46lY2Mg.png" /></figure><p>The first question anyone asks at the start of working with seismic data is: in what format is it stored and how to open it? And most likely you will meet the SEG-Y format. If you are a geophysicist, then most likely you already have access to special software like Petrel to deal with seismic data with lots of embedded instruments. However, with the evolution of artificial intelligence and machine learning, the need to work with SEG-Y arose among software developers and data scientists who work in the geology domain. Without knowledge of the internal structure of the format, working with it becomes difficult.</p><p>In this article, we will talk about</p><ul><li>how binary SEG-Y files are structured</li><li>what tools are available in Python for working with them</li><li>typical problems with SEG-Y loading</li></ul><h3>A brief reminder of seismic acquisition</h3><p>The source of seismic data is seismic surveying. We have <a href="https://medium.com/data-analysis-center/making-seismic-interpretation-easy-with-seismiqb-ac62d01a477">already talked </a>about it in previous articles, so let’s just say that as a result, we have a huge number of seismic traces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*U-4KvY6hR7jfUuPG" /></figure><p>After seismic surveying, you do not immediately obtain just a realistic representation of the earth’s internals; for this, you need to go through <a href="https://medium.com/data-analysis-center/seismic-processing-the-earth-shattering-truth-92dddee89f43">a seismic processing pipeline</a>. Only after the stage of processed data stacking you can obtain a post-stack cube, which will be a representation of the earth&#39;s structure.</p><p>In either case, both pre-stack and post-stack data are represented as a set of seismic traces. The major difference is the way these traces are indexed and what meta information they have. Therefore, it is reasonable to have one standard format for storing and transporting the traces and corresponding information about them.</p><h3>Internals of SEG-Y</h3><p>SEG-Y was developed by the SEG Technical Standards Committee to store seismic digital data on magnetic tapes. The first version (rev 0.0) was proposed in 1975 and the latest (<a href="https://library.seg.org/pb-assets/technical-standards/seg_y_rev2_0-mar2017-1686080998003.pdf">rev 2.0</a>) in 2017. For decades, the largest oil &amp; gas companies have been accumulating vast archives of data for future revision: for example, hard-to-recover reserves that were previously out of reach. To keep older versions supported and avoid data conversions, new revisions of SEG-Y are backward compatible, so they retain a huge amount of legacy. For example, the optional first byte of the file stores the magnetic tape label, which, of course, is not used now. Besides, when the format was developed, the IBM-float format was in use, and SEG-Y still provides the ability to store data in different data types, nevertheless IEEE-float is currently standard. Therefore, when loading data, you have to add a time-consuming conversion procedure to IEEE for CPU processing. Despite new seismic data storage formats being constantly proposed, SEG-Y remains the industry standard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*_j9NNuLomj1Wut2o" /><figcaption>SEG-Y internal description. Standard says that files can include extended binary file/trace headers, but we exclude them from consideration here because they are quite rare. M (the number of traces) and N (the length of the trace multiplied by the number of bytes per sample) are stored in a binary file header.</figcaption></figure><p>The SEG-Y is a binary file divided into several blocks:</p><ul><li>file-wide information block, which in most cases takes the first 3600 bytes<br>- <strong>textual header</strong>: the first 3200 bytes are reserved for textual info about the file. Most of the software uses this header to keep acquisition meta, date of creation, author, etc. Fun fact about another legacy: because of the archaicness of the format and backward compatibility, the textual header can still be encoded in EBCDIC encoding (have you ever heard of this?)<br>- <strong>binary header</strong>: 3200–3600 bytes contain file-wide headers, which describe the number of traces, the format used for storing numbers, the number of samples for each trace, acquisition parameters, etc.<br>- (optional) 3600+ bytes can be used to store the extended textual information. If there is such a header, then this is indicated by the value in one of the 3200–3600 bytes.</li><li>a sequence of traces, where each trace is a combination of its header and signal data<br>- <strong>trace header</strong> takes the first 240 bytes and describes the meta info about its trace: shot/receiver coordinates, the method of acquisition, current trace length, etc. Analogously to binary file header, each trace also can have extended headers.<br>- <strong>trace data</strong> is usually an array of amplitude values, which can be stored in various numerical types. Its length is defined by a file binary header. Traces can have lengths that differ from one that can be inferred from a trace header. We will further assume that all traces have the same length.</li></ul><p>The number of traces can be in the hundreds of millions, so post-stack data can reach hundreds of gigabytes in size. Pre-stack datasets can be even larger, like a few terabytes.</p><p>Thus, the storage unit is an individual trace, but most likely we want to either organize them into seismic gathers for the following processing or into a 2D/3D array for the seismic interpretation. For example, to get the traces of one common shot gather from raw pre-stack data, one should</p><ul><li>read FieldRecord header for all traces</li><li>group trace indices by the value of that header</li><li>select one source and load data for corresponding traces (usually traces are already sorted in the channel order)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/0*sVlJAOstCA_q77a7" /><figcaption>Seismic traces organized in common shot gather (from <a href="https://wiki.seg.org/wiki/2D_Vibroseis_Line_001">Geofizyka Torun’s survey</a>)</figcaption></figure><p>As specified in <a href="https://library.seg.org/pb-assets/technical-standards/seg_y_rev2_0-mar2017-1686080998003.pdf">SEG-Y revision 2.0</a>, FieldRecord is stored at bytes 9–12 of the trace header as an int32 value. For convenience, hereinafter we will use byte numbering starting with 1, as it was done in the revision. We also will use the same header names as in <a href="https://segyio.readthedocs.io/en/latest/segyio.html?highlight=TraceField#segyio.TraceField">segyio</a>.</p><p>To compose traces of the 3D post-stack cube into an array:</p><ul><li>read INLINE_3D (bytes 189–192) and CROSSLINE_3D (bytes 193–196) headers for all traces. These headers are created for the numeration of lines and can be thought of like X-Y coordinates.</li><li>create a 3D array from the loaded traces according to the headers</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*ijTe_QDZOsO0_P1V" /><figcaption>Inline and crossline from 3D seismic cube (from <a href="https://terranubis.com/datainfo/F3-Demo-2020">F3</a> open dataset)</figcaption></figure><p>It turns out that all you need to work with SEG-Y is to read the necessary headers and traces. In the next step, we will show how you can do this in Python.</p><h3>Python tools</h3><p>Now, let’s work with the open well-known dataset: <a href="https://terranubis.com/datainfo/F3-Demo-2020">F3</a> Netherlands. This is a post stack cube that is only ~1GB in size. To read segy you could use <a href="https://github.com/equinor/segyio">segyio</a> by Equinor, but we will demonstrate how to work with our open-source library <a href="https://github.com/analysiscenter/segfast">segfast</a>. It is <a href="https://medium.com/data-analysis-center/making-seg-y-fast-fda702894907">optimised</a> to make trace loading faster because it is crucial for fast seismic processing and training of ML models. Two main methods which we need are load_headers and load_traces. Let’s see how to use them.</p><p>This is a snippet to create pandas.DataFrame with two headers for all traces:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f534a0a72a91a98fe1243191e01a533/href">https://medium.com/media/3f534a0a72a91a98fe1243191e01a533/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/0*HvxzY6C668OgqzaO" /><figcaption>Resulting pandas.DataFrame</figcaption></figure><p>Here you can see that each trace has its unique index TRACE_SEQUENCE_FILE, and two coordinates in the coordinate system in the range (100, 750) and (300, 1250). Here we hope that the traces in the file are in lexicographic order (by INLINE_3D/CROSSLINE_3D), we will discuss it in the next section).</p><p>The length of traces is stored in the attribute segy_file.n_samples which is 463 for that cube. Thus, we can store all the traces organized as 3D array of shape (651, 951, 463).</p><p>To load inline with the index 100, we should choose traces corresponding to inline 100 and call load_traces method:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Nv60dtJ1iq7iunnv" /><figcaption>Resulting inline</figcaption></figure><p>In a similar way, you can load data and put it into a 3D array. Note that seismic data lives not only in line coordinates but also in world coordinates. So each trace has CDP coordinates (common depth point) which are stored in CDP_X / CDP_Y headers. These coordinates are very important to relate seismic data with other data sources like well logs.</p><h3>Typical problems</h3><p>Everything seems to be quite simple, but unfortunately, unexpected problems may arise during the loading of data. Most of them are connected with non-compliance with the standard.</p><ul><li><strong>storing the trace headers in the non-standard bytes:</strong> for example, the North Viking Graben dataset includes post-stack data where INLINE_3D and CROSSLINE_3D are stored at bytes 17–20 and 25–28 despite the fact that these bytes are meant for other data (“Energy source point number” and “Trace number within the ensemble”). Sometimes headers can be stored even in totally arbitrary positions, which does not coincide with any standard header. Note that SEG-Y file does not store any information about what value certain bytes correspond to. This information can only be obtained from the standard, the attachment to the file, or from SEG-Y text file header.</li><li><strong>wrong units of measurement: </strong>for example, revision 2.0 says that the sample interval for the trace must be measured in microseconds. Despite this, you can often find values in milliseconds there without any notice about it (you can detect that situation if the sample interval is in the order of ones instead of thousands).</li><li><strong>endianness (byte order)</strong>: probably you will have to reload a file with another, big or little, endianness if your file can’t be read.</li><li><strong>ordering of traces in the file: </strong>as we said earlier, we often rely on the ordering of traces we need. Fortunately, everything is mostly fine, but it is worth checking that everything is organized exactly as it should. For example, when we loaded inline, we took the traces in the order in which they are given in the file. At the same time, we believed that they were ordered by crosslines.</li><li><strong>the speed of reading:</strong> now we are processing huge fields, when loading traces for a single gather may result in random memory access, which is way slower than sequential reading. Besides, we train neural networks on batches of crops randomly sampled from a cube, and it is also random access. The widely used segyio allows you to do everything described, but it is not fast enough. segfast is <a href="https://medium.com/data-analysis-center/making-seg-y-fast-fda702894907">designed more appropriately</a> for fast data loading and therefore shows better results in benchmarks.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G9FmyMcuRtFbctHZ" /><figcaption>Comparison of inline/crossline/depth slice loading for post-stack data.</figcaption></figure><h3>Conclusion</h3><p>In this article, we talked about how the SEG-Y format is structured, how you can work with it in Python, and also talked about some problems that we encountered in our practice and how to solve them. Stay tuned with us and we will tell you more about the latest methods for storing and analyzing seismic data!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad3b3999c0ed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analysis-center/the-splendors-and-miseries-of-seg-y-an-introduction-for-software-developers-ad3b3999c0ed">The Splendors and Miseries of SEG-Y: an introduction for software developers</a> was originally published in <a href="https://medium.com/data-analysis-center">Data Analysis Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rock on with LLMs: Comparing GeoModels]]></title>
            <link>https://medium.com/data-analysis-center/rock-on-with-llms-comparing-geomodels-f2feb11b27cf?source=rss----9faef2f3af0f---4</link>
            <guid isPermaLink="false">https://medium.com/p/f2feb11b27cf</guid>
            <category><![CDATA[oil-and-gas]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[energy]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sergey Tsimfer]]></dc:creator>
            <pubDate>Tue, 05 Mar 2024 17:58:28 GMT</pubDate>
            <atom:updated>2024-03-21T11:19:11.167Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y9w4N9qVcwmfLIjyDGVV5g.jpeg" /></figure><p>In our previous article, we’ve covered usage of LLMs in geology. It has recipes for using open-sourced models, common gotchas and techniques for augmenting the model knowledge. We also describe potential use cases (general assistance, planning, document analysis), which we extensively tested in our workflow of seismic exploration.</p><p>Today we want to assess quality of open-sourced language models. You will learn about different types of metrics and benchmarks, and, finally, we’ll discuss results of our evaluation of popular networks.</p><h3>The why</h3><p>Before we begin, let’s consider whether we really need our own method to evaluate models. After all, there are already several comprehensive and thorough ratings (<a href="https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard">HuggingFace</a>, <a href="https://toloka.ai/llm-leaderboard/">Toloka</a>). Why can’t we just rely on them?</p><p>The main part of the answer is specificity. We are interested not just in general performance; instead, we want to compare LLMs on tasks that are closely related to our geoscience domain. Usage of your own dataset leads to the need of self-evaluating performance metrics, giving a much better assessment of quality.</p><p>Furthermore, if the plan is to fine-tune models to a specific data, metrics are an absolute must. As usual in ML, having an automated numerical evaluation is crucial for making progress, and training models is meaningless without it.</p><h3>The how</h3><p>To start evaluating LLMs, we would need questions and corresponding reference answer(s). For this purpose, we’ll use the <a href="https://huggingface.co/datasets/daven3/geobench">GeoBench</a> dataset: a collection of such pairs, which was open-sourced by the creators of <a href="https://arxiv.org/abs/2306.05064"><em>K2 model</em></a>. Let’s look at an example from it:</p><blockquote><strong>Question: The umbrella theory explaining the Earth’s movement, contact, and flattening of large land plates is known as:<br>A. </strong>the Coriolis effect<br><strong>B. </strong>plate tectonics<br><strong>C. </strong>hotspots<br><strong>D. </strong>the Richter Magnitude Scale<br><strong>E. </strong>the subduction zone</blockquote><p>What are the possible answers? Depending on the model, generated output may look like any of the following:</p><ul><li>B</li><li>(B)</li><li>plate tectonics</li><li>B. plate tectonics</li><li>The correct answer to your question is (B) Plate Tectonics. This theory explains the movement, collision, and flattening of large land plates on Earth. The Coriolis effect (A) refers to the apparent deflection of objects (like wind or ocean currents) due to Earth’s rotation, while hotspots (C) are areas of intense volcanic activity caused by mantle plumes. The Richter Magnitude Scale (D) measures the energy released during earthquakes, and a subduction zone (E) is a region where one tectonic plate slides beneath another, leading to earthquakes and volcanic activity.</li></ul><p>Even with all the answers being technically correct, the evaluation process is not easy. Depending on the context, a more concise or detailed answers are desired. Correspondingly, there are metrics for assessing either very short (based on accuracy) or wordier outputs (BLEU, ROUGE, …). So how do we judge our models?</p><h3>Accuracy-based metrics</h3><p>The easiest approach is to specify instructions more precisely. As usual, we use prompt for it, and by prefacing each question with <strong><em>“This is a multiple choice question. Answer with one letter and one letter only”</em></strong> we can narrow down the correct answers to exactly one.</p><p>This is the most common approach: the most popular benchmark <a href="https://arxiv.org/abs/2009.03300">MMLU</a> is implemented exactly like that. Under the hood, hovewer, there is an additional parameter of metric computation, that depends on used tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/0*QGKuYFy77Q5tm12Y.png" /><figcaption>image from: <a href="https://blog.allenai.org/a-guide-to-language-model-sampling-in-allennlp-3b1239274bc3">https://blog.allenai.org/a-guide-to-language-model-sampling-in-allennlp-3b1239274bc3</a></figcaption></figure><p>Language models generate a probabilistic distribution for the next token over the set of known tokens. Given the sentence <strong><em>“2 + 2 =”</em></strong>, we hope that the probability of token <strong><em>“4”</em></strong> is higher, than for any other number-like token, but that does not eliminate other probabilities. <br>At the same time, we also hope that token <strong><em>“3”</em></strong> has a higher probability, than something completely unrelated, for example, token<strong><em> “the”</em></strong>. With the answer <strong><em>“3”</em></strong>, the output of a model would still be incorrect, but at least the question and the instruction are understood correctly.</p><p>For our evaluation, this means that there are multiple approaches. Token, corresponding to the correct letter, should have the highest probability among:</p><ul><li>all of the model tokens (often referred as HELM implementation);</li><li>letter-tokens only (ORIGINAL implementation of this metric);</li><li>something in-between the two previous options: in our evaluation, we use all of the model tokens except technical ones (like symbols of linebreaks or generation start).</li></ul><p>Those different options also correspond to somewhat different flavors of testing: the first one requires correct understanding of the instruction and correct output, while analysing only a few letters is much easier and assesses raw model knowledge.</p><blockquote><em>Following the most common implementations, we also add a few examples in the model prompt and finish it with </em><strong>“Answer: “</strong><em>.<br>Our full implementation will be available shortly.</em></blockquote><h3>Benchmark</h3><p>We have computed the accuracy metric (with all of the mentioned implementations) and ranked models according to it. At this point, we’ve included the most popular models (<em>llama-2</em> family), trending now (<em>Mixtral </em>and <em>miqu</em>), as well as models that directly target geology (<em>K2 </em>and <em>geogalactica</em>). To put things into perspective, we’ve also evaluated some <em>llama-1</em> LLMs. Take a look:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2t5HgZwuipA_wRb7Z-Uojg.png" /><figcaption>Evaluation results. Sorted by the rightmost (ORIGINAL) column.</figcaption></figure><p>Other that just saying <em>“model A is better than model B on GeoBench”</em>, we can see that:</p><ul><li>as expected, values for HELM are considerably smaller, than for ORIGINAL implementation. This is due to the lower probability of correct token being chosen among all of the model tokens, not only the letter ones. The difference between those two columns can be seen as the difference between raw knowledge (ORIGINAL) and actually being able to apply it appropriately (HELM);</li><li>fine-tuned geological models (K2, geogalactica) have significantly worse ability to follow instructions, than their base LLMs. As we pointed out in the previous article, that is one of the common pitfalls of the model fine-tune procedures;</li><li>fine-tuned geological models are behind general ones by a landslide. Partly, this may be attributed to fewer parameters and not being able to follow the latest trends in architectures;</li><li>while table is sorted by the rightmost column (accuracy on letter-only tokens), model size is clearly a big factor. Also, the discrepancy between accuracy implementation is big for small models, which are notoriously worse at following instructions;</li><li>overall, progress during just one year (even less, if you think of it) is mind-blowing. If you compare the <em>llama-2</em> and <em>llama-1</em> in terms of error rates (25.6% vs 31.4%), there is almost a 20% improvement! Notably, this growth is even more pronounced for smaller sizes;</li><li>quantized models are losing a few percentages of accuracy, with small models losing more;</li><li>finally, Mixture of Experts models (<em>Mixtral </em>and <em>miqu</em>) are doing exceptionally well, while also being considerably faster.</li></ul><p>It should be noted that this kind of benchmark is not perfect: just like the MMLU, the used dataset has some percentage of incorrect or incoherent examples. Nevertheless, until we approach very high (90+%) accuracy, we can still use it to rank LLMs.</p><blockquote>We plan on regularly updating this benchmark with new model, so don’t hesitate to suggest which ones to add!</blockquote><h3>Metrics for generation</h3><p>The accuracy-based metrics have a huge downside: they work exclusively with quiz-like tasks, where the correct answer is short, unambiguous and known in advance. When it comes to comparing quality of larger (and more diverse!) outputs, your options are quite limited:</p><ul><li><strong>BLEU and ROUGE </strong>are comparing word matches between target sentence and the generated one;</li><li><strong>BERTScore and MoverScore</strong> are computing distance between embeddings of the target/generated sentences. Unlike the BLEU and ROUGE, these two are less sensible to synonyms;</li><li><strong>perplexity and cross-entropy </strong>evaluate the model certainty in the answer, given the input sequence. These are the same ones, that are used during model training, and are more about confidence in the answer, not its quality.</li></ul><p>As you can see, the nature of these metrics also limits their applicability — some are better for summarization, others for translations or raw generation. Also, when dealing with longer outputs, metric values can fluctuate a lot due to the sampling randomization.</p><p>The final metric we want to mention has a human in loop: <a href="https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard">LLM Arena</a>. Essentially, it is an ELO-based leaderboard of models, both open-sourced and proprietary ones. Due to human feedback and closeness to the actual use cases of chat systems, this may be the most accurate estimate of the performance. If you have narrowed the set of tested model to just a few items, then this setup may be the best.</p><h3>Conclusion</h3><p>In this article, we’ve talked extensively about measuring model performance: mainly, on multiple answer questions. As we can see, open source models are outperforming specifically fine-tuned ones by a huge margin.</p><p>We hope to add new models into the table and update it periodically to reflect the progress in LLM community. If you have an idea of which model to include, be sure to leave it in the comments.</p><p>Now, as we know how to evaluate models properly, time nears for talking about actual fine-tuning: this will be the theme of our next article.</p><p>In either case, stay (sic!) tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2feb11b27cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analysis-center/rock-on-with-llms-comparing-geomodels-f2feb11b27cf">Rock on with LLMs: Comparing GeoModels</a> was originally published in <a href="https://medium.com/data-analysis-center">Data Analysis Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rock on with LLMs: Geological Applications]]></title>
            <link>https://medium.com/data-analysis-center/rock-on-with-llms-geological-applications-c291d10a041c?source=rss----9faef2f3af0f---4</link>
            <guid isPermaLink="false">https://medium.com/p/c291d10a041c</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[geoscience]]></category>
            <category><![CDATA[oil-and-gas]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sergey Tsimfer]]></dc:creator>
            <pubDate>Wed, 28 Feb 2024 22:15:46 GMT</pubDate>
            <atom:updated>2024-02-28T22:15:46.164Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gbj_jIJPqpyDUQQxqdtmZg.jpeg" /></figure><p>After the release of ChatGPT and GPT-4, the internet has been buzzing with excitement over language models. These models have rapidly evolved from simple fun tools to fullу fledged assistants capable of performing various complex tasks.</p><p>While LLMs are looking exciting, the question still remains: how can we use them in a specific domain and what exactly can we automate with their help? Our team has lots of experience in geophysics and overall seismic exploration, and in the past months we’ve made language models work for a number of everyday scenarios. In this article, we want to share them, as well as some insights into how such chat systems are built.</p><p>Let’s dig in!</p><h3>Starting with..</h3><p>A good start for working with LLMs is to just poke around with some questions. Importantly, we want to use only <a href="https://huggingface.co/models?pipeline_tag=text-generation&amp;sort=trending">open sourced models</a>: after all, Oil&amp;Gas is still a very private and restrictive industry. Take a look:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xvlpQGloAG4s43l6-BY1Sw.png" /></figure><p>Unlike very impressive ChatGPT outputs, this one is rather silly. Fortunately, we can easily guess the problem with the generation: input should also contain model-specific instructions, collectively known as prompt. We can also alter the behaviour of the model by making it “impersonate” an expert in field. Adding those instructions substantially improves the quality of answers!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*47DbvTEOwyFS-tbJCBlVjg.png" /></figure><p>While this simple fix is helpful, it also raises a ton of questions. Which prompts are allowed? Which one is the best? Is the best prompt unique for each question? Answering them has created an entirely new area of prompt engineering, and the tricks from it are widely used behind the scenes of every popular chat system. A few links to key results to get you started:</p><ul><li><a href="https://prompthero.com/">a collection of prompts for wide variety of tasks</a>;</li><li><a href="https://arxiv.org/abs/2205.11916">let’s think step by step</a>, <a href="https://arxiv.org/abs/2201.11903">chain of thought</a>;</li><li><a href="https://arxiv.org/abs/2305.03495">automatic prompt discovery and optimization</a>.</li></ul><h3>Boundaries of knowledge</h3><p>Let’s give a model another, slightly more involved question. Suppose we are doing <a href="https://medium.com/data-analysis-center/facies-analysis-on-seismic-data-31a356446c27">facies analysis on seismic data</a>: this is where geological attributes are coming in handy, and original formula together with their numerical approximations are required. What if instead we just ask about this?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FoBvqIMRnzP1LJ0lhWCFKg.png" /></figure><p>The result is very insightful! First, the model communicates in perfectly viable LaTeX: this is a huge plus when working with any math or physics related questions. If you have ever tried communicating equations in plain text, you definitely know the pain of it.</p><p>Next, we can clearly see consequences of model training on scraped web pages: it not only gives intro into instantaneous frequency, but finished up with the comment section about it. Formatting generated texts and keeping the unwanted bits from the end user of a chat system is a must, and that is where a lot of engineering solutions are needed.</p><p>Finally and most importantly, while the model answer is correct, it lacks the necessary details. Even if prompted with additional questions, the LLM is uncapable of clarifying those: it just does not know all the ins and outs of numerical computation of a particular attribute.</p><p>As you can see, we can extract a ton of info about the inner workings of the model and potential problems just by analyzing a few examples. Out of the box, models are quite well versed in both processing and interpretation, which are our specialization, as well as subdomains of geology (oceanography, mineralogy, etc). For example, following questions start meaningful conversations with in-depth understanding:</p><ul><li><em>explain details of first break picking;</em></li><li><em>describe sedimentation conditions for X type of facies;</em></li><li><em>compare well tie algorithms in seismic time versus depth domain.</em></li></ul><h3>Document excitement</h3><p>Testing the limits of a model with a slew of questions is fun and all, but if we are talking about a truly helpful assistant, we should think about ways to include new knowledge into it. For example, a small selection of carefully picked books on the most important subject; a few important documents and regulations in the field; finally, a database of corporate files.</p><p>Importantly, this also alleviates one of the already encountered problems: some things are just unknown to the LLM. By targeting a specific domain and providing the necessary context search tools, our assistant can be much more informative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FNIj887utypEfOw_U6T6GQ.png" /></figure><p>Under the hood, the machinery for adding info into the network splits into two parts. The first one is to actually have a database, that can be queried with questions: in a way, it is very similar to wikipedia and the likes. By preparing such database with your own data, you can easily get added context for any given question. The preparation process often involves some kind of optical recognition to work with raw scans and photos. For actual storage, the embedding-based ones are the way to go.</p><p>The second part is to deliver this retrieved knowledge into the model, and, you’ve guessed it, we do it via prompt. Just by inserting a few pages about the subject of interest, we arm the model with all the details it needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*44Qt8BnVRniazUrTgpv5hA.png" /><figcaption>Example of prompt with added context. Additional info is found based on user query</figcaption></figure><p>After generating the model answer with that (much improved) prompt, we can also cite the sources and even score each one of them. This makes it trivial to refer the user into expert literature and latest papers.</p><h3>Assistant usecases</h3><p>With the introduction of additional contexts into the model, we have actually provided the tools to use our chat assistant in a wide variety of scenarios. Just by supplying the necessary documents (and, maybe, some task-specific prompts) our system is now capable of:</p><ul><li><strong>General geological assistance</strong><br>With the added knowledge base of books, our system is capable of explaining core concepts, providing historical and background information about given field, advising on methodology and even plan parts of workflow with detailed timings;</li><li><strong>Q&amp;A with document(s)</strong><br>By using provided documents as a knowledge base, we can simplify interaction with large documents. With the ability to cite used sources, we can point to the exact place in a document where the information can be found for ease of verification. Essentially, this is a better search tool, powered by LLM;</li><li><strong>Check that given documents meets standards / regulations / templates</strong><br>Oil&amp;Gas companies are large, and large companies tend to produce a lot of paperwork. These files must follow some guidance and standards, and automating the verification can save thousands of work hours;</li><li><strong>Compare documents and highlight key differences</strong><br>Again, all the paperwork usually has a few versions that are iterated upon. Following with the updates is relatively easy while you are at it; but coming back to them even after a few months is a major pain;</li><li><strong>Paper grading</strong><br>By iterating over the submitted papers, our model can evaluate them using our very carefully designed criteria. Wink-wink: some of the associations of geoscientists and engineers are already using our LLM tool to rate papers!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jnh2lHEEmxsFSImCiil6lQ.png" /><figcaption>An example of paper grading. Note that the article text can be either user input or an external document</figcaption></figure><p>Other than those, our system is also capable of regular LLM stuff:</p><ul><li>text summarization;</li><li>extracting information from a text according to a provided template;</li><li>text writing and editing.</li></ul><p>Combining all of those capabilities, we’ve got ourselves an impressive assistant to help with a huge number of daily routines. A great thing about the rapidly developing world of LLMs is that the list of well-behaved and explored use cases is ever-growing, and having the basic building blocks in place allows to seamlessly benefit from the common developments.</p><h3>A word about models</h3><p>We turned our model into a better assistant by improving our prompts: providing contexts and more detailed instructions is a great way to enhance quality. There is an alternative approach: model fine-tuning.</p><p>Very similar to the model training, we are collecting a dataset of the examples to learn from; the main difference being the focus on a particular texts. Also, the scale is usually much smaller: fine-tuning takes a fraction of iterations, and, with the introduction of <a href="https://arxiv.org/abs/2106.09685">LoRA</a>, a fraction of model parameters.</p><p>While the perspective of having a smarter, better adapted to your needs, model is fascinating, it requires much more expertise and resources to do so, compared to promp-based improvements. Besides, during the fine-tune models tend to forget some of the previous skills: in most cases, the chat component goes out the window first. The resulting model may know the subject better, but is much worse at following instructions, therefore leaving a bad impression overall.</p><p>On the topic of models, we also need to select the base language model from a large zoo of open sourced projects. An easy (and not so terrible!) approach is to go with the current flavor of the month; to make the selection better motivated, we have conducted research to evaluate 10+ of the most popular models of varying sizes on the corpus of geological questions. This article is coming up shortly, so keep your eyes peeled:)</p><blockquote>Update: <a href="http://link/">article with model comparison</a> is ready!</blockquote><h3>Conclusion</h3><p>Throughout the article, we’ve discussed and showcased a non-exhaustive list of LLM capabilities, applied to the domainof Oil&amp;Gas. While most of our examples were related to the geological exploration, it is easy to alter them (mainly, the prompts) to be working in any other particular area of knowledge.</p><p>We’ve also shared some of the more technical details on howto implement such a system: the building blocks that allow us to control the general format of the output messages, to import external contexts and documents in the model, the making of a base model. Some of them require more attention, and in the future we plan to post more about particular elements of a geological chat assistant. Also, we would like to expand on some of the proposed use cases: let us know which are the most interesting to you.</p><p>Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c291d10a041c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analysis-center/rock-on-with-llms-geological-applications-c291d10a041c">Rock on with LLMs: Geological Applications</a> was originally published in <a href="https://medium.com/data-analysis-center">Data Analysis Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hands-on the desk with First Break Picking]]></title>
            <link>https://medium.com/data-analysis-center/hands-on-the-desk-with-first-break-picking-a5aeb6f669e3?source=rss----9faef2f3af0f---4</link>
            <guid isPermaLink="false">https://medium.com/p/a5aeb6f669e3</guid>
            <category><![CDATA[oil-and-gas]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[geology]]></category>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[seismic]]></category>
            <dc:creator><![CDATA[Altynanke]]></dc:creator>
            <pubDate>Fri, 19 Jan 2024 14:27:55 GMT</pubDate>
            <atom:updated>2024-01-19T14:27:55.027Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8NqY7EZ0CQ_QHR8WzxphMQ.jpeg" /></figure><p>As an old adage goes, <em>“If there is something wrong with your stack, check the first breaks”.</em></p><p><strong>First break picking</strong> is a task of tracking a time when a receiver starts to record a meaningful signal from a source. It is one of the initial steps of the seismic processing pipeline, and only the knowledge of first break times allows one to model the velocities and thicknesses of the first layers of the earth. You can find more details on this stage in the seismic processing pipeline in our previous <a href="https://medium.com/data-analysis-center/seismic-processing-the-earth-shattering-truth-92dddee89f43"><em>feature</em></a>. Usually, it requires <em>months</em> of painstaking manual work from an experienced geologist to carry out picking of the whole survey.</p><p>If you have been following us closely, you probably know we’ve already touched on this task and discussed why conventional methods aren’t enough for success. We captured the working principle of auto pickers and why even 1D UNet can handle the task of first break picking (or <strong>FBP</strong>) substantially better. For more details, please follow this <a href="https://medium.com/data-analysis-center/advancing-the-first-break-picking-with-neural-networks-4d7ea2ea7c2"><em>link</em></a>. Now it’s time for us to introduce our <strong>2D</strong> first break picking approach based on a <strong>linear moveout correction</strong> and a <strong>neural network</strong>.</p><p>Let us provide a little more context to the narrative. Our journey to this method started with an algorithm we’ve been experimenting with for some time. It was designed as an extension of the existing 1D algorithm to two spatial dimensions.</p><p>It follows a <em>two-staged recipe</em>:<br> 1. <strong>1D</strong> UNet processes <em>each trace separately</em> at the first stage and returns a <em>binary mask</em> following a first break.</p><p>2. In the second stage,<strong> 2D</strong> UNet <em>refines the obtained mask</em> by accounting for <strong>spatial</strong> information, see <em>Figure 1</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*idgXkdQZgEZFY984EfgNsg.png" /><figcaption>Figure 1: two-staged FBP approach. From left to right: shot gather, binary mask obtained after the first stage, refined mask after the second stage, gather with first breaks.</figcaption></figure><p>While this approach allows for the successful elimination of the artifacts of noise and phase inconsistency, it comes with a couple of drawbacks. Namely, in the second stage, the model<em> does not rely upon trace data explicitly</em>, which makes the result harder to interpret and debug. Secondly, the 2D model needs to be relatively <em>large</em> and thus wheezingly <em>slow</em>, as model inputs<em> cannot be substantially cropped</em> along the time axis, and have to be<em> broad enough along the channel axis </em>to represent the trend across different reclines. As a result, a train plus inference stage could take up to <strong>4 hours</strong> to complete on a large field. Still much faster than the manual approach, but there is definitely room for improvement!</p><p>We’ve experimented with various ways to shorten this procedure down to one 2D model. However, <strong>feeding signal data instead of a binary mask to a 2D model is a significant complication</strong>. The more <em>sophisticated</em> the data is for the model, the greater should be the size of the training <em>dataset</em> and the number of model <em>parameters</em>.</p><p>But FBP <em>datasets are relatively small</em>, namely 15–50 shot gathers, which is not nearly enough for the proper training of a sufficiently large model. Training datasets for FBP are mostly prepared manually with little help from auto pickers, and to make the process of first break picking <em>efficient</em> and less laborious, training data preparation should not be a mammoth task. Moreover, FBP models should be trained (or finetuned) for each survey <em>individually</em>, as every field has its specific traits. In big companies, first break picking is performed routinely and frequently, so the preparation of new datasets<strong> should not become yet another bottleneck</strong> in the seismic processing pipeline.</p><p>However, one scheme met our requirements, and thus we’re eager to describe a radiant end-to-end approach utilizing <em>linear moveout correction</em> and only one <em>UNet</em>, see Figure 2.</p><p>The idea is straightforward: <br>1. To simplify the data we perform <strong>linear moveout correction</strong>, after which the first breaks are roughly located along a <strong>straight line</strong>, giving the model a solid prior on where they might be.</p><p>2. As the first breaks are located along a straight line, the <strong>crop size</strong> on time and channel dimensions can be substantially <strong>reduced</strong>, allowing us to also shrink the number of <em>parameters</em> and <em>time</em> for train and inference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EpMiIR73ErPSfW2luU-Jfw.png" /><figcaption>Figure 2: shot gather with first breaks(left), shot gather after linear moveout correction applied(right)</figcaption></figure><p><strong>Linear moveout correction </strong>is an operation of straightening up the hodographs according to the velocities of <strong>refractors</strong>, which are <em>piecewise linear</em> functions of offset. To calculate <em>refractor velocities</em>, we utilize the first breaks available in the training set, and the fact that after a certain offset, the first useful signal is always coming from the refractors. At the inference stage, calculated refractor velocities are interpolated over the whole field, see Figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RkBAC3d789P7e4rVBbba_g.png" /><figcaption>Figure 3: refractor velocity field on the left, picked velocities for one shot gather on the right.</figcaption></figure><p>However, introducing the model to a linear prior might result in not sensitive enough predictions. To tackle this issue we introduce a specific <strong>augmentation method</strong>, which is simple yet effective. At each training step, roll up or down a random set of <em>consequent traces</em> for a <em>random time</em> with some <em>probability</em>, see an example in Figure 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*UDW9Ggn_a3ZW9aPRYlDBPA.png" /><figcaption>Figure 4: Augmentation example</figcaption></figure><p>This little trick allows the model to learn how to properly <em>react to abrupt changes</em> in the scenery and also prevent itself from <em>overfitting</em>. Have a look at Figure 5 to see the effect of employing augmentation in the training process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*oz8_Ixudc1pZ4VW50CqDFw.png" /><figcaption>Figure 5: The effect of introduced augmentation. Blue dots correspond to no augmentation.</figcaption></figure><p>This approach allows us to perform the whole stage of FBP in <strong>less than an hour</strong> for almost any field using a single GPU! For instance, model training with 13k traces takes 15 minutes, and inference on a survey with 18 million traces takes only about 40 minutes.</p><p>The last step is to ensure that the <strong>quality</strong> of obtained first breaks meets certain requirements. Depending on whether it’s the minimum or maximum phase we’re interested in, we want to make sure the picked <strong>phase is consistent</strong> across the field. Moreover, we don’t want any <strong>outliers</strong> to be present in the final picking.</p><p>To tackle this challenge, we provide various<strong><em> interactive QC maps</em></strong>, that detect <em>outliers</em> from expected time, <em>phase</em> and <em>amplitude</em> inconsistency, calculate signal <em>correlation</em> around the first break across the gather, and even guess an <em>offset</em> after that the first breaks are most likely to <em>diverge</em> from the expected time. Gathers with deviations are <strong>highlighted</strong> on the map, allowing an expert to quickly identify the most problematic areas, see Figure 6.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4xfvehXH9Ga9GP8j_crPZA.png" /><figcaption>Figure 6: Metric map of first breaks outliers on the left, visualization of the metric for one gather on the right. Outliers are dots beyond the red area, and the metric is calculated as a fracture of outliers in individual gather.</figcaption></figure><p>Map construction is also highly optimized: calculation of five different metrics for a survey of 4.5 million traces takes only<strong> 2.5 minutes</strong> on CPU.</p><p>Conventionally to detect anomalies experts had to meticulously wade through a plethora of gathers on a <em>sparse grid</em>, the number of which might tally to <em>thousands</em> for a large field. The proposed approach not only greatly accelerates the <strong>time</strong> needed to assess FBPs, but also makes estimation more <strong>accurate</strong>, as it assesses every single gather on a field.</p><p>Moreover, the ability to promptly assess FBPs enables one to check the quality of the<strong> training set</strong>. That is a crucially important step, as any glitches in the training set would result in similar ones during inference. Our approach allows one to automatically <strong>screen out </strong>any inferior picks, leaving only representative data on board. See Figure 6 for an example of phase inconsistency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*hqrip-oaHeueGqYox3LS-w.png" /><figcaption>Figure 6: Here we detected the first breaks tracked at max phase instead of min.</figcaption></figure><p>To sum up, we’ve shared our most successful approach on first breaks. Being able to carry out this stage swiftly and accurately is highly likely to boost the quality of the final stack and eliminate the sweated labor connotated with it. We hope our experience will save you some time struggling and evoke new ideas for handling your challenges. See ya next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5aeb6f669e3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analysis-center/hands-on-the-desk-with-first-break-picking-a5aeb6f669e3">Hands-on the desk with First Break Picking</a> was originally published in <a href="https://medium.com/data-analysis-center">Data Analysis Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seismic Processing: The Earth-Shattering Truth]]></title>
            <link>https://medium.com/data-analysis-center/seismic-processing-the-earth-shattering-truth-92dddee89f43?source=rss----9faef2f3af0f---4</link>
            <guid isPermaLink="false">https://medium.com/p/92dddee89f43</guid>
            <category><![CDATA[seismic-processing]]></category>
            <category><![CDATA[statics-correction]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[oil-and-gas]]></category>
            <dc:creator><![CDATA[Altynanke]]></dc:creator>
            <pubDate>Mon, 23 Oct 2023 16:05:18 GMT</pubDate>
            <atom:updated>2023-10-23T16:05:18.628Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*ygUG3JWWSuOnnwQG77B16A.png" /></figure><p>A good seismic processing is like a four-leaf clover, hard to find and lucky to have. Everybody involved in oil and gas knows that the processing stage is weighty and complex, yet not many get the picture. Our team fully covers some of the main processing steps, and today we will show you the ropes. So let’s get down to it!</p><h4>Why do we need seismic processing?</h4><p>The processing stage lies between seismic data <em>acquisition</em> and <em>interpretation</em>, as raw acquired data is unsuited for interpretation analysis. It usually requires a lot of meticulous labor and can linger for months. Thankfully, some of the stages can already be enchanced with ML, others are bent to remain algorithmic, and the rest are yet to be studied.</p><p>Let’s start with the data description and our main goal. At the processing stage, we work with a <strong>pre-stack seismic cube</strong>, it is like a post-stack, but with a little bit more traces in it.</p><p>Every pre-stack seismic cube is unique not only in location but also in <em>survey design</em>, which is in the hands of the seismic acquisition team. In short, the objective of seismic acquisition is to emit and record waves at various locations around each inspected point to get complete information about it. As a result, there is much more than one trace recorded to describe each point of the survey. Seismic waves are generated by <strong>sources</strong> (also called <em>shots</em>) and get recorded by <strong>receivers</strong> that can be of various types, spacing, quantity, sampling rate, record length, etc. The two main types of sources for field data are <em>explosives</em> and <em>vibrators</em>, and for marine fields, it’s <em>air guns</em>. Positions of sources and receivers are also a big deal. Usually, they are settled in lines, with sources perpendicular to receivers. See the single-shot acquisition schema in Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4lAc89XQmTUdbJyf.jpg" /><figcaption>Figure 1: Seismic aquisition. Source: <a href="https://www.scisnack.com/2019/10/23/what-seismic-data-tell-us-about-rocks-below-our-feet/">https://www.scisnack.com/2019/10/23/what-seismic-data-tell-us-about-rocks-below-our-feet/</a></figcaption></figure><p>We will illustrate our narrative with the examples made on open <a href="https://wiki.seg.org/wiki/2D_Vibroseis_Line_001"><strong>Geofizyka Torun’s survey</strong></a>. It’s a 16 km long vibroseis 2D line with 251 sources, 781 receivers, and 70531 traces. Here on the left side of the Figure 2, you can see its map of all sources (red stars) and a set of receivers (blue dots) activated for a particular shot (black cross). The picture on the right displays this <strong>shot gather.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KOT84PSqEbbixo34utMtqA.png" /><figcaption>Figure 2: Geometry of Geofizyka Torun’s survey. Red stars denote shots, blue dots denote receivers activated for a particular shot(black cross)</figcaption></figure><p>If we index survey traces on the common midpoint (<a href="https://wiki.seg.org/wiki/CMP_sorting"><strong>CMP</strong></a>) and sort by offset, we get gathers like the one on Figure 3:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yAU3wVppZ2oMVkp6OZAmpw.png" /><figcaption>Figure 3: CMP gather. Blue dots denote start of the useful signal.</figcaption></figure><p><strong><em>Remark</em></strong><em>: </em>As you may have already guessed, every seismic processing step is bounded by <strong>field geometry</strong>. If the shots and receivers coordinates are greatly miscalculated, we cannot get, for instance, proper CMP gathers. Luckily, large <em>geometry errors</em> are singular and we have a quality control method for them, which we’ll discuss later in this post.</p><p>To get this survey line <strong>stacked</strong>, one has to size up CMP gathers down to <em>one</em> trace. We cannot sum or average them right away as the horizons aren’t aligned and too much noise is present. So that is literally why we need a seismic processing stage: <strong><em>to perform a sequence of certain operations beneficial for the final stack</em></strong>.</p><h3>Normal Moveout Correction</h3><p>We can notice in the CMP gathers that the useful signal starts later at further offsets, see Figure 3. Under the assumption that <a href="https://wiki.seg.org/wiki/NMO_for_a_flat_reflector"><strong><em>reflection traveltimes follow hyperbolic trajectories</em></strong></a> as a function of offset, we can fit these hyperbolic hodographs coefficients and straighten them up on the seismogram such that the <em>offset effect</em> is removed,<em> </em>see Figure 4. These coefficients represent some kind of “mean” velocities from the surface up to the given reflection boundary, they <em>increase with depth</em> and <em>vary between gathers</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d-H5WQ3yZQW6TQKxKPqbHg.png" /><figcaption>Figure 4: from left to right: original CMP supergather, its NMO-corrected version, and the semblance colormap</figcaption></figure><p>This process of flattening the hyperbolic events is called <strong>normal moveout correction (NMO)</strong> and is essential for seismic processing. So how do we manage the velocities?</p><p>For each of the reflectors, we can manually <em>choose the velocity</em> of the signal reflecting from it so that it <em>fits its hyperbola</em> on the seismogram, see Figure 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*gZ3VRgs4Urg3xNJO1LIZTg.gif" /><figcaption>Figure 5: Demonstration of NMO correction with constant velocity picked for single hodograph (blue)</figcaption></figure><p>As we go deeper, we have to <em>meticulously correct </em>our velocities, as velocity increases with depth and actually may surprise us depending on the earth’s structure. Usually, velocity picking at this stage is done <em>manually</em> via narrowing one’s lids while looking at several random gathers and hoping they would <em>extrapolate to the whole survey</em>.</p><p>To <strong><em>automate and correct</em></strong> this process, we utilize the following algorithm:</p><ul><li>Firstly, we define a<em> time grid</em> covering the full-time range of the survey, and for each time we define a <em>grid of possible velocities</em>. We also have to choose some <em>metric</em> to evaluate the coherency of flattened hodographs, such as <a href="https://wiki.seg.org/wiki/Semblance"><strong>semblance</strong></a>.</li><li>Secondly, for each gather we <em>precompute</em> semblance according to the defined grid.</li><li>Next, for each gather we construct a<em> directed acyclic graph</em> so that:<br> 1. Each (time, velocity) pair defines a node of the graph. An edge between nodes A and B exists <em>if and only if </em>time and velocity at node B are greater than at A. <br> 2. Edge weight is defined as a <em>sum of semblance values</em> along its path.</li></ul><p>Then a <strong><em>path with maximal velocity spectrum sum</em></strong><em> </em>along it between any of the starting and ending nodes is found using the <strong>Dijkstra</strong> algorithm and is considered to be the required stacking velocity.</p><p>To reduce the amount of noise it is also beneficial to aggregate several close CMP gathers into <strong>supergathers</strong> and perform the algorithm on them.</p><p>In the Figure 4 on the left, you can see the original CMP supergather, its NMO-corrected version in the center, and the semblance colormap on the right with picked velocities for each time grid (yellow).</p><p>Now we can try to take the <em>mean</em> along the time axis of these NMO-corrected gathers and see our <em>NMO </em>stack in Figure 6:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gRMYH08420u5KO7TN2CcNw.png" /><figcaption>Figure 6: NMO stack</figcaption></figure><p>Well, that resembles earth’s underlying structure, however, the horizons seem to be a bit deformed: we can see abrupt breaches of linear continuity in various places. Can we do better?</p><h3><strong>Static correction</strong></h3><p>Let’s look closer at the interior of the earth. The first layer, or <em>weathering</em>, is never flat. Surveys might contain rivers, mountains, hills, or other god knows what, so sources and receivers are located at different heights above sea level. Yet we are trying to stack the flattest 3D cube without any mountains, and our <em>traces don’t contain explicit information about the landscape</em>. Moreover, elevation changes are often accompanied by sudden velocity changes. Naturally, we want to <strong><em>shift sources and receivers as if they were located on a flat plane with nearly constant velocity</em></strong><em>.</em></p><p>Then we have to understand how the<em> </em>traces should be modified corresponding to this kind of correction. Let us notice that <em>the deeper the reflector is, the steeper the wave should travel to reflect from it</em>. In general, we have more interest in deep horizons than shallow ones, as the latter are much harder to align on the stack. This means that we can utilize a<em> vertical approximation</em> of the raypaths in the upper layers. This leads us to a <strong>static trace correction</strong>: we need to shift traces for a vertical traveltime from the source and receiver to the reference datum and back. This traveltime depends on<em> </em><strong>velocities and thicknesses</strong> of the layers above. See the schema of the static correction in the following Figure 7:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FU3iBys2xI6UIkNFTmI-9g.png" /><figcaption>Figure 7: Static correction schema. Source: <a href="https://utheses.univie.ac.at/detail/3353#">https://utheses.univie.ac.at/detail/3353#</a></figcaption></figure><p>So, how can we calculate these velocities and thicknesses? To do so, one might take into account that apart from reflected waves there are <a href="https://en.wikipedia.org/wiki/Seismic_refraction"><em>refracted</em> ones</a>. They travel <em>faster</em> than reflectors and appear after a<em> certain offset</em>. This indicates that after this offset the<strong> <em>first useful signal is always coming from the refractors</em></strong>. In general, all prominent refractors on a seismogram can be counted with fingers on one hand, and their velocities normally change not so abruptly. That makes refractors great candidates for making our trace shift.</p><p>To introduce such a correction for one of the refractors, we must have the <em>times of first breaks</em>, the <em>velocity of that refractor</em>, and finally <em>velocities and thicknesses of the layers above</em>.</p><blockquote>As for the first breaks, that is a great task for <strong><em>UNet</em></strong>. Check out our other <a href="https://medium.com/data-analysis-center/advancing-the-first-break-picking-with-neural-networks-4d7ea2ea7c2"><em>blogpost</em></a><em> </em>about first break picking.</blockquote><blockquote>Also, we have recently switched to a brand new approach utilizing only one model and <strong>linear moveout correction</strong>. Subscribe in order not to miss the upcoming post about it!</blockquote><p>The next step is to get refractor velocities. Having the first break times it’s not that hard: unlike in NMO, now we know exactly which hodographs to straighten up. Moreover, <strong><em>refractor hodographs are linear as a function of offset</em></strong>. As a result, we get fitted piece-linear regression, with coefficients corresponding to the <em>velocities of refractors</em>, see Figure 8:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1HdfZUHBuwWab-sNiq2VhA.png" /><figcaption>Figure 8: Colormap with first refractor velocity and fitted refractor velocity model for one shot.</figcaption></figure><p>Now we utilize these velocities to fit a layered model to calculate layers’ thicknesses. The <strong><em>fitted model has information about the thickness and velocity of each layer </em></strong>(see Figure 9), which means we can use it to predict the first breaks (yes, again).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LD9SJZLQAYl-bRwcQFURCQ.png" /><figcaption>Figure 9: On the left side colormap of layer 1 thickness with black line denoting a profile on the right side. Profile shows layers’ margins and velocities according to the fitted layered model.</figcaption></figure><p>If we then compare these first breaks with the former ones from UNet, they should be nearly <em>identical</em>. However, for some shots we may encounter a <strong>“sinusoidal shift”</strong> of one picking compared to another: it delays at the receivers on one side and outruns on the other. The example is shown in a Figure 10 below: for traces on the left we observe a lag of the model picking (<em>orange</em>) compared to UNet (<em>blue</em>), and vice versa on the right. For this example we used an open <a href="https://wiki.seg.org/wiki/Stratton_3D_survey"><strong>Stratton 3D</strong></a><strong><em> </em></strong>survey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WQFtUwtSErLYz-l_QqsA9Q.png" /><figcaption>Figure 10: Colormap with Geometry error metric on the left and gather corresponding to red dot on the right withscattered first break picks.</figcaption></figure><p>Pattern like this may indicate that the <em>factual location</em> of the shot was different from the <em>registered</em> one. Thus we can <strong><em>identify geometry errors and recover the actual shot locations</em></strong> by utilizing the structure of the error.<br>Geometry errors are infrequent, nevertheless, their automatic detection is a killing feature, as conventionally geologists do it by scrutinizing every shot gathered, which takes <em>weeks of laborious work</em>.</p><p>Coming back to our pipeline, we can finally perform vertical statics correction, apply NMO, and get our new stack (Figure 11):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KGSn_nv7VS-SsC7fWN-HCA.png" /><figcaption>Figure 11: Stack with static correction applied.</figcaption></figure><blockquote>Follow <a href="https://cdn.knightlab.com/libs/juxtapose/latest/embed/index.html?uid=fe0b4a92-5d44-11ee-b5be-6595d9b17862">this</a> link and compare these two stacks with a slider! Or just enjoy the GIF at Figure 12. Can you spot the difference? Write in the comments which one you like the most!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*jJeZK994iihSbkYmCZZkqA.gif" /><figcaption>Figure 12: Stacks comparison. Left: NMO stack, right: stack with Statics correction applied.</figcaption></figure><h4>Summary</h4><p>Static correction is essential for seismic processing, and as you can see, it consists of many smaller steps. Even first break picking would take a geologist several <em>weeks</em> for a large survey. In contrast, <strong><em>our approach takes only a few hours to go through all the steps and get a stacked cube</em></strong><em>.</em></p><p>Of course, this is not nearly the end of the seismic processing pipeline. There are also essential <em>migration</em>, <em>deconvolution</em> and denoising steps such as <em>multiples </em>and<em> groundroll </em>attenuation.</p><blockquote>For more information about <strong>noise attenuation</strong> on seismic data with <strong><em>ML</em></strong> please check out our previous articles <a href="https://medium.com/data-analysis-center/acquisition-footprint-attenuation-6d7b8365568f">here</a> and <a href="https://medium.com/data-analysis-center/designing-a-metric-for-seismic-denoising-af8fbcf9f809">there</a>.</blockquote><p>Most of these steps are being done with nearly retired software, either with a lot of interpolations or an arduous manual effort. Augmentation of the geology field with computer science methods has great potential to speed up and enhance the quality of all seismic exploration steps, and for now, we are at the very beginning of this route.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92dddee89f43" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analysis-center/seismic-processing-the-earth-shattering-truth-92dddee89f43">Seismic Processing: The Earth-Shattering Truth</a> was originally published in <a href="https://medium.com/data-analysis-center">Data Analysis Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jupyter Notebook: beyond prototyping with NBTools and Research]]></title>
            <link>https://medium.com/data-analysis-center/jupyter-notebook-beyond-prototyping-with-nbtools-and-research-9de635fb3b04?source=rss----9faef2f3af0f---4</link>
            <guid isPermaLink="false">https://medium.com/p/9de635fb3b04</guid>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[resource-management]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Alexey Kozhevin]]></dc:creator>
            <pubDate>Tue, 05 Sep 2023 11:16:06 GMT</pubDate>
            <atom:updated>2023-09-05T11:16:06.187Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g4rGqjVN3qiYOFQOXUj1vw.png" /></figure><p>Jupyter Notebook is a very handy and ubiquitous ML prototyping tool. However, at some point you have to perform reproducible multiple experiments and transfer the code to scripts and lose interactivity. To work with the same notebook at all stages, from baseline to production, our team has developed several tools:</p><ul><li><a href="https://github.com/analysiscenter/nbtools">NBTools</a>: collection of instruments for running Jupyter Notebooks and interacting with them,</li><li><a href="https://github.com/analysiscenter/batchflow/blob/master/examples/research_tutorials/01_introduction.ipynb">Research</a>: a tool for multiple parallel experiments.</li></ul><p>These tools make possible to:</p><ul><li>create reproducible and well-described baseline solution in Jupyter Notebook,</li><li>use it to produce hundreds and thousands of relatively similar experiments in parallel with different parameters,</li><li>make the final Jupyter Notebook and use it in production solution.</li></ul><h3>Baseline</h3><p>Imagine that you received a new task from a customer: for example, to identify <a href="https://paperswithcode.com/dataset/imagewoof">the breed of a dog</a> from a photograph. So, in the course of discussions, you find more details about the problem and get a labeled dataset. Data acquisition is a separate issue that we shall leave out of the equation and, for example, you can read about it <a href="https://medium.com/data-analysis-center/automatic-lithology-description-a-history-of-failure-871d54140650">here</a>. Most likely, you have to iteratively improve the dataset along with the solution.</p><p>Surely you will create a new Jupyter Notebook and write the code in it using your favorite libraries and frameworks. Already at this stage, it is important to have a well-described solution for further work. There are several reasons for this:</p><ul><li>it makes it easier for your team members to collaborate,</li><li>it will be easier to return to code after a while,</li><li>a notebook is immediately a report on the solution.</li></ul><p>We try to maintain a single standard for all notebooks:</p><ol><li>Description of the dataset (number of elements, their statistics, etc.) and data loading code,</li><li>Model description (model type, hyperparameters) and model build code,</li><li>Training process with derived statistics: loss function values, available resources, their utilization,</li><li>Model validation: examples of inference, metrics,</li><li>(optional) Further model improvements and ideas.</li></ol><p>You can find an example of a notebook designed according to the described standard at <a href="https://colab.research.google.com/drive/1tKpE7xfG310HvPC9ySJuhG36SkSM00Dz?usp=sharing">Google Colab</a>. We made it with our BatchFlow library, which has PyTorch wrappers that use configs to describe neural network architectures. This possibility is extremely useful for studying the influence of various parameters on the result.</p><p>In the beginning, you can see necessary imports and the nbtools.set_gpus function, which allows you to select a free available GPU for training the model. While it is easy enough to do yourself, it makes managing multiple kernels and experiments that much easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xxLvB3Zhq6Y9S1sfYrMF6A.png" /></figure><p>Then the description of the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wg6RUa2MSwDxVrlQ_4lwUA.png" /></figure><p>If you need to preprocess the entire dataset and analyze it, it is reasonable to do it in a separate notebook, and add a link to it here. Only now we are writing code for lazy loading of the dataset and show a few examples for clarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XlheCpS5UuQA6M_Qn7lq-w.png" /></figure><p>If a code block in the notebook is complex, move it to a separate class in a different module (e.g., loading and processing data from a specific format). Then the notebook will remain structured and easy to read. This is what we did when solving the problem of detecting cancerous nodules on a CT scan. We moved the designed loaders and handlers to a separate <a href="https://medium.com/data-analysis-center/how-to-make-ct-scan-preprocessing-fast-and-easy-e7e06e51baa3">RadIO</a> library.</p><p>Then we describe model architecture in a model config. Model training in cell 7 is called not just with the tqdm progress bar, but with batchflow.Notifier. It allows you to track resource usage, which suggests potential optimizations (for example, GPU memory is not utilized to the fullest, so we can increase batch size)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NtuAB57EAPgsuhrxlj3uQA.png" /></figure><p>You can do the same if you are not using BatchFlow directly to train the model. It is enough to train the model in batchflow.Monitor as a context manager.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8383d736e7f1a3112fb0b59f979f047c/href">https://medium.com/media/8383d736e7f1a3112fb0b59f979f047c/href</a></iframe><p>Then we provide the code to validate the model and compute metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*qVeFEqPQ7ZqSVfZrosIC6g.png" /></figure><h3>Linter for notebook</h3><p>If you write the code in a separate script, it can be passed through pylint to avoid stylistic and other errors, but it does not work with notebooks. To do this, we have created a function nbtools.pylint_notebook that you can run directly from a notebook. As a result, you will have a full review of the code in cells.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iYQJKfPWG7PQlwoyvhXinQ.png" /></figure><h3>Run notebook with arguments</h3><p>Well, there is a baseline solution: everything works, the data is loaded, the model is trained and even produces some kind of reasonable result. Although the model is one of the state-of-the-art architectures (isn’t it?), but its hyperparameters, most likely, are not chosen in the best way. And maybe we need to perform some data processing procedures (such as normalization) differently. How to try different parameters and evaluate the result?</p><p>There may be several options:</p><ul><li>add some kind of grid search to an existing notebook,</li><li>transform all the code to the script and save all artifacts (statistics, metrics and data examples) to disk or to the experiment tracker</li></ul><p>Both options are reasonable, but we need to rewrite the existing solution sufficiently, with a chance of breaking something when rewriting the code. We propose another option.</p><p>Note that we defined some parameters as constants at the beginning of the notebook. We can define certain parameters separately, like batch size and number of epochs, but we can also set other parameters that we want to test. Ah, if it were possible to make arguments for starting a notebook…</p><p>nbtools.run_notebook does exactly that: it substitutes variables in the desired cell, executes and saves its top-to-bottom executed version in the specified path. In the provided example, we define channels sequence, attention and loss as constants. So now we will substitute different values for the variables in the cell 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fb-VzhPCDTlVMOtkvnkcpg.png" /><figcaption>Cell of the notebook to rewrite.</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5417b527cd5fb249382c0d0007041bc2/href">https://medium.com/media/5417b527cd5fb249382c0d0007041bc2/href</a></iframe><p>In addition, nbtools.run_notebook can return values from any notebook (for example, metric values). In this approach, the baseline notebook requires minimal intervention, and after choosing the most suitable configuration, the notebook that produces the best model is ready to use.</p><h3>Multiple experiments</h3><p>Now we can launch notebooks with arguments, it remains only to do it in parallel and batchflow.research helps us here. We’ve already told about it that <a href="https://medium.com/data-analysis-center/research-a-tool-for-multiple-ml-experiments-fae6206c5d2b">article</a>. All combinations of parameters under study can be conveniently described using Domain class and each call of nbtools.run_notebook will take one of the parameters combinations from it. Parallel execution of experiments on different GPUs will be taken over by Research. See the full example <a href="https://colab.research.google.com/drive/1M5pgZJmHMWP5Otil63E8cnhXgLuB0WMZ?usp=sharing">here</a>.</p><p>GPU utilization is conveniently monitored using nbwatch from the console. There, we can immediately see the memory consumed by various processes and the overall resource utilization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3d2L9DnBilimU_twvaD-kA.png" /></figure><p>As a result, we will get a lot of completed notebooks: you can save them, or you can only get metric values from there. In addition, within the notebook itself, you can implement an experiment tracker, and additionally, for convenience, transfer the ID of the experiment to the notebook (research generates them).</p><h3>Conclusion</h3><p>This paper discusses the use of Jupyter Notebooks in software development and machine learning projects. We present our instruments <a href="https://github.com/analysiscenter/nbtools">nbtools</a> and <a href="https://github.com/analysiscenter/batchflow/blob/master/examples/00_project_from_scratch.ipynb">Research</a>. They help to make workflow, which involves creating a reproducible baseline solution, conducting parallel experiments, and using the final notebook in a production solution. This approach allows us to reduce our labor costs for rewriting the baseline code into code for experiments and further rewriting in production</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9de635fb3b04" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analysis-center/jupyter-notebook-beyond-prototyping-with-nbtools-and-research-9de635fb3b04">Jupyter Notebook: beyond prototyping with NBTools and Research</a> was originally published in <a href="https://medium.com/data-analysis-center">Data Analysis Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making SEG-Y fast]]></title>
            <link>https://medium.com/data-analysis-center/making-seg-y-fast-fda702894907?source=rss----9faef2f3af0f---4</link>
            <guid isPermaLink="false">https://medium.com/p/fda702894907</guid>
            <category><![CDATA[geoscience]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ml-in-geoscience]]></category>
            <category><![CDATA[oil-and-gas-industry]]></category>
            <category><![CDATA[seismic-interpretation]]></category>
            <dc:creator><![CDATA[Sergey Tsimfer]]></dc:creator>
            <pubDate>Mon, 15 May 2023 12:09:09 GMT</pubDate>
            <atom:updated>2023-05-15T12:09:09.881Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T-OeQLDAO4c0nBmqpVZMww.png" /></figure><p>Machine learning community has come a long way in terms of working with data. In the past, we could have stored everything in RAM and enjoy blazingly fast access to inputs and targets for model training. Nowadays, with data becoming much bigger and spanning terabytes of disk space, researchers have to employ optimized ways of loading chunks of datasets into memory, utilizing parallelization for I/O operations and preprocessing.</p><p>Unfortunately, models for seismic exploration are not at the same level of optimization; that is, until now. To squeeze those speed ups for our automated processing and interpretation pipelines, we’ve developed a library <a href="https://github.com/analysiscenter/segfast">segfast</a>, that outperforms other SEG-Y loaders when it comes to raw reading time. Moreover, it is developed with machine learning in mind, making API convenient for continuous loading of training batches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*izNzkTtcvAGtVfPkDj2Sng.png" /><figcaption>Spoiler: average time taken for loading a huge batch of data from SEG-Y for different methods</figcaption></figure><blockquote>Throughout the article, we will be using examples in <a href="https://github.com/analysiscenter/segfast">segfast</a> and our other library, <a href="https://github.com/gazprom-neft/seismiqb">seismiqb</a>, which is based upon it and automates seismic interpretation with machine learning.</blockquote><h3>SEG-Y standard intro</h3><p>SEG-Y is the most popular format for transferring seismic data, both at processing and interpretation stages. Under the hood, each file consists of two logical parts:</p><ul><li>binary (possibly, extended) header, that contains file-wide information about the survey. Data type, number of samples, sample interval and even some textual information are all stored there;</li><li>trace-wise information, that consists of fixed-length header (again, describing meta-info about that trace) and actual values.</li></ul><p>As traces are stored separately, loading a continuous chunk of data <br>(2D slice or 3D sub-volume) would not be very effective, especially if done within Python, where most of ML models live. Fortunately, an excellent library by Equinor, <a href="https://github.com/equinor/segyio">segyio</a>, somewhat alleviates that problem by providing a Python binding to the C++ core functions. But, while providing faster way to load particular patterns from SEG-Y file, this approach lacks flexibility.</p><h3>Bufferization is key</h3><p>One of the best ways to speed up things is to avoid unnecessary copies and read everything directly into the same memory space, which will be later used for model training or inference. Consider a trivial example:</p><pre>paths = ...<br><br>images = [load_image(path) for path in paths]<br>batch = np.stack(images, axis=0)         # used for model training</pre><p>Here we first load images as separate objects, which are later stacked into continious array. That requires (at least!) two allocations of batch-sized memory blocks: one for the list of separate images and the other for batch.</p><p>One of those allocations can be removed: by specifying, which block of memory to load the image into, we could have spared one copy. Imagine if we could have been able to do something like this:</p><pre>n = 128<br>crop_shape = (1, 512, 512)<br>dtype = np.float32<br>paths = ...<br><br>buffer = np.empty((n, *crop_shape), dtype=dtype)<br>for i, path in enumerate(paths):<br>    load_image(path, buffer=buffer[i])   # make buffer[i] equal the image</pre><p>This second type of API, where we can target specific memory block to load the data into instead of allocating new memory, may be significantly faster and migrating to it is well-worth the effort. While working with seismic data, processing-stage cubes can span terabytes of disk space and take a lot of time just to scan through: making even one additional copy on the way is very prohibitive. Sad to say, this fine control over the loading process is not something provided by the current <a href="https://github.com/equinor/segyio">segyio</a>.</p><blockquote>It is worth noting that by using some not-so-public methods of <a href="https://github.com/equinor/segyio">segyio </a>(looking at you, <a href="https://github.com/equinor/segyio/blob/de1db4e4501b919d1b79723e521fb3a291999a77/python/segyio/segyio.cpp#L991">xfd.gettr</a>) one can achieve bufferization at the level of traces: that is, we can specify a memory address for one trace to be loaded in. Unfortunately, that would make us query each trace load at Python level and be at the mercy of GIL. That would be still faster than public methods of the same library, at least for some shapes and number of loaded subvolumes, but <a href="https://github.com/rhettinger">there must be a better way!</a></blockquote><h3>Data conversion</h3><p>Before digging deeper into faster SEG-Y loads, are there any easy (and better) ways to make I/O quick? Indeed there are! As we’ve discussed before, the main drawback of SEG-Y format is that it is trace-wise: any other loading pattern would inherently be slow.</p><p>Thankfully, we don’t have to stick to that one format! There are well-designed and optimized scientific formats like <a href="https://www.hdfgroup.org/solutions/hdf5/">HDF5</a> and <a href="https://zarr.readthedocs.io/en/stable/">Zarr</a>, that have convenient Python bindings, provide chunking of arbitrary shape and compression, which can both speed up reads and conserve disk space.</p><p>This approach is great for researches but pretty much unusable in production or applications. For example, our <a href="https://medium.com/data-analysis-center/seismic-horizon-detection-with-neural-networks-part-1-5dec9ff7361a">horizon detection</a> model takes just a few (4–7) minutes to complete. Adding the conversion time, which can easily take up another 15 minutes for formats with compression and tight chunking, would kill any interactivity. The same goes for fault probability computation: instead of plugging in your seismic data and getting images nearly in real-time, one would need to wait for the conversion step to be done. Last but not least, managing multiple files in addition to the usual hundreds of horizons, faults, wells, facies etc in your project is incredibly cumbersome and you want to avoid it at all costs.</p><blockquote>Note that all points about memory bufferization still apply: data loading can be significantly speed up by preallocating memory and using it as the target buffer for interacting with either of scientific storages.</blockquote><h3>Making data go brrr</h3><p>Going back to the SEG-Y, we’ve noticed some missed optimization opportunities in regards to memory preallocation. Moreover, except for the first ~3600 bytes the file is a huge well-structured binary. To our luck, <a href="https://numpy.org/">numerical python</a> provides great tools for working with such mediums! <br>As you might have already guessed, efficient work with such a file comes to creating a special memory map over it.</p><p>A memory map is, essentially, an instruction on how the memory is laid out. By carefully selecting parameters of its creation, we can extract requested headers and/or amplitude values from the memmap, treating it as huge 2D array of <strong><em>(traces, samples)</em></strong><em> </em>dimensions. The parameters to initialize memory mapping are:</p><ul><li><strong><em>offset</em></strong>, which is fixed to 3600 bytes almost always for our case. This signals to skip the binary header and get straight to the trace-wise data;</li><li><strong><em>dtype</em></strong>, which tells memory map how big each element is and how to interpret it. In our case, each element is a trace and we want the dtype to describe the size of trace header (240 bytes) plus the number of samples multiplied by item size bytes overall. <br>For example, if each trace consists of 1500 float32 samples, correct dtype would have 240 + 1500 * 4 = 6240 bytes in total;</li><li><strong><em>size/shape</em></strong>, which tells how many dtype-sized elements there are. You can compute it just by dividing the total size of the file (minus the binheader) by the size of your selected dtype.</li></ul><p>By using numpy’s memory map, you can also conveniently name different parts of your dtype. Combining everything, we get:</p><pre>import numpy as np<br>offset = 3600                      # all of those can be found by<br>n_samples = 1500                   # inspecting binary header of SEG-Y<br>dtype = &#39;&gt;f4&#39;                      # and file properties (size)<br><br># mmap_dtype: each trace is 240 bytes of headers and then amplitudes<br>mmap_dtype = np.dtype([(&#39;headers&#39;, np.void, 240),<br>                       (&#39;data&#39;, dtype, n_samples])<br>mmap = np.memmap(path, mode=&#39;r&#39;, dtype=mmap_dtype,<br>                 shape=n_samples, offset=offset)<br><br>mmap[&#39;data&#39;] # convenient access to data, returns (N, 1500)-shaped array<br>mmap[&#39;headers&#39;]         # access to headers as bytes</pre><p>This freshly created memory map can be used for both headers and data retrieval. Importantly, as it is a part of one of the core packages of Python, it is much better optimized than custom solutions; out of the box, it is already 10–15% faster even for slice loading (which is a forte of <a href="https://github.com/equinor/segyio">segyio</a>)!</p><p>Another benefit is that the entire reading process is in pure Python and takes just a few lines of code. Higher level of abstraction makes it trivial to implement new ideas analogously to the already developed.</p><p>The last piece of the puzzle is the indexing: suppose we want to get traces on the inline #345. How do we know which trace indices to use?</p><p>For now, the indexing is not a part of <a href="https://github.com/analysiscenter/segfast">segfast</a> library: instead, we create indexers in our <a href="https://github.com/gazprom-neft/SeismicPro">processing</a> and <a href="https://github.com/gazprom-neft/seismiqb">interpretation</a> modules. The reason is simple: the way you structure your file depends on the stage you are working in. In interpretation, there is a rigorous 3D structure imposed: the file can be oriented in <strong><em>(inlines, crosslines, samples) </em></strong>dimensions. For processing tasks, that is usually not the case and changes from task to task.</p><p><a href="https://github.com/analysiscenter/segfast">segfast</a> wraps the entire memory mapping machinery into oneMemmapLoader class. As a fallback and for a fair comparison, we also implement SegyioLoader and SafeSegyioLoader, which follow the exact same API but rely on private and public functions of <a href="https://github.com/equinor/segyio">segyio</a>. Each of them provides tools for loading trace headers, traces by indices and depth slices.</p><blockquote>Under the hood, we rely on time-proven <a href="https://github.com/equinor/segyio">segyio</a> routines to infer stats from file-wide header. This way, we don’t reinvent the wheel and make sure that everything works with different revisions and flavors of SEG-Y.</blockquote><h3>Timings, timings, timings!</h3><p>All that is left is to compare performance of different ways to interact with the same SEG-Y file. For out tests, a 21-GB sized amplitude data with IEEE float32 format is used. We test following methods for reading:</p><ul><li>public <a href="https://github.com/equinor/segyio">segyio</a> functions. .ilines, .xlines, .trace.raw are used;</li><li>private <a href="https://github.com/equinor/segyio">segyio</a> functions, wrapped into SegyioLoader;</li><li><a href="https://github.com/analysiscenter/segfast">segfast</a> with MemmapLoader, that relies on memory map mechanisms;</li><li><a href="https://github.com/gazprom-neft/seismiqb">seismiqb</a>, our <a href="https://medium.com/data-analysis-center/making-seismic-interpretation-easy-with-seismiqb-ac62d01a477">interpretation library</a>. It relies on MemmapLoader for interacting with SEG-Y data and provides additional layer of bufferization for batch dimension;</li><li>HDF5, also provided by <a href="https://github.com/gazprom-neft/seismiqb">seismiqb</a>. The conversion takes about 4 minutes, during which 3 projections in all orientations are created with no chunking nor compression. In our experience, the performance is relatively similar to Zarr storage so we don’t compare it separately.</li></ul><p>Lastly, we include the <em>segfast-quantized</em> loader: it takes the best from both worlds of convenient geological transporting and fast scientific formats. Essentially, the original SEG-Y standard allows to store values of other than the usual float32 types. By carefully quantizing the data, we can use one-byte unsigned integers instead, reducing the disk usage 4x times and speeding up the loading comparably. Moreover, many geological programs do work with uint8 cubes, which allows us to rip the benefits during both research and production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1kATDj6Ei_MY06jUl1RdBA.png" /><figcaption>The same inline for original float32 dtype and quantized uint8. Visually, there are hardly any changes!</figcaption></figure><p>Each loader was tested against multiple tasks: loading slices in different (inline, crossline, depth) projections, loading 3D crops and even loading a batch of 3D crops. The shape of crops is fixed to (256, 256, 500), and batch consists of 20 of such crops.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6BMA2JLTQVgNz_43nO7CFQ.png" /><figcaption>Average time taken for slide loading across different orientations</figcaption></figure><p>Each experiment is repeated a few hundred times to get reliable averages. The key takeaways are:</p><ul><li>as expected, memory map based methods are much faster, achieving at least 30% speedup compared to traditional <a href="https://github.com/equinor/segyio">segyio</a> usage even before quantization;</li><li>crossline-oriented slices are almost twice slower than inline ones: that is natural due to the data alignment;</li><li>depth-oriented slices are waaay slower, as SEG-Y format is completely inadequate for working with them. HDF5 absolutely smokes the competition here, and if you need those slices frequently (for example, while performing attribute analysis) conversion may be your best bet;</li><li>quantization provides a massive speedup, while not incurring any meaningful information loss for a lot of tasks. For example, all kinds of structural interpretation can be done on quantized cubes;</li><li>curiously, the public methods of <a href="https://github.com/equinor/segyio">segyio</a> are faster than the private for inline slide loading. That is due to a slightly better optimization for this exact task and this advantage is lost for other settings.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oyIREmX4V_XIcoJewdilww.png" /><figcaption>Time taken for data loading</figcaption></figure><blockquote>Another way of speeding up your model is to do all CPU and I/O bound operations in a separate thread or process. Our main library <a href="https://github.com/analysiscenter/batchflow">batchflow</a> does so seamlessly: be sure to give it a go!</blockquote><h3>Even more timings</h3><p>Before loading any data from SEG-Y, we need to infer its structure via analyzing trace-wise headers. By utilizing the same memory mapping mechanism, we speed up this part of workflow massively, allowing to load all headers for the entire file in a fraction of time taken by other packages.</p><p>Another crucial step of most algorithms is to export the data in SEG-Y format: for example, our models compute fault probability for each voxel in the original file, resulting in the same-shaped cube with same trace-wise meta. Again, memmap is far superior than other approaches; it is also trivial to parallelize for large tasks like header reading and array export.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4BHqnKVheGkXCZRIEUUFGQ.png" /><figcaption>Time taken for misc tasks</figcaption></figure><p>Finally, the SEG-Y files can come with legacy IBM floats inside: that happens surprisingly often as a result of old processing software. Working with such files requires data conversion to regular IEEE floats at each access, which takes a lot of time. To circumvent this, we have an utility function to convert the file once into faster data format.</p><h3>Conclusion</h3><p>Today you learned about new library for working with the most popular seismic transport format. As it is developed for the needs of machine learning, it compares favorably against other packages on the most demanding task: data reading, as well as on working with trace headers and exporting results in the same SEG-Y format.</p><p>On the github page of <a href="https://github.com/analysiscenter/segfast">segfast</a>, you can find tutorials and notebooks with examples. Be sure to star the repo and stay in touch for more updates on our processing/interpretation advances!</p><blockquote>Check out our packages that automate <a href="https://github.com/gazprom-neft/SeismicPro">seismic processing</a> and <a href="https://github.com/gazprom-neft/seismiqb">interpretation</a>, and learn how to use neural networks for <a href="https://medium.com/data-analysis-center/advancing-the-first-break-picking-with-neural-networks-4d7ea2ea7c2">first break picking</a>, <a href="https://medium.com/data-analysis-center/seismic-fault-detection-ef78c1f7307c">structural interpretation</a>, <a href="https://medium.com/data-analysis-center/facies-analysis-on-seismic-data-31a356446c27">facial analysis</a> and <a href="https://medium.com/data-analysis-center/solving-differential-equations-using-neural-networks-with-pydens-e831f1a115f">more</a>!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fda702894907" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analysis-center/making-seg-y-fast-fda702894907">Making SEG-Y fast</a> was originally published in <a href="https://medium.com/data-analysis-center">Data Analysis Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Convolutions and Self-Attention or There and Back Again]]></title>
            <link>https://medium.com/data-analysis-center/convolutions-and-self-attention-or-there-and-back-again-a9729bd34faa?source=rss----9faef2f3af0f---4</link>
            <guid isPermaLink="false">https://medium.com/p/a9729bd34faa</guid>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[convnets]]></category>
            <dc:creator><![CDATA[Altynanke]]></dc:creator>
            <pubDate>Mon, 27 Mar 2023 12:44:03 GMT</pubDate>
            <atom:updated>2023-03-28T09:41:12.747Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZK9WDWcsev2L36NIFgys8A.png" /></figure><p>Competition between transformers and convolutional neural networks in computer vision has provoked a rapid evolution of both since the beginning of 2020s. In the last few years, community discovered not only that attention-based architectures are a good fit for CV tasks, but also that convolutional networks still have a great potential for modernization.</p><blockquote>Our team is using CV architectures for Oil&amp;Gaz exploration, and lots of them are already implemented in our <a href="https://github.com/analysiscenter/batchflow">Batchflow</a> library. Check out our other <a href="https://medium.com/data-analysis-center">articles</a> on the matter!</blockquote><p>In the first part of this article we’ll revise the Self-Attention mechanism in Vision Transformers and compare its capabilities with those of convolutions. In the second part, let us briefly discuss the ideas of monumental architectures from both attentional and convolutional worlds which further extended our knowledge of computer vision. Finally, we will examine some of the cutting-edge designs that battle for state-of-the-art results almost right now.</p><h3>Intro</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xtvmxQckanDFdGjVSnbaZA.png" /><figcaption><strong>source: </strong><a href="https://paperswithcode.com/sota/image-classification-on-imagenet">https://paperswithcode.com/sota/image-classification-on-imagenet</a></figcaption></figure><p>Have you met <a href="https://www.image-net.org/">ImageNet</a>? The ImageNet classification benchmark has been a staple for ranking CV models since their dawn. As the accuracy score on ImageNet was growing, we were observing the rise of classic ConvNets like ResNet up until the mid-10s. Then for some time, the community was mostly utilizing computational resources to build smart visual models by applying techniques such as Neural Architecture Search, transfer learning, large model scaling, pretraining, etc. Suddenly in 2020, amid Covid, <a href="https://arxiv.org/abs/2010.11929">Dosovitskiy A. et al</a>. knocked out new state-of-the-art results with Visual Transformer, or ViT. Thus ViT outperformed ConvNets although Transformers came from NLP and work on a completely different mechanism. Eventually, these architectures were all over the computer vision field. Let’s talk about what the hell happened.</p><h3>ViT vs ResNet</h3><h4>Attention</h4><p>Transformers are driven by an Attention mechanism, the idea of which is always the same: calculate the significance of information from the input data and then aggregate it.</p><p>In the picture, you can see an example of a simple weighting a feature map on one of the axes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vRLSiRqms4u6HugZY2C3-Q.jpeg" /><figcaption><a href="https://jalammar.github.io/illustrated-transformer/">Simple Attention example.</a></figcaption></figure><p>Attention is often described as smart pooling: don’t just average over the features (or find the maximum), but calculate a weighted sum with scores that vary based on the input.</p><h4>Query-Key-Value Attention</h4><p>Query-Key-Value Attention in Transformers is a variation of the attention mechanism from retrieval systems. There we need to find a matching key by query and return its value. For example, a search query on YouTube is a query, clip titles are keys, and videos themselves are values. Yet this is an abstraction. In Transformers <em>queries</em>, <em>keys</em> and <em>values</em> are tensors, which are computed from an input, that’s why it’s called <em>self</em>-attention. These tensors are obtained after multiplying feature map X by trainable parameters W. Note that there is no activation function after multiplication by W!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1H1oem1O5dl1urc5znVJXQ.jpeg" /><figcaption><a href="https://jalammar.github.io/illustrated-transformer/">Query-Key-Value Attention</a></figcaption></figure><p><strong>Visual Attention</strong></p><p>What is X if we have a picture as the network input? Let’s look at the architecture of ViT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/0*a_55EdM_eadTysC-" /><figcaption><a href="https://d2l.ai/chapter_attention-mechanisms-and-transformers/vision-transformer.html">ViT architecture</a></figcaption></figure><p>Firstly, an image is partitioned into non-overlapping patches of fixed size, usually 4x4. Then it’s projections to emebedding space are built and fed into a transformer block, which iteratively transforms them (note the skip connections). In other words, the goal of the ViT encoder is to <em>extract an informative embedding for each image patch</em>. This idea comes <em>as is </em>from NLP, where embeddings are built for word tokens (and quite successfully).</p><p>As we found out, Attention layer input in ViT is a matrix with rows corresponding to patch embeddings (and actually one extra row for entire image embedding).</p><h4>Linear Projection</h4><p>Embeddings initialization is also important, so let’s briefly discuss it.</p><p>The default variant is Linear Projection: cut the picture into patches, then flatten the patches into vectors. Next, transform vectors into embedding space using a linear layer, but without the activation function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QeUEZBhU5ZbPLxFhPeXxfA.png" /><figcaption><a href="https://jalammar.github.io/illustrated-transformer/">Linear Projection of one patch</a></figcaption></figure><p>Once this is done, we lose the inductive bias that the patches in the picture have neighboring patches above and below, left and right. This local relationship between patches can be very useful. To add this inductive bias, each embedding is summed up with position vector from matrix E_pos. E_pos is either learned or pre-calculated with engineering hacks to preserve the position information.</p><h4>Visual self-attention</h4><p>Our image patches are represented as a matrix X, so when self-attention is calculated, one obtains a query, key, and value representation for each patch. Then for each patch, one counts a weighted sum of the value representations <em>of all the other patches</em>, where weights depend on the similarity of the corresponding q and k. Finally, this weighted sum is used to update patch embedding.</p><p>You can meditate on this, and doggie will help you meditate: coming query patch, have a look at the keys of other patches, find similar, aggregate, then update the embedding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l2I6LpEC0nQBVsQrreOb8Q.jpeg" /><figcaption>Source: <a href="https://arxiv.org/abs/2203.11926">https://arxiv.org/abs/2203.11926</a></figcaption></figure><p>If you are still confused about attention mechanism, you can just have in mind the basic differences from the convolution, which we discuss in the next section.</p><h4>What are the differences between ViT and ResNet?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cdo67ULHjjNzs0q_9aLWvA.png" /><figcaption>ViT vs ResNet</figcaption></figure><p>Although ViT is capable of long-range dependencies and dynamic change of weights according to the input, it certainly has its limitations, such as implicit information about patch positions and quadratic Attention complexity.</p><p>Moreover, it doesn’t build embeddings for features of different scale, which makes it unsuitable for downstream tasks like image segmentation. One could upscale embeddings of small patches, as did <a href="http://openaccess.thecvf.com/content/CVPR2021/html/Zheng_Rethinking_Semantic_Segmentation_From_a_Sequence-to-Sequence_Perspective_With_Transformers_CVPR_2021_paper.html">SETR</a> for Cityscrapes dataset. However, it will be not so good:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eci0NQAcaKJTpGXq-I1wGQ.png" /><figcaption><strong>Source:</strong> <a href="http://openaccess.thecvf.com/content/CVPR2021/html/Zheng_Rethinking_Semantic_Segmentation_From_a_Sequence-to-Sequence_Perspective_With_Transformers_CVPR_2021_paper.html">SETR</a></figcaption></figure><p>Finally, <a href="https://arxiv.org/abs/2010.11929">ViT</a> needs a lot of pretraining to get a noticeable improvement over ResNet :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*48fMiXqcuYMtcOeRW1FKUw.png" /><figcaption><strong>Source:</strong> <a href="https://arxiv.org/abs/2010.11929">ViT</a></figcaption></figure><p>So it’s not really that impressive, but we already know that it works well. Maybe self-attention and convolution have something in common?</p><h4>Self-Attention vs Convolution</h4><p>There are several papers pondering the equivalence and interchangeability of self-attention and convolutions in some sense. Let us mention two important studies.</p><p><a href="https://arxiv.org/abs/1911.03584">In the first one </a>authors proved the theorem that self-attention can simulate (almost) any convolution, i.e. the self-attention class of layers encloses the class of convolutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6NfPqhCSRhgcwoErVLwhoQ.png" /></figure><p>Do self-attention layers behave like convolutions in NNs? Attention scores visualization showed that there are at least similarities in interpretability: the attention scores on the first layers are suspiciously reminiscent of learned convolutional filters in the last layers of ConvNets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JuRuDgf3XhfBOroIFyosGA.png" /><figcaption><strong>Source</strong>: <a href="https://arxiv.org/abs/1911.03584">https://arxiv.org/abs/1911.03584</a></figcaption></figure><p><a href="https://proceedings.neurips.cc/paper/8302-stand-alone-self-attention-in-vision-models">Another article</a> (a year before ViT!) replaces convolution with a local attention layer in a ResNet-like architecture and confirms that these things are somewhat interchangeable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oJDmTw3ezBv7T77Lf5whvA.png" /></figure><p>Their implementation was at the pixel level in a 7x7 sliding window so that it replaced convolutional layers. Such an approach does not support our self-attention interpretation in terms of the extraction of informative patch embeddings.</p><blockquote>Do a simple exercise and write in the comments what this module can represent!</blockquote><p>Despite the promise it showed, authors found that the resulting model runs slowly due to the inefficient implementation of this local self-attention module.</p><p>All in all, self-attention is similar to convolutions, that is why transformers beat the ConvNets, and it makes sense to dig deeper.</p><h3>Swin vs ConvNext</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iIhLT0zrvhcZ0WE1ojON8g.png" /><figcaption>Source: <a href="https://paperswithcode.com/sota/semantic-segmentation-on-ade20k">https://paperswithcode.com/sota/semantic-segmentation-on-ade20k</a></figcaption></figure><h4>Swin</h4><p>Here comes 2021, and the state-of-the-art on ADE20k segmentation benchmark goes to <a href="http://openaccess.thecvf.com/content/ICCV2021/html/Liu_Swin_Transformer_Hierarchical_Vision_Transformer_Using_Shifted_Windows_ICCV_2021_paper.html">Hierarchical Swin Transformer</a>.</p><p>Swin modifies two things in ViT: firstly, it calculates attention not over the entire image, but in a local window to ease the time complexity. At the next layer, the window is shifted to increase the receptive field:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*Bwjcv4pGwP_C0VJoK2Pv5w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*WHPFf8XPYLKyOb-uiykG-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Quqcg2yJDhhPCMfoMGvpeA.jpeg" /></figure><p>Secondly, SWIN performs patch merging between the blocks, so that one can extract hierarchical representations of different scales for downstream tasks.</p><p>It does get better on segmentation, but usually if one puts in a lot of effort and does a pretraining of a large model.</p><p>Notice that we’ve lost<em> global spatial interactions</em> so that the receptive field is limited to a local window… just like in ConvNets. So their starting point is the same?</p><h4>ConvNext</h4><p>A little later appears <a href="http://openaccess.thecvf.com/content/CVPR2022/html/Liu_A_ConvNet_for_the_2020s_CVPR_2022_paper.html">ConvNext</a>, which is a completely convolutional architecture, and beats Swin across many CV tasks. The authors were iteratively making ResNet as similar to Swin as possible. ConvNext is the first model from today’s zoo that’s already implemented in Batchflow, <a href="https://github.com/analysiscenter/batchflow/tree/master/batchflow">enjoy</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*wDBIeqCrBoFHYIhjQgIsjg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*Gu_9gCZdHEcFF2VsnZUpUg.png" /></figure><p>What did the trick:</p><p>1. Modifying the training process to more Swin-like resulted in additional 2% accuracy, which is great for just training hacks: AdamW, more warmup, 300 epochs instead of 90, augmentations, less aggressive initial downsampling (i.e. 1/4 instead of 1/16),</p><p>2. Larger kernel size, a couple more tricks, and <em>depthwise convolution!</em></p><p>Which, as it turns out, is no accident.</p><h3>Local Self-Attention vs Dynamic Depthwise Conv</h3><p>According to this curious <a href="https://arxiv.org/abs/2106.04263">paper</a>, depthwise convolution and local self-attention are very alike in terms of sparse connectivity (existence of non-global receptive field ), weight sharing(but spatial vs channel-wise), and connectivity pattern.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YENmr_vAircP2h4PBrVyvw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-KUoKbEjrXKlRWD--RxEoA.png" /></figure><p>The only thing that’s missing is dynamic weights. Wait! Dynamic convolutions are already a <a href="http://openaccess.thecvf.com/content_CVPR_2020/html/Chen_Dynamic_Convolution_Attention_Over_Convolution_Kernels_CVPR_2020_paper.html">thing</a>. The idea is to convolve input with a weighted sum of several kernels, where weights are input-dependent.</p><p>Simply adding dynamic kernels does the trick, and authors get similar to Swin’’s result by replacing Self-Attention with Dynamic Depthwise Convolutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aa5NkR0vweMsRpe6C7Lv5A.png" /></figure><p>I’m not sure why ConvNext did not experiment with this version of convolutions, my best guess is the absence of dynamic convolution in PyTorch.</p><p>Local Attention is defeated. Can we do better?</p><h3>Frankensteins</h3><p>You have probably already met them. They have been training among regular models for some time. Their creators tried to make a perfect architecture by combining modules from both worlds. They bind convolution and self-attention. They are great.</p><blockquote>Pause for a second and try to guess: what parameter of a convolution would bring benefits to Swin in terms of expanding the receptive field?</blockquote><p>Dilation!</p><h4>Dilated Neighborhood Attention Transformer</h4><p>One of the most recent Transformer models with new SOTAs achieved is <a href="https://arxiv.org/abs/2209.15001">Dilated Neighborhood Attention Transformer</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*n4zhVnYFWr_HVFSt3EV9NA.png" /><figcaption><strong>Source</strong>: <a href="https://arxiv.org/abs/2209.15001">Dilated Neighborhood Attention Transformer</a>.</figcaption></figure><p><em>Dilated</em>: calculate the attentions in the window with spaces, and obtain a global receptive field!</p><p><em>Neighborhood</em>: like local windows in Swin, but define a neighboring window for each query patch:</p><p>What’s the catch? One has to put in a lot of effort to just launch it and would have to use the <a href="https://www.shi-labs.com/natten/"><em>NATTEN</em></a><em> </em>package. However, authors promise that it runs 40% faster than Swin!</p><p>However, DiNAT did not outplay ConvNets on ADE20k. Despite having a very decent 58 mIoU, state-of-the-art 62 mIoU still belongs to convolutional InternImage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*esRD1wU2KREhjWQkVL8_1A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZZIq6-ELOkeFjenIiFI1g.png" /><figcaption><strong>Source</strong>: <a href="https://arxiv.org/abs/2209.15001">Dilated Neighborhood Attention Transformer</a>.</figcaption></figure><h4>InternImage</h4><p><a href="https://arxiv.org/abs/2211.05778">InternImage</a> presented Deformable convolution v3. The second version contributed to convolutions learnable grid sampling locations, finally bringing the input-adaptive, potentially global receptive field to ConvNets. Dynamic weights were also implemented via learnable modulation scalars. Modulation can be interpreted as another form of dynamic convolution, but with weighting over the spatial dimensions of single convolution kernel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*HwME3Tt4aoh4_oEbIxB7wQ.png" /><figcaption><strong>Source</strong>: <a href="https://arxiv.org/abs/2211.05778">InternImage</a></figcaption></figure><p>V3 augmented depthwise convolution into deformable convolution, and (who would have thought) it became better. Deformable Convolution layers and InternImage blocks are also implemented in the Batchflow.</p><h4>DiNA vs Deformable Convolution V3</h4><p>There should be a chapter comparing DiNA and Deformable Convs, but alas, these are new and such research has not been conducted yet. IMHO they are excellent candidates to compare.</p><h3>Outro</h3><p>There are of course many other tracks of vision models development, take masked transformers for example, but that’s a completely different topic.</p><p>Many ML engineers since 2017 are trying to find out what is that we all need: attention, <a href="https://arxiv.org/abs/2201.09792">patches</a>, <a href="https://arxiv.org/abs/2207.14284">recursive gated convolution</a>, <a href="https://arxiv.org/abs/2109.04553">matrix decomposition</a>… It’s all about the same thing: we need Frankenstein of Convolution and Self-Attention.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9729bd34faa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analysis-center/convolutions-and-self-attention-or-there-and-back-again-a9729bd34faa">Convolutions and Self-Attention or There and Back Again</a> was originally published in <a href="https://medium.com/data-analysis-center">Data Analysis Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automatic lithology description: a history of failure]]></title>
            <link>https://medium.com/data-analysis-center/automatic-lithology-description-a-history-of-failure-871d54140650?source=rss----9faef2f3af0f---4</link>
            <guid isPermaLink="false">https://medium.com/p/871d54140650</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[petrophysics]]></category>
            <category><![CDATA[oil-and-gas]]></category>
            <dc:creator><![CDATA[Alexey Kozhevin]]></dc:creator>
            <pubDate>Fri, 27 Jan 2023 09:42:39 GMT</pubDate>
            <atom:updated>2023-11-13T12:27:05.742Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vh_Wd_PgxCjtfO1jqHIbDA.png" /></figure><p>Unfortunately, not all launched projects are destined successfully. Sometimes difficulties arise unexpectedly which make it impossible to solve the problem, at least for the moment. We will walk you through our attempt to produce an automatic lithology description and highlight the major difficulties that did not allow to find a solution.</p><p>In our opinion, sharing about the “failed” project is none less important than shouting about the successful ones. Such stories enable readers to learn about feasible approaches and deep dive into the task without any commitment. This way future researchers can be aware in advance what things to consider at the very beginning and this article will help to draw conclusions that will save your time and effort.</p><h3>Core analysis as a part of oil exploration</h3><p><a href="https://medium.com/data-analysis-center/seismic-fault-detection-ef78c1f7307c">Seismic exploration</a> is one of the first steps in field exploration. Then comes the stage of a more detailed study of the internal structure <br>To do this, it is necessary to drill exploratory wells to make <a href="https://en.wikipedia.org/wiki/Well_logging">log measurements</a> and extract <a href="https://en.wikipedia.org/wiki/Core_sample">core samples</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I9qTjfs4cL1-soWJF8YPtA.png" /><figcaption>Example of a core 1 meter long</figcaption></figure><p>A core is the only source of ground truth information about the entire field structure, but drilling several wells and conducting such studies is comparable in cost to the entire seismic survey. For this reason, it is important to choose the correct location for the exploration wells and conduct the most accurate analysis of the well data. Lithology description is a part of that analysis.</p><h3>Lithology description</h3><p>The problem of automatic core description attracted our attention because it fits ideally into classification or semantic segmentation task: image as input, lithology class as target. It is enough to get photos of the core, their annotation by experts, and train vanilla ResNet/UNet. What can go wrong?</p><p>We’ve got a dataset of 163 wells from 21 fields. Each well includes daylight (DL) and ultraviolet (UV) photos of core, a CSV file with depth binding, and depth-wise lithology description. UV photos are a significant source of information because hydrocarbons glow in ultraviolet light.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QppcHYZSmXUvA_Ghpkseog.png" /><figcaption>DL and UV photos of core</figcaption></figure><p>Total, 10 kilometres of annotated core photos! Sounds great to train a neural network, doesn’t it? Let’s dig in!</p><h4>Challenge 1: Unstructured description</h4><p>We expected to see several classes of lithologies (sandstone, mudstone, etc.) and their properties (e.g., color and grain size), but the description looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*v9rXO3b8bJFgwqsPYMglPA.png" /><figcaption>Sample from initial unstructured description</figcaption></figure><p>Unfortunately, each description is unique and does not fit any structure to extract the main lithology and its properties. So first we have to figure out how to extract lithology classes. There were two possible ways:</p><ul><li>make an algorithm to structure the description</li><li>structure the description manually</li></ul><p>We took the second path, since the development of the algorithm could take a significant amount of time, and the repetition of this procedure is unlikely. It was decided to extract the formation, its color and grain size and, if exists, the same properties for secondary and tertiary formation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YiAE-iaXKHArr7elYH2SuA.png" /><figcaption>The same structured description</figcaption></figure><p>Vigilant reader can notice that it has errors (see Color2 in the first line). At first it is enough to keep only prime properties to predict (Formation1 and Grainsize1) and convert meters to centimeters for convenience. All descriptions and core photos we import to format which allows to work with data in our open-source library <a href="https://pypi.org/project/petroflow/">PetroFlow</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*2RL19DQYaA-uaaLc2_VNwg.png" /><figcaption>Lithology description in PetroFlow format</figcaption></figure><p>We concatenate lithology and grain size to make classes and also merge some small classes. Totally we’ve got 13 classes (wait… maybe that was the reason for the failure?)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*ZaqqKoPGZocyM0WMzOjCuQ.png" /><figcaption>Statistics of the transformed annotation where FORMATION_MERGED is a resulting label of the class</figcaption></figure><p>Nice, now we can start training. Let’s stack DL and UV photos into an array of shape height x width x 6 where 6 is the doubled number of RGB channels to use as model inputs. The task can be formulated in two ways. The first one is a classification task where crops of a fixed size belong to one class or another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tGiq-h0XHa9dwjuw5IhToA.png" /><figcaption>Classification task scheme</figcaption></figure><p>The second one is a segmentation task where each image is classified depth-wise and the target/prediction is a 1D mask with class label for each depth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*o_lHn8wv1EeGHQ6Uf9BYpQ.png" /><figcaption>Segmentation task scheme</figcaption></figure><p>Needless to say that even though both approaches were tested the results were very poor still (e.g., macro f1-score 0.126). Next thing that came to mind is researching the data that revealed a number of problems.</p><h4>Challenge 2: DL and UV photo mismatch</h4><p>Good pair of images must correspond to each other like that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*57D9Uy2_-mknYZlbtK_xxg.png" /></figure><p>But we note that there are a lot of flipped, stretched or shifted images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*KS_JUP826LVuLJY16MjdSA.png" /></figure><p>It turned out that we were training the model on data that has significant errors. So the data set has to be filtered somehow. Since flipped and stretched core photos still look like core photos, we can only detect a discrepancy between the DL and UV.</p><p>It is important to note that such filtering does not guarantee that there are no problems in the remaining images. For example, both images could be flipped relative to lithology at the same time. However, assuming that there are relatively few such cases, even a pessimistic researcher at this stage hopes to get a good result.</p><p>To find all pairs of DL and UV core photos which don’t correspond to each other perfectly we manually labeled “bad” pairs in the dataset of 768 pairs of photos and trained ResNet18 to classify photos. The resulting model allowed to remove obviously problematic images from the dataset of 26083 images. We trained the model again and found out that the quality is still very low.</p><h4>Challenge 3: Controversial annotation</h4><p>A hypothesis was born that there could be errors in the lithology description. To confirm it, experts labeled 150 correctly classified and 150 incorrectly classified photos. This is what the confusion matrix of the results looked like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*0v1vGbxPkDMY1RImiU2jqw.png" /></figure><p>As you can see, two experts can label the same data very differently! At the same time, we have got the results of the <a href="https://en.wikipedia.org/wiki/Optical_granulometry">granulometry</a>. It is a result of a measurement of different grain sizes from cylinders cut out of the core columns. Based on these numbers, the sample can be attributed to one of the types of lithology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*P1LIL77-hUKtNq8Iep0Lkw.png" /><figcaption>Example of granulometry. Each column is percentage of each grain</figcaption></figure><p>There are criteria that make it possible to attribute a sample to one or another class of lithology by granulometry. We’ve investigated granulometry points with tSNE and get the plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*A55S4aiegzgbx2hCYpkpPA.png" /><figcaption>Spatial axes are 2D tSNE of granulometry. Colored labels are lithology from granulometry.</figcaption></figure><p>By design, points are ideally divided into classes. But if one takes lithology from the initial description, then the result will be much worse:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*e6T5C9M1v8c9ksqTofttrg.png" /><figcaption>Spatial axes are 2D tSNE of granulometry. Colored labels are lithology from lithologucal description.</figcaption></figure><p>This implies the conclusion that the lithological description has a significant discrepancy with granulometry data. At this stage, we realized that the historical dataset is not suitable for training a quality model.</p><p>The first way out of the situation is to develop lithology labeling criteria and perform this labeling by several experts in order to aggregate their results and get an accurate average ground truth. The second is to predict grain size directly and then use it to get a lithology class in sample points by some criteria.</p><h4>Challenge 4: Image resolution</h4><p>A historical huge dataset includes low-quality images, approximately 75 dpi. The fact is that the data was quite old, and the quality of the photo on more modern projects is much higher. Together with the experts, it was decided that the available resolution is not enough to predict such a detailed characteristic as the lithology. If you start labeling lithology data again, wouldn’t it be better to do it on high-quality photos? The team managed to get access to several kilometres of an annotated core with a resolution of more than 200 dpi. We were even able to find experts willing to take over the annotation process. However, it turned out that there are no convenient tools for labeling the core.</p><h4>Challenge 5: Annotation tools</h4><p>Such markup can be done either with specialized proprietary software or by manually filling tables with depths and classes. Then came the realisation that it is better to spend some time developing a special tool, and only then use it to label. Since we already had such experience in <a href="https://medium.com/data-analysis-center/how-to-easily-train-a-3d-u-net-or-any-other-model-for-lung-cancer-segmentation-fc88f970d53">our medical projects</a>, the technology stack was clear: JSX/React frontend + python backend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*s_djVglY7-4vkiqAksEhOQ.png" /><figcaption>Interface of core annotator</figcaption></figure><p>In addition to developing a convenient tool for setting boundaries on the core, we have added the ability to select formation and grain from a predefined list of classes to make structured lithology description in <a href="https://github.com/gazprom-neft/petroflow">PetroFlow</a> well format out of the box.</p><p>Now there is high-quality data, a high-quality annotation tool, it remains to label data and return to training models. Hope it happens someday.</p><h3>Conclusion</h3><p>When the dataset of the annotated core photos was at our disposal, we expected that solving the problem of automatically compiling a lithological description would be possible at least within a year. It will only be necessary to experiment with model training procedures: deal with sample imbalance, choose model architectures and optimizers, post-process and aggregate predictions, etc.) However, it turned out that it was impossible to finish the project, because</p><ul><li>the existed description is unstructured and can be subjective</li><li>there are obviously defective photos in the dataset</li><li>low image resolution which is not enough to train an accurate model</li><li>it would be possible to create a new annotated dataset, but there is no convenient tool to do it quickly, uniformly, and accurately</li></ul><p>These problems can also be encountered in other tasks. For example, we went through similar stages when solving the problem of seismic interpretation.</p><p>A well-known principle says “Garbage in — garbage out”. However, it is not always immediately obvious that there are problems with the input data, especially when their volume is large.</p><p>The failure of this case and the success of some others have taught us that it is often better to spend time and resources to create a high-quality dataset and then jump into model training.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=871d54140650" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analysis-center/automatic-lithology-description-a-history-of-failure-871d54140650">Automatic lithology description: a history of failure</a> was originally published in <a href="https://medium.com/data-analysis-center">Data Analysis Center</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>