<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Elucidata - Medium]]></title>
        <description><![CDATA[Elucidata accelerates drug discovery by applying its AI-enabled platform, Polly, to vast heterogeneous semi-structured biomedical data. Polly makes available biomedical data FAIR and transforms it into a ready-to-use form. - Medium]]></description>
        <link>https://medium.com/elucidata?source=rss----ddc0f60dbf8a---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Elucidata - Medium</title>
            <link>https://medium.com/elucidata?source=rss----ddc0f60dbf8a---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 11:45:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/elucidata" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling Data Pipelines for High-throughput Bioinformatics]]></title>
            <link>https://medium.com/elucidata/scaling-data-pipelines-for-high-throughput-bioinformatics-006a815c8376?source=rss----ddc0f60dbf8a---4</link>
            <guid isPermaLink="false">https://medium.com/p/006a815c8376</guid>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[pipeline]]></category>
            <dc:creator><![CDATA[Saiful Khan]]></dc:creator>
            <pubDate>Tue, 18 Jun 2024 08:54:11 GMT</pubDate>
            <atom:updated>2024-06-18T10:44:27.889Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mq7ocdH71mAf7m2g2TKjSQ.png" /></figure><p>Data pipelines are an essential tool in bioinformatics. They enable the standardization of data-cleaning and complex data-processing which in turn results in better reproducibility. Most data pipelines use some kind of framework for executing its tasks. Modern pipeline orchestration frameworks do a good job of abstracting and separating pipeline logic from infrastructure complexity. But that complexity remains to be managed at some level.</p><h3>Latent Challenges in Pipeline Management</h3><p>Before we talk about the specifics of scalability, let us first consider a simplified model of the lifecycle of a bioinformatics data pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w10i7xEVoMxabfisFS90Ew.png" /><figcaption>Lifecycle of a Bioinformatics Pipeline</figcaption></figure><p>Bioinformaticians are primarily trained and skilled at developing algorithms, performing data analysis, and interpreting the biological implications of data. Their expertise is central to understanding and solving biological problems through computational methods. Therefore, they will be highly involved in the development and interpretation stages of the pipeline. However, it is <strong>not a good use of their time to be responsible for managing deployment and configuring execution environments</strong> as well.</p><p>Deployment and maintenance of pipelines in a cloud environment requires understanding of cloud services, configuration management, containerization technologies (like Docker) and CI/CD principles. These tasks are time-consuming and can divert bioinformaticians from their primary role – research and data analysis. Bioinformatics pipelines have highly dynamic storage and computational needs as well. Efficiently managing computational resources to handle variable workloads, optimizing costs, handling scaling issues, and ensuring data security involves skills generally associated with cloud solutions experts. Getting involved with this will be yet another distraction.</p><h4>A Collaborative Solution</h4><p>This is where <strong>Polly Pipelines</strong>, Elucidata’s managed pipelines hosting and execution service can help. You write the code, we take care of deployment and provide a fully-managed &amp; secure infrastructure to run it. You can monitor your executions and finally download the output and reports for further analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-UAs1HtVqg2pxyHXKJkqxQ.png" /><figcaption>Polly Pipelines manages deployment &amp; execution stages completely</figcaption></figure><p>Providing such a service in a multi-tenant (or even a single-tenant) environment is not easy. We need to support highly concurrent workloads w.r.t. both development and execution of pipelines. In this post we will talk about our pipeline deployment and execution architecture to cater to the required throughput and concurrency.</p><h3>Scaling Pipeline Deployment with Polly</h3><p>The deployment of pipelines is often the most boring part of its development phase. It involves compiling or packaging the code, building and pushing execution containers, configuring environment variables and registering or updating the pipeline’s metadata with the orchestration framework. It is a multi-step process that is automated in most production environments.</p><p>In this section we will describe how our unique deployment strategies enable teams to scale both the number of pipelines and the number of contributors without being choked by operational blockers.</p><h4>Isolated Deployment</h4><p>Polly Pipelines gives us the ability to host all our pipelines in a central pipelines repository. This pattern can be replicated for any number of organizations, although we have not done it outside of Elucidata yet. Enterprises often adopt a “<a href="https://en.wikipedia.org/wiki/Monorepo">monorepo</a>” approach to help centralize code governance as well as maintain common utilities &amp; coding standards across pipelines. However, such repositories come with a major challenge – coupled deployment of pipelines.</p><p>At any given point of time, different pipelines in the repository can be in different stages of development. If we deploy the repository’s code because Alice’s Pipeline-1 has finished adding a new feature while Bob’s Pipeline-2 is still testing out a new feature, then we have Pipeline-2 being deployed to production with unverified behavior. This is non-ideal. Trying to manage this situation through people processes, often results in deliveries of one pipeline being blocked by another.</p><p>This is why we have carefully designed our continuous-deployment workflow such that <strong>each pipeline’s code (along with its dependencies) and container image are deployed in isolation from each other</strong>. If a developer makes changes to a pipeline and commits it to a live branch, then they are presented with a prompt for approving the deployment of that pipeline. Therefore, if Alice and Bob make some changes to Pipeline-1 and Pipeline-2, respectively, and commit them to the same branch, then they will be presented with two separate approval holds. Alice will simply approve only Pipeline-1’s deployment, leaving Pipeline-2 unchanged in the live environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eALpjofV1EAlKRPcTIAG-w.png" /><figcaption>Isolated deployment ensures that only intended changes are deployed</figcaption></figure><p>If all pipeline developers practice this simple approach – approve only the pipelines that they have made changes to – then it eliminates the need for any kind of manual coordination between members of a team or between members across teams.</p><h4>Multi-stage Deployment</h4><p>While we support and recommend local testing of pipeline code before pushing it to the main repository, some pipelines can simply not be tested offline on the developer’s local machine. This is most often due to a pipeline’s resource-intensive nature. Additionally, many teams also need a way to stage a new version of a pipeline in “testing” mode before moving it to production. This pre-production stage is where the pipeline may be used by internal peers or QA teams to make sure that the pipeline’s behavior satisfies the acceptance criteria set by the stakeholders.</p><p>To help with these requirements, <strong>Polly Pipelines&#39; registry supports three stages for each pipeline</strong>. The repository has three live branches, one for each stage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eh2tdDs0m376f7oV3VwlMA.png" /><figcaption>Each group (devs, QA specialists &amp; end-users) gets a separate deployment</figcaption></figure><p>Merging a pipeline’s feature branch to the develop branch deploys the pipeline in “dev” mode. When executed, a pipeline in “dev” stage will be run on the development environment. The development environment is completely isolated from other environments. It supports small-scale execution of pipelines in a true cloud environment. It is ideal for testing the pipeline behavior during development.</p><p>Merging a pipeline&#39;s feature branch to the staging branch deploys the pipeline in “test” mode. When executed, these pipelines will run in a “test” environment, once again isolated from other environments. This environment is useful for providing a pre-production replica of the pipeline for verification by peers and QA team members.</p><p>Finally, once the team is confident that the pipeline is ready for production, they can merge their feature branch to the master branch, which will deploy the pipeline to a main production environment. This environment, while isolated from the dev and test environments, has all the compute capacity that Polly Pipelines has to offer, which is what we will discuss next.</p><h3>Scaling Pipeline Execution with Polly</h3><p>Although bioinformatics pipelines are specialized and adapted to the unique demands of biological data analysis, they can still be considered as a type of <a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">ETL</a> workload:</p><ol><li><strong>Extract</strong> – This often involves retrieving raw biological data from various sources, such as publicly available sequencing data (GEO, etc.), public databases (like GenBank, ENSEMBL, etc.), or private lab-generated datasets.</li><li><strong>Transform</strong> – This is the main analytical part of the pipeline. It includes a series of computational steps and tools that process raw data into meaningful insights. This might include quality control, sequence alignment, variant calling, gene prediction, and other bioinformatics analyses.</li><li><strong>Load</strong> – Finally the results, which could be annotated expression data, models or analytical reports, are often stored in repositories and databases for further use and interpretation. Optionally, this output may itself be input to yet another pipeline for other specific use cases.</li></ol><p>We are constantly innovating at each phase of the ETL. This section describes how we are solving the most challenging issues encountered by practitioners when running their bioinformatics pipelines.</p><h4>Seamless Data Import</h4><p>One of the bigger challenges of the data extraction phase is the migration of data from different sources. Each source will have its own authentication mechanism. Additionally, a pipeline that is run often for different customers may need to use different source of the same type (e.g., different S3 bucket) in each run. Managing the code and authentication logic for so many disparate sources for each pipeline is a minor headache for the developers.</p><p>Fortunately, Polly Pipelines comes with the ability to automatically import data from a variety of sources and make it easily accessible to your pipeline runs. In short, the user flow looks like:</p><ol><li>The user opens the pipeline execution form.</li><li>For parameters that can accept files or folders as inputs, the user opens a file browser.</li><li>If the source of files or folders has not been used previously, the user registers the new source with necessary authentication details. For security purposes, we may invalidate auth tokens at pre-defined intervals, after which they must be updated again by the user.</li><li>Once the registered, the user can browse files or folders from their source of choice. These sources remain available for all future runs.</li><li>The user starts the pipeline run and behind the scenes we bring in the specified data into an S3 bucket path accessible to their pipeline run. Each run can only access data imported for their run.</li><li>After the import is completed, the pipeline’s actual execution begins, where they can simply do an aws s3 cp on the path (passed in the place of the file or folder parameter).</li></ol><p>This approach makes writing pipelines a lot simpler for bioinformaticians. They no longer have to manage dozens of secrets and authentication mechanisms. They can focus on the transformation processes of their pipeline instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_JkM1lH23hV2UDmHOc3Y7A.png" /><figcaption>The import feature does all the heavy-lifting of your data extraction process</figcaption></figure><p><strong>For pipelines</strong> <strong>where multiple (potentially large) files are needed, we enable data import on a fleet of serverless workers</strong>. This greatly reduces the amount of time it would take to import large batches of files or folders.</p><p>As of this writing, we are starting with <a href="https://aws.amazon.com/s3/">Amazon S3</a>, <a href="https://polly-docs.readthedocs.io/en/latest/Getting%20Started/Workspaces.html">Polly Workspaces</a> and <a href="https://www.illumina.com/products/by-type/informatics-products/basespace-sequence-hub/order.html">Illumina BaseSpace</a> as our first few sources. However, the Polly Pipelines architecture ensures that we are agile in bringing support for new sources.</p><h4>Uniform Interface Across Executors</h4><p>Most data pipelines are defined in a DSL or framework specifically crafted for that purpose. One such DSL is <a href="https://www.nextflow.io/">Nextflow</a>. It has become increasingly popular in the scientific community due to its flexibility, scalability and ability to streamline complex computational workflows. For this reason, we also chose to go with Nextflow as our primary executor for all bioinformatics workloads.</p><p>However, since then, we have realized that not everyone will want to use Nextflow for their needs. For example, within Elucidata, we have data engineers who need to automate secondary or tertiary transformations &amp; loading of processed data. They prefer a python-native stack and do not need any bioinformatics specific utilities. To cater to their use cases, we created our own pipeline executor called PWL (which stands for “Polly Workflow Language”) written in Python.</p><p>At the same time, we often talk to customers who have already written and verified their data pipelines internally. Their choice of the workflow language could be something else like <a href="https://snakemake.readthedocs.io/en/stable/">Snakemake</a> or <a href="https://github.com/openwdl/wdl">WDL</a> but we still want them to benefit from all the features and scalability that Polly Pipelines has to offer.</p><p>Very soon, it was evident that to support the diversity of skilled professionals we will need a multi-framework platform. Therefore, we have carefully crafted programmatic and graphical interfaces without letting one framework dictate any of our choices. <strong>These interfaces will allow us to bring in new pipeline executors based on the demands of customers</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WpImzo8iMW7EsY4zBWk6fg.png" /><figcaption>Polly Pipelines’ API abstracts all executors behind a uniform interface</figcaption></figure><p>UX patterns are a neglected aspect of scalability. We firmly believe that providing a unified interface for running pipelines in any executor is the best way to serve the diverse and rapidly evolving field of bioinformatics.</p><h4>Horizontally Scaled Execution</h4><p>Modern bioinformatics projects often deal with massive datasets, such as genomic sequences, proteomic data, and large-scale imaging data. Scaling out allows the distribution of these large datasets across multiple computational nodes, facilitating efficient processing and analysis. This scale out can happen on any given axis based on the pipeline logic (most commonly “tasks” or “processes”). For example, transcriptomic datasets often come with multiple samples. The processing of each sample can be pushed a separate compute node, thereby significantly reducing the overall time required to complete the analysis. This is crucial for time-sensitive research.</p><p>Polly Pipelines automatically implements this scale out pattern based on the definition of your workflow. Any tasks that can be run parallelly will be run as such without the pipeline developer having to specify anything other than how compute intensive their tasks are. A child task only moves forward when all its parent tasks have finished executing. So the infrastructure is built in a true <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a> fashion, making the most effective usage of available compute.</p><p>And since we are availing compute from large cloud providers, our reserve is potentially limitless. However, in practice we do put certain restrictions on the number of tasks that we run parallelly. This is purely a cost-control mechanism and we will increase this limits should we observe that workloads are reaching our maximum capacity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*huPZ5FY2JP7cB-QEPdnhYA.png" /><figcaption>The pipeline execution infrastructure scales relative to the demands of running jobs</figcaption></figure><p>To give a sense of the current scale of our infrastructure, we have estimated that <strong>Polly Pipelines can successfully process 5k GB-samples of transcriptomics data per week</strong> through our custom RNA-Seq pipeline, which uses kallisto for (pseudo-)alignment &amp; quantification. In simple terms, this means we can process 5000 samples weekly where each sample is 1 GB in size.</p><p>Apart from raw compute power, we ensure that each executor has its own priority queues and execution clusters. Not only that, each stage of a pipeline gets its own execution cluster as well. Pipeline developers can expect each of these clusters to exhibit similar scalability semantics.</p><h4>Bottomless Storage per Node</h4><p>Bioinformatics pipelines have high storage capacity needs due to the nature of the data they handle and the computational processes involved. Each process may itself generate more data for the next process to be used. Moreover, this I/O happens on “local” storage of the nodes participating in pipeline execution. It is difficult to estimate how much storage a given pipeline run needs. It is difficult still to provision disks dynamically based on each run’s demands. So jobs frequently used to fail with out-of-disk errors. We therefore had to come up with a new design for the storage attached to compute nodes.</p><p>All compute instances in the cloud come with block storage of some kind. Either it will be a disk directly attached to the instance or a volume connected over the local network. These volumes have a fixed capacity. If a process continues writing more data to the volume, it will eventually reach the volume’s capacity. Normally, we would expect the process to fail at this point. However, if we are running our process on Polly Pipelines&#39; infrastructure, the user will never really find out whether they ran out of capacity. Internally, as soon as the instance’s disk space is exhausted we switch to a remote volume (using the <a href="https://en.wikipedia.org/wiki/Network_File_System">NFS</a> protocol). This remote volume can store an extreme amount of data and our <strong>processes effectively never run out of disk space</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hWYgMrG6RAZnsvk1E5WMCg.png" /><figcaption>A tandem arrangement of local &amp; NFS volumes maximizes storage capacity</figcaption></figure><p>The caveat here is that the NFS volume has a maximum throughput limit. If too many nodes are writing data concurrently to this volume, we may see slowdowns in pipeline runtime. To counter this effect, we simply provision more NFS volumes based on the load on our execution cluster. Tasks or processes from the same pipeline run, however, do share the same NFS volumes.</p><h4>S3 for Intermediate Data</h4><p>Data on disk is not the only produce from pipelines. Any non-trivial pipeline will have multiple tasks that execute in parallel as well as in succession. Each successive task needs the output from its parent task. So we need a data-exchange layer for storing this intermediate data. Finally, once a run completes, the output is generated which also needs to be saved durably.</p><p>For both of these needs we use Amazon S3. Intermediate data can be written by hundreds of tasks simultaneously, totalling terabytes in size. Most traditional storage devices will not able to handle this kind of write throughput. The same goes for reads as well. Amazon S3, being a distributed file-system, has no problem under this kind of load. It is <strong>built precisely to handle large number of concurrent read and writes</strong>. Moreover, it provides <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html">eleven 9s</a> of durability guarantee.</p><p>Similarly, we also store the pipeline’s final output in S3. But based on the instructions of the pipeline, the developer is free to upload that data wherever they want and in whatever format they want.</p><p>So far we have never faced an issue while using Amazon S3 for any of our pipeline runs, in terms of concurrency as well as durability.</p><h3>Network and Security in Polly Pipelines</h3><p>One final aspect of Polly Pipelines worth highlighting is security through network isolation. Bioinformatics often involves dealing with sensitive data. It is imperative to make sure that pipelines running on the cloud, especially in a multi-tenant systems, are operating with strict access controls in place. Therefore, we make sure that:</p><ul><li>Pipelines runs remain unaware of other pipeline runs. They can never access input, intermediate data or output from other pipelines. Even logs generated by the pipelines are kept in dedicated log streams.</li><li>Once your data is imported into the execution system, it never leaves our private network. Of course, the pipeline developers can still choose to push it to other remote data stores should they wish to do so.</li><li>For production environments, all Elucidata personnel (except system administrators) are restricted from viewing or downloading any data processed or generated by the pipeline execution.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c-YaeLHb9pgHp70hfiyzwA.png" /><figcaption>Polly Pipelines enforces strict access control and data isolation for each run</figcaption></figure><p>We also understand that there are organizations that deal with <a href="https://en.wikipedia.org/wiki/Personal_data">PII</a> data and have to follow stricter security policies. We recommend them to go with a single-tenant (enterprise) deployment of Polly Pipelines. This way, their data and their compute is completely owned and governed by the organization.</p><h3>Future Work</h3><p>There is still much work that we want to do to make sure that Polly is the best place to develop and run bioinformatics pipelines. A few ideas that we are currently entertaining are:</p><ul><li><strong>Enable a bring-your-own-pipeline journey</strong> – As mentioned previously, as of this writing, all pipelines we run are managed in a central repository by us. But often bioinformaticians would want to control their own code and <a href="https://en.wikipedia.org/wiki/Systems_development_life_cycle">SDLC</a> for pipelines. Letting them upload their pipeline code &amp; register it through our programmatic interfaces (library or CLI) will help us achieve that outcome.</li><li><strong>Support more executors</strong> – We plan to support more pipeline definition and orchestration frameworks, based on popular demand. The one we are currently ideating is Snakemake. Apart from Nextflow, it is probably the most widely used workflow management system in bioinformatics.</li><li><strong>Improve local storage throughput</strong> – Because of the shared nature of our NFS volumes, they are prone to becoming points of slowdown for our pipelines. Worker nodes will divide the overall throughput amongst themselves and depending on the number of nodes, their read/write speed can suffer. To counter this, we want to further introduce a storage layer for our nodes such that the I/O throughput remains independent on the number of concurrent workers. We have some ideas, but we will need a whole another post to talk details.</li></ul><p>In this post we have shown how Polly Pipelines is a versatile, scalable, and developer-friendly platform for writing bioinformatics pipelines. Its rich feature set and attention to every stage of a pipeline’s lifecycle makes it a good choice for bioinformaticians.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=006a815c8376" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elucidata/scaling-data-pipelines-for-high-throughput-bioinformatics-006a815c8376">Scaling Data Pipelines for High-throughput Bioinformatics</a> was originally published in <a href="https://medium.com/elucidata">Elucidata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Biomedical Data & ML-ops Platform: From Collection to Discoveries]]></title>
            <link>https://medium.com/elucidata/building-biomedical-data-ml-ops-platform-from-collection-to-discoveries-e0ef8a40b2a5?source=rss----ddc0f60dbf8a---4</link>
            <guid isPermaLink="false">https://medium.com/p/e0ef8a40b2a5</guid>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Sahil Rai]]></dc:creator>
            <pubDate>Tue, 14 May 2024 08:07:36 GMT</pubDate>
            <atom:updated>2024-05-14T08:08:47.629Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AW1z2uCPUhwCSblWBi8Ozg.jpeg" /></figure><p>Life science research presents a series of hurdles from the moment you collect your data to the moment you gain meaningful insights. Biomedical data is messy, and complex, and comes in all shapes and sizes. Turning this data into groundbreaking discoveries takes some serious work, and each step has its own hurdles.</p><p>First, we gotta clean the data… think about cleaning typos, fixing inconsistencies and ensuring everything speaks the same language.</p><figure><img alt="Biomedical Data &amp; ML-ops Platform" src="https://cdn-images-1.medium.com/max/1024/1*0GwPi6ZVwmChFO7_RWtG9g.jpeg" /></figure><p>Then we gotta annotate it with metadata… adding content to data to give it meaning and relevance.</p><figure><img alt="Biomedical Data &amp; ML-ops Platform" src="https://cdn-images-1.medium.com/max/1024/1*4bzx6x5lWj5nmrE6rFtrVQ.jpeg" /></figure><p>Next, a huge problem is to do this at scale, which demands robust systems… to ensure it’s organized, accessible and secure.</p><figure><img alt="Biomedical Data &amp; ML-ops Platform" src="https://cdn-images-1.medium.com/max/1024/1*UOSYuKx00f27FgiFzaECCA.jpeg" /></figure><p>Just when you think you’ve weathered the storm, there’s the complex task of running machine learning models or statistical analyses, each with its own set of intricacies and nuances to navigate.</p><h3>Challenges Associated with Data Cleaning &amp; Data Harmonization</h3><p>At Elucidata, our extensive collaboration with pharmaceutical companies of varying sizes spans numerous years. Over this period, we have undertaken the tasks of ingesting, processing, curating, and securely storing vast quantities of both public and proprietary data. Our efforts have empowered clients to extract invaluable insights from this data. Throughout the evolution of our ML-ops platform, Polly, we have encountered and <strong>recognized key challenges that present significant hurdles</strong> in data processing, management, and insight generation.</p><p>Let’s delve into an overview of these challenges below:</p><ul><li>Scalable, cost-effective data processing</li><li>Ensuring high-quality data curation</li><li>Effective data storage and management</li></ul><p>Let’s look at it, one by one!</p><h3>Data Processing: Going from Solo Server to Supercharged Chaos!</h3><p>So, you’ve got your pipeline humming along nicely on your trusty single server or personal computer. It’s all smooth sailing until you start dreaming big — like, a thousand times bigger. Suddenly, you’re facing a whole new set of challenges that make your initial setup look like child’s play. Sure, setting up your own pipeline seemed pretty straightforward at first. But when you try to replicate that magic on a much grander scale, things start to get interesting. And by interesting, I mean complicated.</p><p><strong>One of the first speed bumps you’ll hit is infrastructure scaling issues.</strong> Your once-mighty server is now struggling to keep up with the tsunami of processing demands. And let’s not forget about storage. As your datasets balloon in size, storage limitations rear their ugly heads. Suddenly, you’re playing a game of Tetris with your data, trying to cram it all into whatever space you have left.</p><p>To tackle these challenges head-on, you’ll need to roll up your sleeves and get down to some serious optimization at the infrastructure level. We’re talking about fine-tuning every nook and cranny of your setup to squeeze out every last drop of efficiency. And let’s not forget about the elephant in the room — <strong>cost</strong>. As your operations expand, so do your expenses.</p><h4>Solution? Polly Pipelines</h4><p>Polly Pipelines represents a specialized workflow orchestration system tailored to the complexities of biomedical data processing. Given the vast and intricate nature of biomedical data, a pivotal aspect of the system is its capacity to seamlessly expand in computing power and storage capabilities to meet escalating data requirements.</p><p>For a visual overview of the high-level architecture of Polly pipelines, refer to the diagram below:</p><figure><img alt="Biomedical Data &amp; ML-ops Platform" src="https://cdn-images-1.medium.com/max/1024/1*OFp-DP_Kr6NpjAgEOLtiXA.jpeg" /><figcaption>HLD — Polly Pipelines</figcaption></figure><p>Some key features of Polly Pipelines:</p><ul><li><strong>Flexible Scripting:</strong> Craft your pipelines with your choice of language — Polly Pipelines supports Nextflow and Polly Workflow Language (PWL). We have plans to support Snakemake too in the future!</li><li><strong>Elastic storage:</strong> No need to pre-allocate storage while running your pipelines! Our elastic file storage seamlessly adapts to handle massive datasets and is performant</li><li><strong>Cost-effective processing: </strong>Leveraging Spot instances, the infrastructure reduces expenses by as much as 70%, with fail-safe mechanisms in place to handle instances reclaimed by AWS.</li><li><strong>Effortless scalability: </strong>The infrastructure dynamically adjusts its capacity to accommodate heavier workloads, including intensive pipelines and parallel executions, scaling down to zero when idle.</li><li><strong>Data Importers:</strong> Effortlessly import data from diverse sources like S3 and Basespace, streamlining your workflow.</li><li><strong>Resume post failure:</strong> The infrastructure automatically saves the pipeline’s state, allowing it to resume from the last saved checkpoint in case of failures.</li><li><strong>GUI and Programmatic Interface:</strong> Choose between a user-friendly graphical interface or a powerful programmatic interface (Python SDK) for automated workflows.</li><li><strong>Clear Visibility:</strong> Gain complete insights with our dedicated monitoring dashboard. It visualizes detailed logs for each pipeline run, ensuring transparency and smooth troubleshooting.</li></ul><h3>High-quality Data Curation at Scale</h3><p>Welcome to the world of life science research, where knowledge is endless and data is everywhere. In this busy field, organizing data is crucial. Think of it like sorting through a huge library to find the right books. Detailed information about each dataset helps scientists find what they need quickly. Structured and annotated metadata emerge as the unsung heroes, empowering researchers to navigate with precision and clarity, facilitating seamless retrieval and utilization of invaluable data resources. To do this, researchers resort to curating their data.</p><p><strong><em>But, curation is hard.</em></strong></p><p>Especially for biomedical data. Biomedical data, with its genomic, proteomic, and clinical types, presents unique complexities for curation. Each type has distinct formats and standards, making curation challenging. Research projects often need customized data solutions, viewing curation as a bespoke process. This precision-driven approach extends curation time, demanding significant effort. Additionally, quality concerns like missing values and errors require meticulous attention and rigorous quality control.</p><p><strong>Solution? AI-assisted Curation on Polly</strong></p><p>High-quality data is paramount for deriving accurate and meaningful insights. While manual curation ensures superior quality, it is hindered by its time-consuming nature and lack of scalability. On the other hand, automatic curation offers scalability but may compromise on accuracy compared to manual curation. Recognizing the need for both efficiency and precision, Polly curation adopts a hybrid approach. Data is first auto-curated using AI models and then curators verify the curation done by the model. By leveraging artificial intelligence to aid curators, Polly expedites the curation process while upholding rigorous quality benchmarks, such as adherence to FAIR data principles, ontological accuracy, and comprehensive coverage. This hybrid model combines the strengths of manual and automatic curation, ensuring optimal data quality and scalability for enhanced insights.</p><figure><img alt="Biomedical Data &amp; ML-ops Platform" src="https://cdn-images-1.medium.com/max/1024/1*e45SBMOgTwut6cNtEwPsbA.jpeg" /><figcaption>HLD — Polly Curation</figcaption></figure><h3>Effective data storage and management</h3><p>How data is stored profoundly impacts the depth of insights it yields. Even with access to top-tier data, finding answers to all inquiries isn’t guaranteed. To excel in this arena, two key components are indispensable:</p><ol><li>A superior data model adept at organizing the wealth of information at your disposal.</li><li>An efficient storage infrastructure capable of accommodating vast volumes of data, while also functionally equipped to aid in discovery and analysis processes.</li></ol><p><strong>Solution? Polly Atlases</strong></p><p>Polly Atlas serves two primary functions: Firstly, it aids in the organization and management of data. Secondly, it facilitates the analysis and exploration of data in optimal ways.</p><p>Bioinformaticians rely on a diverse range of tools and methodologies to extract meaningful insights from biological data. For example, a bioinformatician exploring gene expression patterns under environmental stress conditions requires access to a search interface capable of retrieving relevant datasets matching specific experimental parameters and associated keywords. Similarly, in another scenario, bioinformaticians may develop machine-learning models to identify disease biomarkers. These are just a few examples of the many potential applications. Polly Atlases offer flexibility to accommodate numerous use cases, enabling users to effectively consume data and derive insights.</p><figure><img alt="Biomedical Data &amp; ML-ops Platform" src="https://cdn-images-1.medium.com/max/1024/1*paM5mTf3NqkLMAZCNvPIrQ.jpeg" /><figcaption>HLD — Polly Atlases</figcaption></figure><h3>Polly as an ML-ops Platform</h3><p>Polly’s robust infrastructure makes it an excellent ML-ops platform for curation, QCing, and for downstream consumption. Its architecture is designed to streamline the entire machine learning lifecycle — from data preparation to model training, deployment, and monitoring foundational models across your harmonized data for further downstream use cases and analyses- like patient stratification, meta-analysis, biomarker predictions, target identifications, and more.</p><p>Drive successful ML initiatives for 75% faster insights and unlock deep biomolecular insights from your harmonized data. Work with our domain experts to perform metadata-based exploration, differential expression, build knowledge graphs, develop interactive dashboards, etc., to deep-dive into data for robust insights.</p><p>To conclude, I’d just say that the challenges discussed above are just the tip of the problems. A well-established biomedical data platform would require many more components that are not discussed as part of this blog. We have not even covered the data security and governance part of such a platform. Maybe an idea for the next blog?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0ef8a40b2a5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elucidata/building-biomedical-data-ml-ops-platform-from-collection-to-discoveries-e0ef8a40b2a5">Building Biomedical Data &amp; ML-ops Platform: From Collection to Discoveries</a> was originally published in <a href="https://medium.com/elucidata">Elucidata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How PollyBERT Outperforms Large Language Models in BioNER]]></title>
            <link>https://medium.com/elucidata/how-pollybert-outperforms-large-language-models-in-bioner-9dc072542c9b?source=rss----ddc0f60dbf8a---4</link>
            <guid isPermaLink="false">https://medium.com/p/9dc072542c9b</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[gpt-3]]></category>
            <dc:creator><![CDATA[Elucidata]]></dc:creator>
            <pubDate>Mon, 21 Nov 2022 13:57:29 GMT</pubDate>
            <atom:updated>2022-11-21T13:57:29.120Z</atom:updated>
            <content:encoded><![CDATA[<p>Large Language Models (or LLMs) are humongous models usually in the order of tens or hundreds of gigabytes which are trained on huge amounts of training data, which are generally in terabyte to the petabyte scale.</p><p>One of the most famous LLMs rights is <a href="https://en.wikipedia.org/wiki/GPT-3">GPT-3</a> with about 175 billion parameters and a 45TB big corpus. Some researchers calculated that if we were to train GPT-3 again using 1024 A100 GPUs it will still take us about 34 days [ <a href="https://arxiv.org/abs/2104.04473">1</a>]. It’s easy to see why a lot of individuals and organizations cannot afford to train an LLM.</p><blockquote><em>The silver lining with LLMs is that they are few-shot learners, which means that due to the sheer number of parameters and size of the training dataset they are able to perform tasks without being fine-tuned or with minimal tuning.</em></blockquote><p>Recently, we came across an <a href="https://towardsdatascience.com/a-quiet-shift-in-the-nlp-ecosystem-84672b8ec7af">article</a> that showed some interesting results using prompt design. The purpose of the article was to highlight the task-solving capabilities of LLMs which meet two conditions — the number of parameters in the model exceeds 100 billion and the amount of training exceeds 10 23FLOPS. This leads to the emergence of the model which gives it the capability to solve tasks without being specifically trained for them [3].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IwVlD4s_DPCqc4GO.png" /></figure><p>In the past, we have seen that vanilla natural language models tend to struggle in the biomedical domain. They usually need some fine-tuning or pre-training with biomedical data to give good results. Due to this our hypothesis is that the Quality of our dataset is more important than both the size of the model and the size of the corpus.</p><p>This intrigued us to explore LLMs with biomedical text data, specifically for Named Entity Recognition (NER) task and compare them against our BERT models which we use for NER on textual metadata from biomolecular datasets.</p><p>Before we jump to the experiments and numbers, let’s first define two more terms — <strong>domain-specificity</strong> and <strong>task-specificity</strong>, and what we mean by <strong>Quality of a dataset</strong>.</p><p><strong>Domain-specificity </strong>can be defined as the similarity between the domain of our dataset and the domain of the use case or problem statement we are developing a model for.</p><p><strong>Task-specificity </strong>can be defined as the similarity between the task that needs to be performed for our use case or problem statement and the task our dataset is built for.</p><p>According to our definition of the <strong>Quality of a dataset</strong>, it is a property of a dataset that will affect how well can a model learn from that dataset and this quality can be increased for a dataset by increasing domain-specificity and task-specificity.</p><p>Task-specificity should be an intuitive factor that can increase model performance. To learn patterns, a model needs a task to perform, and learn from the mistakes it makes while performing that task. While <strong>it is possible for a model to learn different tasks by using just one general task, empirically we have seen that fine-tuning for specific tasks yields better results</strong>.</p><p>Why domain specificity can help us train a better model, can also be explained using the distribution of our training and test sets. For Empirical Risk Minimization learning algorithms (which are almost all the supervised learning models we use), we need our training and test sets to be sampled from the same distribution. In the real world, after we serve a model we don’t have control over the test set but by <strong>increasing domain-specificity according to the use case, we can increase the similarity between the distribution of train and test examples</strong> since examples from the same domain will be similar. This becomes a bigger factor for biomedical data due to words that are unique to this domain.</p><h3>What Is Prompt-based Learning and How Do We Make Our Prompt?</h3><p>Prompt-based learning is an ML model training method in which we describe our task to perform with/without some examples. This description is then given to a pre-trained LLM for filling the missing information which is used as the output for the said task.</p><p>To create our prompt, we used the paper <a href="https://arxiv.org/pdf/2203.01543.pdf">QaNER</a> as a reference. In their paper, they suggested a Question Answering based NER framework and a method to convert NER problems to QA problems which we have used to create our prompts [ <a href="https://arxiv.org/abs/2203.01543">4</a>].</p><p>Along with methods from QaNER we also created 10 prompts and empirically selected the best prompts using a small test set. This was done to get the best prompt we can get so that the prompt is not a bottleneck and we can have an apples-to-apples comparison.</p><h3>How Do We Fine-tune PollyBERT?</h3><p>The task we want our BERT model to perform is NER. and for that, we use BERT in a token classification setting. In token classification, the model classifies each token as a certain entity or a regular token.</p><p>To fine-tune our model, we used a manually labelled dataset we created with Huggingface models and training APIs. The manually labelled corpus we used for fine-tuning PollyBERT was also a relatively small corpus with about 4k samples from <a href="https://www.ncbi.nlm.nih.gov/geo/">GEO</a> and only a few MBs in size.</p><h3>Experiments</h3><p>To test prompt design we picked GPT-J-6B and OPT-175B to represent two different sizes of LLMs, but their size is not the only difference between GPT-J-6B and OPT-175B. GPT-J-6B and OPT-175B were trained with the <a href="https://arxiv.org/abs/2101.00027">Pile Dataset</a> which is an 800GB big corpus, which includes natural language data from a lot of different sources. An important fact to note here is that OPT-175B was trained on a subset of Pile and this subset didn’t include PubMed Central and some other categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/0*YlwzCbGKxBrnf4iu.png" /></figure><p>For our BERT model, we used <a href="https://arxiv.org/pdf/2007.15779.pdf">PubMedBERT</a> weights for initializing a BERT model, PubMedBERT is a BERT-based model which has been pre-trained with a 21GB big PubMed abstract corpus. The PubMedBERT model was then fine-tuned for NER using a manually labelled training dataset. This dataset was created for extracting tissues from a given piece of text.</p><p>The task we chose for comparing these models was to perform NER and extract tissue entities from a piece of text. We are using the same prompt that was designed in the earlier section in a <strong>0-shot learning setting with OPT-175B and GPT-J-6B (due to API limitations)</strong>.</p><p>Since we had the capability to perform 2-shot learning with GPT-J-6B we also did that with the same test set to see if using 2-shot learning can help the model perform better.</p><p>To quantify performance for any model we are using both accuracy and F1-score as our metrics to measure performance for NER. One thing to note for these scores is that we are <strong>considering partial matches between the prediction and ground truth</strong>.</p><p>The test set we will be using is made using textual metadata from <a href="https://www.ncbi.nlm.nih.gov/geo/">GEO</a> which has been manually labelled by Subject Matter Experts (or SMEs). The test set is a perfectly balanced set with 100 examples containing tissue entities and 100 examples not containing any tissue entities.</p><h3>Results</h3><p>Now coming to numbers with PollyBERT we saw an accuracy of 87% and an F1 score of 0.84, GPT-J-6B gave us an accuracy of 51.7% and an F1 score of 0.515 and OPT-175B was not able to give any predictions hence accuracy and F1 score both stand at 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r2MZmOdsIjTObmeK.png" /></figure><p>Plot to show as F1 score increases the model parameter count increases</p><p>For 2-shot learning with GPT-J-6B, we saw a further drop in performance, we got an accuracy of 47% and an F1 score of 0.38. Why this drop in performance happened is not clear to us, we suspect this might just be because of the test set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/0*JC2fh6jOS_jX5JcO.png" /></figure><p>Comparing 2-shot learning with GPT-J-6B with PollyBERT</p><p>From these results, it’s clear that <strong>PollyBERT is the better performing model by a huge margin</strong> even though it has 10–1000 times lesser parameters and a significantly smaller training corpus than both GPT-J-6B and OPT-175B.</p><h3>Conclusion</h3><h4>Effect of Quality of the Dataset</h4><p>From our results, we have seen that PollyBERT performs much better than LLMs when it comes to NER for tissue entities. Also, we have a decrease in performance with an increase in the number of parameters. <strong>These results are completely opposite of what we expected given the condition of emergence was only met by OPT-175B</strong>.</p><p>According to us, this was due to the difference in the training dataset for these models. Biomedical data is very different from general language data, it has a lot of named entities that are only found in the biomedical domain. To identify these entities for NER, a model needs them in its corpus and since the occurrence of these entities will be negligible in a general language corpus it leads to poor performance.</p><p>If we recall the corpus for all these models, we used data from <strong>GEO for PollyBERT, GPT-J-6B had PubMed Central data and OPT-175B lacked any biomedical data</strong> while we were testing them on GEO. On top of that, while the dataset for <strong>PollyBERT was made for tissue NER, both GPT-J-6B and OPT-175B were trained using an autoregressive task</strong>.</p><p>If we look at domain-specificity and task-specificity here, the dataset for GPT-J-6B was more domain-specific than OPT-175B due to which GPT-J-6B performed better than OPT-175B. The dataset for PollyBERT was more task-specific and domain-specific (since they share the same source i.e., GEO) than GPT-J-6B which lead to a huge jump in performance.</p><h4>Manual Labelling vs Prompt Design</h4><p>After the quality of data, there is also a case to be made for using manual labelling instead of spending more time creating and experimenting with prompts, specifically for biomedical NER tasks.</p><p>Now, let’s assume we have an LLM which is trained a lot in biomedical data and is capable of performing a variety of tasks with prompt-based learning. If we go with prompt-based learning, we will decide on a framework for designing prompts like QaNER and then begin experimenting with different prompts on a test set. The problem with this method is that we are not guaranteed that we can design a prompt in the first place or the best possible one.</p><p>On the other hand, manual labelling has a defined process for getting labelled data with consistent results. Due to the reliability and consistency of manual labelling, it should be the preferred method of fine-tuning the language models for biomedical NER tasks.</p><blockquote><em>We are focussed on improving not just the accuracy but also the speed of the PollyBERT model. </em><a href="https://www.elucidata.io/blog/making-pollybert-run-10x-faster"><em>Read this blog to find out how some optimizations which were tried out recently made PollyBERT 10x faster!</em></a></blockquote><p><em>Originally published at </em><a href="https://www.elucidata.io/blog/how-pollybert-outperforms-large-language-models-in-bioner"><em>https://www.elucidata.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9dc072542c9b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elucidata/how-pollybert-outperforms-large-language-models-in-bioner-9dc072542c9b">How PollyBERT Outperforms Large Language Models in BioNER</a> was originally published in <a href="https://medium.com/elucidata">Elucidata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing search through ontology-driven knowledge graphs]]></title>
            <link>https://medium.com/elucidata/enhancing-search-through-ontology-driven-knowledge-graphs-1d4cfc1325cd?source=rss----ddc0f60dbf8a---4</link>
            <guid isPermaLink="false">https://medium.com/p/1d4cfc1325cd</guid>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[graph]]></category>
            <dc:creator><![CDATA[Pawan Verma]]></dc:creator>
            <pubDate>Fri, 04 Nov 2022 12:02:24 GMT</pubDate>
            <atom:updated>2023-02-14T05:36:38.979Z</atom:updated>
            <content:encoded><![CDATA[<h3>Motivation</h3><p>At present, there is a vast compendium of biomedical data being made publicly available from multiple sources. However, a major cause of concern for the majority of the users is the challenge of finding the <strong>right</strong> data for their research. Biomedical datasets are stored in various data repositories that fulfil different functions. Users may need to query various data repositories to collect all desired information. As a result, these biological datasets exist in fragments that do not “talk” to each other, reducing their value.</p><p>The current biomedical data retrieval systems have some major flaws. We, at <a href="https://www.elucidata.io/">Elucidata</a>, have identified a few below:</p><ol><li>Lack of cross-repository search</li><li>Lack of semantic search</li></ol><p>Therefore users spend more time searching datasets in known repositories than doing actual science. The growing amount of heterogeneous data makes it impossible to know for sure where some data of interest will be.</p><p>An example of a dataset search application case is when a dataset search engine delivers datasets in response to a user query. Such dataset search engines are currently in use, including the Mendeley Data search engine, run by Elsevier and Google Dataset Search.</p><p><strong>The drawback of such query-driven search engines is that they rely on the user to provide an accurate search phrase.</strong></p><p>The major driving factor towards making search more user-friendly is, to build a recommendation paradigm for search phrases for enabling better data findability and reducing false negatives. Moreover, context-driven search is achieved by modelling the relationships that exist between several biomedical molecular entities that are often unknown to a user or are not apparent by looking at datasets in silos.</p><p>In order to represent such relationships we utilise the capabilities of a network-based approach called a <strong>“Knowledge Graph” (KG).</strong></p><h3>Graph representation of biomedical molecular data</h3><p>In order to represent linked biological entities, we tend to utilise a schema or a structure that is a result of how the data is being modelled. More often, such models are defined by relationships that are known to exist between the given entities. For example, a Protein is a “product” of a Gene, therefore these relationships are known to exist despite the heterogeneity of the data.</p><p>But, not all relationships that exist between multiple biological entities are well-defined or “universally” accepted. There are several examples where the relationships between biological entities can change depending on how the data is being modelled.</p><p>In order to model the “universal” relationships, we utilise relationships between entities using existing <strong>ontologies (<em>O</em>)</strong>.</p><blockquote><strong>What is an Ontology?</strong></blockquote><blockquote>The term “ontology” was borrowed from philosophy to computer science to signify a <strong>machine-readable</strong> formalisation of a conceptualisation pertaining to a particular domain of knowledge. Simply put, an ontology is a digital object that can be interpreted by both humans and machines.</blockquote><blockquote>Below is an example of a linked view of 7 molecular entities linked into a consolidated ontology. The relationships depicted between the entities are consistent with the universally accepted ontologies linked to each of the entities (Details in next section).</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XBYJlMdLJHkmcrVXaz391w.png" /><figcaption>Figure 1: PollyGraph Schema</figcaption></figure><h3>Modelling biological ontologies in Neo4J</h3><p>Neo4j is a graph-native database which performs fast, dynamic and transactional queries over graph data. The inherent structure of Neo4j is a label-property graph model.</p><p>The information is organised as nodes, relationships and properties.</p><p><strong><em>Nodes</em></strong> are entities in the graph:</p><ul><li>Nodes can be tagged with <em>labels</em>, representing their different roles in your domain.</li><li>Nodes can hold any number of key-value pairs, or <em>properties</em>. (For example, disease).</li><li>Node labels may also attach metadata (such as index or constraint information) to certain nodes.</li></ul><p><strong><em>Relationships</em></strong> provide directed, named, connections between two node entities (e.g. cell-line sampled_from tissue).</p><ul><li>Relationships always have a direction, a type, a start node, and an end node, and they can have properties, just like nodes.</li><li>Nodes can have any number or type of relationships without sacrificing performance.</li><li>Although relationships are always <em>directed</em>, they can be navigated efficiently in any direction.</li></ul><p>There are key advantages of modelling graph data in Neo4j:</p><ol><li>Has its own querying language, CYPHER, similar to SQL but optimised for graphs.</li><li>Its flexible schema allows rapid materialisation and adding new relationships as per requirement.</li><li>Enables traversals across a wide range of depth and breadth with moderate computing.</li><li>Is offered as a managed service by AuraDB which is essentially a graph database.</li></ol><blockquote><em>A “native” graph database means that Neo4j implements a true graph model till the storage level, as in it is not stored as a “graph abstraction” over another technology.</em></blockquote><h3>Overview of the architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zBYzti7m66B40UNBpxAJPA.png" /><figcaption>Figure 2: PollyGraph Architecture</figcaption></figure><p><a href="https://www.elucidata.io/blog/polly-elucidatas-ml-ops-platform-for-biomedical-r-d">Polly</a>, which is home to over 1.6 million biomedical molecular data, is standardised for both data and metadata. There are two major advantages to this effort. First, data standardisation using uniform file formats such as GCT, h5ad, vcf etc. enables large-scale consumption of data for analysis. Second, metadata harmonisation enables users to perform data findability at scale without ever worrying to provide exact keywords for search.</p><p>To make the search even more powerful for scientists, we introduce <strong>PollyGraph</strong> to perform semantic searches using several biological concepts.</p><p>The architecture mainly comprises 4 components: <em>Source, Data Layer, Semantic Layer </em>and<em> Graph Layer</em></p><ol><li><strong>Source</strong>: The entities and relationships that form a knowledge graph are available in repositories which maintain domain-specific ontologies. These sources provide ontologically linked data in <em>.obo</em> files that are built on the Web Ontology Language (OWL) Framework. <br>Certain sources exist that map biological entities and store them in a tabular format, but do not maintain an ontology. For such sources we define rules that formalise the relationships between the entities, thereby making them machine actionable.</li><li><strong>Data Layer:</strong> There are 2 key processes in the data layer:<br> — <strong>Import</strong> the entities and relationships from multiple sources and generate data models or loadings. These loadings contain data that describes an entity using a unique identifier and relationships between entities.<br> — <strong>Store</strong> the data obtained from multiple heterogeneous data sources in a uniform database management system using a well-defined schema.</li></ol><p>3. <strong>Semantic Layer: </strong>The semantic layer organises the mapped entities stored in multiple tables in the DB store to a flat file containing RDF triples using a pre-defined ontology. The biomedical ontology (O) used here forms a set of rules but does not contain the traditional graph data represented as a subject—predicate—object or ‘triplet’. In order to generate these triplets we define a set mapping (M) assertions of relationships between an ontology and a data source.</p><p>In order to facilitate this, the Ontology-Based Data Access (OBDA) framework [1] is used, which allows querying arbitrary data sources using SPARQL. The ontology together with the mappings exposes the RDF graph.</p><p>4. <strong>Graph Layer: </strong>The materialised graph triples in RDF and cannot be directly imported on Neo4j. The reason is that the inherent graph data structure of Neo4j is not the same as RDF. RDF acts as a persistent store for the graph data but is not dynamic as is not meant to handle transactional workloads.</p><p>Neo4j is however a graph native that performs well with dynamic datasets and transactional workloads. Here, the graph data is defined as a label-property graph (LPG) where each class acts as a single node in a graph defined by its properties.</p><p>In order to export our RDF-based graph data into Neo4j, we utilise Neosemantics (n10s). n10s is a plugin provided by Neo4j that enables the use of RDF in Neo4j.</p><h4><strong>Detailed view of the semantic layer</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*sfjtxvcX2M0wz5PiqYZk4A.png" /><figcaption>Figure 3: Semantic Layer</figcaption></figure><p>The Semantic Model comprises 2 main components; Ontology<strong><em>(O)</em></strong> and Mappings<strong><em>(M).</em></strong></p><ol><li><strong>Ontology <em>(O)</em></strong>: This component contains the multiple biomedical ontologies consolidated in a single data model called Resource Description Framework (RDF). RDF is a W3C standard meant for data interchange and is used for storing graph data in a lossless manner. Semantics defined in RDF is just ruling, from which no new information can be derived from the triples. These rules are called ontologies that are an optional layer on top of RDF. A crucial step here is to combine ontologies from multiple sources.<br> — <strong>From existing ontologies: <br> (i) </strong>Existing biological ontologies such as BTO etc. are created based on W3C standards, therefore can be imported as is. Figure 4a gives an example of BTO ontology that contains data as RDF triples therefore they need not be modified.<br> — <strong>Defining custom rules between entities: <br> (i) </strong>Cross-entity relationships are seldom defined in one single biomedical ontology and therefore need to be defined explicitly.<br> <strong>(ii)</strong> The Ontology allows the addition/modification of new/existing rules when a new relationship is defined.</li></ol><p>Therefore, an ontology can be updated as per the requirement or to model a specific consumption journey for users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yrlHmP6Ussield6xkwqlzQ.jpeg" /><figcaption>Figure 4: Landscape of the Brenda Tissue Ontology (BTO). Highlighting tissue “midbrain central grey “ (left). KG representation of the tissue ontology for “midbrain central grey “ after loading the relationships on Neo4J. (right)</figcaption></figure><p><strong>2. Mappings <em>(M)</em>: </strong>The OBDA paradigm exposes a conceptual view of the domain knowledge with the purpose of querying, in a manner, where users DO NOT need an understanding of the data sources or the relationship between them. Simply put, OBDA provides a high-level querying interface without exposing the structure of the relational database to the users.</p><p>The mapping contains a series of declarations that defines relationships (predicates) that exist between a subject and object using a unique mapping identifier that populates the ontology using a querying language specific to the relational database. The query imports the required mappings from the DBMS store in order to <strong>materialise</strong> graph triples with the intent to populate an ontology(O).</p><p>Ontologies are used as a conceptual representation of data stored in RDBMS. The basic approach for mapping databases to ontologies and vice versa is Direct Mapping, which suggests such basic mapping rules as:</p><p>i. mapping tables into <em>Classes</em>;</p><p>ii. mapping columns into <em>Data</em> <em>Properties</em>;</p><p>iii. mapping foreign keys into <em>Object</em> <em>Properties</em></p><p>The following is an example of a mapping declaration specified in an .obda file.</p><pre>[PrefixDeclaration]<br>:		http://example.org/voc#<br>ex:		http://example.org/<br>owl:		http://www.w3.org/2002/07/owl#<br>rdf:		http://www.w3.org/1999/02/22-rdf-syntax-ns#<br>xml:		http://www.w3.org/XML/1998/namespace<br>xsd:		http://www.w3.org/2001/XMLSchema#<br>foaf:		http://xmlns.com/foaf/0.1/<br>obda:		https://w3id.org/obda/vocabulary#<br>rdfs:		http://www.w3.org/2000/01/rdf-schema#<br><br>[MappingDeclaration] @collection [[<br>mappingId	&lt;db_name&gt;-cell_line-tissue<br>target		:&lt;db_name&gt;/thing/{cell_line} :sampled_from :&lt;db_name&gt;/thing/{tissue} . <br>source		SELECT * FROM &lt;db_name&gt;.&lt;table&gt;<br>.<br>.<br>.<br>]]</pre><h3>Detailed view of the graph layer</h3><p>The materialised RDF triples are just a flat form of graph data which are non-transactional in nature. To efficiently perform dynamic graph operations, the data needs to be exported to Neo4J using neosemantics (n10s).</p><p>n10s provides control over the ontology that is serialised in RDF to be imported and configured into a neo4j database.</p><p>The main method for importing RDF is n10s.rdf.import.fetch. It imports and persists into Neo4j, the triples returned by a URI. For ontologies serialised as RDF, n10s.onto.import.fetch provides more control over the RDF.</p><p>Once the ontology has been configured based on requirements, it can be fetched directly using the URI for the OWL file and a serialisation format using the following command.</p><p>CALL n10s.rdf.import.fetch(&lt;RDF_PATH&gt;, &lt;format&gt;);</p><p>The first thing we notice is that dataType properties in the RDF have been converted into node properties and object properties are now relationships connecting nodes.</p><h3>Querying PollyGraph</h3><p>One of the main mechanisms behind this ability of semantic search is to utilise publicly available biomedical ontologies to provide more meaningful results using knowledge graphs. Linking biomedical ontologies by mining associations between them.</p><p>By leveraging ontologies, semantic search is able to provide a suitable response even if the results don’t contain the exact wording of the query.</p><h4>Find datasets related to Brain Injury where BRD1 gene is regulated</h4><blockquote>Note: Here, Brain Injury is not a valid <strong>MeSH</strong> term. However, it’s able to fetch datasets tagged with the relevant MeSH term for brain injury.</blockquote><blockquote>Such search enable users to worry less about the specificity of input keywords and can perform queries that are biology focussed.</blockquote><pre>MATCH (n:ns0__disease) <br>WHERE ANY (x in n.ns0__name + n.ns0__synonyms where toLower(x) CONTAINS(&quot;brain injury&quot;)) <br>WITH n.ns0__name as disease <br>UNWIND disease as d <br>MATCH (m:ns0__gene {ns0__gene_symbol : [&#39;BRD1&#39;]})--(n:ns0__disease {ns0__name : [d]})--(p: dataset) <br>RETURN m,n,p;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IQHYoymCp60PR7NWxM4LEw.jpeg" /></figure><h4>Fetch drugs that up-regulate genes that are down-regulated in a given disease (e.g. sarcomas)</h4><blockquote>Similar to above example, sarcoma is an extremely generic search for which the query is able to fetch datasets on relevant diseases which may be of interest to users, thereby obtaining results that are non-obvious.</blockquote><pre>MATCH (p:ns0__disease)-[:ns0__downregulates]-(q:ns0__gene)-[:ns0__drug_upregulates]-(r:ns0__drug) <br>WHERE ANY(x IN p.ns0__name + p.ns0__synonyms WHERE tolower(x) CONTAINS(&#39;sarcoma&#39;))<br>WITH r.ns0__name AS drug, p.ns0__name AS disease, q.ns0__gene_symbol as gene<br>RETURN drug, gene, disease</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Zqj24byzMfxOW7BNZwcQA.png" /></figure><h4>Fetch other drugs which perturb the expression of the same genes as colistin</h4><pre>MATCH (p:ns0__drug)-[r1]-(q:ns0__gene)-[r2]-(r:ns0__drug)<br>WHERE ANY(x IN p.ns0__name + p.ns0__synonyms WHERE tolower(x) CONTAINS(&quot;colistin&quot;))<br>RETURN p, q, r</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2tQrVueLuL_xZSwPVlpVNA.png" /></figure><h3>Conclusion</h3><p>Building an extensive biomedical knowledge graph is an iterative process. Further developments will allow users to make more complex queries and also allow cross-referencing to multiple ontology IDs for a given biological entity. Adding more relationships will therefore make the KG richer in context which will enable users to execute more robust queries on <a href="https://www.elucidata.io/blog/polly-elucidatas-ml-ops-platform-for-biomedical-r-d">Polly</a>.</p><h3>References</h3><ol><li>Bagosi, T., Calvanese, D., Hardi, J., Komla-Ebri, S., Lanti, D., Rezk, M., … &amp; Xiao, G. (2014, August). The ontop framework for ontology based data access. In <em>Chinese Semantic Web and Web Science Conference</em> (pp. 67–77). Springer, Berlin, Heidelberg.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d4cfc1325cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elucidata/enhancing-search-through-ontology-driven-knowledge-graphs-1d4cfc1325cd">Enhancing search through ontology-driven knowledge graphs</a> was originally published in <a href="https://medium.com/elucidata">Elucidata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Benchmarking data stores for event-based monitoring]]></title>
            <link>https://medium.com/elucidata/benchmarking-data-stores-for-event-based-monitoring-f77fd9a01eac?source=rss----ddc0f60dbf8a---4</link>
            <guid isPermaLink="false">https://medium.com/p/f77fd9a01eac</guid>
            <category><![CDATA[timestream]]></category>
            <category><![CDATA[benchmark]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Saiful Khan]]></dc:creator>
            <pubDate>Mon, 19 Sep 2022 11:13:15 GMT</pubDate>
            <atom:updated>2022-09-19T11:13:15.482Z</atom:updated>
            <content:encoded><![CDATA[<p>Monitoring a process using the events it generates is a very common application in computing. Based on the complexity of the process and the target observer, this can be implemented in different ways.</p><p>The most basic form of event-based monitoring, that most developers will be familiar with, is logging. In such cases, the “process” is usually a computer program either running locally or in the cloud. The “events” are messages written to a log. Looking at the messages appear in this log over time, the developer is able to tell what is going on with their program.</p><p>An ETL job is another kind of process that data teams might want to monitor. Our data ingestion pipeline at <a href="https://www.elucidata.io/">Elucidata</a> has hundreds of such ETL jobs running at any given point of time. Jobs can be of different types, of varying scope with different responsibilities and even run on distributed compute units. One of our near-term goals is to let our users add and manage their data on <a href="https://www.elucidata.io/platform/features">Polly</a> by themselves. For this to be a successful feature, the users also have to be able to see the progress of their data jobs and its logs as it goes through various stages of processing. This calls for a sort of job monitoring service.</p><p>We built this job monitoring service using <a href="https://aws.amazon.com/timestream/">Amazon Timestream</a> as our event store. It looked like a promising product and we were excited to try it out for something useful. This post talks a little bit about our experience with Timestream and our benchmarks against another strong contender in this space — <a href="http://elastic.co/">Elasticsearch</a>. We included <a href="https://aws.amazon.com/cloudwatch/">Cloudwatch</a> (Logs &amp; Insights) in our benchmarks as well, because our internal monitoring dashboard is running on this. <a href="https://aws.amazon.com/rds/postgresql/">Postgres on RDS</a> was also included as a reference OLTP database.</p><h3>Motivation</h3><p>Let us quickly go over a simplified version of our requirements. For that, lets define the two main concepts in the system:</p><ul><li><strong>Jobs</strong> — A job transforms some piece of source data to generate some desired output data. Depending on the type of work being done, a job goes through a certain set of <strong>stages</strong>.</li><li><strong>Batch</strong> — A collection of jobs, grouped based on some common relationship between them. The relationship itself is not important other than the fact that this gives the user an easy way to collectively treat a large number of jobs as a single unit. Batches have a <strong>status</strong> that is determined by the stages of its underlying jobs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GXEEqGlKSN1l7oaIyp1lRg.png" /></figure><p>Using the events generated from each job, we want to be able to do:</p><ul><li>Filtering — on static attributes of a batch, which is present in each event of each job in that batch</li><li>Aggregation — deduce the status of a batch, from the collective stages of its jobs</li><li>Filtering on aggregations — on dynamic attributes like status of a batch</li><li>Sorting — batches ordered based on creation time</li><li>Some time series — fetch batches created in a certain date range</li></ul><p>Currently, our internal data teams are monitoring the progress of their ETL jobs on a <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html">Cloudwatch Dashboard</a>. Our jobs are already generating logs and from those logs, we are able to get the relevant stats. But the latency observed on every refresh of the dashboard is not great at all. We want to be able to show jobs as old as 3 months. Fetching stats from 3 months of logs is very inefficient.</p><p>We were looking for a data store on top of which we can build a custom job monitoring dashboard as a part of Polly itself. That’s when we discovered Timestream. It is primarily built for timeseries data and supports regular and irregular timeseries. Ours is irregular timeseries (events) data. Apart from being serverless, it supports SQL based querying on its schemaless tables. This was a very attractive proposition and we ended up choosing TImestream for our job monitoring service.</p><h3>The problem</h3><p>We had done some basic benchmarking of Timestream by running it on a large number of dummy events and calculating the average response time. Most of the queries gave us sub-second averages. So it was a surprise to us that our GET APIs (running Timestream queries internally) were routinely taking anywhere from 2 to 5 seconds to respond. If you made subsequent API calls, the latency used to improve, but the 5 second max was still hurting. Our dashboard is supposed to have an auto-refresh rate of 1 minute, and it was those refresh requests that took the longest time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ns0raBI19ZmygkfTWzKKWg.png" /><figcaption>Timestream’s contribution to our latency</figcaption></figure><p>It seems like Timestream has a bit of a “cold-start” problem which is not uncommon in serverless systems. The gap between two cold-starts seems to be at least one minute. In the worst case, if we have just one user with their dashboard open and refreshing once a minute, they will notice this long wait every single time. At this rate, our custom dashboard, even though it looked prettier, would be no more performant than the existing Cloudwatch dashboard.</p><p>The reason we missed this behavior in our preliminary benchmarks earlier was because we were looking at average times over a large number of requests. Knowing this, we now wanted to perform a more thorough benchmark. This time we will also throw in a few other data stores to see how Timestream fares against them.</p><h3>Benchmarking setup</h3><p>Our requirements are fairly common and prevalent in most analytical workloads. We have plenty of options today for storing and querying on event data. But we are a small team and we want to stick to services which we can get started with quickly and are easy to maintain. Also, at this point our entire data infrastructure is hosted on AWS, so it makes sense to start with AWS services too. Therefore, the benchmark was performed on the following data-stores:</p><ul><li><strong>Timestream</strong> — We were already using it.</li><li><strong>Elasticsearch (on AWS)</strong> — Not just a search engine. Excellent support for all our requirements.</li><li><strong>Cloudwatch (Logs + Insights)</strong> — Our existing internal dashboard is running on Cloudwatch Logs being queried through Cloudwatch Insights. It made sense to put this in the comparison.</li><li><strong>Postgres (RDS)</strong> — Just added as a reference transactional database. Without any plugins, we are not expecting vanilla Postgres to perform well in this context.</li></ul><p>Please note that of the four above only Timestream and Elasticsearch are true contenders for our use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YV-L-Ys7yVFmBKu_Ajli8Q.png" /></figure><p>Events were generated by a “writer” and written to each of data stores. Events were then periodically consumed by a “reader”. They will write their measurements of write and read performances to a DynamoDB table.</p><h3>Benchmark design</h3><p>A series of artificial batches of jobs were generated and their events were written to each data store. But this was done in a systematic manner:</p><ul><li>8 sets of 20 batches each were generated. This was done so as to increase the scale of the data after each iteration.</li><li>Each batch had about 7000 jobs on average. Each job generates 7 events at most. Ultimately, we had <strong>about 4 million records in total, in each data store</strong>.</li><li>The jobs spanned over two days. The 566k jobs is a fair representation of the kind of workload we expect to see on our production infra.</li><li>Write performances were recorded for each batch of records written. Most databases perform better with batch writes. We chose batches of 100 messages each for Cloudwatch Logs. For Elasticsearch and Postgres we had batches of 500 records per write. For Timestream, we had 10 concurrent threads writing 100 records each and the cumulative time taken by the threads was taken as one write measurement.</li><li>After each iteration we ran the query benchmarks, giving us read performance at each scale (from 1x to 8x). Each query was run 10 times to get a distribution of read times.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZM7qBLcl3CyGYErksi_poA.png" /></figure><p>Each event contained a few static values and a few (dynamic) stage-based values. Normalization of data is clearly not our goal here.</p><h3>Performance results</h3><p>For all the quantitative benchmarks, we will plotting the performance data as <a href="https://en.wikipedia.org/wiki/Box_plot">box-plots</a>. This not only shows us the “typical” write or read times but also shows the extremes. Knowing these extremes is important. They can tell us whether we need to take extra measures to maintain a good write performance or low response latencies. Sometimes the worst-case performances may simply make it unviable to use a service for your requirements.</p><h3>Write performance</h3><p>Comparing write performance is not the primary purpose of this purpose. However, it would still be interesting to see how the different data stores fare against one another. The following plot shows the distributions of write times per 100 records.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iy4RJCINO6uigZgqG3ZPJA.png" /><figcaption>Write performance of each data store</figcaption></figure><p>Key observations:</p><ul><li>The average write performance for Cloudwatch logs is pretty good, but at its slowest, it is many times worse. It is these kinds of observations that are lost in simple bar plots.</li><li>Elasticsearch has the healthiest write performance numbers, with 100 ms at its worst.</li><li>Timestream write performances show a large variation. It should be noted that with more concurrent writers, Timestream might be able to achieve much better throughput.</li></ul><h3>Query performance, Type I</h3><p>We finally come to the query benchmarks. For each type of query, we will show an example SQL query to formally elucidate our intention.</p><p>The first query type is not at all useful for our monitoring dashboard, but its something simple that we can start with — <strong>get the total number of events per batch</strong>:</p><pre>  <strong>SELECT</strong> batch_id, COUNT(*) <strong>AS</strong> num_events<br>    <strong>FROM</strong> monitoring_events<br><strong>GROUP BY</strong> batch_id</pre><p><em>Note: </em>For each of the query benchmark types, we will show two plots. The first plot shows the distribution of query performances at 8x scale (highest number of records). <strong>The orange dot in the first plot is the read time when the first query request was made</strong>. The second plot shows how average query performance changes with scale (from 1x to 8x). In both plots, lower values are better.</p><p>Performance profile for Type I queries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K8p2lzbtfwdBGtfuI9V93Q.png" /><figcaption>Query performance at 8x scale, Type I</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j2jrylwfJHXW1S9i1UGTAQ.png" /><figcaption>Average query performance with increasing scale, Type I</figcaption></figure><p>Key observations:</p><ul><li>Elasticsearch returns every single time in less than one-third of a second.</li><li>Timestream shows a large variance in query performances. The first call (cold-start?) is noticeably worse.</li></ul><h3>Query performance, Type II</h3><p>In the second query, we will be doing multiple aggregations by batch IDs. This is more useful query and one that we may run in certain situations — <strong>get the latest stats of each batch</strong>:</p><pre>  <strong>SELECT</strong> batch_id,<br>         COUNT(DISTINCT(job_id)) <strong>AS</strong> num_jobs,<br>         SUM(CAST(finished <strong>AS</strong> integer)) <strong>AS</strong> successful_jobs,<br>         SUM(CAST(errored <strong>AS</strong> integer)) <strong>AS</strong> errored_jobs,<br>         MIN(created_at) <strong>AS</strong> creation_time,<br>         MAX(time) <strong>AS</strong> last_updation_time<br>    <strong>FROM</strong> monitoring_events<br><strong>GROUP BY</strong> batch_id<br>   <strong>LIMIT</strong> 100</pre><p>Performance profile for Type II queries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*esSOqWPQnrdQVTOElULigw.png" /><figcaption>Query performance at 8x scale, Type II</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zoWOmIW6oVeqFwk81Yd_RA.png" /><figcaption>Average query performance with increasing scale, Type II</figcaption></figure><p>Key observations:</p><ul><li>Elasticsearch still shows the best performance, with 75% queries returning under half a second.</li><li>Timestream gives more predictable performance this time. The worse case performance is still not good though.</li><li>Postgres has to likely due do full table scans this time around.</li></ul><h3>Query performance, Type III</h3><p>In this query, we will be doing multiple aggregations but also applying filters on static as well as aggregated attributes. Finally we will be applying a sorting based on an aggregated attribute. This sort of query will routinely run on our dashboard — <strong>get batches that have completed successfully, sorted by creation time</strong>:</p><pre>  <strong>SELECT</strong> batch_id,<br>         COUNT(DISTINCT(job_id)) <strong>AS</strong> num_jobs,<br>         SUM(CAST(finished <strong>AS</strong> integer)) <strong>AS</strong> successful_jobs,<br>         SUM(CAST(errored <strong>AS</strong> integer)) <strong>AS</strong> errored_jobs,<br>         MIN(created_at) <strong>AS</strong> creation_time,<br>         MAX(time) <strong>AS</strong> last_updation_time<br>    <strong>FROM</strong> monitoring_events<br>   <strong>WHERE</strong> user_id = &#39;1110&#39;<br><strong>GROUP BY</strong> batch_id<br>   <strong>WHERE</strong> successful_jobs = num_jobs<br><strong>ORDER BY</strong> creation_time</pre><p>Performance profile for Type III queries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tJpvjYXfxd5Fi8_obuwrfA.png" /><figcaption>Query performance at 8x scale, Type III</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7FhzwVvLtWA9xJX_q4r0bw.png" /><figcaption>Average query performance with increasing scale, Type III</figcaption></figure><p>Key observations:</p><ul><li>Every single Elasticsearch query returns under half-a-second.</li><li>Postgres and Timestream are doing much better this time. This is likely due to the filters being applied, which probably narrows down the search space.</li></ul><h3>Query performance, Type IV</h3><p>This fourth query is unique in the way that it finds out values unique based on the maximum value of another dimension. This is another query we will run often, when viewing jobs within a batch — <strong>get the latest stage and progress for each job</strong>:</p><pre>  <strong>SELECT</strong> job_id,<br>         MAX_BY(stage, time) <strong>AS</strong> stage,<br>         MAX_BY(stage_progress, time) <strong>AS</strong> stage_progress,<br>         MAX_BY(errored, time) <strong>AS</strong> errored,<br>         MAX(time) <strong>AS</strong> last_updation_time<br>    <strong>FROM</strong> monitoring_events<br>   <strong>WHERE</strong> batch_id = &#39;16554252419__1661749456__1110&#39;<br><strong>GROUP BY</strong> job_id</pre><p>Performance profile for Type IV queries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fP-6YUVZCTJ5N9HBnUoFuQ.png" /><figcaption>Query performance at 8x scale, Type IV</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sGgDU7fYm0xAeYjH39oPmw.png" /><figcaption>Average query performance with increasing scale, Type IV</figcaption></figure><p>Key observations:</p><ul><li>All queries, except those run on Cloudwatch logs, are performing extremely well. The search space here has been reduced drastically — to a single batch.</li><li>Cloudwatch insights queries taking 3 seconds at this point a consistent trend.</li></ul><h3>Query performance, Type V</h3><p>This final query is again not needed for our dashboard, but we want to test how each of the data stores perform for a common timeseries use case — <strong>get number of jobs finished per repo, each minute, for a small set of users</strong>:</p><pre>  <strong>SELECT</strong> repo_id,<br>         BIN(time, 1m) <strong>AS</strong> interval,<br>         COUNT(DISTINCT(job_id)) <strong>AS</strong> num_jobs<br>    <strong>FROM</strong> monitoring_events<br>   <strong>WHERE</strong> stage = &#39;Finished&#39;<br>     <strong>AND</strong> user_id <strong>IN</strong> (&#39;0011&#39;, &#39;0111&#39;, &#39;1111&#39;, &#39;1110&#39;, &#39;1100&#39;)<br><strong>GROUP BY</strong> repo_id, interval<br><strong>ORDER BY</strong> interval <strong>DESC</strong></pre><p>Performance profile for Type V queries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KjGRKLWm954ujAqy06agyg.png" /><figcaption>Query performance at 8x scale, Type V</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yCFBVKKy4pySM_EUrEt3HA.png" /><figcaption>Average query performance with increasing scale, Type V</figcaption></figure><p>Key observations:</p><ul><li>Elasticsearch is once again excellent. Far superior to Timestream, which is supposed to be a timeseries database.</li><li>Postgres starts off being better than Timestream but with scale it eventually gets left behind.</li><li>Timestream’s performance, although not as good as expected, is quite stable even with increasing amounts of data.</li></ul><h3>Qualitative comparison</h3><p>Good performance is definitely important but it is not everything. Depending on one’s use-cases and the amount of time one can afford, some of these data-stores will be more suitable than others. Let us compare our data stores based on some commonly desired qualities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JZYbQQDHF9Ot62neHUvA0g.png" /><figcaption>Feature matrix comparing the data stores</figcaption></figure><p>Based on the amount of green and yellow, Timestream seems to be the most featureful option among the lot.</p><h3>Cost comparison</h3><p>No benchmark is complete without a cost comparison. The pricing of a service should not be unjustifiable for the performance it provides. Predicting costs over some constant period of time would be difficult and prone to inaccuracy. So, here we will just attach the cost incurred by each instance of the service for the duration of the benchmark, which took place over a span 4 days:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z7I-fSjFGZ_Xsv6BgCr4oQ.png" /><figcaption>Service costs incurred throughout the benchmark</figcaption></figure><p>Cloudwatch wins here, since it is built to store vast number of messages and per query cost is very low. Elasticsearch costs the most but it should be noted that those costs will remain constant per day for up-to a certain threshold of data volume (which is pretty high). Timestream also seems to be very economical service for our purposes.</p><h3>Caution!</h3><p>Before we conclude this post, we would like to mention some possible biases that may make this benchmark “not true” or completely inapplicable to your work:</p><ul><li><strong>Scale of data</strong> — The amount of data we performed this benchmark with, may not have been large enough to notice some emergent scalability patterns.</li><li><strong>Variety of queries</strong> — We started off with explaining why we were doing this exercise. The types of queries we ran were specific to our data model and use cases. They were not exhaustive and your primary use case may not have been covered at all.</li><li><strong>Minimal optimizations</strong> — While we did pay attention to model the data so as to support our queries well, some of the data stores (especially Postgres and Elasticsearch) may be further optimized to beat the competition on certain metrics. But the idea here is that, the data store should be capable of achieving “good enough” performance without a whole lot of tinkering and optimizations on the developers part. We did spend some time on trying to come up with more performant queries for each use case.</li></ul><h3>Conclusion</h3><p>There is definitely a winner here for <em>us</em>. However, your requirements might be different enough that one of the others may be more attractive to you. We will end this post with some notes on the strengths and weaknesses of each:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eULd_cvYOtZw6FtzZ8ziWw.png" /></figure><p>We really like what Elasticsearch has to offer in terms of predictable performance and costs. It also checks most of the boxes in terms of features. Therefore, we are planning to replace Timestream with Elasticsearch in our job monitoring service, sometime in the near future.</p><p>With that, we come to an end of our benchmarks. If you have other data stores in mind that you want to compare against this plan, you can do so by adding to the source code. The infra configuration, source and analysis scripts are all available in this <a href="https://github.com/saiful-elucidata/event-based-monitoring-benchmarks">GitHub repository</a>. In case you want to suggest changes or face issues with reproducibility, please raise an issue there itself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f77fd9a01eac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elucidata/benchmarking-data-stores-for-event-based-monitoring-f77fd9a01eac">Benchmarking data stores for event-based monitoring</a> was originally published in <a href="https://medium.com/elucidata">Elucidata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build a scalable backend to query and search billions of records?]]></title>
            <link>https://medium.com/elucidata/how-to-build-a-scalable-backend-to-query-and-search-billions-of-records-16a532433fda?source=rss----ddc0f60dbf8a---4</link>
            <guid isPermaLink="false">https://medium.com/p/16a532433fda</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Saiful Khan]]></dc:creator>
            <pubDate>Thu, 05 May 2022 15:26:29 GMT</pubDate>
            <atom:updated>2022-05-11T03:12:37.441Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ET4_XRRD-AIHMj5aqzCr_A.jpeg" /></figure><p>Every day, Elucidata’s Polly ingests gigabytes(GB) of biomolecular data and transforms it into analysis-ready format for the biologists around the world. This rate of data ingestion presents us with problems of scale and storage. We have recently written about how Elucidata handles biomolecular data at scale with <a href="https://elucidata.io/how-to-build-highly-effective-etl-pipelines/">highly effective ETL pipelines</a>. In this article, we shall look at how engineers at Elucidata handled the challenges of storage and figured out a way to effectively store and make data consumable.</p><p>Historically, Polly’s data platform has relied solely on <a href="https://aws.amazon.com/opensearch-service/the-elk-stack/what-is-elasticsearch/">Elasticsearch</a> to store consumable data. And, It has worked well. Response times for most queries were swift. We were even storing large numerical matrices, serialized as documents, in Elasticsearch. The AWS Elasticsearch service (now OpenSearch service) allows users to use the <a href="https://opendistro.github.io/for-elasticsearch-docs/docs/sql/">SQL plugin</a> to make SQL queries on our indices.</p><p>Moreover, Elasticsearch — -just like other NoSQL data stores — -does not require a predefined schema for data indexing. But, there is a flip side to this feature. Elucidata deals with a lot of semi-structured data coming from heterogeneous sources. Still, we ended up choosing something that mandated the definition of schemas before starting ingestion. We slowly realized that Elasticsearch has several limitations that makes it unsuitable to serve as our query engine. But, before we go further, we need to explain what data means in this context.</p><h3>What do we mean when we say “data“</h3><p>A majority of data on Polly exists in the form of <strong>datasets</strong>. A typical dataset comprises the following:</p><ul><li>A single dataset-level metadata</li><li>A set of <em>n</em> <strong>samples</strong>, each having its metadata</li><li>A set of <em>m</em> <strong>features</strong> measured per sample, each feature having its metadata</li><li>An (<em>n</em> ⨯ <em>m</em>) <strong>data-matrix</strong>, storing the actual measurements</li></ul><p>Each entity’s <strong>metadata is a dictionary of key-value pairs</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0XI770lgbULirT3eA0HCKQ.png" /><figcaption>Layout and contents of a typical dataset</figcaption></figure><p>For most Polly datasets, the <strong>data matrix constitutes &gt;95% of the </strong>file contents and information volume. One final concept to remember is “<strong>repositories</strong>,” which are collections of datasets. Datasets in a repository share some common attributes due to exact origin (source), same data type, or even based on the presence of a particular tissue.</p><h3>Our requirements</h3><p>Ultimately, our users need to consume the data present in these datasets. And we have three primary use cases here:</p><ul><li><strong>Search on metadata</strong> for finding datasets and samples of interest.</li><li><strong>Query metadata</strong> of datasets, samples, and features using SQL.</li><li><strong>Query data-matrix</strong> of datasets and features of interest using SQL.</li></ul><p>We need an API that accepts full-text search (FTS) queries and an API that accepts SQL queries. The data size on which these queries will run is already in terabytes, increasing every day.</p><p>Before you make any guesses on how we addressed all of these requirements with just Elasticsearch, we should first take a look at the limitations of Elasticsearch.</p><h3>Elasticsearch scaling issue</h3><p>For the past few months, Polly has been ingesting hundreds of datasets every day. Some of these datasets have hundreds of thousands of features. The combined size of Elasticsearch indexes quadrupled(from 1 to 4 TB) very quickly. At Elucidata, we anticipated that this trend would continue.</p><p>Usually, cloud storage is not very expensive. But with AWS Elasticsearch service (now OpenSearch service), for a given class of nodes, there’s a limit on the size of EBS volume one can attach to each node. When a size limit is reached, users either have to increase the number of nodes or upgrade the nodes to a more powerful and costlier instance class.</p><p>Typically, the Elasticsearch cluster would scale with the surge in data to seamlessly handle the increasing load during data search. But in Elucidata’s case, data volumes were increasing at an alarming rate. Very often, a majority of the data matrices ended up not being queried. A user would have to compromise by narrowing down on a small number of datasets of interest and then fetch the measurement data present in their matrices. We concluded that querying on data matrices was sub-optimal.</p><p>This meant that while we were horizontally or vertically scaling our Elasticsearch cluster, because of storage demands, we were not actually using the compute that was coming with it. <strong>We were paying for both storage and compute, when we needed only one</strong>.</p><h3>Limitations in querying</h3><p>Elasticsearch cluster has a reputation for being an excellent search engine with the added advantage of vast community support. There are various plugins available that augment the abilities of an Elasticsearch cluster that enable index querying using SQL as if they were a traditional database table. The engineers at Elucidata leveraged the plugins available via Elasticsearch to query using SQL. Elasticsearch was being used for both the FTS and SQL APIs until the engineers realized that the Elasticsearch is an FTS engine and is not the best choice for SQL activities. Few of the challenges that we have faced while performing SQL activities:</p><ul><li>Getting more than 10,000 rows for any non-trivial query.</li><li>Limited scope for joins and aggregations on joins.</li><li>Querying nested data like in the data-matrix — one document per row where each sample’s name and measurement was in an array of nested documents.</li></ul><p>These limitations were very problematic for a platform whose purpose is to add value and expose large amounts of analysis-ready biomedical data.</p><h3>Absence of schemas</h3><p>Elasticsearch allows users to dump data and make it easily queryable. Except with a limit on how many fields you can have per index. And for a good reason, too — otherwise, there will be no upper bound on how much memory a query needs, which is not a good thing for a reliable distributed system.</p><p>Polly’s repositories can have datasets coming in from heterogeneous sources, which makes different metadata fields easy to access. However, the union of these fields can easily exceed Elasticsearch’s field limit. And, it is not recommended to index everything. This issue can be managed through proper processes.</p><p>However, writing structured queries (SQL) on Polly’s data turned out to be the real challenge without schemas. The system that was put into use initially didn’t need any schemas. And, writing SQL queries with no documented schema is challenging. <strong>Users had to tediously figure out columns and their types f</strong>rom thousands of available fields.</p><h3>Our solution</h3><p>While our search APIs were working very well, <strong>it soon became apparent that we needed a separate backend to store and serve SQL on structured data</strong>. Polly needed something that:</p><ul><li>has native SQL support,</li><li>scales well with very large amounts of data per table,</li><li>scales with a large number of tables,</li><li>enforces a schema for each table.</li></ul><h4>A distributed file-system</h4><p>First, the engineers at Elucidata figured that using a data-lake format and serializing our tables on a distributed file system would help with scalability since storage is cheap. <strong>S3 is a great choice here</strong>, given that the rest of the cloud platform already runs on AWS.</p><h4>The data format</h4><p>Second, there is a need to select a data format that can support on-disk SQL querying and enforces schema as well. Engineers evaluated a few choices:</p><ul><li><a href="https://docs.tiledb.com/main/">TileDB embedded</a> — Looks great but is yet to be adopted widely. There’s limited community support information.</li><li><a href="https://iceberg.apache.org/">Apache Iceberg</a> — An Apache project (originally developed at Netflix). Decent community support and adoption. But is not a good choice for workloads that involve frequent updates and deletions to your tables, which is the case for us.</li><li><a href="https://hudi.apache.org/">Apache Hudi</a> — Another Apache project. Decent community support and adoption. Hudi was almost a right fit.</li><li><a href="https://docs.delta.io/latest/delta-intro.html">Delta Lake</a> — Developed and maintained by <a href="https://databricks.com/">Databricks</a>, the open-source version checked all the boxes. Delta lake is widely adopted and also provides an option to move to the more robust enterprise version if the need arises.</li></ul><p>We concluded that <strong>using Delta Lake is the right fit for all our data storage issues.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PpIcjZMJ2x6ziKqB0Ca-GA.png" /><figcaption>Delta is based on Parquet and augments it</figcaption></figure><h4>Separate backends for search and query</h4><p>That said, complete migration to Delta Lake (running on S3) and ditching Elasticsearch wasn’t the plan either. We still wanted to keep our already excellent search interface based on Elasticsearch. It works exceptionally well. We decided to <strong>separate search and query backends, even at the cost of data duplication between these backends to</strong> optimize the backends separately so as to serve their respective roles in the consumption journey.</p><p>Moreover, only SQL queries needed to be run on data matrices, and there was no need for a <em>search</em>. This allowed the engineers to keep the data matrices in Delta Lake as individual tables and not index them in Elasticsearch. This greatly alleviated Polly’s Elasticsearch cluster’s storage requirements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aFJtsrOwTiCJ6X0VnfzrQg.png" /><figcaption>Separation and duplication of data in Elasticsearch and Delta Lake</figcaption></figure><h4>Running SQL on Delta Lake</h4><p>Consuming data through SQL requires a query engine supported by Delta Lake on S3. Here are a few options to choose from:</p><ul><li><a href="https://spark.apache.org/">Spark</a></li><li><a href="https://prestodb.io/">Presto</a></li><li><a href="https://aws.amazon.com/athena">Athena</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hWca9uPd1a_5KD1xGX5QsQ.png" /></figure><p>We <strong>chose to go with Athena because it requires the least effort to get started</strong>. Athena is based on Presto (v0.217), and they both <strong>require the definition of all external queryable tables to be present in a Hive-compatible metastore, which</strong> is how the query engine will know what fields (and their types) are available for querying and what files on S3 should be read to get the actual data. For the complete documentation for the table registration process, check <a href="https://docs.delta.io/latest/presto-integration.html">here</a>. Furthermore, in the future, if the current service reaches its threshold for simultaneous query executions, we will move to a self-managed infrastructure running Presto or Spark.</p><p>Once again, <strong>instead of setting up and maintaining the Hive instance, </strong><a href="https://docs.aws.amazon.com/glue/latest/dg/populate-data-catalog.html">AWS Glue Data Catalog</a> was used. The Glue Data Catalog service is supported by Delta Lake and Athena and has a pay-per-table object model, saving us the effort and upfront cost of maintaining our Hive metastore. In case, the Glue Data Catalog is getting either too restrictive or expensive in the future, there is always an option to move to a self-managed solution.</p><h3>Final Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jRMSDF8zm1RGgW4HEwp05A.png" /><figcaption>The final architecture of our storage and consumption infrastructure</figcaption></figure><p>As shown in the figure above, two different backends serve Polly’s search API and SQL query API. They both go through a Lambda whose job is to perform some common tasks such as:</p><ul><li>Repository-level access control — each user has access to a certain set of read-only datasets.</li><li>Capturing usage metrics for getting insights on the usage of data repositories.</li></ul><p>Polly’s new storage infrastructure enables entity metadata to be present in both Elasticsearch and Delta Lake. From our initial estimates, the persistent increase in storage cost amounts to just ~<strong>$3 per terabyte</strong> of data ingested. Though a bit expensive, the risks associated with scaling Elasticsearch vertically or horizontally frequently have been eliminated by accommodating the disproportionately sized data matrices.</p><p>The new consumption infrastructure of Polly’s new storage backend allows users to make powerful queries on biomolecular data. It also forced the engineers and scientists internally to set up processes for schema definition for each source and type of data, which led to better documentation and management of data complexity and significantly enhanced a user’s querying experience.</p><p>What do you think about our scalable backend storage solution? Do you think you have a better solution? Then come work with us at Elucidata. We are hiring!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16a532433fda" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elucidata/how-to-build-a-scalable-backend-to-query-and-search-billions-of-records-16a532433fda">How to build a scalable backend to query and search billions of records?</a> was originally published in <a href="https://medium.com/elucidata">Elucidata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build Highly Effective ETL Pipelines]]></title>
            <link>https://medium.com/elucidata/how-to-build-highly-effective-etl-pipelines-dd08019ec7fe?source=rss----ddc0f60dbf8a---4</link>
            <guid isPermaLink="false">https://medium.com/p/dd08019ec7fe</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Sahil Rai]]></dc:creator>
            <pubDate>Mon, 21 Feb 2022 12:49:57 GMT</pubDate>
            <atom:updated>2022-02-24T06:06:13.487Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/u/76437b8297d7">Rohit Jagre</a>| <a href="https://medium.com/u/2411edeeafee">Sahil</a>| <a href="https://medium.com/u/e4c858101129">Ram Sagar</a></p><p>Inconsistency in data inflows can be a huge challenge to engineers who oversee ETL pipelines. Last month, one of our biggest clients introduced 32TB of biomedical molecular data on our platform. Up until that point, our cloud infrastructure was processing a maximum of 15TB every month. Handling such a large surge requires a superior architecture that makes the best use of all the available tools. We implemented this. The result is, today our platforms can ingest more than 600 GB per day, up from 60GB per day, a month ago. That’s a 10x performance improvement. Not only that, but we also brought down the cost associated with handling such data by up to 90%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c83ti1F_VuaE4eY8.jpg" /></figure><p>Elucidata’s platform ‘Polly’ currently hosts close to 1.5 million datasets. Each dataset goes through various data processing layers before it comes onto the Platform i.e. cleaning &amp; preparation, validation &amp; sanity checks, and lastly, indexing &amp; loading into our storage infrastructure. It is imperative that the ETL pipeline is designed to be fast, cheap, reliable, and easily scalable. In this article, we take a look at ways to achieve such exceptional performance from ETL pipelines.</p><h3>What does it take to build an effective ETL pipeline</h3><p><strong>1| Solving for the inconsistent flow of data</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a62iXvQ0DbEQtCHV.jpg" /><figcaption>ECS Cluster with custom scaling</figcaption></figure><p><strong>ECS cluster with custom scaling lambda</strong></p><p>An effective ETL infrastructure should be able to add compute resources elastically — from almost zero resources to multiple when the flow of data increases. Thankfully, in today’s world, this is a small challenge that can be solved by using the right tools, effectively.</p><p>What you need here is a “Container Orchestration” tool. There are many options available in the market — Kubernetes, AWS ECS, AWS EKS, Nomad to name a few. For Polly’s ETL pipelines, we have chosen AWS ECS because of the following reasons 1) Our remaining infrastructure is already on AWS 2) ECS is easier to manage compared to Kubernetes (AWS EKS).</p><p>As ECS is a managed container orchestration service that takes care of scheduling workers on ECS instances, associated with a capacity provider for auto-scaling, it makes an ideal solution for running our ETL workers. Since all we have to do now is invoke ECS to run the specified number of tasks. In case more tasks are needed, we can invoke ECS and the capacity provider will automatically add instances to incorporate those. But for our use case, instead of using the ECS service’s auto-scaling, we prefer to use AWS lambda that manages the scaling of the ECS workers.</p><p><strong>Why do we use a custom scaling solution</strong></p><p>The primary responsibility of the ECS service is to maintain a set number of replicas. When the load on the cluster is reduced, it starts scaling in i.e.terminating the replicas i.e terminate any of the replicas. But in any ETL pipeline, workers are not exact replicas since they’re tasked to process different jobs. Therefore, we prefer not to terminate them until the job is completed. Additionally, terminating at this stage could start a chain of events where new tasks are being created and killed simultaneously.</p><p>That is why we opted to write a custom scaling logic for ECS tasks, that would deliver on our need to scale when required, without terminating unfinished jobs. We try to emulate the target tracking strategy by continuously monitoring how many workers are required to process the available jobs by using a combination of Cloudwatch alarm + Cloudwatch EventRule + SNS + lambda.</p><p><strong>2| Building a cost-effective ETL pipeline</strong></p><p>While dealing with the management and processing of big data, infrastructure costs can skyrocket easily. It is essential to employ cost savings techniques at every opportunity.</p><p>We use two of them for Polly:</p><p><strong>Employing AWS EC2 Spot instances</strong></p><p>Spot instances are 90% cheaper compared to regular on-demand instances and are usually available at heavy discounts. However, they are not completely reliable as they are susceptible to being terminated by AWS with a 2-min warning. To tackle this, we have employed a fault-tolerant system (details below).</p><p><strong>Scale to complete zero</strong></p><p>Scale to zero is achieved by using Scaler lambda which terminates all the available instances when there is no data to be processed.</p><p><strong>3| Incorporating checks and balances for pipeline monitoring</strong></p><p>When the number of upload jobs in the pipelines increases significantly, it becomes challenging to keep track of the status of each job. To solve it, we have employed a NoSQL database (AWS DynamoDB) to store the status and related metadata of every job coming into the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/0*lPjbcpDSQXyTs111.png" /><figcaption>Jobs table schema</figcaption></figure><p>Eventually, this DB can be connected to an analytics dashboard like AWS Quicksight.</p><p><strong>4| Building a fault-tolerant architecture</strong></p><p>As we opted to use Spot instances for running our ETL workers, it’s inevitable for a worker to be unexpectedly terminated between jobs. In such a case, the job should be retried a set number of times before being labelled as FAILED. To identify an incomplete job, we used SQS’s visibility timeout feature. Then a new worker is assigned to retry that failed job.</p><h3>The Final Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*56Ko5rKAC_kkfFDaBmptEw.png" /><figcaption>Final Architecture</figcaption></figure><p>Here’s a step by step guide to building an ideal architecture:</p><ol><li>S3 sends an event to controller lambda when files are created/modified/deleted.</li><li>Controller lambda saves the metadata of the job in DynamoDB and marks the status as IN-QUEUE.</li><li>Then, controller lambda pushes the job ID onto the queue (SQS) for processing.</li><li>When there are new messages in the queue, a Cloudwatch Alarm (ApproximateNumberOfMessagesVisible&gt;0) triggers an SNS topic which in turn triggers the scaling lambda.</li><li>Scaling lambda starts spawning new workers based on how many messages are present in the queue. It does one other thing also which we’ll discuss in point 9.</li><li>Workers start fetching job IDs from the queue. Metadata is fetched from Jobs DynamoDB and at the same time, the status is marked as PROCESSING.</li><li>Workers start transforming and loading the datasets onto the storage layers.</li><li>On completion of the job, the status is either marked as SUCCESS or FAILURE. If there are no messages to process in the SQS, the workers start self-terminating.</li><li>On its first execution, Scaling lambda also enables a Cloudwatch Event Rule that’ll trigger it every 5 minutes. This cron job keeps on invoking Scaling lambda as long as the Cloudwatch Event Rule is enabled. On any execution, if Scaling lambda sees there are no more messages to process, it disables the 5 minutes Cloudwatch Event Rule and finishes the execution.</li></ol><h3>Conclusion</h3><p>Today, there are many tools and services available, which can be used to create custom ETL pipelines. In this blog, we have demonstrated a clean and reusable pattern for a highly scalable ETL pipeline. We showed how Polly’s pipelines can accommodate for the inconsistent flow of data in a cost-effective manner while incorporating a fault-tolerant architecture.</p><p>What do you think of the solutions built by our engineers? Do you think there are better alternatives? Then come join us. We at Elucidata are always looking for engineers who are passionate about solving cutting-edge problems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd08019ec7fe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elucidata/how-to-build-highly-effective-etl-pipelines-dd08019ec7fe">How to Build Highly Effective ETL Pipelines</a> was originally published in <a href="https://medium.com/elucidata">Elucidata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dataset of the week — GSE102956]]></title>
            <link>https://medium.com/elucidata/dataset-of-the-week-gse102956-2dcd03d41c0e?source=rss----ddc0f60dbf8a---4</link>
            <guid isPermaLink="false">https://medium.com/p/2dcd03d41c0e</guid>
            <category><![CDATA[homo-sapiens]]></category>
            <category><![CDATA[drug-targets]]></category>
            <category><![CDATA[alzheimers-disease]]></category>
            <category><![CDATA[omics]]></category>
            <category><![CDATA[crispr-cas9]]></category>
            <dc:creator><![CDATA[Elucidata]]></dc:creator>
            <pubDate>Thu, 27 May 2021 18:48:58 GMT</pubDate>
            <atom:updated>2021-05-27T18:50:11.188Z</atom:updated>
            <content:encoded><![CDATA[<h3>Dataset of the week -GSE102956</h3><p>by Shruti Mehrotra</p><p><em>Our ‘Dataset of the Week’ series features publicly available omics datasets of scientific value, intending to promote data sharing and reuse.</em></p><p><strong>Publication</strong>: Neuron, 2018.</p><p>This week, we feature RNA-seq profiles of three iPSC-derived brain cell types from the publication ‘<a href="https://www.cell.com/neuron/fulltext/S0896-6273(18)30380-5?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0896627318303805%3Fshowall%3Dtrue">APOE4 Causes Widespread Molecular and Cellular Alterations Associated with Alzheimer’s Disease Phenotypes in Human iPSC-Derived Brain Cell Types</a>’(1).</p><p><strong>Clinical Problem</strong>:</p><p>The apolipoprotein E4 (APOE4) has gained much attention in recent years as the most significant genetic risk factor for sporadic Alzheimer’s disease (sAD)(2). Conformational changes in the secretion of APOE4 proteins by astrocytes, neurons, and microglia lead to various pathological effects unique to sAD. However, prior studies have failed to evaluate the significance of APOE4 expression in brain cells in the context of humans. Researchers have used the CRISPR/Cas9 technology to produce human-induced pluripotent stem cells (hiPSCs) mimicking AD phenotypes in 3 types of brain cells to bridge this gap. The paper also explores the physiopathological effects of removing the sAD causing factor from these stem cells.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*pLY3KPARt_5490NTKp8eRA.png" /><figcaption>Graphical Abstract</figcaption></figure><p><strong>Experimental Design for the datasets</strong>:</p><p>Parental APOE3 cells from an unaffected subject were used to generate APOE4 iPSCs using the CRISPR/Cas9 method. These cells were further differentiated into neurons, astrocytes, and microglia-like cells in replicates of 3. Further, exome-seq data (76-bp paired-end) were generated for each type of brain cells. Gene expressions underlying the cellularization were detailed using these RNA-seq datasets.</p><p><strong>Key findings of the publication</strong>:</p><ul><li>APOE4 caused accelerated maturation of neurons as well as upregulated genes related to neuronal differentiation.</li><li>Significant downregulation of the BDNF gene associated with the synaptic function was noted. This is consistent with the cognitive impairment phenotype associated with the BDNF gene in previous studies (3,4).</li><li>A cell-specific reduction in APOE protein transcription was noted in astrocytes indicating that the APOE4 variant can impact its own transcription.</li><li>The APOE4 variant regulates the expression of both metabolism and transport of lipids such as<br>cholesterol, with potential effects on each of the three brain cell types.</li><li>APOE4 glial cells were observed to be impaired in their ability to uptake and eliminate extracellular beta- amyloids (Aβ) peptides.</li></ul><p><strong>Significance of the dataset</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*SMWhFfwNZ8Ov0Nb9KxUzzg.png" /><figcaption>A PCA plot depicting clusters of iPScs from a normal individual (Purple), iPSCs derived from neurons (Olive), and iPSCs derived from astrocytes(Brown). Plot generated on Polly.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*VFmgdeO_jN8vnj7cjM_gJA.png" /><figcaption>Volcano plot showing the statistical significance of several genes, including OSMR (Oncostatin M Receptor). OSMR is expressed in the glial cells and is involved with the formation of amyloids in patients with neurodegenerative diseases. Plot generated on Polly.</figcaption></figure><p>Links: <a href="https://www.cell.com/neuron/fulltext/S0896-6273(18)30380-5?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0896627318303805%3Fshowall%3Dtrue">Full Article</a> | <a href="https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE102956">Dataset: GSE102956</a> | <a href="https://pubmed.ncbi.nlm.nih.gov/29861287/">PMID</a> | <a href="https://linkinghub.elsevier.com/retrieve/pii/S0896627318303805">DOI</a> | <a href="https://omixwiki.elucidata.io/507/gse102956/apoe4-causes-widespread-molecula">OmixWiki</a> | <a href="https://elucidata.io/discover/">Polly</a></p><p><strong>References:</strong></p><ol><li>Lin YT., et al. APOE4 Causes Widespread Molecular and Cellular Alterations Associated with Alzheimer’s Disease Phenotypes in Human iPSC-Derived Brain Cell Types. <em>Neuron</em>. vol. 98(6): 1141–1154 (2018).</li><li>Cataldo, A., et al. Endocytic Pathway Abnormalities Precede Amyloid β Deposition in Sporadic Alzheimer’s Disease and Down Syndrome: Differential Effects of APOE Genotype and Presenilin Mutations. <em>The American Journal of Pathology</em>, vol. 157(1): 277–286 (2000).</li><li>Zhao J., et al. APOE4 exacerbates synapse loss and neurodegeneration in Alzheimer’s disease patient iPSC-derived cerebral organoids. <em>Nature Communications</em>. vol. 11(1): 5540 (2020).</li><li>Zhang L., et al. Brain-Derived Neurotrophic Factor Ameliorates Learning Deficits in a Rat Model of Alzheimer’s Disease Induced by Aβ1–42. <em>PLoS ONE</em>. vol. 10(4): e0122415 (2015).</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dcd03d41c0e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elucidata/dataset-of-the-week-gse102956-2dcd03d41c0e">Dataset of the week — GSE102956</a> was originally published in <a href="https://medium.com/elucidata">Elucidata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hiring Engineering Leaders for Scaling Bioinformatics 101]]></title>
            <link>https://medium.com/elucidata/hiring-engineering-leaders-for-scaling-bioinformatics-101-10a91b4c5fa2?source=rss----ddc0f60dbf8a---4</link>
            <guid isPermaLink="false">https://medium.com/p/10a91b4c5fa2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[hiring]]></category>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[full-stack]]></category>
            <dc:creator><![CDATA[Elucidata]]></dc:creator>
            <pubDate>Mon, 24 May 2021 12:01:39 GMT</pubDate>
            <atom:updated>2021-05-24T12:01:39.592Z</atom:updated>
            <content:encoded><![CDATA[<p>by Swetabh Pathak</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZsNVUyiZaWGbW3x6.png" /></figure><p>We, at <a href="https://elucidata.io/">Elucidata</a>, work with many small to mid-size biopharma companies that use large amounts of high throughput data. They generate this data in-house or through CRO partners. Apart from this, they are also looking at multiple public data sources — data repositories, databases and papers.</p><p>For a while, the teams are composed of just bioinformatics scientists. If you are lucky or have hired well — your team can make and deploy production-ready pipelines on their own. But at some point, that isn’t enough. You need a way to link your data store to notebooks, connect interactive environments to production-ready pipelines and build dashboards. Like lots of dashboards!</p><p>This is when many of our partners identify the need for an engineering leader. You need someone who can think of software maintenance, building scalable infrastructure, enforcing code practices and more.</p><p>As pharma companies reorient themselves to tackle the <a href="https://www.nature.com/articles/498255a">biological big data revolution</a>, bioinformatics leaders often find themselves tasked with filling technical/engineering positions within the organization to help meet their team’s growing computational needs. Hiring for a position that you have no background or training in can be very challenging. Ask me! I have hired for many roles where I couldn’t pass the most basic first round.</p><p>If you’re a bioinformatics scientist looking to hire an engineering leader for your team, here’s a handy guide to help you get started:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E-Ej5vtJmMR1glWm.png" /></figure><p><strong>1. Identify the technical gap.</strong></p><p>The first step of the hiring process is to identify the gaps in knowledge on your team. Different roles will require different areas of expertise (see infographic). Narrowing down on the exact technical requirements will help you in your search for potential candidates and simplify the selection process. For instance, if your team is required to build a large number of dashboards, you can anticipate a big front-end gap. On the other hand, if your team deals with a high throughput of next-generation sequencing (NGS) data, you probably want to hire someone with significant experience in cloud computing services like AWS, GCP or similar platforms of your choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ucnVQzVK8tcJysm6.png" /></figure><p><strong>2. Remember that you are a biopharma company, not a tech start-up.</strong></p><p>If you’re hiring an engineering leader at a tech company, you’re looking for someone who can build scalable technology and grow and manage teams of developers. On the contrary, at a biopharma company, the focus is on enabling bioinformaticians/bench-scientists to be more efficient. So advice which is excellent for tech companies, such as <a href="https://a16z.com/2017/05/26/hiring-vp-engineering-why-what/">this post</a> from a famous technology venture firm, might not necessarily work for you. The person you are looking for is more of an individual contributor who can also play a bit of the product manager role. Tech companies have specialists to do product/project management. In your case, it’s more likely that the head of engineering is also talking to users to understand requirements and translate them into products. This is NOT what a traditional head of engineering at a tech start-up would do. They would focus a lot more on hiring, building engineering culture and be answerable for the final engineering delivery.</p><p><strong>3. Know when to buy versus build.</strong></p><p>Once you’ve figured out what kind of technical expertise your team needs, you may be able to work with contract tech professionals instead of hiring a full-time employee. This is a common trend in tech; <a href="https://www.forbes.com/sites/johanmoreno/2019/05/31/google-follows-a-growing-workplace-trend-hiring-more-contractors-than-employees/?sh=4f96ef15447f">Google reportedly employs 102,000 temporary/contract workers worldwide</a>. The question is when to opt for a contractor. The most common reason is when you require someone with a niche specialization for a fairly high-level need. Hiring a subject matter expert for a short-term project, such as a cloud migration specialist to get your cloud infrastructure up and running, would serve you better economically, as opposed to hiring a full-time employee. Outsourcing could also be a good long-term strategy when the requirement is for a small team that doesn’t have to be scaled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GhKtd3VIcEDiOEIM.png" /></figure><p><strong>4. Hire someone comfortable with working with contractors or outsourcing companies.</strong></p><p>Leading up from the previous point, It’s unlikely that you can hire for all the diverse skills that a bioinformatics platform needs. Budgets won’t allow it. In that case, your leadership might need to more open to hiring contractors or outsourcing companies. However, If your engineering leader wants to do everything in-house, you can run into problems. In tech companies, engineering leaders primarily focus on growing their teams and managing them. On the other hand, biopharma companies would require them to work with a limited team of engineers and be more involved in project and stakeholder management. In this scenario, you’re better off hiring a candidate who is amenable to leading a fairly small team of engineers while working with and managing contractors as required.</p><p><strong>5. Provide context to attract and retain the right tech talent.</strong></p><p>Engineers and developers who lack pharma industry experience may not understand how their work will contribute to the team and its growth. This may <a href="https://www.oracle.com/us/media1/talent-retention-6-best-practices-1676595.pdf">make it hard to attract and retain talent.</a> It’s important to provide this context through a well-crafted job description that talks about specific technical requirements, day-to-day responsibilities as well as how the potential hire’s contributions will fit into the larger vision of the company. The better you are at telling your story and specifying what you want, the easier it will be to find the right candidate.</p><p><strong>6. Look for someone who is on top of biopharma SaaS</strong></p><p>Over the past decade, the intersection of technology and biomedical research <a href="https://www.nature.com/articles/d41586-018-05267-x">has considerably accelerated the process of drug discovery</a>. As Pieter Peeters of Janssen Research &amp; Development put it, “AI is affecting every aspect of the pharmaceutical industry, from the very early drug discovery aspects, all the way through to how we do clinical development and bring medicines to patients.” There are a plethora of SaaS products on the market today that cater specifically to the data-intensive needs of biopharma companies. This ranges from access to ML-ready biomedical data (<a href="https://elucidata.io/">Elucidata</a>) to cloud-based services for processing (<a href="https://databricks.com/">Databricks</a>) and deploying machine learning models (<a href="https://aws.amazon.com/sagemaker/">AWS SageMaker</a>) on massive quantities of high-throughput data. In order to be a good fit, an ideal candidate would be someone who is excited about harnessing technology to solve problems in the pharma sector and keeps up with the latest technology trends in this space.</p><p><strong>7. Leadership qualities are as important as technical capabilities.</strong></p><p>While it’s a given that a technical lead should have superior coding skills, you should also be looking for someone with prior project and people management experience, the ability to serve as a good mentor, and has great communication and conflict resolution skills. Another underrated quality is the ability to delegate. It may be tempting for a tech lead to tackle every challenging problem hands-on, but this might leave them with limited bandwidth for their other responsibilities. Look for someone who not only surpasses the technical criteria but is also open to delegating important tasks to other team members without micromanaging them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KMQ_u05Voas2QfM2.png" /></figure><p><strong>8. Involve someone with prior experience in hiring for similar technical roles.</strong></p><p>It’s never a bad idea to ask for help when you need it. Talk to your friends, family, mentors, etc. to find someone who has prior experience hiring for similar positions. Get them involved in the process at some level. They will have a better sense of what to look for in terms of technical abilities as well as cultural fit for your team. If you have questions or would like to know more about hiring engineering leaders for a pharmaceutical company, feel free to reach out to me on <a href="https://www.linkedin.com/in/swetabh-pathak-b4154558/?originalSubdomain=in">Linkedin</a> or via email at swetabh.pathak@elucidata.io.</p><p>— <a href="https://www.linkedin.com/in/swetabh-pathak-b4154558/?originalSubdomain=in">Swetabh Pathak</a> is a co-founder of <a href="https://elucidata.io/">Elucidata</a>, the world’s largest repository of ML-ready biomedical multi-omics data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10a91b4c5fa2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elucidata/hiring-engineering-leaders-for-scaling-bioinformatics-101-10a91b4c5fa2">Hiring Engineering Leaders for Scaling Bioinformatics 101</a> was originally published in <a href="https://medium.com/elucidata">Elucidata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dataset of the week — GSE70955]]></title>
            <link>https://medium.com/elucidata/dataset-of-the-week-gse70955-28fe91ac31f1?source=rss----ddc0f60dbf8a---4</link>
            <guid isPermaLink="false">https://medium.com/p/28fe91ac31f1</guid>
            <category><![CDATA[bioinformatics]]></category>
            <category><![CDATA[muscular-dystrophy]]></category>
            <category><![CDATA[microarray-techniques]]></category>
            <category><![CDATA[omics]]></category>
            <category><![CDATA[drug-discovery]]></category>
            <dc:creator><![CDATA[Elucidata]]></dc:creator>
            <pubDate>Fri, 21 May 2021 20:23:27 GMT</pubDate>
            <atom:updated>2021-05-21T20:23:26.943Z</atom:updated>
            <content:encoded><![CDATA[<h3>Dataset of the week — GSE70955</h3><p>by Shruti Mehrotra</p><p><em>Our ‘Dataset of the Week’ series features publicly available omics datasets of scientific value, intending to promote data sharing and reuse.</em></p><p><strong>Publication</strong>: Cell Reports, 2016.</p><p>This week, we feature a microarray gene expression profile generated on an Affymetrix PrimeView human oligonucleotide array from the publication titled ‘<a href="https://www.cell.com/cell-reports/fulltext/S2211-1247(16)30584-8?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS2211124716305848%3Fshowall%3Dtrue">Concordant but Varied Phenotypes among Duchenne Muscular Dystrophy Patient-Specific Myoblasts Derived using a Human iPSC-Based Model</a>’(1).</p><p><strong>Clinical Problem</strong>:</p><p>Duchenne Muscular Dystrophy (DMD) is characterized by deteriorated muscle functioning caused by degeneration of myocytes (muscle cells). This genetic disorder is caused by a dysfunction of the DMD gene that regulates the production of dystrophin, contributing to the reinforcement of muscle fibers. Although previous studies have documented histological pathologies and molecular mechanisms of DMD in animal models, human cell DMD models have been rarely assessed(2,3). Researchers have used hiPSCs to demonstrate an in-vitro DMD model in humans that exhibited patient-specific dystrophin mutations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*Hb72D6T0Bc00dpQbc6n9JA.png" /><figcaption>Graphical Abstract</figcaption></figure><p><strong>Experimental Design</strong>:</p><p>Human embryonic stem cells were differentiated into skeletal muscle cells and compared to non-differentiated cells in triplicates. Microarray gene expression profiles were generated for the confirmation of differentiated cell characteristics. Global mRNA profiles were used to detail the gene expression underlying cellularization and to identify up-regulated genes involved in this process. Furthermore, microarray datasets were used for comparing control and disease gene expression profiles.</p><p><strong>Key findings of the publication</strong>:</p><ul><li>DMD myoblasts cultured in this experiment displayed patient-specific phenotypes — including altered transcriptional profiles, aberrant intracellular signaling, and defective myotube formation.</li><li>Using genetic correction and pharmacological “dual-SMAD” inhibition, the DMD-hiPSC-derived myoblasts formed “rescued” myotubes.</li><li>Affected myoblasts were observed to have increased BMP4 and TGFB signaling levels with aberrant expression of interleukins 6 and 8 and collagen 3 forming genes.</li><li>SPP1 (Osteopontin), BMP4, CASPASE1, BMP2, and TGFβ3 were reported to be the most positively correlated genes in the case of DMD myoblasts. SPP1 has been shown to be a potent genetic modifier of disease severity in DMD in previous studies(4).</li></ul><p><strong>Significance of the dataset</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*HGyNWBq1Pg2SH7GZDF4nWA.png" /><figcaption>PCA plot depicting clusters of normal embryonic stem cells (olive) and DMD-hiPSC-derived cells (purple). The plot is generated on Polly.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*q1kaSebBRTqqx6lBzcZY7A.png" /><figcaption>Volcano plot showing the statistical significance of several genes including MYOZ1 (Myozenin 1) which is involved with muscle strength. Plot generated on Polly.</figcaption></figure><p><strong>Links</strong>: <a href="https://www.cell.com/cell-reports/fulltext/S2211-1247(16)30584-8?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS2211124716305848%3Fshowall%3Dtrue">Full Article</a> | <a href="https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE70955">Dataset: GSE70955</a> | <a href="https://pubmed.ncbi.nlm.nih.gov/27239027/">PMID</a> | <a href="https://www.cell.com/cell-reports/fulltext/S2211-1247(16)30584-8?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS2211124716305848%3Fshowall%3Dtrue">DOI</a> | <a href="https://omixwiki.elucidata.io/506/gse70955/human-ipsc-based-model-for-duchenhttps:">OmixWiki </a>| <a href="https://elucidata.io/discover/">Polly</a></p><p><strong>References:</strong></p><ol><li>Choi IY., et al. Concordant but Varied Phenotypes among Duchenne Muscular Dystrophy Patient-Specific Myoblasts Derived using a Human iPSC-Based Model. <em>Cell Rep</em> vol. 15, 2301–12 (2016).</li><li>Barberi, T., Bradbury M., et al. Derivation of engraftable skeletal myoblasts from human embryonic stem cells. <em>Nat. Med.</em>1vol. 3, 642–648 (2007).</li><li>Li, H.L., Fujimoto, N., et al. Precise correction of the dystrophin gene in Duchenne muscular dystrophy patient induced pluripotent stem cells by TALEN and CRISPR-Cas9. Stem <em>Cell Reports</em> vol. 4, 143–154 (2015).</li><li>Pegoraro, E., Hoffman, E.P., et al. Cooperative International Neuromuscular Research Group SPP1 genotype is a determinant of disease severity in Duchenne muscular dystrophy. <em>Neurology</em> vol. 76, 219–226 (2011).</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28fe91ac31f1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elucidata/dataset-of-the-week-gse70955-28fe91ac31f1">Dataset of the week — GSE70955</a> was originally published in <a href="https://medium.com/elucidata">Elucidata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>