<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by xmoleslo on Medium]]></title>
        <description><![CDATA[Stories by xmoleslo on Medium]]></description>
        <link>https://medium.com/@xmoleslo?source=rss-1f4dbd47844b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*c3uh6cETgMukQ-79-5jxQg.jpeg</url>
            <title>Stories by xmoleslo on Medium</title>
            <link>https://medium.com/@xmoleslo?source=rss-1f4dbd47844b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 16:14:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@xmoleslo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a solution based on Machine Learning]]></title>
            <link>https://xmoleslo.medium.com/building-a-solution-based-on-machine-learning-4c454b375390?source=rss-1f4dbd47844b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c454b375390</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[hr]]></category>
            <dc:creator><![CDATA[xmoleslo]]></dc:creator>
            <pubDate>Wed, 19 May 2021 07:52:02 GMT</pubDate>
            <atom:updated>2021-05-19T21:51:27.437Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why a Machine Learning model is not a product if there is no MLOps.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vvhgrUX4YLrMZ1FQ" /><figcaption>Photo by <a href="https://unsplash.com/@martinadams?utm_source=medium&amp;utm_medium=referral">Martin Adams</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In <a href="https://xmoleslo.medium.com/how-we-improved-our-core-business-with-machine-learning-35be3293ceb4">a previous post</a>, we described the steps we took to implement a machine-learning solution to improve our core business. As a quick reminder, our goal is to help our colleagues from the recruitment department in their quest to find good candidates given a job description, which we formalized as follows:</p><blockquote>“Recruiters want a system that takes a job description as input, and orders the CVs from our database by relevance”</blockquote><p>Formulated in this way, the problem essentially boils down to a <strong>“sort by similarity”,</strong> which can be solved by constructing an adequate representation of the documents in a vector space.</p><p>In this post, we will describe the algorithm we chose to compute that similarity, why we chose it, and how we trained and evaluated it’s performance. However, training the model is only one of the many steps involved and we can confidently state that it is probably the best documented step.</p><p>Therefore, we will focus on the step 4 at the junction between the <em>Implement</em> and <em>Operate</em> stages of the MLOps cycle and expose <strong>our approach to implement the training as a reproducible process, and how this process intertwines with our CI/CD pipeline</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*O_W6FQSYNj0t7SeF-d1stw.png" /><figcaption>Figure 1: Mapping of the 5 steps presented to the MLOps cycle; Image by Author inspired by <a href="https://ml-ops.org/img/mlops-loop-en.jpg">ml-ops.org</a></figcaption></figure><p><strong>It’s time to get our hands dirty!</strong></p><h4>Choosing an Embedding method</h4><p>Among many, a very popular method of evaluating document similarity relies on word embeddings. In order to pick one of the several embedding algorithms that exist, we started by looking at our data, exclusively made of text extracted from CVs: sentences were very short, and many did not even contain verbs (e.g., lists of competences). Furthermore, the layout in a CV is very important, i.e., the content doesn’t necessarily speak for itself. In addition to the fact that CVs are not really written in “natural language”, they also contain many terms that are not “in the dictionary”, such as <em>C#</em>, <em>.NET</em>, <em>BABOK</em>, or <em>CISSP</em>.</p><p>Because of those two characteristics, we considered training a model from scratch. However, the size of our dataset was relatively small in terms of number of sentences, which is why we did not jump on large models such as those based on Transformers, but rather looked at more classical word-embedding representations. Among those, <strong>we picked Stanford’s GloVe word vector representation</strong> because of its construction based on the word-word co-occurrence count matrix [1], which we considered better suited to situations like ours where the text content is not made of “natural” sentences (lists of technologies, competences, or experiences).</p><h4>The embedding pipeline</h4><p>Unlike other embedding methods, GloVe requires the computation of a word-word co-occurrence count matrix (which will refer to as <em>“co-occurrence matrix” </em>for brevity). The co-occurrence matrix relies on the counting of words, which itself relies on the tokenization/text pre-processing stage (what do we consider as a “word”). Which can be summarized graphically as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*BtVIsPyWKEFGqSCHYcjRCA.png" /><figcaption>Figure 2: Steps of an embedding pipeline. Pre-processing steps are shown in blue and embedding step is shown in green. Produced artefact are shown in vivid colors. Image made by the author.</figcaption></figure><p>During the training stage, we divided the pipeline in two parts: the pre-processing stage illustrated in blue, and the embedding stage illustrated in green. Both pre-processing and embedding stages have hyper-parameters that may affect the results of the “sort by similarity” function we ultimately aim to build, and that is based on the cosine similarity between embedded document vectors, illustrated in red.</p><p>In order to evaluate the ranking produced using the cosine similarity distance on the embeddings that we just trained, we applied the Mean Average Precision and more generally the Mean Precision as described in <a href="https://medium.com/swlh/rank-aware-recsys-evaluation-metrics-5191bba16832">the excellent article by Moussa Taifi [2]</a>.</p><h4>Experiment tracking</h4><p>Because there are many hyper-parameters and combinations thereof, we need an experiment tracking mechanism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*YQdAJzbs79nulspRhgPprw.jpeg" /></figure><p>One of the go-to solution for experiment tracking is <a href="http://mlflow.org/">MLflow</a>. Once that you have setup a server and a database, which is a must if you wish to collaborate on the creation of models, you can log the results of your experiments using their API. Many resources exist about how to setup and use MLflow for experiment tracking, and a very good summary was made by <a href="https://quantum-inc.medium.com/tracking-experiments-with-mlflow-969b50e83d89">Quantum [3]</a>.</p><p><strong>MLflow was a good solution when the objective was mainly to obtain a satisfying model for a PoC and helped us reach the target after only a few weeks of work.</strong> However, in order to apply our setup to a larger dataset and to a larger set of hyperparameters, it exposed us to a challenging issue: the API proposed by MLflow to track experiments is interfaced in such a way that the code which manages the tracking of models’ performances is the same than the code which manages the training. In concrete terms, this means that specific MLflow code needs to be added inside your training loop to benefit from the tracking.</p><p>This way of passing training outputs through the tracking API has two consequences:</p><blockquote><strong>1. It induces tight coupling between pre-processing and training steps.</strong></blockquote><p>The current API of MLflow does not embrace well the concept of pipeline (1). This limitation makes the tool not quite adequate for cases where the pre-processing stage has such a large impact on the results. In those cases, experiments must be designed so that they span the whole pipeline, hindering modularity. Solving this shortcoming requires developing a lot of custom code in order to reuse intermediate results of the pipelines and often ends up breaking the lineage of the results, ultimately making the deployment of so trained models difficult.</p><blockquote><strong>2. Training and evaluation scripts need to be adapted to enable the tracking.</strong></blockquote><p>Although it may be seen as a good practice to start your ML project with tracking in mind, you may want to train a model implementation from GitHub or elsewhere for comparison purpose, and that project may not be using MLflow. Henceforth, you must modify, build, and possibly maintain a fork of that other project for the time of your experiments.</p><h4>Alternative solutions to scale-up the infrastructure: MLOps</h4><p>Having not only these limitations but also our need to scale-up in mind (both in terms of data ingestion capacity and of ability to try different model architectures), we decided to test two alternatives: <a href="https://www.kubeflow.org/">Kubeflow</a> and <a href="https://www.valohai.com/">Valohai</a>.</p><p>Kubeflow is the most present of the two solutions. It relies heavily on Kubernetes and the fact that it embraces the concept of pipeline better fits our need of modularity. However, Kubeflow is more difficult to setup than MLflow and still suffers from the fact that its definition of pipelines relies on the use of an SDK (based on decorators), which makes code updates mandatory in order to manage training flows.</p><p>On the contrary, <a href="https://docs.valohai.com/topic-guides/architecture/">Valohai is a SaaS solution</a>, hence very easy to setup. The pipelines are defined as steps in a dedicated YAML file (<em>valohai.yaml</em>) where you specify commands to be executed in Docker containers. In our case, the containers needed to run on a server configured with two high-end GPUs on-premises; the configuration to add our on-premises machine to the management interface of Valohai as a worker was very easy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*8pcKxnVQWD9FeGpbB4DJgQ.png" /><figcaption>Valohai’s value proposition. Image source: <a href="https://valohai.com/blog/how-to-track-machine-learning-experiments/">https://valohai.com/blog/how-to-track-machine-learning-experiments/</a></figcaption></figure><p>Valohai also comes with a very intuitive CLI, which enabled us to fully automate the Continuous Integration process by linking our version control system (Gitlab) with Valohai and triggering events based on merge requests.</p><p>If you’re curious about what it takes to track a machine-learning project with Valohai, <a href="https://github.com/Qiminfo/glove-mlops">we prepared a public Github repository</a>: the steps and the pipeline are defined in the <em>valohai.yaml</em> file, and execution of vh(Valohai’s CLI) are given in the <em>README.md</em>.</p><h4>Hyperparameter tuning</h4><p>Hyperparameters influence the output of each step of the pipeline, and consequently influence the final output of the pipeline. Similar to <a href="https://www.kubeflow.org/docs/components/katib/overview/">Kubeflow’s Katib</a>, Valohai proposes a hyperparameter tuning component helping us to find the best hyperparameter values for each step of the pipeline. There are basically two search modes available: grid search and Bayesian optimization.</p><p>One current downside is that the range of values has to be specified in the Web UI, and therefore requires manual intervention, or by using Valohai’s REST API, at the expense of additional code. Ideally, this range could be provided in the <em>valohai.yaml</em> file, similarly to multistep definitions in MLflow (1).</p><p>In terms of parameter tuning, one extremely useful feature of Valohai is <em>Tracing</em>, which enables to analyze execution lineage visually. For example, in Figure 4, we see that pipeline 30 is constituted from executions #156 and #157, that these executions originate from commit 905a491dd5, that execution #156used a pickled glove embedder as input and then outputted the file <em>embedded_cvs_labeled_df.json</em>, which in turn was used as input for execution #157.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*Hpk_TJTijhKyHq1J6TTtyA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bisbeD2UID_9r8wBU6dYSA.png" /><figcaption>Figure 4: Visual information provided by the tracing module. Above: the simplified lineage information for pipeline 30 (description in the text). Below: hovering on the “glove_embedder.pkl” node shows were it comes from.</figcaption></figure><h4>CI: Pipeline automation</h4><p>As mentioned earlier, there is more to a model than the training: you need to add preprocessing methods, evaluation metrics, the boilerplate to manage the execution of related experimentations and the serialization of the model, and of course, the unit-tests. Needless to say, all this code is subject to change.</p><p>After having achieved a first PoC, we needed to optimize various steps to cope with the increased volume of data to build the final model, leading us to change some implementations to make good use of our on-premises GPUs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/1*SMwXN7azKgM3ARaFw-VJ-A.jpeg" /><figcaption>Time to adapt the code to make good use of that hardware.</figcaption></figure><p>At this point, having a reliable way to run our experiments and integrate them in our CI/CD pipeline was more than a nice to have. In this regard, Valohai’s CLI was an extremely useful feature: it allowed us to simply run a few commands in Gitlab’s runner in order to update Valohai’s project and run the newly built docker images containing the latest code’s version and its dependencies on our on-premises ML rig!</p><p>In Figure 3, we illustrate our CI/CD pipeline based on 2 branches, the <strong>valohai</strong> branch, which triggers new runs of the training pipeline, and the <strong>master</strong> branch, which triggers the packaging and deployment on UAT environment for each new merge (and on PRD environment for each new version tag).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*HzvT3v2MuF5NtCJlprewzw.png" /><figcaption>Figure 3: Above: CI / CD pipeline interconnecting Gitlab and Valohai. Below: steps of the CI/CD pipeline mapped to git events. Image made by the author.</figcaption></figure><p>An important thing to note, is the ability to perform ad-hoc execution on Valohai, which sends a local copy of the code to be executed on worker host without the need to push. This can be extremely handy when developing the training pipeline <em>itself</em> or testing some optimizations.</p><h4>CD: Automated packaging and deployment</h4><p>Even if Valohai offers an easy solution to deploy models, the deployment happens in a Kubernetes cluster, <a href="https://docs.valohai.com/reference-guides/valohai-yaml/endpoint/">which can be managed by Valohai or yourself</a>. As a reminder from <a href="https://xmoleslo.medium.com/how-we-improved-our-core-business-with-machine-learning-35be3293ceb4">our previous post</a>, the purpose of our model is to be integrated in a business application managed internally. Therefore, it was easier for us to export and package the best performing model using an alternative tool, namely BentoML, with which we already had prior experience.</p><p>BentoML, which presents itself with the appealing statement of “<a href="https://docs.bentoml.org/en/latest/">Model Serving Made Easy</a>”, is a library that really shines in packaging and deploying model to a variety of target infrastructures, such as Docker Container, AWS lambda, AWS ECS, Azure Function, <a href="https://docs.bentoml.org/en/latest/deployment/index.html">and many more</a>. Because we expect this pattern to be frequent for, e.g., deploying a model to a client’s network, we took the time to integrate BentoML into our CI/CD pipeline. The following snippet show how the code to retrieve the best model from Valohai and packaging it happens using BentoML framework.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d15d3245ebfbf4389835cbd23e755cd/href">https://medium.com/media/9d15d3245ebfbf4389835cbd23e755cd/href</a></iframe><p>After packaging, we can deploy the model to one of the different targets proposed by BentoML. In our case, we used the Dockerfile generated by BentoML, and then built the image using Kaniko (for security reasons). The deployment to our AWS-based Kubernetes cluster is done via helm charts triggered by the Autodevops functionality of Gitlab that was properly configured by the DevOps team of Qim’s Center of Expertise.</p><h4>Conclusion</h4><p>Even when you are dealing with “simple” models, complexity augments rapidly with the number of pre-processing steps. Managing efficiently the different steps involved in the design and training of a model is an important building block for reproducible experimentation. Furthermore, a solution that preserves traceability even when reusing artefacts from older runs helps improving the pace of iterations.</p><p>Without these capabilities, building and maintaining a product based on machine learning in a collaborative, reproducible, and traceable way becomes really cumbersome and is nearly impossible in an ever-changing team such as ours. Of course, you can easily deliver a model, but auditing it and continuously improving it based on user feedbacks is a never-ending process that can only be sustainable if you have some level of automation and embrace MLOps practice.</p><p>We tested several solutions during our Proof of Concept: MLflow, Kubeflow, Valohai and BentoML each with their strength and weaknesses. Since our goal was to create a pipeline that would fit into our existing landscape for ease of maintenance, and still be generic enough to be used with our customers without loss of automation, we picked the tools that felt better suited for that goal.</p><p>The ability to link all the pieces together: code versioning, unit-testing, training, evaluation, packaging and deployment is not straightforward and requires mastering many of these tools. However, depending on your setup it may be possible that other tools are more integrated or covering a broader scope, which could ease this process. The solutions presented here are being actively developed, and some are still in beta at the time of writing this article. Therefore you might want to verify the status of these products before making your call.</p><h4>Conflicts of interests disclosure</h4><p>Qim info S.A. is not Valohai’s affiliate. The experimentations reported in this article were conducted during the trial period of Valohai’s MLOps platform.</p><h4>Notes</h4><p>(1) Multistep workflow capabilities do exist in MLflow, though it’s a more advanced feature. A good example of its usage can be found <a href="https://github.com/mlflow/mlflow/tree/2592314d4b731b052ae192052a693383095069c9/examples/multistep_workflow">here</a>. However, even though it reduces the coupling in the code, it does not yet offer the same level of abstraction as pipelines. A quick look at the <a href="https://github.com/mlflow/mlflow/blob/2592314d4b731b052ae192052a693383095069c9/examples/multistep_workflow/main.py">main.py file</a>, the script file responsible for gluing all the steps, shows that it is mainly made of boilerplate code needed to deal with orchestration, retries and so forth.</p><h4>References</h4><p>[1] <a href="https://nlp.stanford.edu/projects/glove/">Jeffrey Pennington, Richard Socher, Christopher D. Manning, GloVe: Global Vectors for Word Representation (2014</a>), <a href="https://nlp.stanford.edu/projects/glove/">https://nlp.stanford.edu/projects/glove/</a></p><p>[2] Moussa Taifi, MRR vs MAP vs NDCG: Rank-Aware Evaluation Metrics And When To Use Them (2019), <a href="https://medium.com/swlh/rank-aware-recsys-evaluation-metrics-5191bba16832">https://medium.com/swlh/rank-aware-recsys-evaluation-metrics-5191bba16832</a></p><p>[3] Quantum inc., Tracking Experiments with MLflow (2020), <a href="https://quantum-inc.medium.com/tracking-experiments-with-mlflow-969b50e83d89">https://quantum-inc.medium.com/tracking-experiments-with-mlflow-969b50e83d89</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c454b375390" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we improved our core business with Machine Learning]]></title>
            <link>https://xmoleslo.medium.com/how-we-improved-our-core-business-with-machine-learning-35be3293ceb4?source=rss-1f4dbd47844b------2</link>
            <guid isPermaLink="false">https://medium.com/p/35be3293ceb4</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[recruiting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[devops-and-mlops]]></category>
            <dc:creator><![CDATA[xmoleslo]]></dc:creator>
            <pubDate>Fri, 09 Apr 2021 13:04:12 GMT</pubDate>
            <atom:updated>2021-05-06T10:30:35.235Z</atom:updated>
            <content:encoded><![CDATA[<h3>A step-by-step guide to a real-world implementation of a machine learning solution.</h3><figure><img alt="Both theory and experience matter when you start your journey to apply machine-learning to your business" src="https://cdn-images-1.medium.com/max/1024/0*R3IG9Je72bwMDIHy" /><figcaption>Photo by <a href="https://unsplash.com/@benwhitephotography?utm_source=medium&amp;utm_medium=referral">Ben White</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Do you remember that rush and excitement after leaving a conference talk or ending an article describing a successful story about Machine Learning (ML) or AI? How easy and straightforward it seemed? And then, back at the office, when trying to replicate the story, feeling frustrated by the lack of time of your team, the absence of quality — “Kaggle-grade” — data, the lack of computing power at your disposal, or experiencing the feeling of a daunting task to implement all best practices such as TDD, CI/CD, DevOps, MLOps, Agile project management, let alone ethical AI?</p><p><strong>This is our first post of a series of articles about our journey: how we formalized the business problem, the challenges we faced, technical details about our data pipelines, ML techniques used, validation methods, and how we implemented some of the best practices (CI/CD, MLOps) to deliver a full-blown product to our colleagues. In this first post, we describe our first NLP project applied to one important aspect of our core business, which is recruitment.</strong></p><p>Because this experience summed up many aspects of ML projects, from stating the problem to the deployment of the solution, and because it was conducted in the usual conditions SMEs would face (time and resource constraints), we were willing to share it with a broader audience.</p><p><strong>So let’s dive in!</strong></p><h4>Step 1: Problem Statement</h4><p>To relate a journey that leads us somewhere, it is useful to describe where we come from. Qim is a consulting company operating in Switzerland (Geneva and Lausanne, and recently Zürich) and France. With about 300 employees, it ranks in the top 10 of the consulting firms in Romandy. Our clients send us requests for many different types of profiles: analysts, developers, data engineers, architects, DevOps, project managers, …</p><p>As new technologies emerge and, more importantly, are adopted by companies and agencies, the need for professionals mastering them becomes more pressing. As a consulting firm, we need to hire the best profiles to successfully meet our customers’ expectations and needs. Therefore, our recruiters have the difficult responsibility to find the perfect match between customer demands and candidates’ skills and interests.</p><p>Considering the type of profiles sought after, it requires very seasoned recruiters to understand all these CVs in detail and spot the right candidate. Recruiters often get to skim through many — sometimes hundreds — CVs to filter out potential candidates. As you can imagine, this process is time-consuming and can be frustrating in some cases.</p><p>The team of Data Scientists and ML Engineers at Qim has a strong experience supporting our customers in their data and analytics projects and our newly created Center of Expertise enabled us to apply our NLP skills to this recruitment challenge.</p><p>Because good data science begins with a good understanding of the business needs, we began by working closely with the recruiters: for every job description they work on, recruiters must find relevant candidates.</p><p>Therefore, we expressed our recruiters’ need as follows:</p><blockquote>“Recruiters want a system that takes a job description as input, and returns the relevant CVs from our database”</blockquote><h4>Step 2: Assess possibilities</h4><p>Now that we formulated the problem so that it states clearly “<strong>what </strong>needs to be done”, we will quickly check what we have at our disposal to solve it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*5qP7Sz3ddItSpJJWJX1lng.png" /><figcaption>Table 1: Checklist of ingredients for a successful AI project</figcaption></figure><p>From Table 1, we observe that we have most of what is needed to conduct a successful AI project except <em>Clean Data</em> and <em>Labeled Data</em>. Even though a good data processing pipeline will help us get clean data (we will dedicate an article to the pipeline we implemented on AWS), the lack of labeled data is definitely an issue. Indeed, how are we supposed to train a machine learning system without valid examples?</p><p>The first, obvious, solution to this issue consists of manually selecting relevant candidates for a large and representative set of job descriptions, and record those matches as labels. Although this option would help us solve the issue directly, it would require a lot of time and effort. We will therefore consider a second option before going down that road: redefine the problem to use more accessible information, as described in the next section.</p><h4>Step 3: Formalize the problem</h4><p>Now that we know <strong>what</strong> needs to be done and have validated that we have <em>the appropriate elements (commitment, skills, data, resources)</em>, let’s see <strong>how</strong> we can achieve our goal.</p><p>Because we lack proper <em>Labeled Data</em>, we adapted the original recruiters’ need as follows:</p><blockquote>“Recruiters want a system that takes a job description as input, and orders the CVs from our database by relevance”</blockquote><p>Doing so, we preserve most of the value for our recruiters, who will <strong>still be able of finding relevant candidates more quickly, while reducing the overall effort to build a <em>Labeled Dataset</em></strong>. Indeed, we can now formalize our problem as a <strong>sort</strong>, which implies that we should somehow evaluate the similarity between documents (job description and CV).</p><p>Among many, a very popular method of evaluating document similarity relies on word embeddings, <a href="https://web.stanford.edu/~jurafsky/slp3/6.pdf"><em>which are learned representations of the meaning of words directly from their distributions in texts. These representations are used in every natural language processing application that makes use of meaning</em> (Jurafsky &amp; Martin, 2020)</a>. In a future post, we’ll describe the process of designing, training, and evaluating document sorting methods relying on word embeddings in a controlled setup. For the time being, let’s assume that we have a method to compute a similarity between two documents. How can we make sure that this measure is helpful to solve our problem?</p><h4>Step 4: Train and evaluate models</h4><p>We still miss a proper way to validate that we correctly sort our entire CV database given a job description. Again, it would require a large amount of time and effort to produce a dataset directly.</p><p>We started looking at some of the CVs we had access to, and because Qim is active in the IT space, most of the candidates could be associated with one or more of these <strong>job roles</strong>: front-end developer, back-end developer, architect, project manager, analyst, security professional, system administrator, support staff, or data professional. Because job roles are very relevant to recruiters at the time of considering a candidate, we used those as labels to evaluate our models. In essence, two documents sharing a common label are deemed to be similar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*6E-9h0OKmaaHCt1q_KyZaw.png" /><figcaption>Figure 1: Evaluate sorting models with labeled data</figcaption></figure><p>Although we now have a simple way of creating <em>Labeled Data</em> and evaluate models, these are still <em>proxy values</em> and proper evaluations on actual job descriptions must be conducted. However, this requires the help of our recruiters, whose time is best used at finding candidates for actual job descriptions than at evaluating models, which is why <em>only the best model according to our proxy benchmark was evaluated using their help</em>.</p><p>To conduct this real-world evaluation, we designed and developed a minimal UI that would enable participating recruiters to use our model for their actual research and enable them to provide feedback about the results.</p><h4>Step 5: Integrate the model in the workflow</h4><p>There are many ways to integrate a model into existing workflows and many considerations drive the decision. In our case, because the tool used by Qim’s recruiters is developed by Qim, we have more liberty when it comes to adapt the workflow to best integrate our new model.</p><p>For example, recruiters add the new job description into the system before starting the search for candidates to link the interesting CVs to that job description. In this case, we could call the model as soon as the job description’s PDF is uploaded, providing recruiters with results ordered by relevance during their search. With additional modifications of the existing UI, relevant candidates could even be shown as soon as the model’s results are available, saving even more time for the recruiter!</p><blockquote>This kind of integration is very different from the minimal UI discussed in the previous section because that interface was made to collect feedback and not to streamline the job. However, it is a good step to take to evaluate if the model is worth the integration development costs.</blockquote><p>When integrating the model, it is useful to plan for the monitoring of the model and prepare for its retraining. Even though this significantly increases integration costs, planning for issues such as new job roles, techniques and frameworks, and future improvements such as accepting CVs in German or Italian in addition to French and English will lower maintenance and model improvement costs. We will present the techniques and tools we used to implement this MLOps cycle in a future post.</p><h4>Conclusions</h4><p>In this article, we briefly presented our 5 steps approach applied to our business problem. We expose some of the considerations we had to build our MLOps framework. If we had to map those 5 steps to the traditional MLOps cycle, it would be looking as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*iH1uvUyjLufGdTnMwTv6AA.png" /><figcaption>Figure 1: Mapping of the 5 steps presented to the MLOps cycle; Image by Author inspired by <a href="https://ml-ops.org/img/mlops-loop-en.jpg">ml-ops.org</a></figcaption></figure><p>This introductory post will be followed by:</p><ul><li>an in-depth description of our pdf processing pipeline</li><li>the algorithm and evaluation techniques that we used</li><li>the MLOps tools used to implement the cycle.</li></ul><p>We hope that you’ll appreciate those posts too!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35be3293ceb4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>