<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alfonso Farruggia on Medium]]></title>
        <description><![CDATA[Stories by Alfonso Farruggia on Medium]]></description>
        <link>https://medium.com/@alfarruggia?source=rss-a981c1de8866------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_GvUw9z7oPG-T1Dso3aGTQ.jpeg</url>
            <title>Stories by Alfonso Farruggia on Medium</title>
            <link>https://medium.com/@alfarruggia?source=rss-a981c1de8866------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:57:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alfarruggia/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Token-classification in ReactNative]]></title>
            <link>https://alfarruggia.medium.com/token-classification-in-react-62cc2cba1a37?source=rss-a981c1de8866------2</link>
            <guid isPermaLink="false">https://medium.com/p/62cc2cba1a37</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[react-native]]></category>
            <category><![CDATA[tensorflow-lite]]></category>
            <dc:creator><![CDATA[Alfonso Farruggia]]></dc:creator>
            <pubDate>Mon, 11 Mar 2024 09:21:49 GMT</pubDate>
            <atom:updated>2024-03-13T13:08:20.975Z</atom:updated>
            <content:encoded><![CDATA[<h3>Token-classification in Expo ReactNative</h3><p>This post shows how to integrate the machine learning models into a Expo ReactNative App. Firstly, the fine-tuned DistilBERT model will be converted into TensorFlow Lite format. Subsequently, it will be loaded into your mobile app to execute token classification.</p><h3>Series Table of Contents</h3><ul><li><a href="https://medium.com/p/ec1140d75085">How to create a Workout Domain Synthetic Dataset Synthetic Dataset</a></li><li><a href="https://alfarruggia.medium.com/1dc0acb0115d">Token Classification Fine-tuning with a Synthenic Dataset</a></li><li><a href="https://alfarruggia.medium.com/62cc2cba1a37"><strong>Token-classification in Expo ReactNative</strong></a></li></ul><p>Convert model to TensorFlow Lite format</p><p>To convert it to TFLITE format, we will use <strong>optimum-cli.</strong> The following script allows you to install the Python library and convert the fine-tuned DistilBERT model into the TFLITE format. The converter saves the TFLITE model in the <em>tflite_mobile</em> folder. It’s worth noting that we need to pass the <strong>sequence_length</strong> and task <strong>parameters</strong> to optimum-cli. In our case, we passed 512 as the sequence_length and ‘token-classification’ as the task.”</p><pre>$&gt; pip install &quot;optimum[exporters-tf]&quot;<br>$&gt; optimum-cli export tflite --model ./model_output --sequence_length 512 --task token-classification ./tflite_mobile</pre><h4>ReactNative App Configuration</h4><p>The React Native app we are going to discuss is very simple. It allows the classification of a text using the ML model we refined earlier. It enables users to input text using a text box and assigns a label to each word according to the dataset we have already created. You can find the full code of the React Native app on this <a href="https://github.com/Arfius/ReactNative-Token-Classification">GitHub</a> repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b2mRcwZMWOR1zcN0v0ihUQ.png" /></figure><p>After creating the React Native app, we need to install the react-native-fast-tflite library to interact with the TFLite model. Please follow these instructions to install the react-native-fast-tflite library. Next, we need to copy the tflite_mobile folder into the assets folder inside the React Native project for the mobile app and update the metro.config.js file to load the necessary files from the model folder.</p><pre>// Learn more https://docs.expo.io/guides/customizing-metro<br>const { getDefaultConfig } = require(&#39;expo/metro-config&#39;);<br><br>/** @type {import(&#39;expo/metro-config&#39;).MetroConfig} */<br>const config = getDefaultConfig(__dirname);<br><br>config.resolver.assetExts.push(&#39;tflite&#39;)<br>config.resolver.assetExts.push(&#39;json&#39;)<br>config.resolver.assetExts.push(&#39;txt&#39;)<br><br>module.exports = config;<br></pre><h4>Load the TensorFlow Lite model in the mobile App</h4><p>To load the model, we first need to import the react-native-fast-tflite library. In this example, we will work on the App.js file. Then, we need to load the model by passing the URI of the TFLite file to the loadTensorflowModel function. You can see the full changes by examining the loadModel function in the App.js file.</p><pre>import {<br>  loadTensorflowModel<br>} from &#39;react-native-fast-tflite&#39;<br><br>[...]<br><br>export default function App() {<br>  const [model, setModel] = useState(null);<br>  const [tokenizer, setTokenizer] = useState();<br><br>[...]<br><br>  const [assets, error] = useAssets([require(&#39;./assets/tflite_mobile/model.tflite&#39;), <br>                                     require(&#39;./assets/tflite_mobile/vocab.txt&#39;)]);<br><br>  const config = require(&#39;./assets/tflite_mobile/config.json&#39;)<br><br>  const loadModel = async () =&gt; {<br>    try {<br>        const loadedModel = await loadTensorflowModel({url:assets[0].uri});<br>        setModel(loadedModel);<br>        let tokenizer = new MyLightTokenizer(vocab)<br>        setTokenizer(tokenizer)<br>[...]</pre><p>For simplicity, I’ve implemented a MyLightTokenizer class that allows tokenization using the vocabulary carried over from the ML model. This class will enable us to generate input for the inference, which we&#39;ll explore in the next step.</p><h4>Run Inference</h4><p>Before running the TFLite model, I examined its inputs and outputs to align the data accordingly. As we can see, the model expects two [1, 512] Int64 arrays as input and returns a float32 as output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WupH56EMyeMqNbRYQrsvvQ.png" /><figcaption>Image from Netron.app</figcaption></figure><p>The actual inference is implemented in the process_text function in the App.js file, as shown in the code below. This function first tokenizes the text to prepare the data according to the model&#39;s input format.</p><pre>const process_text = async ()=&gt;{<br>    // text tokenisation<br>    const sentence = text;<br>    const tokens_info = tokenizer.tokenize(sentence);<br>    const paddedTokens = tokenizer.padTokens(tokens_info.tokenIds, 510);<br>    const inputIdsIDS = tokenizer.addSpecialTokens(paddedTokens)<br>    const inputIdsIDS_b64 = BigInt64Array.from(inputIdsIDS, x =&gt; BigInt(x));<br>    const inputAttentionMaskIDS = new BigInt64Array(512).fill(BigInt(1))<br></pre><p>Next, it runs the inference and reshapes the output according to the model’s output format. Each word is represented by a vector of 8 values. Then, for each vector, it extracts the index with the maximum probability. This index represents the label for the corresponding word.</p><pre>try{<br>      // run inference<br>      // 8 is the number of labels <br>      const outputData =  model.runSync([inputAttentionMaskIDS, inputIdsIDS_b64])<br>      let proPos = _.chunk(outputData[0], 8)<br>      let indexs = []<br>      proPos.forEach(x =&gt; indexs.push(_.indexOf(x, _.max(x))))</pre><p>The final step is to convert the label index into textual labels and update the UI with the list of tokens. The mapping between labels and indices is defined in the config.js file of the ML model.</p><pre>  // get label for each token<br>  indexs = indexs.slice(1, tokens_info.tokens.length+1)<br>  const words = indexs.map(index =&gt; config.id2label[index]); <br>   <br>  // update UI<br>  var tags = words.map((k, i)=&gt;{ <br>    return &lt;TagLine tag={k} text={tokens_info.tokens[i]}&gt;&lt;/TagLine&gt;<br>  })<br><br>  setTags(tags)<br>      <br>    }catch(e){<br>      console.error(e)<br>    }<br>  } // end process_text function</pre><p>To display the list of labels, I’ve implemented the TagLine React Native component, which shows a different icon and color for each label. For example, a purple # represents the label &quot;Number&quot;, while a green bicycle indicates that &quot;jumps&quot; has the label &quot;Workout&quot;.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*T2DlOJSf8HB9rRBUkO7_0w.png" /></figure><p>You can find the full code of the React Native app on this <a href="https://github.com/Arfius/ReactNative-Token-Classification">GitHub</a> repository.</p><h4>Conclusion</h4><p>In this series, we’ve explored how to generate, fine-tune, and embed an ML model for token classification into a mobile application. Despite the availability of LLM services, which are now a reality and straightforward to integrate into mobile applications, there is still a market and a need to pay attention to scenarios where ad-hoc ML models are used in embedded systems or smartphones.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62cc2cba1a37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Token Classification Fine-tuning using Synthenic Dataset]]></title>
            <link>https://alfarruggia.medium.com/token-classification-fine-tuning-using-synthenic-dataset-1dc0acb0115d?source=rss-a981c1de8866------2</link>
            <guid isPermaLink="false">https://medium.com/p/1dc0acb0115d</guid>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[synthetic-data]]></category>
            <dc:creator><![CDATA[Alfonso Farruggia]]></dc:creator>
            <pubDate>Mon, 11 Mar 2024 09:20:57 GMT</pubDate>
            <atom:updated>2024-03-13T13:09:56.853Z</atom:updated>
            <content:encoded><![CDATA[<p>This Medium series covers the content published in the Hugging Face blog about refining the Token Classification Task. While the original post utilizes the well-known wnut_17 dataset to refine the DistilBERT model, here, we refine DistilBERT using the synthetic dataset wout generated by AI.</p><h3>Series Table of Contents</h3><ul><li><a href="https://medium.com/p/ec1140d75085">How to create a Workout Domain Synthetic Dataset</a></li><li><a href="https://alfarruggia.medium.com/1dc0acb0115d"><strong>Token Classification Fine-tuning with a Synthenic Dataset</strong></a></li><li><a href="https://alfarruggia.medium.com/62cc2cba1a37">Expo ReactNative App for Token Classification</a></li></ul><h3>Token Classification Task</h3><p>Token classification involves labeling individual tokens within a sentence, with Named Entity Recognition (NER) standing out as a prevalent task within token classification. NER aims to assign labels to entities in a sentence, such as identifying persons, locations, or organizations. You need to install the following Python packages to proceed.</p><pre>pip install transformers datasets evaluate seqeval</pre><h3>Load WMOUT dataset and Distilbert model</h3><p>Let’s load the WOUT dataset from the Huggingface Hub.</p><pre>from datasets import load_dataset<br>from transformers import AutoTokenizer<br><br>wmout = load_dataset(&#39;alfarruggia/wmout&#39;)<br>tokenizer = AutoTokenizer.from_pretrained(&quot;distilbert-base-uncased&quot;)</pre><h3>Dataset Alignment</h3><p>The Synthetic Dataset is not compatible with the <em>DistilBERT</em> network and needs to be tokenized properly. After tokenization, we need to align the dataset tags because the transformer tokenizer introduces some variations, such as the special tokens <em>[CLS]</em> and <em>[SEP]</em>, which may not match the input and labels. To fix this, I have used the code suggested by the HuggingFace post.</p><pre># https://huggingface.co/docs/transformers/en/tasks/token_classification<br>def tokenize_and_align_labels(examples):<br>    tokenized_inputs = tokenizer(examples[&quot;tokens&quot;], truncation=True, is_split_into_words=True)<br><br>    labels = []<br>    for i, label in enumerate(examples[f&quot;ner_tags&quot;]):<br>        word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.<br>        previous_word_idx = None<br>        label_ids = []<br>        for word_idx in word_ids:  # Set the special tokens to -100.<br>            if word_idx is None:<br>                label_ids.append(-100)<br>            elif word_idx != previous_word_idx:  # Only label the first token of a given word.<br>                label_ids.append(label[word_idx])<br>            else:<br>                label_ids.append(-100)<br>            previous_word_idx = word_idx<br>        labels.append(label_ids)<br><br>    tokenized_inputs[&quot;labels&quot;] = labels<br>    return tokenized_inputs<br><br>tokenized_mout = wmout.map(tokenize_and_align_labels, batched=True)</pre><h3>Fine-tuning and Evaluation</h3><p>After the fine-tuning process using the Huggingface Trainer, the resulting model is saved within the ./model_output directory. This model will subsequently be imported into the React Native Application in the nex Medium post.</p><pre>[...]<br>trainer = Trainer(<br>    model=model,<br>    args=training_args,<br>    train_dataset=tokenized_mout[&quot;train&quot;],<br>    eval_dataset=tokenized_mout[&quot;test&quot;],<br>    tokenizer=tokenizer,<br>    data_collator=data_collator,<br>    compute_metrics=compute_metrics,<br>)<br><br>trainer.train()<br>trainer.save_model(&#39;./model_output&#39;)</pre><p>The model has been trained for 10 epochs with a learning rate of 2e-5 and a batch size of 16. The evaluation metrics, including Eval Loss, Eval Recall, Eval F1 Score, and Eval Accuracy, were measured. The graph below depicts the trend of these metrics over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*blhpXcGdAg9BFKYb6tSr8g.png" /></figure><p>Taking into account these metrics collectively, it seems that the ML model exhibits a satisfactory performance within the scope of our objectives.</p><p>In the <a href="https://alfarruggia.medium.com/token-classification-in-react-62cc2cba1a37">upcoming post</a>, we’ll explore how to leverage our refined DistilBERT model within a React Native app.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1dc0acb0115d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to create a Workout Domain Synthetic Dataset]]></title>
            <link>https://alfarruggia.medium.com/how-to-create-a-workout-domain-synthetic-dataset-ec1140d75085?source=rss-a981c1de8866------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec1140d75085</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[datasets-for-ml]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Alfonso Farruggia]]></dc:creator>
            <pubDate>Mon, 11 Mar 2024 09:19:06 GMT</pubDate>
            <atom:updated>2024-03-13T13:09:28.685Z</atom:updated>
            <content:encoded><![CDATA[<p>In this serie we are going to <strong>fine-tuning a Token Classifier Model </strong>trained using a <strong>Synthetic dataset for the Workout Domain</strong>. Dataset has been generated by AI and embedded in a ReactNative Mobile Application.</p><h3><strong>Serie Table of Contents</strong></h3><ul><li><a href="https://medium.com/p/ec1140d75085"><strong>How to create a Workout Domain Synthetic Dataset</strong></a></li><li><a href="https://alfarruggia.medium.com/1dc0acb0115d">Token Classification Fine-tuning with a Synthenic Dataset</a></li><li><a href="https://alfarruggia.medium.com/62cc2cba1a37">Expo ReactNative App for Token Classification</a></li></ul><h3>Synthetic Dataset Generation</h3><h4>Synthetic Data Generation</h4><p>The synthetic dataset was generated using ChatGPT 3.5. To validate the workflow, I employed four entities: <strong>Workout, Frequency, Duration, and Number</strong>. ChatGPT was prompted with a concise list of variations to facilitate dataset creation.</p><pre>- Workout=[ Plank, Pushup, Lunges, Burpees, Squats, Walking lunges, Frog squats, Jumping jacks, Knee push-ups, Mountain Climbers, Regular lunges, Bodyweight squat, Box jumps, Dips, Glute bridge, Jump Rope, Pike pushups, Butterfly sit up, Shoulder taps, High knees running in place, Plank with arm lift, Side plank]<br><br>- Frequency=[ Never, Seldom, Always Every hour, Everyday, Often Constantly, Ever, Eventually, Daily, Frequently, Hourly, Yearly, Generally, Monthly Occasionally, Regularly, Sometimes Rarely, Usually, Normally Hardly ever, Scarcely, Now,  Then, Every Day, Every Month, Every Year, Every Minute]<br><br>- Duration=[minute, hour, day, month, year]<br><br>- Number=[1,2,3,4,5,6,7,8,9,10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150]</pre><p>The command was initiated to generate approximately 800 unique dataset entries. Each entry has been tagged by AI according to the specified entities. Below is an example of a dataset entry:</p><pre>Achieving better flexibility requires ((Number) [ten]) ((Workout) [Bodyweight squat]) sessions ((Frequency) [every day]) for the next ((Duration) [week]).</pre><p>After a text processing task, the dataset has been spit in 3 json file for train, test, and validation tasks. An example of train.json file look likes this:</p><pre>{<br>&quot;id&quot;: 620, <br>&quot;ner_tags&quot;: [0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 1, 2, 4, 3, 0, 6, 7], <br>&quot;tokens&quot;: [&quot;Improving&quot;, &quot;hip&quot;, &quot;mobility&quot;, &quot;and&quot;, &quot;flexibility&quot;, &quot;,&quot;, &quot;I&#39;ll&quot;, &quot;engage&quot;, &quot;in&quot;, &quot;45&quot;, &quot;Hip&quot;, &quot;Circles&quot;, &quot;twice&quot;, &quot;weekly&quot;, &quot;for&quot;, &quot;15&quot;, &quot;minutes&quot;]<br>}<br></pre><h4>Upload Dataset to Huggingface Hub</h4><p>The dataset has been divided into training, testing, and validation sets. It has been uploaded to the Hugging Face platform. For further details on the dataset, please refer to thie <a href="https://huggingface.co/datasets/alfarruggia/wmout"><strong>Dataset Hugging Face Page</strong></a>. The dataset entries has been tagged following the <a href="https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)"><strong>IOB Tagging Format</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NX84WQOwvWIsJ_QUARC00g.png" /></figure><h4>Load and Test Dataset</h4><p>First you need to install <strong>datasets</strong> python library with following command.</p><pre>pip install datasets</pre><p>Instead, to load the dataset in your python code you need to run the code below:</p><pre>from datasets import load_dataset<br>wmout = load_dataset(&#39;alfarruggia/wmout&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cx51eKzOr7du3NZUb7w8sg.png" /></figure><p>In the next step, we are going to do a <a href="https://alfarruggia.medium.com/token-classification-fine-tuning-using-synthenic-dataset-1dc0acb0115d">Fine-tuning of Token Classification Task using this Synthenic Dataset</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec1140d75085" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Explore Grocery Transactions using a Knowledge Graph]]></title>
            <link>https://alfarruggia.medium.com/explore-grocery-transactions-using-a-knowledge-graph-db04bf013dde?source=rss-a981c1de8866------2</link>
            <guid isPermaLink="false">https://medium.com/p/db04bf013dde</guid>
            <category><![CDATA[knowledge-base]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[grocery-shopping]]></category>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[automatic-extraction]]></category>
            <dc:creator><![CDATA[Alfonso Farruggia]]></dc:creator>
            <pubDate>Fri, 03 Feb 2023 16:01:29 GMT</pubDate>
            <atom:updated>2023-02-03T17:19:29.317Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How to build a knowledge graph from the well-known Groceries Dataset and explore it for getting insights. The code is in python programming language and exploits the pandas, and netwkorx libraries.</em></p><p>We are going to implement it in three steps:</p><ol><li><em>Dataset manipulation.</em> The dataset is composed of a list of transactions, in this first step, we re-build the receipts grouping the products by member and date.</li><li><em>Building the Graph.</em> In this step, we are going to extract nodes and edges in order to create and plot the graph.</li><li><em>Exploring the graph.</em> As the last step, we are going to interrogate the graph in order to discover a combination of products to create a promotion.</li></ol><h4>Dataset manipulation</h4><p>We are going to process the transactions in the popular <a href="https://www.google.com/search?q=groceries+dataset+kaggle&amp;rlz=1C5CHFA_enGB946GB946&amp;sxsrf=AJOqlzVvq6MwZrPoeRToBc84Ww3qvhBo5Q%3A1675423757505&amp;ei=DfDcY-nIHoahgQbegru4Bg&amp;oq=groceries+dataset+ka&amp;gs_lcp=Cgxnd3Mtd2l6LXNlcnAQAxgAMgUIABCABDIFCAAQhgM6CggAEEcQ1gQQsAM6BAgAEEM6BggAEBYQHjoKCAAQFhAeEA8QCkoECEEYAEoECEYYAFCLA1iVC2D7E2gBcAF4AIABPogBtgGSAQEzmAEAoAEByAEIwAEB&amp;sclient=gws-wiz-serp"><strong>Groceries Dataset</strong></a><strong>. </strong>The first step is re-build the recepits assuming that one user had one purchase in a day. To do this, we create the <em>Recept_id</em> concatenating the <em>Member_number</em> and the <em>Data.</em></p><pre>groceries = pd.read_csv(&#39;Groceries_dataset.csv&#39;)<br>groceries[&#39;Receipt_id&#39;] = [f&quot;{d}_{i}&quot; for d,i in zip(groceries[&#39;Date&#39;], groceries[&#39;Member_number&#39;])]<br><br># To check a recepit we can run the code below.<br>receipts = groceries.sort_values(by=[&#39;Date&#39;]).groupby(by=[&#39;Receipt_id&#39;])<br>r_id = list(receipts.groups.keys())[0]<br>print(receipts.get_group(r_id))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*Qiwi-7uwb8XhYiZv-iQlXQ.png" /></figure><h4>Build the Graph</h4><p>From the grocery list, we are going to extract the nodes for the network: items, members, and receipts.</p><pre># list product nodes<br>items_unique = groceries[&#39;itemDescription&#39;].unique()<br>node_items = [(x, {&#39;type&#39;:&#39;item&#39;}) for x in items_unique]<br><br># list member nodes<br>members_unique = groceries[&#39;Member_number&#39;].unique()<br>node_members = [(x, {&#39;type&#39;:&#39;member&#39;}) for x in members_unique]<br><br># list member nodes<br>receipt_id_unique = groceries[&#39;Receipt_id&#39;].unique()<br>node_receipts = [(x, {&#39;type&#39;:&#39;receipt&#39;}) for x in receipt_id_unique]</pre><p>Now, we create the relations between these nodes. We consider two relations:</p><ol><li>A member has one/more receipts. We call this relation has_purchased.</li><li>A receipt has one/more item. We call this relation has_item.</li></ol><p>The code below generates the relations from the groceries Dataframe. Each relation is a Tuple composed of three values, the first two are the nodes, and the last is the relationship between these nodes.</p><pre># Create Relations<br>relations_has_item = []<br>relations_has_purchased = []<br><br>for index, row in groceries.iterrows():<br>    relations_has_purchased.append((row[&#39;Member_number&#39;], row[&#39;Receipt_id&#39;], {&#39;relation&#39;:&#39;has_purchased&#39;}))<br>    relations_has_item.append((row[&#39;Receipt_id&#39;], row[&#39;itemDescription&#39;], {&#39;relation&#39;:&#39;has_item&#39;}))<br><br>relations = relations_has_item + relations_has_purchased</pre><p>At this point, we are ready to build the graph. First, we create an empty MultiGraph, then we populate it with nodes and relations.</p><pre># create an empty Graph<br>G = nx.MultiGraph()<br><br># populate the Graph with nodes and relations<br>G.add_nodes_from(node_items)<br>G.add_nodes_from(node_members)<br>G.add_nodes_from(node_receipts)<br>G.add_edges_from(relations)</pre><p>If we plot this graph taking into account only the first 10 entries in the groceries Dataframe, it looks like the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*50lht2qZXKyQmsdbaXKriQ.png" /></figure><p>Where the yellow nodes are the members, the green ones are the products and the red ones are the receipts. We can see, for example, that user <em>1808</em> had the purchase <em>21–07–2015</em> buying <em>whole milk</em>.</p><pre># Plot the graph<br>plt.rcParams[&quot;figure.figsize&quot;] = [10, 10]<br>plt.rcParams[&quot;figure.autolayout&quot;] = True<br><br>pos = nx.circular_layout(G)<br><br>nx.draw_networkx(G, pos=pos, nodelist=[x[0] for x in node_items] ,node_color=&#39;g&#39;, node_size=2000)<br>nx.draw_networkx(G, pos=pos, nodelist=[x[0] for x in node_members] ,node_color=&#39;y&#39;, node_size=2000)<br>nx.draw_networkx(G, pos=pos, nodelist=[x[0] for x in node_receipts], node_color=&#39;r&#39;, node_size=2000)<br>plt.show()</pre><h3>Explore the Graph</h3><p>Now, we are going to interrogate the graph looking for insights. We want to create a promotion for a product we have in stock but is not frequently sold.</p><p>To do this, we proceed to compute the degree_centrality of the graph, this metric gives more importance to a node with more connections, then we act in these three steps:</p><p><strong><em>a) Select a less important product</em></strong></p><p>Let’s see what are the 10 less important products according to the degree_centrality. First, we compute the degree_centrality for the network and filter the output only for the node items<em>, </em>then, we sort the values of degree_centrality in ascending mode and print the first 10. At this point, let’s consider that we are interested to do a promotion for the product whisky .</p><pre># compute the degree_centrality measure<br>g_centrality = nx.degree_centrality(G)<br><br># filter node taking only the items<br>items_centrality = {k: v for k, v in g_centrality.items() if k in items_unique}<br><br># sort the degree_centrality in ascending mode<br>items_centrality = sorted(items_centrality.items(), key=lambda x: x[1], reverse=False)<br><br># print products<br>the_10_less_important_products=[x[0] for x in items_centrality[0: 10]]<br><br>print(the_10_less_important_products)<br># [&#39;kitchen utensil&#39;, &#39;preservation products&#39;, &#39;baby cosmetics&#39;, <br># &#39;bags&#39;, &#39;frozen chicken&#39;, &#39;rubbing alcohol&#39;, &#39;make up remover&#39;, <br># &#39;toilet cleaner&#39;, &#39;salad dressing&#39;, &#39;whisky&#39;]</pre><p><strong><em>b) Find the receipts connected to the product in (a)</em></strong></p><p>Now, we are going to analyse what are the receipts where the whisky has been sold with. In this case, we explore the receipts where this product has been sold with. To do this, we extract the neighbor nodes for the node whisky. We found 8 receipts.</p><pre>receipts = list(G.neighbors(&#39;whisky&#39;))<br>print(receipts)<br># [&#39;25-03-2015_1654&#39;, &#39;31-08-2015_1090&#39;, &#39;03-09-2015_2152&#39;, &#39;25-12-2015_2825&#39;, &#39;15-01-2015_2232&#39;, &#39;05-04-2014_3523&#39;, &#39;01-10-2014_2138&#39;, &#39;29-05-2014_4261&#39;]</pre><p><strong><em>c) Find the most important product among those in (b)</em></strong></p><p>As the last step, we need to analyse the products in the list of receipts that we got before, then we compute the degree_centrality only for those products. In this case, we sort the degree_centrality values in descending mode in order to get the most important product at the top of the list. Without any surprise, we found that the most important product sold with the whisky is the soda. Now, as a shop manager, you can do your evaluation and activate a promotion combining that two products.</p><pre>products = []<br>for x in receipts:<br>    products.extend(list(G.neighbors(x))[0:-1])<br>    <br>target_product_centrality = {k: v for k, v in g_centrality.items() if k in products}<br>items_centrality = sorted(target_product_centrality.items(), key=lambda x: x[1], reverse=True)<br>products_for_promotion = items_centrality[0][0]<br>print(products_for_promotion)<br># soda</pre><h3>Conclusion</h3><p>This tutorial aims to show you an alternative way how to exploit a Knowledge Graph. The problem we solved can be faced in several simplest ways, but in this exercise, we have to show how to exploit the potentiality of Graph with the nextworkx python library.</p><p>Links</p><ul><li><a href="https://networkx.org/">Networkx</a></li><li><a href="https://github.com/Arfius/mymedium/tree/master/explore-grocery-transactions-using-a-knowledge-graph">Source code</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db04bf013dde" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Apache-Kafka simple story]]></title>
            <link>https://alfarruggia.medium.com/the-apache-kakfa-simple-story-6ac736580dd6?source=rss-a981c1de8866------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ac736580dd6</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[microservice-architecture]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Alfonso Farruggia]]></dc:creator>
            <pubDate>Fri, 09 Dec 2022 13:58:00 GMT</pubDate>
            <atom:updated>2023-01-02T15:53:21.890Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A copy-and-paste guide on how the Apache-Kafka event streaming platform for a high-performance system works. We are going to use the python-kafka package and docker to run the Apache-Kafka ecosystem.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yOo2f-Fu2zjgk1Y5eAoq_g.png" /><figcaption><em>This image has been generated by DALL-E using the following input text: A futuristic image for a post about Apache Kafka for beginner users. Apache Kafka is the event streaming platform for high-performance system.</em></figcaption></figure><h4>Real-life analogy</h4><p><em>Apache-Kafka is your h24 concierge. Let’s you (Consumer) are waiting for a parcel. The postman (Producer) will not deliver it to your hand, but the parcel will be collected by the concierge (Apache-Kafka) who will dispatch it to you asap.</em></p><h4>Why Apache-Kafka is so important</h4><p><em>The key point is the Consumer could be busy with some tasks and receiving the message could be delayed creating, for example, network congestion or timeout exception.</em></p><p><em>From my point of view, who is approaching Apache-Kafka for the first time, one of the most important lessons to get is that: Apache-Kafka decouples Producer and Consumer. Decoupling, the producer will be able to stream high-frequency data and the consumer will receive them when resources are free to use.</em></p><p><em>In the middle, between Producer and Consumer, Apache-Kafka plays its role. The secret is that Apache-Kafka has an internal queue mechanism that facilitates data dispatching.</em></p><h4>How to translate that in python</h4><ul><li>Let’s install the python package. For this exercise, we are going to install python-kafka. We also need docker composeto run the Apache-Kafka platform.</li></ul><pre>% pip install kafka-python</pre><ul><li>Save the file below as docker-compose.yml . We will run it later to set up the cluster of docker containers.</li></ul><pre># This docker-compose file is ready to interact with external components.<br># The Producer and Consumer can be out of the docker cluster.<br><br>version: &#39;3.8&#39;<br>services:<br>  zookeeper:<br>    image: &#39;bitnami/zookeeper:latest&#39;<br>    ports:<br>      - &#39;2181:2181&#39;<br>    environment:<br>      - ALLOW_ANONYMOUS_LOGIN=yes<br><br>  kafka:<br>    image: &#39;bitnami/kafka:latest&#39;<br>    ports:<br>      - &#39;9093:9093&#39;<br>    environment:<br>      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181<br>      - ALLOW_PLAINTEXT_LISTENER=yes<br>      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT<br>      - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://0.0.0.0:9093<br>      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9093<br>      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT<br><br>    volumes:<br>      -  /var/bitnami/kafka<br>    depends_on:<br>      - zookeeper</pre><ul><li>Before sending or receiving a message, the Producer and the Consumer have to decide on a common place where store and receive the message. To deal with this, Apache-Kafka introduces the concept of the Topic: <em>a Topic serves as a place where the messages are temporarily stored before the Consumer read and processes them.</em></li><li>Let’s create the python files: administrator.py, producer.py and consumer.py. Theadministrator.py will serve to create the topic. The producer.py and consumer.py will be run to send and receive a message respectively.</li></ul><pre># administrator.py<br># create new topic if it not exists<br><br># import the KafkaAdminClient and NewTopic classes<br>from kafka.admin import KafkaAdminClient, NewTopic<br><br># instance a KafkaAdminClient <br>administrator = KafkaAdminClient(<br>        bootstrap_servers=&#39;localhost:9093&#39;,<br>        api_version=(0, 10, 1))<br><br># get topics from kafka ecosystem<br>existing_topics = administrator.list_topics()<br><br># create the new topic if it is not already created<br>if &#39;example_topic&#39; not in existing_topics:<br>    topic = NewTopic(name=&#39;example_topic&#39;,<br>                     num_partitions=1,<br>                     replication_factor=1)<br><br>    administrator.create_topics(new_topics=[topic])</pre><pre># producer.py<br># send a message<br><br># import the KafkaProducer class<br>from kafka import KafkaProducer<br><br># instance a KafkaProducer object<br>producer = KafkaProducer(bootstrap_servers=&#39;localhost:9093&#39;, <br>                         api_version=(0, 10, 1))<br><br># send the message to &#39;Hello, World!&#39; using the topic &#39;example_topic&#39;<br>producer.send(&#39;example_topic&#39;, b&#39;Hello, World!&#39;)<br>producer.flush()<br>print(&#39;message sent&#39;)</pre><pre># consumer.py<br># receive and print a message<br><br># import the KafkaConsumer class<br>from kafka import KafkaConsumer<br><br># instance a KafkaConsumer producer that listen for example_topic messages<br>consumer = KafkaConsumer(&#39;example_topic&#39;,<br>                          bootstrap_servers=&#39;localhost:9093&#39;,<br>                          api_version=(0, 10, 1))<br><br># consumer object is a python cursor<br># consumer waiting for new messages forever<br>for m in consumer:<br>    print(m)</pre><h4>Run the code in 4 steps</h4><p>After creating the files, this is how your folder should look like:</p><pre>├── administrator.py <br>├── consumer.py <br>├── docker-compose.yml<br>└── producer.py</pre><ol><li>Run the Apache-Kafka cluster. Open a shell and run % docker compose up . Let’s wait until the containers are up and running.</li><li>Create the topic % python administrator.py. As we explained before, the message doesn’t go directly from the producer to the consumer, but at first, it is received by Apache-Kafka and stored in the topic example_topic.</li><li>Make the Consumer listen for messages, run% python consumer.py</li><li>Send a message. Open a new shell and run % python producer.py</li></ol><h4>Conclusion</h4><p>This simple tutorial aims to introduce you to Apache-Kafka and how the Producer and Consumer work together. Apache-Kafka has many other parameters to be set, for example, we didn’t talk about how topics are spit in partitions and replica factors, but if you are interested to get more information please follow these links:</p><ul><li><a href="https://kafka.apache.org/intro">https://kafka.apache.org/intro</a></li><li><a href="https://kafka-python.readthedocs.io/en/master/#">https://kafka-python.readthedocs.io/</a></li><li><a href="https://hub.docker.com/r/bitnami/kafka">https://hub.docker.com/r/bitnami/kafka</a></li><li>👉 <a href="https://github.com/Arfius/mymedium">SOURCE CODE</a></li></ul><h4>Thanks!</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ac736580dd6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apijet CLI — Fast prototyping tool to deploy APIs]]></title>
            <link>https://alfarruggia.medium.com/apijet-cli-fast-prototyping-tool-to-deploy-apis-de089348c498?source=rss-a981c1de8866------2</link>
            <guid isPermaLink="false">https://medium.com/p/de089348c498</guid>
            <category><![CDATA[framework]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[fastapi]]></category>
            <dc:creator><![CDATA[Alfonso Farruggia]]></dc:creator>
            <pubDate>Mon, 19 Sep 2022 09:44:41 GMT</pubDate>
            <atom:updated>2022-09-19T09:44:41.757Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apijet CLI — Fast prototyping tool to deploy APIs</h3><p><a href="https://github.com/Arfius/apijet">Apijet</a> is a Python framework for building APIs via the command line. In this tutorial, we are going to create the backend for the ToDo app for managing Notes.</p><p><em>We are going to run a couple of commands from the terminal and update just one python file. You will not struggle with how to pass the data, query the database or validate the JSON, apijet will do all of this for you.</em></p><h4>Requirements:</h4><p><em>Install the apijet package.</em> I am going to use conda as the package manager , but feel free to use that you like. Please make sure you are using a python version that matches <em>python≥3.8+.</em></p><pre>$&gt; conda create --name todoapp python=3.9</pre><pre>$&gt; conda activate todoapp</pre><pre>$&gt; python -m pip install apijet</pre><h4>Step 1 — Create the ToDo project</h4><p>First, running the command create. <strong>Apijet</strong> generates the skeleton folders and framework files, and then you do cd into the <strong>ToDo</strong> folder.</p><pre>$&gt; apijet create --name ToDo --port 9090 --address 127.0.0.1<br>$&gt; cd ToDo</pre><h4>Step 3— Create the Note endpoint</h4><p>The endpoint command generates the files for the endpoint <strong>Note</strong>. With the parameter <strong>— database, apijet </strong>creates the support for the basic <strong>CRUD</strong> operations.</p><pre>$&gt; apijet endpoint --add Note --database</pre><p>If you run <strong>ls, </strong>you have the <em>apijet.json</em> configuration file and the <em>ToDo</em> folder — with the generated python code. The configuration file looks like this below and keeps the information to run the server, please you can update it if you prefer a different setup.</p><pre>{<br>    &quot;name&quot;: &quot;ToDo&quot;,<br>    &quot;port&quot;: 9090,<br>    &quot;address&quot;: &quot;127.0.0.1&quot;,<br>    &quot;workers&quot;: 1,<br>    &quot;mongo&quot;: {<br>        &quot;auth&quot;: false,<br>        &quot;address&quot;: &quot;127.0.0.1&quot;,<br>        &quot;port&quot;: 27017,<br>        &quot;username&quot;: &quot;&quot;,<br>        &quot;password&quot;: &quot;&quot;<br>    },<br>    &quot;endpoints&quot;: [<br>        &quot;Note&quot;<br>    ]<br>}</pre><h4>Step 4— Run the project</h4><p>Before running the server, we need a MongoDB instance running. You can launch it via docker by running the command below.</p><pre>$&gt; docker run -d -p 27017:27017 --name ToDoApp mongo:latest</pre><p><strong>Now, let’s run the server.</strong></p><pre>$&gt; python -m ToDo.app</pre><p>The endpoints can be reached by opening your browser at <a href="http://127.0.0.1/"><em>HTTP://</em></a><a href="http://127.0.0.1/"><em>127.0.0.1</em></a><em>:9090/docs</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T4pEgnXJQO7hAKx7elY-Eg.png" /><figcaption>Swagger</figcaption></figure><h4>Step 5 — Update Note Model to store the Note status</h4><p>Because we want to keep the status of a note — <em>done</em>, we need to update the data structure in the file <strong>./ToDo/router/Note.py</strong> and add the lines marked with the comments. <em>Line 8</em> adds a boolean attribute to the Note Class, this attribute store the status for the Note. Instead, <em>line 17</em> updates the swagger configuration for the interactive API documentation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/167ae32e90722b3ceed44dd24bfad5ae/href">https://medium.com/media/167ae32e90722b3ceed44dd24bfad5ae/href</a></iframe><h4>Conclusion</h4><p>In this example, we have generated the endpoint for the ToDo in a few steps. You didn’t work on how to pass the data, query the database or validate the JSON. If you want to find how apijet work, you can explore the files in the ToDo folder, the pattern used by apijet for code generating is described in the link below.</p><p><a href="https://github.com/Arfius/apijet">GitHub - Arfius/apijet: A command line tool to deploy RestApi</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de089348c498" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to generate a Knowledge Graph from plain text in few steps]]></title>
            <link>https://alfarruggia.medium.com/how-to-generate-a-knowledge-graph-from-plain-text-in-few-steps-42f3a639155f?source=rss-a981c1de8866------2</link>
            <guid isPermaLink="false">https://medium.com/p/42f3a639155f</guid>
            <category><![CDATA[automatic-extraction]]></category>
            <category><![CDATA[knowledge-base]]></category>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Alfonso Farruggia]]></dc:creator>
            <pubDate>Fri, 15 Jul 2022 09:40:16 GMT</pubDate>
            <atom:updated>2023-01-02T15:35:45.068Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A copy-and-paste guide to generate a Knowledge graph with spacy, textacy and netwkorx python libraries. The example below is processing just one sentence as plain text. you can replace it with the text you like.</em></p><h3>Requirements</h3><ul><li>Install <strong>spacy</strong>, <strong>textacy</strong> and <strong>netwkorx</strong> libraries</li></ul><pre>$&gt; pip install <strong>spacy</strong><br>$&gt; pip install <strong>netwkorx<br></strong>$&gt; pip install <strong>textacy</strong></pre><ul><li>Import the libraries in your project</li></ul><pre>import spacy<br>import textacy<br>import matplotlib.pylab as plt<br>import networkx as nx</pre><h3>Extract Triples</h3><p>There are different approaches in the literature to extract triples. In this tutorial, we are going to use the <strong>Subject-Verb-Object (SVO)</strong> approach.</p><ol><li><em>Load the spacy ‘en_core_web_s’ english pipeline and process the plain-text</em></li></ol><pre># Load spacy model<br>nlp = spacy.load(&#39;en_core_web_sm&#39;)</pre><pre># Plain text<br>text = &quot;I am going to extract SVO&quot;</pre><pre># Process plain text with spacy<br>doc = nlp(text)</pre><p><em>2. extract triples from spacy object</em></p><pre># Extract SVO list from spacy object<br>triples = list(textacy.extract.subject_verb_object_triples(doc))</pre><pre>print(triples)<br># [SVOTriple(subject=[I], verb=[am, going], object=[to, extract, SVO])]</pre><h3>Populate Graph</h3><p>Now, we are going to populate the graph. There are a few steps between us and our graph. First, we extract the nodes <em>Subject</em> and nodes <em>Objects</em>, and then the relation between these. We tag the relation as ‘<em>action</em>’. The second step is to create the Graph using nodes and relations, whereas the last step is to display the graph.</p><p>Let’s start:</p><ol><li><em>Generate node list and relations</em></li></ol><pre>nodes = []<br>relations = []</pre><pre># iterate over the triples<br>for triple in triples:</pre><pre>    # extract the Subject and Object from triple<br>    node_subject = &quot;_&quot;.join(map(str, triple.subject))<br>    node_object  = &quot;_&quot;.join(map(str, triple.object))</pre><pre>    nodes.append(node_subject)<br>    nodes.append(node_object)</pre><pre>    # extract the relation between S and O<br>    # add the attribute &#39;action&#39; to the relation<br>    relation = &quot;_&quot;.join(map(str, triple.verb))<br>    relations.append((node_subject,node_object,{&#39;action&#39;:relation}))</pre><pre># remove duplicate nodes<br>nodes = list(set(nodes))</pre><pre>print(nodes)<br># [&#39;to_extract_SVO&#39;, &#39;I&#39;]</pre><pre>print(relations)<br># [(&#39;I&#39;, &#39;to_extract_SVO&#39;, {&#39;action&#39;: &#39;am_going&#39;})]</pre><p>2. Create the graph</p><pre>G = nx.DiGraph()<br>G.add_nodes_from(nodes)<br>G.add_edges_from(relations)</pre><p>3. Show the graph</p><pre># extract the attribute &#39;action&#39; from edges<br>edge_attribute = nx.get_edge_attributes(G, &#39;action&#39;)<br>edges, weights = zip(*edge_attribute.items())</pre><pre># resize figure<br>plt.rcParams[&quot;figure.figsize&quot;] = [5, 2]<br>plt.rcParams[&quot;figure.autolayout&quot;] = True</pre><pre># set figure layout<br>pos = nx.circular_layout(G)</pre><pre># draw graph<br>nx.draw(G, pos, node_color=&#39;b&#39;, width=2, with_labels=True)</pre><pre># draw edge attributes<br>nx.draw_networkx_edge_labels(G, pos,edge_attribute, label_pos=0.75 )<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*cyFBbk-SIJzATb1kNdE-dg.png" /></figure><h3>Conclusion</h3><p>This simple tutorial aims to introduce you to Automatic knowledge extraction from documents. There are different ways to exploit this NLP approach, one usage is to aggregate statistics knowledge from large collections of documents or for the Question-Answering task.</p><p>👉 <a href="https://github.com/Arfius/mymedium">SOURCE CODE</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42f3a639155f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to link pytest to your Jira Board]]></title>
            <link>https://alfarruggia.medium.com/how-to-link-pytest-to-your-jira-board-906f37c464c1?source=rss-a981c1de8866------2</link>
            <guid isPermaLink="false">https://medium.com/p/906f37c464c1</guid>
            <category><![CDATA[pytest]]></category>
            <category><![CDATA[jira]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[report]]></category>
            <dc:creator><![CDATA[Alfonso Farruggia]]></dc:creator>
            <pubDate>Thu, 20 Jan 2022 23:23:16 GMT</pubDate>
            <atom:updated>2022-01-20T23:28:43.799Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>A simple guide on how to link your pytest to Jira task using pytest-html report.</em></h4><h3>Requirements</h3><ul><li>Make sure that you are using <strong>pytest</strong> and <strong>pytest-html</strong> packages</li></ul><pre>$&gt; pip install pytest pytest-html</pre><ul><li>The name of the unit test function must have the <strong>task id</strong> after the keyword <strong>test_. </strong><em>You can find the </em><strong><em>TASK_ID</em></strong><em> in the Breadcrumbs of the Jira web portal or in the last part of the URL. Please, replace the dash with the underscore.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*jZWJIyNy4vgyRs5eXSOwRQ.png" /></figure><pre>def test_<strong>NON_160</strong>_whatever_you_want():<br>...</pre><h3>Let’s do it.</h3><ol><li>Create the <strong>conftest.py</strong> file inside the test folder and import the pytest package. You can add also the information that you want to render in the report, such as the package version.</li></ol><pre><strong># conftest.py<br>from version import __version__<br>import pytest</strong></pre><p>2. Create the variable <strong>TITLE</strong> to personalize the report title, <strong>BASE_URL</strong> to point to your JIRA project and the <strong>JIRA_PREFIX</strong>. This last is the <strong>Project Key</strong> that you find in <strong>Jira board &gt; Project setting.</strong></p><pre># conftest.py<br>from version import __version__<br>import pytest</pre><pre><strong>TITLE = f&quot;Test report - MY PROJECT - v: {__version__}&quot;<br>BASE_URL = &quot;https://myjiraproject.atlassian.net/browse&quot;<br>JIRA_PREFIX = &quot;NON&quot;</strong></pre><p>3. Add the hook <strong>pytest_html_report_title </strong>to update the title</p><pre># conftest.py<br>from version import __version__<br>import pytest</pre><pre>TITLE = f&quot;Test report - MY PROJECT - v: {__version__}&quot;<br>BASE_URL = &quot;https://myjiraproject.atlassian.net/browse&quot;<br>JIRA_PREFIX = &quot;MJP&quot;</pre><pre><strong>def pytest_html_report_title(report):<br>    report.title = TITLE</strong></pre><p>4. Add the function <strong>__get_jira_id. </strong>This function extracts the Task Id from the unit test function name. Make sure that the <strong>JIRA_PREFIX</strong> is that right used by Jira.</p><pre># conftest.py<br>from version import __version__<br>import pytest</pre><pre>TITLE = f&quot;Test report - MY PROJECT - v: {__version__}&quot;<br>BASE_URL = &quot;https://myjiraproject.atlassian.net/browse&quot;<br>JIRA_PREFIX = &quot;MJP&quot;</pre><pre>def pytest_html_report_title(report):<br>    report.title = TITLE<br></pre><pre><strong>def __get_jira_id(_nodeid):<br>    try:<br>        _nodeid = _nodeid.split(&quot;::&quot;)[-1]<br>        jira_id = _nodeid.split(&quot;_&quot;)[1:3]<br>        if jira_id[0] == JIRA_PREFIX:<br>            return &quot;-&quot;.join(jira_id)<br>        else:<br>            return None<br>    except Exception as e:<br>        logger.debug(f&quot;Exception is occurred : {e}&quot;)<br>        return None</strong></pre><p>5. Add the hook <strong>pytest_runtest_makereport. </strong>This function permits to extract the <strong>Task Id </strong>from the <strong>report.nodeid, </strong>build the link as <strong>{BASE_URL}/{jira_key} </strong>and push the link as an extra parameter in the report<strong> pytest_html.extras.url(https://medium.com/f%E2%80%9D%7BBASE_URL%7D/%7Bjira_key%7D%E2%80%9D).</strong></p><pre># conftest.py<br>from version import __version__<br>import pytest</pre><pre>TITLE = f&quot;Test report - MY PROJECT - v: {__version__}&quot;<br>BASE_URL = &quot;https://myjiraproject.atlassian.net/browse&quot;<br>JIRA_PREFIX = &quot;MJP&quot;</pre><pre>def pytest_html_report_title(report):<br>    report.title = TITLE</pre><pre>def __get_jira_id(_nodeid):<br>    try:<br>        _nodeid = _nodeid.split(&quot;::&quot;)[-1]<br>        jira_id = _nodeid.split(&quot;_&quot;)[1:3]<br>        if jira_id[0] == JIRA_PREFIX:<br>            return &quot;-&quot;.join(jira_id)<br>        else:<br>            return None<br>    except Exception as e:<br>        return None</pre><pre><strong>@pytest.hookimpl(hookwrapper=True)<br>def pytest_runtest_makereport(item, call):<br>    pytest_html = item.config.pluginmanager.getplugin(&#39;html&#39;)<br>    outcome = yield<br>    report = outcome.get_result()<br>    extra = getattr(report, &#39;extra&#39;, [])<br>    if report.when == &#39;call&#39;:<br>        jira_key = __get_jira_id(report.nodeid)<br>        if jira_key is not None:<br>            extra.append (<br>             pytest_html.extras.url(https://medium.com/f&quot;%7BBASE_URL%7D/%7Bjira_key%7D&quot;)<br>            )<br>        report.extra = extra</strong></pre><p>6. Verify the output. The URL link will redirect you to the JIRA task.</p><pre>pytest --html=test-report.html --self-contained-html</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*QfHkvqswil1XsvwYuCV_Lw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*-nB83Lrbv0pZD3iqnDknKg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=906f37c464c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-Tuning of a Sentiment Analysis Task with Transformers-TensorFlow on Apple M1 Chip]]></title>
            <link>https://alfarruggia.medium.com/fine-tuning-transformers-of-sentiment-analysis-task-on-apple-m1-chip-a5adda8a3d87?source=rss-a981c1de8866------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5adda8a3d87</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[apple-m1-chip]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Alfonso Farruggia]]></dc:creator>
            <pubDate>Wed, 03 Nov 2021 01:22:18 GMT</pubDate>
            <atom:updated>2023-01-02T15:54:07.582Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A simple guide to fine-tuning a Transformers DistilBert Model using Tensorflow on Apple M1 Chip for a Sentiment Analysis Task.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oNbSEV5kzdMibk4B-KvOsQ.png" /><figcaption><em>This image has been generated by DALL-E using the following input text: </em>a futuristic propaganda poster for an article on English sentiment analysis natural language processing task</figcaption></figure><h4>Requirements:</h4><ul><li>If your Transformers library is working fine, skip the requirements.</li><li>You can install <strong>Transformers</strong> from the <a href="https://huggingface.co/transformers/installation.html">official website</a>.</li><li>If you are in trouble with <strong>Apple M1 Chip</strong>, you can follow my guide <a href="https://medium.com/p/e5d9afa35f5/edit?source=your_stories_page-------------------------------------">From Zero to Transformers on Apple M1 Chip</a>.</li><li>Google a bit and download the dataset “IMDB Dataset.csv”</li><li>👉 <a href="https://github.com/Arfius/mymedium">SOURCE CODE</a></li></ul><h3>⚠️ Force GPU device (macOS &lt;12)</h3><p>During the execution of the <strong>model.fit() </strong>method, I got my Tensorflow crashed because of this error:</p><pre>[MPSGraph adamUpdateWithLearningRateTensor:beta1Tensor:beta2Tensor:epsilonTensor:beta1PowerTensor:beta2PowerTensor:valuesTensor:momentumTensor:velocityTensor:maximumVelocityTensor:gradientTensor:name:]: unrecognized selector sent to instance 0x2e3025230</pre><p>After investigation, I found this solution that works for <strong>TF2.6 </strong>and forces the <strong>GPU</strong> as the only device available to run the network</p><pre>import tensorflow as tf<br>physical_devices = tf.config.list_physical_devices(&#39;GPU&#39;)<br>try:<br>    tf.config.set_visible_devices(physical_devices[1:], &#39;GPU&#39;)<br>    logical_devices = tf.config.list_logical_devices(&#39;GPU&#39;)<br>    assert len(logical_devices) == len(physical_devices) - 1<br>except:<br>   pass</pre><h3>Data preparation</h3><p>Read the CSV file and apply a lambda function to convert labels from text to numbers. Label <strong>positive </strong>is<strong> 1 </strong>and label <strong>negative </strong>is<strong> 0. </strong>The dataset will be split into training, validation, and testing, according to the percentages of 70, 15, and 15.</p><pre>import pandas as pd<br>from sklearn.model_selection import train_test_split<br>dataset = pd.read_csv(&quot;./IMDB Dataset.csv&quot;)</pre><pre>def setLabel(x):<br>    if(x[&#39;sentiment&#39;] == &#39;positive&#39; ):<br>        x[&#39;label&#39;] = 1<br>        return x<br>    elif(x[&#39;sentiment&#39;] == &#39;negative&#39; ):<br>        x[&#39;label&#39;] = 0<br>        return x<br>    return None</pre><pre>dataset = dataset.apply(lambda x : setLabel(x) , axis=1)<br>dataset = dataset.dropna()</pre><pre>train_texts  = dataset[&#39;review&#39;].values.tolist()<br>train_labels = dataset[&#39;label&#39;].values.tolist()</pre><pre>train_texts, test_val_texts, train_labels, test_val_labels = train_test_split(train_texts, train_labels, test_size=.3)</pre><pre>test_texts, val_texts, test_labels, val_labels = train_test_split(test_val_texts, test_val_labels, test_size=.5)</pre><h3><strong>Data Tokenization</strong></h3><p>For this <em>copy-paste</em> tutorial, the <strong>distilbert-base-uncased</strong> has been used, so the <strong>DistilBertTokenizerFast</strong> is used to tokenize the dataset, the output is in numpy form.</p><pre>from transformers import DistilBertTokenizerFast<br>tokenizer = DistilBertTokenizerFast.from_pretrained(&#39;distilbert-base-uncased&#39;)<br>train_encodings = tokenizer(train_texts, truncation=True, padding=True, return_tensors=&#39;np&#39;).data<br>val_encodings = tokenizer(val_texts, truncation=True, padding=True, return_tensors=&#39;np&#39;).data<br>test_encodings = tokenizer(test_texts, truncation=True, padding=True, return_tensors=&#39;np&#39;).data</pre><h3><strong>Setup the model</strong></h3><p>Load the pre-trained model <strong>distilbert-base-uncased </strong>in Tensorflow form using <strong>TFAutoModelForSequenceClassification</strong>. We passed the basic value as parameters: name of the pre-trained model, number of labels, and the tag of each label.</p><pre>from transformers import TFAutoModelForSequenceClassification, DistilBertConfig<br>from tensorflow.keras.layers import *<br>from tensorflow.keras import models<br>model = TFAutoModelForSequenceClassification.from_pretrained(&#39;distilbert-base-uncased&#39;, num_labels=3, id2label={0: &#39;negative&#39;, 1: &#39;neutral&#39;,2:&#39;positive&#39;})<br>optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)<br>model.compile(optimizer=optimizer, loss=model.hf_compute_loss)<br>model.summary()</pre><h3>Model Summary</h3><p>This is how the network looks like.</p><pre>model.summary()<br>_________________________________________________________________<br>Model: &quot;tf_distil_bert_for_sequence_classification&quot;<br>_________________________________________________________________<br>Layer (type)                 Output Shape              Param #   <br>=================================================================<br>distilbert (TFDistilBertMain multiple                  66362880  <br>_________________________________________________________________<br>pre_classifier (Dense)       multiple                  590592    <br>_________________________________________________________________<br>classifier (Dense)           multiple                  1538      <br>_________________________________________________________________<br>dropout_19 (Dropout)         multiple                  0         <br>=================================================================<br>Total params: 66,955,010<br>Trainable params: 66,955,010<br>Non-trainable params: 0</pre><h3>Fine-tuning and Evaluation</h3><p>The model has been trained using <strong>5000 positive</strong> samples and <strong>5000 negative</strong> samples.</p><pre>import numpy as np<br>model.fit(train_encodings, np.array(train_labels) validation_data=(val_encodings, np.array(val_labels)), epochs=3, batch_size=8)</pre><pre>Epoch 1/3<br>219/219 [==============================] - 6441s 29s/step - loss: 0.3129 <br>Epoch 2/3<br>219/219 [==============================] - 6345s 29s/step - loss: 0.1437 <br>Epoch 3/3<br>219/219 [==============================] - 6464s 30s/step - loss: 0.0713 </pre><pre>model.evaluate(test_encodings, np.array(test_labels))</pre><h3>Export Model and Tokenizer</h3><p>Following lines of codes to export the model and the tokenizer in Tensorflow form. The folder <strong>IMDB-distilbert-base-uncased</strong> will be created in your machine with h5 and JSON files.</p><pre>model.save_pretrained(&quot;IMDB-distilbert-base-uncased&quot;)<br>tokenizer.save_pretrained(&quot;IMDB-distilbert-base-uncased&quot;)</pre><pre>+IMDB-distilbert-base-uncased<br>|-config.json<br>|-special_tokens_map.json<br>|-tf_model.h5<br>|-tokenizer.json<br>|-tokenizer_config.json<br>|-vocab.txt</pre><h3>Load model and make a prediction</h3><p>We can exploit the Transformer pipeline to make the prediction. The pipeline for text classification is customized with the model and the tokenizer exported before.</p><pre>from transformers import pipeline<br>pipe = pipeline(&quot;text-classification&quot;, model=&quot;./IMDB-distilbert-base-uncased&quot;, tokenizer=&quot;./IMDB-distilbert-base-uncased&quot;)</pre><pre>pipe(&quot;i love this actor&quot;)<br>[{&#39;label&#39;: &#39;positive&#39;, &#39;score&#39;: 0.9965764880180359}]</pre><pre>pipe(&quot;i don&#39;t like this movie&quot;)<br>[{&#39;label&#39;: &#39;negative&#39;, &#39;score&#39;: 0.8704461455345154}]</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5adda8a3d87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Zero to Transformers on Apple M1 Chip]]></title>
            <link>https://alfarruggia.medium.com/from-zero-to-transformers-on-apple-m1-chip-e5d9afa35f5?source=rss-a981c1de8866------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5d9afa35f5</guid>
            <category><![CDATA[m1-chip]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[conda]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[installation]]></category>
            <dc:creator><![CDATA[Alfonso Farruggia]]></dc:creator>
            <pubDate>Sat, 02 Oct 2021 17:17:14 GMT</pubDate>
            <atom:updated>2021-10-02T17:17:14.900Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A copy-and-paste guide to install Tensorflow and Transformers on Apple M1 Chip.</em></p><h3><strong>Requirements</strong></h3><ol><li>Run a <strong>Terminal.app</strong>, make sure it has<strong> Rosetta2 disabled</strong>. You can check it from <strong>GetInfo</strong> option of <strong>Terminal.app</strong>.</li><li>Install <strong>XCode</strong> and <strong>Command Line Tools Package. </strong>From<strong> Terminal run:</strong></li></ol><pre>xcode-select --install</pre><p>3. Install <strong>Miniforge</strong></p><p>See instructions here : <a href="https://github.com/conda-forge/miniforge"><em>https://github.com/conda-forge/miniforge</em></a></p><h3>Prepare conda Environment and activate it</h3><pre>conda create -name tf-conda-hugging python=3.9<br>conda activate tf-conda-hugging</pre><h3>Install Tensorflow package</h3><pre>conda install -c apple tensorflow-deps<br>python -m pip install tensorflow-macos<br>python -m pip install tensorflow-metal</pre><h4>Verify Tensorflow package Installation</h4><p>Run <strong>$&gt;python </strong>command on your terminal, make sure that <strong>tf-conda-hugging</strong> envirnoment is enabled.</p><pre>import tensorflow as tf<br>tf.__version__ # print the current version<br>&#39;2.6.0&#39;</pre><pre>tf.config.list_physical_devices() # print available physical devices<br>[PhysicalDevice(name=&#39;/physical_device:CPU:0&#39;, device_type=&#39;CPU&#39;), PhysicalDevice(name=&#39;/physical_device:GPU:0&#39;, device_type=&#39;GPU&#39;)]</pre><h3><strong>Install Transformers package</strong></h3><pre>conda install -c huggingface transformers -y</pre><h4>Verify Transformers package Installation</h4><p>Run <strong>$&gt;python </strong>command on your terminal, make sure that <strong>tf-conda-hugging</strong> envirnoment is enabled.</p><pre>import transformers as tr<br>tr.__version__ # print the current version<br>&#39;4.11.2&#39;</pre><h3><strong>Install common packages</strong></h3><pre>conda install -c conda-forge scikit-learn -y<br>conda install -c conda-forge pandas -y<br>conda install -c conda-forge jupyterlab -y</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5d9afa35f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>