<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gabriel Ribeiro on Medium]]></title>
        <description><![CDATA[Stories by Gabriel Ribeiro on Medium]]></description>
        <link>https://medium.com/@gabrielhribeiro?source=rss-cd385a51120f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*PGEjs1AnHRZrOaFP5CeoJQ.jpeg</url>
            <title>Stories by Gabriel Ribeiro on Medium</title>
            <link>https://medium.com/@gabrielhribeiro?source=rss-cd385a51120f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 22:41:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gabrielhribeiro/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Security-in-the-Loop: Why Agentic AI Needs Human Judgment From Start to Finish]]></title>
            <link>https://gabrielhribeiro.medium.com/security-in-the-loop-why-agentic-ai-needs-human-judgment-from-start-to-finish-7606d7d355a1?source=rss-cd385a51120f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7606d7d355a1</guid>
            <dc:creator><![CDATA[Gabriel Ribeiro]]></dc:creator>
            <pubDate>Fri, 13 Nov 2026 07:17:28 GMT</pubDate>
            <atom:updated>2026-11-13T07:17:28.309Z</atom:updated>
            <content:encoded><![CDATA[<p>While reading a paper unrelated to security, I came across a term that stayed with me: clinical<strong>-in-the-Loop</strong>.</p><p>The paper, <em>“Clinician-in-the-Loop Smart Home System to Detect Urinary Tract Infection Flare-Ups via Uncertainty-Aware Decision Support,”</em> describes a system that does more than flag a possible infection. It extracts interpretable behavioral markers, quantifies prediction uncertainty, and explicitly says <strong>“I don’t know”</strong> when confidence is too low, allowing clinicians to make safer decisions.acm+1</p><p>That idea felt immediately familiar.</p><p>In cybersecurity, we are also dealing with a living, adaptive system under pressure. Threats mutate. Signals are incomplete. Legitimate activity can look malicious. Malicious activity can look legitimate. And a wrong automated decision — blocking a critical business process, isolating the wrong endpoint, or escalating an incident incorrectly — can have real operational consequences.</p><h3>Security-in-the-Loop (SITL)</h3><p><strong>Security-in-the-Loop</strong> means embedding security expertise, context, and human judgment throughout the entire agentic process: from <strong>architecture and process design, through detection and investigation, to response, recovery, and learning.</strong></p><p>It is not simply “<strong>human approval before execution.</strong>”</p><p><strong>“It is a continuous collaboration between AI agents and security professionals</strong>.”</p><p>The clinical analogy is powerful because healthcare and cybersecurity share a fundamental challenge: both operate in environments where uncertainty can be dangerous. In the clinical system described by Ugwu et al., the model does not only classify a case as “UTI” or “No UTI.” It can abstain and return <strong>“I don’t know”</strong> when uncertainty is high, giving the clinician room to investigate rather than act blindly.<a href="https://arxiv.org/html/2511.18334v1">arxiv</a></p><p>Security systems should behave the same way.</p><p>An agent should not only say:</p><blockquote><em>“This is malicious.”</em></blockquote><p>It should be able to say:</p><blockquote><em>“This may be malicious. Here is the evidence, here is what I am uncertain about, and here are the possible consequences of acting now.”</em></blockquote><h3>Why Security Needs More Than Detection</h3><p>Traditional security automation often follows a simple pattern:</p><p><strong>Detect → Alert → Respond</strong></p><p>That model is no longer enough.</p><p>Modern security operations require agents that can participate across the full lifecycle:</p><p><strong>1. Architecture and design</strong></p><ul><li><strong>What an agentic system can do:</strong> Identify risky assumptions, missing controls, and threat-model gaps.</li><li><strong>Where Security-in-the-Loop matters:</strong> Security consultants validate business context and risk appetite.</li></ul><p><strong>2. Detection</strong></p><ul><li><strong>What an agentic system can do:</strong> Correlate signals, identify anomalies, and prioritize alerts.</li><li><strong>Where Security-in-the-Loop matters:</strong> Analysts validate ambiguous or high-impact findings.</li></ul><p><strong>3. Investigation</strong></p><ul><li><strong>What an agentic system can do:</strong> Collect evidence, build hypotheses, and explain likely attack paths.</li><li><strong>Where Security-in-the-Loop matters:</strong> Humans assess intent, business impact, and context.</li></ul><p><strong>4. Response</strong></p><ul><li><strong>What an agentic system can do:</strong> Recommend or execute containment actions.</li><li><strong>Where Security-in-the-Loop matters:</strong> Humans approve irreversible or high-impact actions.</li></ul><p><strong>5. Recovery</strong></p><ul><li><strong>What an agentic system can do:</strong> Verify restoration, identify root causes, and update controls.</li><li><strong>Where Security-in-the-Loop matters:</strong> Security teams validate lessons learned.</li></ul><p><strong>6. Improvement</strong></p><ul><li><strong>What an agentic system can do:</strong> Suggest detection, policy, and architecture changes.</li><li><strong>Where Security-in-the-Loop matters:</strong> Humans decide what becomes a durable control.</li></ul><p>This matters because security is not only a technical problem. It is a socio-technical one. Human analysts remain essential for interpreting ambiguous outputs, validating alerts, distinguishing false positives from real threats, and making response decisions under incomplete information.</p><p>Medium does not natively support Markdown tables (pasting one renders raw | characters), and its narrow reading column cramps multi-column grids on mobile. Pasting the formatted text above directly into the Medium editor will automatically preserve the bold labels and bullet spacing.</p><h3>The Real Challenge: Uncertainty</h3><p>The hardest part of security automation is not writing code.</p><p>It is dealing with:</p><ul><li>Edge cases</li><li>Anomalies</li><li>Novel attacks</li><li>Incomplete telemetry</li><li>Conflicting signals</li><li>External factors</li><li>Business context</li><li>Human error</li><li>Adversarial behavior</li></ul><p>All of these introduce entropy and complexity.</p><p><strong>An AI agent may see an unusual login, a strange API call, or an unexpected privilege change. But is it an attack, a misconfiguration, a legitimate developer testing something, or a business process that nobody documented?</strong></p><p><strong>A security agent can analyze the evidence. It can correlate events. It can identify patterns. It can even propose a response.</strong></p><p>But it may not know whether blocking that identity will stop an attacker — or break a critical production deployment.</p><p><strong>That is where the human must enter the loop.</strong></p><p>Research on human-AI collaboration in security operations emphasizes that human judgment is especially important in ambiguous cases, novel attack patterns, and high-stakes incidents. It also highlights the need for explainability, uncertainty indicators, and evidence-based justifications — not just a final verdict.link.springer+1</p><h3>Agentic AI Changes the Stakes</h3><p>Agentic AI is different from a traditional alerting system.</p><p>A conventional tool may flag an anomaly.</p><p>An agentic system can:</p><ul><li>Plan an investigation</li><li>Query multiple data sources</li><li>Correlate evidence</li><li>Execute tools</li><li>Draft a response</li><li>Modify configurations</li><li>Trigger workflows</li><li>Learn from feedback</li></ul><p>That autonomy is powerful, but it also expands the attack surface and the blast radius of a mistake.</p><p>Guidance from the Canadian Centre for Cyber Security notes that human supervision, approval, and review are important controls in agentic AI workflows, particularly when actions are high-impact or difficult to reverse. The UK National Cyber Security Centre similarly identifies human-in-the-loop approval and real-time monitoring as ways to manage unexpected agent behavior.cyber.gc+1</p><p>In other words: <strong>the more autonomous the agent, the more intentional the human oversight must be.</strong></p><h3>SITL Is Not a Bottleneck</h3><p>Security-in-the-Loop should not mean that a human approves every action.</p><p>That would destroy the value of automation.</p><p>Instead, SITL should be adaptive.</p><p>Low-risk, well-understood, reversible actions may be automated. Examples include enriching an alert, correlating related events, drafting an investigation report, or recommending a detection improvement.</p><p>High-risk, ambiguous, or irreversible actions should require human validation. Examples include isolating a production server, disabling an account, changing firewall rules, deleting data, or executing a response that could affect business continuity.</p><p>A useful principle is: <strong><em>Automate confidence. Escalate uncertainty.</em></strong></p><ul><li>The agent should act when the evidence is strong, the action is reversible, and the risk is acceptable.</li><li>It should pause when the evidence is weak, the action is consequential, or the context is unclear.</li></ul><h3>What a SITL System Should Provide</h3><p>For Security-in-the-Loop to work, an agent cannot simply output a score or a binary decision.</p><p>It must provide:</p><ul><li><strong>Evidence:</strong> What signals led to this conclusion?</li><li><strong>Interpretability:</strong> Why does the agent believe this is risky?</li><li><strong>Uncertainty:</strong> How confident is the system, and what does it not know?</li><li><strong>Impact:</strong> What could happen if we act — or fail to act?</li><li><strong>Options:</strong> What are the possible responses, with trade-offs?</li><li><strong>Traceability:</strong> Which data, rules, tools, and decisions produced this outcome?</li><li><strong>Feedback:</strong> How can the analyst correct or improve the agent’s reasoning?</li></ul><p>This is close to the clinical model. The UTI-detection system does not merely produce a prediction; it provides uncertainty intervals and can abstain when confidence is low, helping clinicians make more informed decisions.acm+1</p><p>Security teams need the same discipline.</p><h3>From Consultant to Guardian</h3><p>As someone who has worked in the security industry for a long time, I see the security professional’s role changing.</p><p>We are moving from being the person who manually investigates every alert to being the person who designs the boundaries of autonomy.</p><p>We become:</p><ul><li>Architects of secure agentic workflows</li><li>Designers of escalation paths</li><li>Validators of high-impact decisions</li><li>Interpreters of uncertainty</li><li>Guardians of business context</li><li>Reviewers of irreversible actions</li><li>Teachers of the system through feedback</li></ul><p><strong>The agent may handle scale.</strong></p><p><strong>The human must preserve judgment.</strong></p><h3>The Goal Is Not Full Autonomy</h3><p>The goal of Security-in-the-Loop is not to remove humans from security.</p><p><strong>It is to remove humans from the repetitive work that does not require human judgment, while keeping them close to the decisions that do.</strong></p><p><strong>A well-designed agentic security system should be able to work continuously, correlate signals at scale, and accelerate investigation. But it should also know when to stop and ask for help.</strong></p><p>Because in security, as in medicine, the most dangerous answer is not always:</p><blockquote><em>“No.”</em></blockquote><p>Sometimes, it is:</p><blockquote><em>“Yes, act immediately.”</em></blockquote><p>When the correct answer is actually:</p><blockquote><em>“I am not sure yet.”</em></blockquote><p>Security-in-the-Loop is about building systems that know the difference.</p><h3>Suggested references</h3><ul><li>Ugwu, C. E., Fritz, R., Cook, D. J., &amp; Doppa, J. R. <em>Clinician-in-the-Loop Smart Home System to Detect Urinary Tract Infection Flare-Ups via Uncertainty-Aware Decision Support.</em> arXiv:2511.18334.<a href="https://arxiv.org/html/2511.18334v1">arxiv</a></li><li>Canadian Centre for Cyber Security. <em>Careful adoption of agentic AI.</em><a href="https://www.cyber.gc.ca/en/guidance/careful-adoption-agentic-ai">cyber.gc</a></li><li>UK National Cyber Security Centre. <em>Managing the cyber risk of agentic AI.</em><a href="https://www.ncsc.gov.uk/blogs/managing-the-cyber-risk-of-agentic-ai">ncsc.gov</a></li><li><em>A Unified Framework for Human–AI Collaboration in Security Operations Centers.</em><a href="https://dl.acm.org/doi/10.1145/3837073">acm</a></li><li><em>Human-AI Integration in Cybersecurity: An Industry-Aligned…</em><a href="https://link.springer.com/article/10.1007/s10207-026-01259-5?error=cookies_not_supported&amp;code=222bfb97-60e1-4cff-8e80-c7ee48ef335c">link.springer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7606d7d355a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transform Your Restaurant’s Success through AI-Enhanced Customer Feedback Analysis]]></title>
            <link>https://blog.devgenius.io/transform-your-restaurants-success-through-ai-enhanced-customer-feedback-analysis-a6d65bffdb94?source=rss-cd385a51120f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6d65bffdb94</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[restaurant-review]]></category>
            <dc:creator><![CDATA[Gabriel Ribeiro]]></dc:creator>
            <pubDate>Tue, 07 Nov 2023 22:57:30 GMT</pubDate>
            <atom:updated>2023-11-11T17:17:26.890Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Analyzing the Use of Aspect-Based Sentiment Analysis with Artificial Neural Networks for Generating Feedback Based on Online Restaurant Reviews</strong></p><p>In this contemporary era driven by data, the greater the volume of data you possess, the more precise your decision-making capabilities become. In line with this principle, I’ve developed a model designed to classify comments as either positive or negative. Leveraging Natural Language Processing and a deep learning model, I’ve gathered crucial insights into the most significant topics scraped from the restaurant comments available on Google Maps.</p><p>The model was developed using a public dataset called YELP, and after that, I classified the comments available at Google Maps restaurants from Portugal. I focused on my favorite Portugal dish, the “Francesinhas.” By utilizing the Apify endpoint to scrape comments, I successfully gathered historical feedback from Google Maps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*wWMDZ7uOydOnGTenpEy8dQ.png" /><figcaption>Francesinha — Lego</figcaption></figure><h3>Resource</h3><p><strong>Project</strong>: <a href="https://github.com/gabrielhribeiro/foodandfeedback/blob/main/Foodriven.ipynb">https://github.com/gabrielhribeiro/foodandfeedback</a></p><p>This dataset represents a portion of Yelp’s comprehensive collection, containing businesses, reviews, and user information.</p><p><strong>Yelp</strong>: <a href="https://www.kaggle.com/datasets/yelp-dataset/yelp-dataset">https://www.kaggle.com/datasets/yelp-dataset/yelp-dataset</a></p><p>Scraping data in an easy-peasy way: Nowadays, scrapping and getting data is not more of a big challenge; we have disposed of a variety of resources available on the web, and seeing that, I have chosen APIFY to collect my data(comments) from Google Maps.</p><p><strong>Apify</strong>: <a href="https://apify.com/compass/easy-google-maps">https://www.kaggle.com/datasets/yelp-dataset/yelp-dataset</a></p><h3>The Project</h3><p>In my research, I collected five million reviews from 52,000 restaurants. To start, I focused on a smaller batch of 20,000 entries, making it easier to work with and helping my model learn better.</p><p>To make sure our data isn’t primarily positive or negative reviews, I sorted them into two groups: negative (1, 2, and 3-star ratings) and positive (4 and 5-star ratings). Then, I picked an equal number of reviews from each group and mixed them up to create a balanced sample. This helps our model learn without favoring one sentiment during training.</p><p>After organizing this new dataset, I saved it in our shared folder. This way, we won’t have to redo the sample creation process repeatedly.</p><p><strong>Obs</strong>. Since the project is significant, in the snippets below, I will just show the principal topics that I have identified as essential to pinpoint. You can have access to the entire code on my <a href="https://github.com/gabrielhribeiro/foodandfeedback">GitHub</a>.</p><h4>Data Processing and Balanced Sampling of Yelp Dataset</h4><p>The code snippet pertains to data processing and sampling of a dataset, explicitly targeting a Yelp dataset to generate a sample with 100,000 lines. The code snippet showcases data processing steps, mainly the cleaning, filtering, sampling, and exporting a Yelp dataset to create a balanced 100,000-line sample with an equal representation of different rating categories. The biggest challenge of IA models is to collect a dataset that would be as clean and unbiased as possible by normalizing and balancing the dataset.</p><pre>#@title Generate 100,000 lines for sample (Sample already saved in shared drive folder)<br><br>#save data at the project3, if exist the file at the folder(Project3), dont run it. <br>dfyelpnotnull = business_df[business_df[&#39;categories&#39;].notna()]<br>#select just restaurants<br>dfyelpcleanest = dfyelpnotnull[dfyelpnotnull[&#39;categories&#39;].str.contains(&quot;Restaurants&quot;)]<br><br>dfyelp = pd.merge(reviews_df, dfyelpcleanest, on=[&#39;business_id&#39;,&#39;business_id&#39;])<br>dfyelpnotnull = dfyelp[dfyelp[&#39;categories&#39;].notna()]<br><br><br>samples_per_rating_negative = 50000 // 3<br>samples_per_rating_positive = 50000 // 2<br>sampled_df = pd.DataFrame()<br><br># Extract an equal number of samples for each rating<br>for a rating in range(1, 6):<br>  if(rating &lt; 4):<br>    samples = dfyelpnotnull[dfyelpnotnull[&#39;stars_x&#39;] == rating].sample(samples_per_rating_negative, random_state=42)<br>    sampled_df = sampled_df.append(samples, ignore_index=True)<br>  else:<br>    samples = dfyelpnotnull[dfyelpnotnull[&#39;stars_x&#39;] == rating].sample(samples_per_rating_positive, random_state=42)<br>    sampled_df = sampled_df.append(samples, ignore_index=True)<br><br># Shuffle the sampled DataFrame<br>sampled_df = sampled_df.sample(frac=1, random_state=42).reset_index(drop=True)<br>rating_counts = sampled_df[&#39;stars_x&#39;].value_counts()<br>print(rating_counts)<br><br>sampled_df.to_csv(&#39;/content/gdrive/MyDrive/ISEP/Projeto3/cleaned100000reviews.csv&#39;,index=False)</pre><p><strong>Steps done: <br>Data Cleaning and Filtering: </strong>The code cleans the Yelp dataset by removing entries with missing category values and filters it to include only restaurants.</p><p><strong>Sampling for Balanced Data</strong>: It aims to create a balanced sample by extracting an equal number of samples for each rating from the merged dataset (`dfyelp`). It divides the 100,000 samples into two subsets, negative ratings (1, 2, and 3 stars) and positive ratings (4 and 5 stars). The sampling process involves randomly selecting an equal number of samples for each rating category using `sample` function based on the defined division of samples per rating category.</p><p><strong>Data Imbalance Resolution</strong>: The code resolves potential data imbalance issues by ensuring an equal representation of different ratings within the sampled dataset. It achieves this by adjusting the number of samples taken for each rating category based on a predetermined formula</p><h3>Text Preprocessing</h3><h4>Sentences and words Tokenizer</h4><p>This code defines a function, `tokenize_review`, which tokenizes text into words and sentences using the Natural Language Toolkit (NLTK) library. It then applies this function to a DataFrame column named ‘text’ in a dataframe called ‘dfyelp’, storing the tokenized output in a new column called ‘preprocessed_text’.</p><pre>def tokenize_review(review):<br>    sentences = nltk.sent_tokenize(review)<br>    words = [nltk.word_tokenize(sentence) for sentence in sentences]<br>    return words<br><br>dfyelp[&#39;preprocessed_text&#39;] = dfyelp[&#39;text&#39;].apply(tokenize_review)</pre><h4>Removing stop words and Turn text to lowercase</h4><p>The code snippet creates a function, remove_stop_words, eliminating common English stopwords from a tokenized text. It operates on a DataFrame column named &#39;preprocessed_text&#39; in the dataframe &#39;dfyelp&#39;, modifying the tokens by removing these stopwords and certain suffixes like &quot;&#39;s&quot; or &quot;&#39;t&quot;.</p><pre>#@title Removing stop words<br><br>from nltk.corpus import stopwords<br><br>def remove_stop_words(tokenized_text):<br>    stop_words = set(stopwords.words(&#39;english&#39;))<br>    cleaned_text = []<br>    For sentence in tokenized_text:<br>        cleaned_sentence = []<br>        for word in sentence:<br>            if word.lower() not in stop_words:<br>                # Remove &quot;&#39;s&quot; suffix if present<br>                if word.endswith(&quot;&#39;s&quot;):<br>                    cleaned_word = &quot;&quot;<br>                elif word.endswith(&quot;&#39;t&quot;):<br>                    cleaned_word = &quot;&quot;<br>                elif word.endswith(&quot;\&#39;s&quot;):<br>                    cleaned_word = &quot;&quot;<br>                else:<br>                    cleaned_word = word<br>                if word != &quot;&quot;:<br>                  cleaned_sentence.append(cleaned_word)<br>        cleaned_text.append(cleaned_sentence)<br>    return cleaned_text<br><br>dfyelp[&#39;preprocessed_text&#39;] = dfyelp[&#39;preprocessed_text&#39;].apply(remove_stop_words)</pre><h4>Remove punctuation</h4><p>This code defines a function, remove_punctuation, that filters out punctuation from tokenized reviews. It uses a lambda function to apply remove_punctuation to each sentence in the &#39;preprocessed_text&#39; column within the &#39;yelp’ DataFrame, eliminating punctuation marks and specific characters like ‘\’ or ‘“.’</p><pre>#@title Remove punctuation<br>def remove_punctuation(tokenized_reviews):<br>    cleaned_reviews = [word for word in tokenized_reviews if (word not in string.punctuation) and (&#39;\\&#39; not in word) and (&#39;&quot;&quot;&#39; not in word)]<br>    return cleaned_reviews<br><br>dfyelp[&#39;preprocessed_text&#39;] = dfyelp[&#39;preprocessed_text&#39;].apply(lambda x: [remove_punctuation(sentence) for sentence in x])<br>dfyelp[&#39;preprocessed_text&#39;]</pre><h4>Lemmatizer</h4><p>Lemmatization is a natural language processing technique that reduces words to their base or root form, called a lemma. It considers the context of the word and transforms it to its dictionary form, enabling better analysis by linking words with the same meaning. For instance, “running” would be lemmatized to “run,” “better” to “good,” etc.</p><p>Now, regarding the provided code: The code utilizes the spaCy library’s English model (‘en_core_web_sm’) to perform lemmatization. The function lemmatize_words takes a list of words, combines them into a sentence, and then uses spaCy to tokenize and lemmatize these words. The resulting lemma forms of the words are returned as a list.</p><pre>#@title Lemmatizer<br><br>nlp = spacy.load(&#39;en_core_web_sm&#39;)<br><br>def lemmatize_words(words):<br>    text = &quot; &quot;.join(words)<br>    token = nlp(text)<br>    text = [word.lemma_ for word in token]<br>    return text<br>tqdm.pandas()<br><br># apply lemmatization to &quot;text&quot; column<br>dfyelp[&#39;preprocessed_text&#39;] = dfyelp[&#39;preprocessed_text&#39;].progress_map(lambda x: [lemmatize_words(sentence) for sentence in x])<br><br>dfyelp[&#39;preprocessed_text&#39;]</pre><h4><strong>Topic Modeling</strong></h4><p>Topic modeling is a technique used to identify the main themes or subjects within a group of documents. In this project, the initial step involved converting the preprocessed text into a suitable format for this technique. To achieve this, I utilized various modules from the Python ‘Gensim’ library, which provides a range of tools for topic modeling.</p><p>The first task in this conversion process was to break down the preprocessed text into a list of tokenized words. Using Gensim’s ‘corpora.dictionary’ module, I created a dictionary. This dictionary helped in forming a corpus, which represents documents by their word frequencies. With this in place, I employed the ‘doc2bow’ method to convert each document into a Bag-of-Words (BOW) representation.</p><p>Once the corpus was ready, I trained an LDA (Latent Dirichlet Allocation) model on it. LDA is a popular topic modeling technique that assigns each document a mix of topics based on the words it contains. After experimenting with different parameters, I settled on 10 topics, 15 passes, and 400 iterations. This choice struck a good balance between computational resources and result relevance.</p><p>Following the model training, I extracted and displayed the top words associated with each of the 10 topics. To visualize and finalize the topics, I used the ‘pyLDAvis’ library, which provides a detailed and interactive display of the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*jIBCrlzzmZedwE-Ndp33ZA.png" /></figure><h4>Analyzing LDA model — Latent Dirichlet allocation</h4><p>The purpose of this code is to apply the LDA model to the text data from the ‘preprocessed_text’ column and reveal underlying topics by displaying the most relevant words for each identified topic.</p><pre>#@title Analyzing LDA model - Latent Dirichlet allocation<br>#https://www.analyticssteps.com/blogs/what-latent-dirichlet-allocation-lda-nlp<br>import pandas as pd<br><br>data = []<br><br># Iterate over each row of the data column<br>for row in dfyelp.preprocessed_text:<br>    words = row.strip(&#39;][&#39;).split(&#39;, &#39;)<br>    words = [w.strip(&quot;&#39;[]&quot;) for w in words]<br>    data.append(words)<br><br>dictionarydfyelp = corpora.Dictionary(data)<br>corpus = [dictionarydfyelp.doc2bow(doc) for doc in data]<br><br># train an LDA model on the corpus<br>lda_model = LdaModel(corpus=corpus, id2word=dictionarydfyelp, num_topics=10, passes=15,iterations=400) <br><br>#print the top words in each topic<br>pprint(lda_model.show_topics())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrWEVrjJlKjBc-eXVDr2gA.png" /></figure><p>By selecting the topics, the code evaluates the similarity between text in the ‘preprocessed_text’ column of the ‘yelp’ DataFrame and predefined aspects such as “service,” “food,” “location,” etc. It employs FastText word embeddings to calculate these similarities. The results are stored in new columns corresponding to each aspect in the DataFrame, allowing assessment of how closely the text aligns with these specific aspects commonly found in reviews.</p><h3>The Model</h3><p>A deep learning model, this configuration class encapsulates various settings essential for training a machine learning model, ensuring control over training processes, data handling, and model performance.</p><pre>class config:<br>    warnings.filterwarnings(&quot;ignore&quot;, category = UserWarning)<br>    DEVICE = (&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;) # where the model will use the processing<br>    FOLDS = 20 #number os cross validation<br>    SHUFFLE = True # randomize the values to increase the fitting<br>    BATCH_SIZE = 32 <br>    LR = 0.1 <br>    EPOCHS = 20 <br>    EMB_DIM = 100 #every word will use a 100 size vector<br>    MAX_LEN = 20<br>    MODEL_PATH = &quot;./Models/MyModel.pt&quot;</pre><p>The code defines a neural network model using PyTorch to perform text classification. It consists of an embedding layer, a bidirectional LSTM layer, and two fully connected (linear) layers. The model takes text input, processes it through the layers, and produces a sigmoid-activated output, aiming to classify the text into categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/0*nguuWwjSXdBL0KJk.png" /></figure><p>The bidirectional LSTM captures forward and backward dependencies in the input text, enhancing its understanding of sequential data. The model uses dropout for regularization.</p><p><strong>Model Setup</strong></p><pre>class Model(nn.Module):<br>    def __init__(self, input_dim, embedding_dim, hidden_dim, output_dim, embedding_layer):<br>        super().__init__()<br>#         self.embedding = nn.Embedding(input_dim, embedding_dim)<br>        self.hidden_dim = hidden_dim<br>        self.embedding = embedding_layer<br>        self.lstm = nn.LSTM(embedding_dim, hidden_dim, bidirectional = True) #The bidirectional parameter is an optional parameter that can be set to True or False. If set to True, the LSTM will process the input data in both forward and backward directions, effectively doubling the number of hidden units in the network. This can be useful for capturing bidirectional dependencies in the data.<br>        self.fc1 = nn.Linear(2*hidden_dim, 128)<br>        self.fc2 = nn.Linear(128, output_dim)<br>        self.dropout = nn.Dropout(0.5)<br>        self.sigmoid = nn.Sigmoid()</pre><p>The prowess of LSTMs in classification tasks is a testament to their prowess in understanding and leveraging sequential data patterns, making them a preferred choice for many machine learning practitioners.</p><h3><strong>Results</strong></h3><p>To enhance the performance of the ANN model, I conducted a series of tests aimed at fine-tuning key parameters. Due to time constraints, I focused on parameters known to significantly impact the model’s performance, such as the number of hidden dimensions, learning rate, dropout rate, and batch size.</p><p>Initially, I set basic values for these parameters as a starting point for our experiments. Adjusting the batch size revealed that a value of 16 marginally improved results compared to the starting value of 32. However, for faster computation speed, I decided to retain the value of 32.</p><p>Experimenting with different learning rates showed that smaller values notably reduced the model’s accuracy. From these results, I chose a value of 0.1 as it provided the best results.</p><p>I also examined the impact of altering the dropout rate on the model’s performance. However, our tests didn’t demonstrate any significant change in the results, so I maintained the initial value of 0.5.</p><p>Regarding the number of hidden dimensions, increasing this parameter didn’t notably improve or reduce the performance. Yet, due to the considerably longer computational time, I established the initial value at 128.</p><p>Throughout these experiments, I closely monitored the model accuracy for each epoch. It became evident that the initially set number of epochs, 20, appeared to be the optimal choice. In the later epochs, the model’s accuracy consistently stabilized, indicating that further training wouldn’t significantly enhance performance.</p><p>After training the ANN model and the FastText model, I used them to extract sentiment and identify the most relevant aspects of restaurant reviews, respectively.</p><p>By combining these tools with reviews from a restaurant situated in Porto, Portugal, I successfully created graphs showcasing the strengths and weaknesses of the restaurant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*TNtCx5fhbMz1OaRFClEocg.png" /></figure><p>Which shows the ratio of positive and negative feelings related to each aspect of a restaurant using a bar graph. By examining this chart, restaurant owners can pinpoint the areas where their restaurant performs well and where it might need improvement. For instance, if a restaurant displays a high proportion of positive sentiment related to its service, it signifies that the service is a notable strength. Conversely, a higher negative sentiment might highlight a weakness. By identifying and addressing these specific areas, restaurants can implement targeted improvements that significantly enhance their overall reputation.</p><p><strong>Conclusions</strong></p><p>In this project, I explored the use of Aspect-Based Sentiment Analysis (ABSA) using Natural Language Processing (NLP) and an Artificial Neural Network (ANN), specifically a BiLSTM (Bidirectional Long-Short Term Memory) model, to provide insights for restaurants based on their reviews.</p><p>By training the LSTM model, I achieved over 80% accuracy in sentiment classification. Employing topic modeling with a FastText model to extract the most relevant aspects of reviews, I created charts that allowed restaurant owners to easily identify their restaurants’ strengths and weaknesses across predefined topics without the need for manual review analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*7BZyvdUT210F0SDwPDnp_A.png" /></figure><p>The practical implications of my findings are substantial, empowering me as a business owner or manager to take the necessary steps to improve the restaurant. Additionally, the methodology and model developed can be adapted to other businesses or products with accessible evaluations.</p><p>However, there are areas for improvement in my study. One such aspect is the accuracy of the ANN model, which, despite achieving high accuracy, can still distort results with misclassified reviews. Moreover, enhancing the feedback by capturing specific sentiment keywords for each aspect of the restaurant could provide me with more direct and valuable insights.</p><p>Overall, my study lays a foundation for further enhancing the accuracy of the LSTM model and integrating more sophisticated techniques to improve the quality of the feedback I receive for my restaurant.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6d65bffdb94" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/transform-your-restaurants-success-through-ai-enhanced-customer-feedback-analysis-a6d65bffdb94">Transform Your Restaurant’s Success through AI-Enhanced Customer Feedback Analysis</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate Your Reporting Process: Harnessing the Power of Tableau API and Python]]></title>
            <link>https://blog.devgenius.io/automate-your-reporting-process-harnessing-the-power-of-tableau-api-and-python-d8f34ed3450f?source=rss-cd385a51120f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8f34ed3450f</guid>
            <category><![CDATA[words]]></category>
            <category><![CDATA[tableauserverclient]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[microsoft]]></category>
            <category><![CDATA[tableau]]></category>
            <dc:creator><![CDATA[Gabriel Ribeiro]]></dc:creator>
            <pubDate>Sat, 03 Jun 2023 18:11:26 GMT</pubDate>
            <atom:updated>2023-06-05T12:36:07.929Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L0pKwuAxu3PHcBYW4NY7oA.png" /></figure><h4>Streamline your data analysis and reporting workflow with this step-by-step guide</h4><p>In today’s data-driven world, businesses rely on accurate and timely reports to make informed decisions. However, creating these reports can be a time-consuming and repetitive task. This is where automation comes in. By leveraging the power of Tableau API and Python, you can streamline your reporting process and focus on what truly matters: analyzing the data and making strategic decisions.</p><p>In this article, we’ll walk you through the process of automating reports using Tableau API and Python. We’ll cover the basics of connecting to the Tableau API, extracting data, and generating reports in Word format. By the end of this guide, you’ll have a solid foundation for automating your reporting workflow.</p><p>Please, you can reach this code at:</p><p><a href="https://github.com/gabrielhribeiro/tableau-pydoc">GitHub - gabrielhribeiro/tableau-pydoc: Automation reports using tableau API and word(docx)</a></p><p>Here’s a detailed description of each part of the code that you can use:</p><p>1. Importing Required Libraries:<br> — The script starts by importing necessary libraries for working with Tableau Server, manipulating data, handling IO operations, working with Word documents, and managing dates.</p><p>2. Setting Tableau Server Connection Details:<br> — The server URL and authentication token are defined to establish a connection with the Tableau Server. The `tableauserverclient` library is used for this purpose.</p><p>3. Helper Functions:<br> — Two helper functions are defined:<br> — `get_workbook_id(desired_workbook_name)`: This function takes the name of a workbook as input and retrieves its ID from the Tableau Server.<br> — `get_view_id(server, tableau_auth, desired_workbook_name, desired_view_name)`: This function takes the server, authentication details, workbook name, and view name as input and retrieves the ID of the specified view from the Tableau Server.</p><p>4. User Input:<br> — The script prompts the user to provide input for the range of dates and the target for reporting.</p><p>5. Retrieving View IDs:<br> — The script calls the helper function `get_view_id()` to retrieve the IDs of various views required for reporting. These view IDs are stored in variables for later use.</p><p>6. Establishing Server Connection:<br> — The script establishes a connection with the Tableau Server using the server URL and authentication token.</p><p>7. Retrieving Data as CSV:<br> — Data is retrieved from different views on the Tableau Server as CSV files. The script utilizes the `server.views.populate_csv()` method to populate the views with CSV data.</p><p>8. Loading CSV Data into DataFrames:<br> — The CSV data is loaded into pandas DataFrames for further processing and analysis. The `pd.read_csv()` function is used to read the CSV data into DataFrames.</p><p>9. Substituting Values in Word Document:<br> — The script uses a Word document template with placeholders for data. It substitutes values from the DataFrames into the placeholders using loops and conditional statements.</p><p>10. Retrieving and Inserting Images:<br> — An image is retrieved from the Tableau Server using the `server.views.populate_image()` method. The image is then inserted into the Word document in place of a specific placeholder.</p><p>11. Saving the Modified Document:<br> — The modified Word document is saved with a filename that includes the current date. The `Document.save()` method is used to save the document.</p><p>12. Exception Handling:<br> — Exception handling is implemented to catch any errors that occur during the execution of the script. Any error messages are displayed to the user.</p><p>13. Notification of Successful Completion:<br> — If the script runs without any errors, a success message is displayed to the user, along with the path to the generated report.</p><p>If you need help just let me know.</p><p>Regardds,</p><p>Gabriel</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8f34ed3450f" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/automate-your-reporting-process-harnessing-the-power-of-tableau-api-and-python-d8f34ed3450f">Automate Your Reporting Process: Harnessing the Power of Tableau API and Python</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configure Mongosqld(Mongo) connector in a Tableau environment: Ubuntu Server 20.04]]></title>
            <link>https://blog.devgenius.io/tableau-in-a-ubuntu-server-and-mongosqld-connector-d140f8595b51?source=rss-cd385a51120f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d140f8595b51</guid>
            <dc:creator><![CDATA[Gabriel Ribeiro]]></dc:creator>
            <pubDate>Sat, 03 Jun 2023 17:28:25 GMT</pubDate>
            <atom:updated>2023-06-05T12:36:00.363Z</atom:updated>
            <content:encoded><![CDATA[<p>I have tried to set up this environment for some time, so in the following article, you can find exactly how I did the configuration.</p><p><strong>MYSQL DRIVE</strong></p><p>To MYSQL driver will interact with mongosqld and mongosqld translate to NoSQL language to communicate with MongoDB environment, similar to the image below</p><ul><li>The MySQL(which should be installed on the Tableau server) will make the <strong>JDBC </strong>paper</li><li>and the BI connector is the <strong>Mongosqld</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E3tT3Ht2cWIaXOHP3w7lqA.png" /></figure><p>At the following link, you can have more details: <a href="https://www.mongodb.com/docs/bi-connector/current/what-is-the-bi-connector/">https://www.mongodb.com/docs/bi-connector/current/what-is-the-bi-connector/</a></p><p>First of all, you need to install the MYSQL(PLUGIN) drivers in your environment, in this case, the Tableau server. <a href="https://www.tableau.com/support/drivers">https://www.tableau.com/support/drivers</a></p><ul><li>On Debian and Ubuntu Linux distributions, download the TAR archive and then follow the instructions on the MySQL website:</li><li><a href="https://dev.mysql.com/doc/connector-odbc/en/connector-odbc-installation-binary-unix-tarball.html">https://dev.mysql.com/doc/connector-odbc/en/connector-odbc-installation-binary-unix-tarball.html</a> — Follow the instructions from this connector installation.</li></ul><p>After you install the driver, add entries similar to these to the /etc/odbcinst.ini file if they&#39;re not already there:</p><pre>[MySQL ODBC 8.0 Unicode Driver]<br>Driver=/usr/lib64/libmyodbc5w.so<br>UsageCount=1<br><br>[MySQL ODBC 8.0 ANSI Driver]<br>Driver=/usr/lib64/libmyodbc5a.so<br>UsageCount=1</pre><p>After this procedure, the connector <strong>MongoDB BI Connector</strong> will be available for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*bngRGVQq3ORRmrCjIw4uHA.png" /></figure><p><strong>MONGOSQLD — Connector for Mongo</strong></p><p>After that, you should install the Mongosqld following the procedures in the following link below <a href="https://www.mongodb.com/docs/bi-connector/current/tutorial/install-bi-connector-debian/">https://www.mongodb.com/docs/bi-connector/current/tutorial/install-bi-connector-debian/</a></p><p>It’s possible to install this connector at the Tableau server or in the same server as Mongodb or in a third server as a middleware!</p><pre>systemLog:<br>  path: &#39;/var/log/mongosqld.log&#39;<br>  logAppend: false<br>net:<br>  bindIp: tableau.domain.com<br>  port: port(example:3331)<br>  ssl:<br>    mode: requireSSL<br>    PEMKeyFile: /etc/ssl/certs/star.mycertificate.com.pem<br>    minimumTLSVersion: TLS1_2<br>mongodb:<br>  net:<br>    uri: mongodb://mongo.mymongoaddress.com:port(example:3331)<br>    auth:<br>      username: &quot;User&quot;<br>      password: &quot;Pass&quot;<br>      mechanism: &quot;SCRAM-SHA-1&quot; #check the documentation<br>    ssl:<br>      enabled: true<br>      PEMKeyFile: /etc/ssl/certs/our_self_signed/mycertificate.com.pem<br>      CAFile: /etc/ssl/certs/our_self_signed/rootCA.crt<br>      minimumTLSVersion: TLS1_2<br>security:<br>  enabled: true<br>processManagement:<br>  service:<br>    name: mongosqlc<br>    displayName: mongosqld<br>    description: &quot;BI Connector SQL proxy server&quot;</pre><p>Save the file as config-mongosqld.yaml, for example.</p><pre>mongosqld --config config-mongosqld.yaml</pre><p>**In the doc shared, it’s possible to turn this command into a service.</p><p>Executing the latest command will generate a schema and Tableau will be able to read the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*iD_jzRrmZhxULEOdmGbh6A.png" /></figure><ul><li>*Use the credentials and pass from Mongo; the server and port use the connector as the example shared.</li></ul><p>Doing this procedure, you will be able to reach out tableau environment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d140f8595b51" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/tableau-in-a-ubuntu-server-and-mongosqld-connector-d140f8595b51">Configure Mongosqld(Mongo) connector in a Tableau environment: Ubuntu Server 20.04</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CRISP-DM: A Step-by-Step Guide to Successful Data Mining and Machine Learning Projects]]></title>
            <link>https://blog.devgenius.io/crisp-dm-a-step-by-step-guide-to-successful-data-mining-and-machine-learning-projects-37dd70267041?source=rss-cd385a51120f------2</link>
            <guid isPermaLink="false">https://medium.com/p/37dd70267041</guid>
            <dc:creator><![CDATA[Gabriel Ribeiro]]></dc:creator>
            <pubDate>Tue, 18 Apr 2023 22:23:25 GMT</pubDate>
            <atom:updated>2023-06-05T12:35:47.429Z</atom:updated>
            <content:encoded><![CDATA[<p>Data mining and machine learning projects require a structured approach to ensure that the process is thorough and the results are relevant and actionable. The CRISP-DM methodology, or Cross-Industry Standard Process for Data Mining, is a widely used framework that provides a step-by-step guide to data mining and machine learning projects. By following the CRISP-DM process, data scientists and analysts can ensure that they are addressing the business problem effectively and producing models that provide accurate insights.</p><p>In this blog post, we will explore the six phases of the CRISP-DM process in detail and provide tips and insights on approaching each phase effectively. We will cover the initial business understanding phase, data understanding, data preparation, modeling, evaluation, and deployment. By the end of this post, you will have a thorough understanding of the CRISP-DM methodology and how to use it to drive data mining and machine learning projects forward. Whether you are new to data mining or an experienced data scientist, this post will provide valuable insights and practical tips for success.</p><figure><img alt="CRISP-DM DIAGRAM" src="https://cdn-images-1.medium.com/max/319/0*nzMpBF5NIp4gSxp3.png" /></figure><p>At the following topics you will be able to identify each of these topics detailed:</p><ol><li>Business Understanding: This phase is all about understanding the business problem and identifying how data mining can help solve it. Examples of topics to cover in this phase include:</li></ol><ul><li>Identifying the business objectives and goals.</li><li>Defining the scope of the project.</li><li>Identifying the key stakeholders and their needs.</li><li>Determining the success criteria for the project.</li></ul><p>2. Data Understanding: In this phase, data sources are identified, and data is collected and explored. The goal is to gain a deeper understanding of the data and identify any data quality issues. Topics to cover in this phase include:</p><ul><li>Identifying the relevant data sources.</li><li>Collecting and exploring the data.</li><li>Assessing the quality of the data.</li><li>Identifying any missing data or outliers.</li></ul><p>3. Data Preparation: In this phase, data is cleaned, transformed, and formatted to be used in modeling. This includes selecting relevant features and dealing with missing data, outliers, and other issues. Topics to cover in this phase include:</p><ul><li>Cleaning and transforming the data.</li><li>Handling missing data and outliers.</li><li>Selecting relevant features.</li><li>Normalizing or standardizing the data.</li></ul><p>4. Modeling: In this phase, various modeling techniques are applied to the prepared data. This includes selecting appropriate algorithms and building, testing, and validating models. Topics to cover in this phase include:</p><ul><li>Selecting appropriate algorithms.</li><li>Building and testing models.</li><li>Validating models to ensure they meet the success criteria.</li><li>Iterating on the models to improve their accuracy.</li></ul><p>5. Evaluation: In this phase, the models are evaluated and compared to determine which model is best suited for the business problem. The evaluation criteria are based on the business objectives. Topics to cover in this phase include:</p><ul><li>Evaluating the models against the success criteria.</li><li>Comparing different models to determine which is best suited for the problem.</li><li>Analyzing the results of the models to gain insights.</li></ul><p>6. Deployment: In this final phase, the best model is deployed into production, and the project results are communicated to stakeholders. This includes documenting the process and providing recommendations for ongoing monitoring and improvement. Topics to cover in this phase include:</p><ul><li>Deploying the model into production.</li><li>Communicating the project results to stakeholders.</li><li>Documenting the process for future reference.</li><li>Providing recommendations for ongoing monitoring and improvement.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*L156aso4HtBrYnVF.png" /></figure><p>A good example its regarding a use case where you want to predict customer churn for a telecom company.</p><p>In the Business Understanding phase, you would identify the business objectives, such as reducing customer churn and increasing customer satisfaction.</p><ul><li>In the Data Understanding phase, you would collect and explore data on customer behavior, such as call and data usage, and assess the quality of the data.</li><li>In the Data Preparation phase, you would clean and transform the data, and select relevant features such as call duration and data usage.</li><li>In the Modeling phase, you would build and test models to predict customer churn, using techniques such as logistic regression or decision trees.</li><li>In the Evaluation phase, you would evaluate the models against the success criteria, and compare different models to determine which is best suited for the problem.</li><li>Finally, in the Deployment phase, you would deploy the best model into production and communicate the results to stakeholders.</li></ul><p>#datamining #machinelearning #CRISPDM #datascience #businessanalytics #bigdata #projectmanagement #dataanalysis #predictiveanalytics #datacleaning #deployment #monitoring #improvement #bestpractices #businessobjectives #dataunderstanding #datapreparation #modeling #evaluation #stakeholdercommunication #successcriteria</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37dd70267041" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/crisp-dm-a-step-by-step-guide-to-successful-data-mining-and-machine-learning-projects-37dd70267041">CRISP-DM: A Step-by-Step Guide to Successful Data Mining and Machine Learning Projects</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series ARIMA algorithm to forecast almost 500 stocks from S&P 500 index]]></title>
            <link>https://blog.devgenius.io/using-arima-algorithm-to-forecast-almost-500-stocks-from-s-p-500-index-443679e6f41e?source=rss-cd385a51120f------2</link>
            <guid isPermaLink="false">https://medium.com/p/443679e6f41e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pycaret]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Gabriel Ribeiro]]></dc:creator>
            <pubDate>Sun, 25 Dec 2022 19:23:11 GMT</pubDate>
            <atom:updated>2023-01-13T13:38:05.774Z</atom:updated>
            <content:encoded><![CDATA[<h4>An approach using Pycaret lib, beta variance and yahoo finance to generate metrics to compose a future stock portfolio.</h4><p>I would like to predict and verify future possible values of which stocks are less risky than others; based on this; I could identify some economy theory and statistics metrics; at this post, I’m going to share some snippets that can deliver <strong>insights</strong> to play around with stock predictions, more precisely with a Time Series Model called A<a href="https://www.investopedia.com/terms/a/autoregressive-integrated-moving-average-arima.asp">uto Regressive Integrated Moving Average</a>(ARIMA).</p><p><strong>“</strong>Before the discovery of Australia, people in the old world were convinced that <em>all</em> swans were white, an unassailable belief as it seemed completely confirmed by empirical evidence.<strong>”, </strong><a href="https://www.nytimes.com/2007/04/22/books/chapters/0422-1st-tale.html"><strong>Nassim Nicholas Taleb</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*XGGliZVvTxfOuHI2g5OHfg.png" /><figcaption>font: <a href="https://sketchplanations.com/black-swan">https://sketchplanations.com/black-swan</a></figcaption></figure><p>Firstly, I realized stocks have black swans like covid, the Ukraine War, Elon Musk, and other situation that can disturb the normality and change in some way the variance behavioral of stocks. Based on this, I started to look around for some risk management systems for stocks, and I found an interesting method called <a href="http://capital%20asset%20pricing%20model%20-%20wikipedia/">CAPM</a> <a href="http://capital%20asset%20pricing%20model%20-%20wikipedia/">— Capital asset pricing model.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*REErQ4Hy2QxY6Te1b3k-hQ.png" /><figcaption>SP 500 index — where we can visualize, in the middle of 2020, covid pandemic breaking out the normal flow.</figcaption></figure><p>The CAPM model offers a theory called beta variance. This feature can be used to calculate the systematic risk of a stocks by comparing the variance between stocks and other indexes like S&amp;P500 indicators, for example. This metric can show up as a factor that can be used to select the less unstable stock based on some index; <a href="http://CAPM - Capital asset pricing model.">for more details, you can check it out in this article</a>.</p><h3>No more words, let’s dive into Data!!</h3><p>To organize, I’ve divided this post in through 6 big steps to organize: Sample, Explore, Modify, Model, Conclusion.</p><p>All these steps and the code are available thought this <a href="https://github.com/gabrielhribeiro/PortifolioSP500">git repo</a>.</p><p><strong>Sample</strong></p><p>First, To proceed with that, we need to collect the stocks from SP 500. Through a simple web scraping, we are going to collect SP500 stock symbol values from Wikipedia page.</p><pre>#Identify after how much time they started at the SP500 after their foundation<br>#Show how sectors are at the SP500<br>#show quantity based on security<br>#show GICS Sub-Industry<br>#function to take stocks and values<br>import pandas as pd<br><br>def returnstocks():<br>    table=pd.read_html(&#39;https://en.wikipedia.org/wiki/List_of_S%26P_500_companies&#39;)<br>    stocksdf = table[0]<br>    stocksdf.to_csv(&#39;S&amp;P500-Info.csv&#39;)<br>    stocksdf.to_csv(&quot;S&amp;P500-Symbols.csv&quot;, columns=[&#39;Symbol&#39;])<br>    stocksdf[&#39;Symbol&#39;] = stocksdf[&#39;Symbol&#39;].replace(&#39;.B&#39;,&#39;&#39;, regex=True)<br>    return stocksdf<br><br>def eachstock():<br>  myList = [&#39;^GSPC&#39;]<br>  for stick in returnstocks()[&#39;Symbol&#39;].unique(): <br>    myList.insert(0, stick)<br>  return myList</pre><pre>tickers = [&#39;^GSPC&#39;,&#39;ZTS&#39;,&#39;ZION&#39;,&#39;H&#39;,&#39;RA&#39;,&#39;YUM&#39;..]</pre><p>The function eachstock(), will return a list of stocks initialized by SP500.</p><p><strong>Explore</strong></p><p>During some exploratory analysis, we were possible to identify the main sectors used by SP500, these sectors can be used to diversified an stock portfolio, as more spread less risk, trivial but important to avoid black swans from an specific sector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fLQd_hTTwUJSk6sddjgzPg.png" /><figcaption>SP500 Sectors</figcaption></figure><p>For this step, we are using the component from <a href="https://en.wikipedia.org/wiki/List_of_S%26P_500_companies">Wikipedia</a>, and we are going to introduce a new agent, the <a href="https://pypi.org/project/yfinance/">yahoo finance</a>. Yahoo Finance, has hot data from <a href="https://finance.yahoo.com/">https://finance.yahoo.com</a>, and we can use these data to improve our power to decision making, regarding to create a stocks portfolio.</p><pre>#@title Yahoo Finance request and beta/Alpha calculation<br>#@markdown For this step, we are using the component from Wikipedia, and we are going to introduce a new agent, the yahoo finance. Yahoo Finance, has hot data from https://finance.yahoo.com, and we can use these data to improve our power to decision making, regarding to create a stocks portfolio.<br><br>yfin.pdr_override() #override method from pandas_datareader by importing data as pdr<br><br>tickers = eachstock()<br><br>#tickers = [&#39;ZTS&#39;, &#39;ZION&#39;, &#39;^GSPC&#39;]<br><br>#function for receive list of stocks, source where to search and initial date/lastdate<br>def stockdoublebench_returns(ticker, initialdate, lastdate): #future change ticker for returnstocks()[&#39;Symbol&#39;].unique()<br>    stockdouble = yfin.download(ticker, initialdate, lastdate) #data = yf.download(&quot;SPY AAPL&quot;, period=&quot;max&quot;,)<br>    return stockdouble[&#39;Adj Close&#39;]<br><br>#stockdoublebench_returns #run just this command to check the volatility of each one. <br><br>def stockdoublebench_percent(ticker, initialdate, lastdate): #future change ticker for returnstocks()[&#39;Symbol&#39;].unique()<br>    stockdoublepercent = yfin.download(ticker, initialdate, lastdate)<br>    #return stockdouble[&#39;Adj Close&#39;]] #add this line to use the graph to compare the volatility<br>    return stockdoublepercent[&#39;Adj Close&#39;].pct_change()[1:]<br><br>#alpha, beta and stock<br>abstocks = pd.DataFrame(columns=[&#39;Alpha&#39;, &#39;Beta&#39;, &#39;stocks&#39;]) #created dataframe<br><br>def alphabetastock(df, index): #Function for generate alpha and beta from stocks<br>    for col in df: #for column in dataframe do<br>        x = sm.add_constant(df[index].values) #create a matrix using 1 and the value<br>        model = sm.regression.linear_model.OLS(df[col].values,x).fit() #linear regression to get values https://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.OLS.html<br>        abstocks.loc[len(abstocks.index)] = [model.params[0], model.params[1], col] #add each value to the dataframe<br>    return abstocks #model.params[0], model.params[1], col<br><br>#need to add a new line at the df for show the predicted values from each stock, and its important add the value<br>#from the accuracy from test as well. </pre><p>The Dataframe <strong>abstocksna</strong>, returned the values. Alpha, Beta and Stocks, for more information regarding beta and alpha, please: check this <a href="https://www.investopedia.com/ask/answers/102714/whats-difference-between-alpha-and-beta.asp">article</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/1*uSk7YrnC4inK-rfMGu44iw.png" /><figcaption>Alpha and Beta variance</figcaption></figure><p>Setting up our configuration, we are able to check more useful information’s.</p><p><strong>Modify</strong></p><p>We need to prepare our data to be used, since stocks has a high volatility, than we need to normalize our data, we are going to use the lib MinMaxScaler() from <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html">sklearn,</a> with functions fit_transform and inverse_transform, now we are able to delivery this requirements. Since ARIMA has better results with minmax scalar than z-score method, we will directly choose minmax() to fit and scale our data.</p><pre>stockdfs = pd.DataFrame()<br>def datenormalization(stockdf): <br>  global stockdfs<br>  stockdf = stockdf.groupby(&#39;Date&#39;).sum()<br>  stockdf = stockdf.asfreq(freq =&#39;D&#39;); <br>  stockdfs = stockdf.ffill() #associate frequency by day to get hollidays<br>  stockdfs.sort_index(ascending=True, inplace=True)<br>  return pd.DataFrame(stockdfs)<br>  <br>obj = &#39;&#39; #global variable<br>def scaler_fit(series):<br>  global obj<br>  scaler = MinMaxScaler()<br>  obj = scaler.fit(series)<br>  return obj<br><br>def min_max_fit(series): #normalization<br>  scaler_fit(series)<br>  scaler = MinMaxScaler()<br>  series[list(series.columns)] = scaler.fit_transform(series)<br>  return series<br><br>def min_max_inverse(series, obj): #denormalization<br>  series[list(series.columns)] = obj.inverse_transform(series)<br>  return series</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/2/1*nw5BYqW1X4oKZik9l2rSww.png" /><figcaption><strong>MinMaxScaler converts feature data to a range of [0,1].</strong></figcaption></figure><p><strong>Model</strong></p><blockquote>ARIMA models <strong>use differencing to convert a non-stationary time series into a stationary one, and then predict future values from historical data</strong>. These models use “auto” correlations and moving averages over residual errors in the data to forecast future values. <a href="https://www.capitalone.com/tech/machine-learning/understanding-arima-models/">https://www.capitalone.com/tech/machine-learning/understanding-arima-models/</a></blockquote><p><strong>Setup Model</strong></p><p>During some tests, we identified ARIMA, one of the best algorithm to predict stocks in a short time period, predictions for days. To proceed with these tests, we have used the pycaret to visualize and identify the best fits methods.</p><p>First, we just create our setup, passing the fh(forecast horizon) values and the fold, is used for cross validation. The session_id argument, is used for delivery a random number for this setup.</p><blockquote>fh: The forecast horizon is <strong>the length of time into the future for which forecasts are to be prepared</strong>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w1m2YZnPc2b02IoVgDXfIw.png" /><figcaption>pycaret setup</figcaption></figure><p>Pycaret return some good values regarding the targeted trained and tested dataframe, one thing to be aware, is regarding <a href="https://www.reed.edu/economics/course_pages.archive/red_spots/endogenous_and_exogenous_v.htm">exogenous</a> variable, how you pass the variable to pycarat, if you face problems, check that reference bellow.</p><p><a href="https://github.com/pycaret/pycaret/issues/1588">Add Exogenous Variable Support for Time Series · Issue #1588 · pycaret/pycaret</a></p><p>After setup process, we need to choose a model, as I already have an ARIMA model choose, If you want that Pycaret choose/test the best algorithm for you, you can use the function compare-models(), you can verify deeper through pycaret <a href="https://pycaret.readthedocs.io/en/latest/api/time_series.html#pycaret.time_series.compare_models">documentation</a>.</p><p><strong>Create Model</strong></p><p>As I already have an option selected, we will proceed with ARIMA model, where its categorized as a Time Series forecaster algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*Of0YUWA1AyM7jmoCOEPaSA.png" /><figcaption>pycaret metrics</figcaption></figure><p><strong>Identify and read the metrics</strong></p><p><strong>Similarity between booth metrics(MAE and MRSE)</strong></p><p>1 — Booth of metrics express the average error based on predicted model regarding the original code(train/test).</p><p>2 — They are between 0-infinity and return the magnitude error.</p><p>3 — More near to zero(0), better the value is.</p><p>For more details: <a href="https://medium.com/human-in-a-machine-world/mae-and-rmse-which-metric-is-better-e60ac3bde13d"><strong>MAE and MRSE</strong></a></p><p><strong>Checking the Results — Created Model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hnfw2bt_G97ywdEl229StQ.png" /><figcaption>plot_model(arima)</figcaption></figure><p><strong>Create Model</strong></p><p><strong>Finalizing Model</strong></p><p>At the moment the results shared accomplish the expectations, we will finalize he model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*bfxMm8F_vZ_fmyv1x1hNBg.png" /></figure><p><strong>Plot predicted values — 5 ahead.</strong></p><p>Since we have our model created, we will predict 5 days for future values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*dFKUczMSOVFDHgyFXz-TPw.png" /><figcaption>5 days predicted</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wTsCghzRYYt6Nrk1MzgsaA.png" /><figcaption>result predicted</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OfYb7r7nlnjkitHuVcyMhg.png" /><figcaption>Plot predicting values — Details</figcaption></figure><p><strong>Engineering Everything Together</strong></p><p>After that, we created two functions, one to compute the train and normalization and denormalization and the second function is used to create an array and merge every information in one dataframe, doing that we can proceed with further analysis.</p><p>All the algorithm works following the workflow bellow:</p><p><strong><em>minmax() → setup() →finalize model() → predict model() → inverseminmax() → mergeresults() → Dataframe[] → Validation → Delviery</em></strong></p><pre> #@title Engineering All to Predict Stocks<br> #@markdown Engineering to predict all 500 stocks<br>from pycaret.time_series import *<br><br>pred_unseen_deno = pd.DataFrame()<br>dffdstocks = pd.DataFrame()<br>transpredicted = pd.DataFrame()<br>dfstocks = pd.DataFrame()<br>dfstocks.iloc[0:0]<br><br>def train_and_normalize(stock, initialdate, finaldate, predictdays, predictdaysx, crossvalidation):<br>  global pred_unseen_deno<br>  global transpredicted<br>  stock_returns = datenormalization(stockdoublebench_returns(stock, initialdate, finaldate))<br>  print(stock_returns)<br>  exp2 = setup(min_max_fit(stock_returns[[&#39;Adj Close&#39;]]), fh = predictdays, fold = crossvalidation, session_id = 123)<br>  arima2 = exp2.create_model(&#39;arima&#39;)<br>  #pred_holdout = predict_model(arima2) <br>  pred_unseen_deno = min_max_inverse(predict_model(exp2.finalize_model(arima2), fh = predictdaysx), obj)<br>  #if you need to see the pred_holdout value<br>  #pred_unseen_deno = min_max_inverse(predict_model(finalize_model(arima2), fh = predictdaysx), obj)<br>  transpredicted = pred_unseen_deno.T<br>  transpredicted[&#39;stocks&#39;] = stock<br>  return transpredicted<br><br>def iterate_predicted_stocks(initialdate, finaldate, predictdays,predictdaysx, crossvalidation): <br>  global dffdstocks<br>  global dfstocks<br>  for value in (abstocksna[&#39;stocks&#39;]):<br>    try: <br>      train_and_normalize(value, initialdate, finaldate, predictdays, predictdaysx,crossvalidation) <br>      dffdstocks = dffdstocks.append(transpredicted, ignore_index = True)<br>      dfstocks = pd.merge(dffdstocks, abstocks, on=&#39;stocks&#39;)<br>    except:<br>      continue<br>  return dfstocks<br>  #We are going to use this function to start up our algoritm to predict future values. <br>#initial date, last date, predictdays(for test), predictdate for future to be used after last date, and how <br>#many values fold we are going to use for cross validation.<br><br>abstocks = alphabetastock(stockdoublebench_percent(tickers, &#39;2015-01-01&#39;, &#39;2022-12-25&#39;),&#39;^GSPC&#39;)<br>abstocksna = abstocks.dropna()<br>iterate_predicted_stocks(&#39;2015-01-01&#39;, &#39;2022-12-25&#39;, 30, 10 ,5) </pre><p>The function alphabetastock(), its called first to calculate all values from alpha and beta, after that the values are saved in a data frame abstocksna, after that the function iterate_predicted_stocks() starts to iterate and delivery each value to our function train_and_normalize(), seeing that, we offer the initial date, final date, test set(30), predicted value for future(10) and the 5 is the quantity used for <a href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)">cross validation.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wlr6_MERc5XqLEVw6r9mOA.png" /><figcaption>Predicting 500 stocks</figcaption></figure><p><strong>Conclusion</strong></p><p>As a result we are able to have the predicted values for the days chosen, in this case we will work with 5 days, and we have the Beta Variance and Alpha to help us to develop our stock wallet. It’s possible to increment this data frame with: high values, min value, average and the sector of each to use more statistical inference to identify and apply particle swarm algorithms or it’s possible to use a multi-criteria decision to identify the best future stock portfolio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JSL_kDbk2a9sDIUC2OuzVA.png" /><figcaption>Plot beta variance vs last day predicted stock</figcaption></figure><p><strong>Next Steps</strong></p><ul><li>Increment the final data frame with new values to have a better decision maker.</li><li>Multi criteria algorithm to define a stock portfolio.</li><li>Optimize future portfolio..</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=443679e6f41e" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/using-arima-algorithm-to-forecast-almost-500-stocks-from-s-p-500-index-443679e6f41e">Time Series ARIMA algorithm to forecast almost 500 stocks from S&amp;P 500 index</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create Virtual Machines Using PowerShell from Azure]]></title>
            <link>https://gabrielhribeiro.medium.com/create-virtual-machines-using-powershell-from-azure-cadd72759145?source=rss-cd385a51120f------2</link>
            <guid isPermaLink="false">https://medium.com/p/cadd72759145</guid>
            <dc:creator><![CDATA[Gabriel Ribeiro]]></dc:creator>
            <pubDate>Fri, 26 Mar 2021 01:36:14 GMT</pubDate>
            <atom:updated>2021-03-26T01:36:14.900Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>PowerShell</strong> for a system administrator it’s a “<strong>ace up sleeve</strong>” but its not crucial, you can normally create your own Virtual machine using the portal of <strong>azure</strong> and build in with some clicks, but for a faster productivity you can use <strong>PowerShell</strong>. I gonna show you how to create your VM<strong> step-by-step</strong> with some lines of code. <strong>First of all open your azure command line mode.</strong></p><h3><strong>Create a Resource Group</strong></h3><p>Resource group help us to organize and identify from where the resource comes from, its also usefull to make changes in a group of resoucers in the same time.</p><pre>New-AzResourceGroup -Name RGtio004 -Location brazilsouth</pre><h3><strong>Declare Variables</strong></h3><p>Each variable has keys and values and compose the structure that i need to execute the functions in the final step.</p><pre>#Name of your VM<br>$vmName = ‘tio004vm2’</pre><pre>#vmSize<br>$vmSize = ‘Standard_DS2_v2’</pre><pre>#What resource group it would you like to use<br>$resourceGroup = Get-AzResourceGroup -Name ‘RGtio004’</pre><pre>#When you specify a location for the resource group, you are #specifying where that metadata is stored. For compliance reasons, #you may need to ensure that your data is stored in a particular #region.<br>$location = $resourceGroup.Location</pre><pre>#get AvailabilitySet to the resource group as AVset004<br>$availabilitySet = Get-AzAvailabilitySet -ResourceGroupName $resourceGroup.ResourceGroupName -Name ‘AVset004&#39;</pre><pre>#Gets a virtual network in a resource group.<br>$vnet = Get-AzVirtualNetwork -Name ‘tio004-vnet’ -ResourceGroupName<br>$resourceGroup.ResourceGroupName</pre><pre>#Gets a subnet virtual network from that vnet variable. <br>$subnetid = (Get-AzVirtualNetworkSubnetConfig -Name &#39;subrede0&#39; -VirtualNetwork $vnet).Id</pre><pre>#Set a Security Network group for the resource group. <br>$nsg = New-AzNetworkSecurityGroup -ResourceGroupName $resourceGroup.ResourceGroupName -Location $location -Name “$vmName-nsg”</pre><pre>#Set the public address for the vm.<br>$pip = New-AzPublicIpAddress -Name &quot;$vmName-ip&quot; -ResourceGroupName $resourceGroup.ResourceGroupName -Location $location -AllocationMethod Dynamic</pre><pre>#Set the network interface<br>$nic = New-AzNetworkInterface -Name &quot;$($vmName)$(Get-Random)&quot; -ResourceGroupName $resourceGroup.ResourceGroupName -Location $location - SubnetId $subnetid -PublicIpAddressId $pip.Id -NetworkSecurityGroupId $nsg.Id</pre><pre>#Set Admin user<br>$adminUsername = ‘Administrator’</pre><pre>#Set Password user<br>$adminPassword = ‘qwe,Administrator@123’</pre><pre>#Set Administrator credentials<br>$adminCreds = New-Object PSCredential $adminUsername, <br>($adminPassword | ConvertTo-SecureString -AsPlainText -Force)</pre><pre>#Name of your service to be published<br>$publisherName = ‘MicrosoftWindowsServer’</pre><pre>#Name of you OS<br>$offerName = ‘WindowsServer’</pre><pre>#Name of SKU, value to identify spec about that service. <br>$skuName = ‘2016-Datacenter’</pre><pre>#Get the disk type<br>$osDiskType = (Get-AzDisk -ResourceGroupName $resourceGroup.ResourceGroupName)[0].Sku.Name</pre><pre>#Set VM configuration<br>$vmConfig = New-AzVMConfig -VMName $vmName -VMSize $vmSize -AvailabilitySetId $availabilitySet.Id</pre><h3>Create Your VM</h3><p>In this step we will execute commands to create our VM</p><pre>#Create Network Interface<br>Add-AzVMNetworkInterface -VM $vmConfig -Id $nic.Id</pre><pre>#Set operating system properties for a virtual machine<br>Set-AzVMOperatingSystem -VM $vmConfig -Windows -ComputerName $vmName -Credential $adminCreds</pre><pre>#Specifies the platform image to use for a virtual machine<br>Set-AzVMSourceImage -VM $vmConfig -PublisherName $publisherName -Offer $offerName -Skus $skuName -Version &#39;latest&#39;</pre><pre>#Operating system disk properties on a virtual machine<br>Set-AzVMOSDisk -VM $vmConfig -Name &quot;$($vmName)_OsDisk_1_$(Get-Random)&quot; -StorageAccountType $osDiskType -CreateOption fromImage</pre><pre>#Modifies boot diagnostics properties of a virtual machine.<br>Set-AzVMBootDiagnostic -VM $vmConfig -Disable</pre><pre>#Creates a virtual machine<br>New-AzVM -ResourceGroupName $resourceGroup.ResourceGroupName -Location $location -VM $vmConfig</pre><p>This all process can be execute in 3 steps with just a copy and past. First we create the resource group than create the variables for finally create the virtual machine. Ok. Now you can visit your dashboard at Azure and your resources are all be created.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cadd72759145" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Audit Azure using Tags and Policy]]></title>
            <link>https://gabrielhribeiro.medium.com/audit-azure-using-tags-and-policy-ec6f40a6acb4?source=rss-cd385a51120f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec6f40a6acb4</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-policies]]></category>
            <category><![CDATA[azure-tags]]></category>
            <dc:creator><![CDATA[Gabriel Ribeiro]]></dc:creator>
            <pubDate>Tue, 23 Mar 2021 17:17:29 GMT</pubDate>
            <atom:updated>2021-03-23T17:17:29.700Z</atom:updated>
            <content:encoded><![CDATA[<pre>Compliance and Security</pre><p><strong>How to set a policy to show us what kind of service is enabled without a tag.</strong></p><p><strong>Tag is usefull to show us who is using certain service on azure, tags help us to mark each crew or a operational department and enable us to generate repports to identify resources and costs.</strong></p><p>1ª- Open Powershell on azure painel and create tags:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*1bhGHW0luW0aILu3xWDpbA.png" /><figcaption>power shell</figcaption></figure><p><strong>a - </strong>Create tag:<strong> az tag create — name Equipe</strong></p><p><strong>b - </strong>add a value to that tag, in this case its my crew of development(dev01)<strong>:</strong></p><p><strong>az tag add-value — name Equipe — value dev01</strong></p><p>2ª- Inside of your resource group, select “definitions” and search for “tag”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*0r4o4Y4bBP6VbUfFpeq7Jw.png" /></figure><p>This Policy force who is in the resource group need to have a tag configured, but we will alter this definition policy to show us who <strong>dosent have tag</strong>.</p><p>Choose: <strong>require a tag on resource</strong></p><p>3ª- Duplicate definition</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*2VrH7FeCOdnFDW1MyOYRHQ.png" /></figure><p>Fill the values with:</p><ul><li>Definition location: choose your subscription</li><li>Name: can choose what you want</li><li>Description: can choose what you want</li><li>Category: Create new</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*4ggdq9MsZY0qkA7IbPbiqQ.png" /></figure><p>Scroll down the page and search for json configuration “effect” parameter and alter “deny” to “audit”, like the screenshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/1*kFdMqCoTKFxbp1vHvYbSXQ.png" /><figcaption>change from deny to audit</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*GUcEOlC8_F4uxmVnHKreaw.png" /><figcaption>we want to audit and not deny</figcaption></figure><p>Ok. And now you can save this definition.</p><p>4ª- Now its the time to “Assign Policy” to the resource group that you want to audit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*pDiQqB4uZbHboJFus-PMbg.png" /></figure><p>5ª- Choose in <strong>policy definition</strong>, the policy that we have created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*MRYRCEYyU_Jx1WAzzb6Agw.png" /></figure><p>6ª- Choose a name for your rule in parameters. And save it.</p><figure><img alt="asdasdasd" src="https://cdn-images-1.medium.com/max/637/1*JqYh6JoW3MbO3TLATLUtDw.png" /><figcaption>qwewqe</figcaption></figure><p><strong>Conclusion:</strong></p><p>After creation the policy and assign to the compliance in the resource group, the policy will be activate. As you can see i had 6 objects in my resource group without tag associate, this overview its essentially to identify resources and help to calculate costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C8ry0Q0RT5z0h0xqBBKjww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zu4qxH5LOuM9AlNDJEoCmw.png" /></figure><p>See you in the next. :D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec6f40a6acb4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Security aspects of Deno RunTime]]></title>
            <link>https://gabrielhribeiro.medium.com/security-aspects-of-deno-runtime-c24fa4420222?source=rss-cd385a51120f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c24fa4420222</guid>
            <category><![CDATA[node]]></category>
            <category><![CDATA[deno]]></category>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[node-js-tutorial]]></category>
            <dc:creator><![CDATA[Gabriel Ribeiro]]></dc:creator>
            <pubDate>Sun, 14 Jun 2020 19:28:44 GMT</pubDate>
            <atom:updated>2020-06-14T20:07:14.137Z</atom:updated>
            <content:encoded><![CDATA[<h3>Security aspects about Deno RunTime</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*wp3u15CFt2MsxC9Gan28EQ.jpeg" /><figcaption>https://deno.land/</figcaption></figure><p><a href="https://deno.land/">Deno - A modern runtime for JavaScript and TypeScript</a></p><h4><strong>Deno its a new engine JS/TS that come with some improvements about Node.js and one of the most important feature from Deno its your security aspects, also its important remember that Deno is written in Rust.</strong></h4><h4>Lesser Privilege Principle</h4><p>Deno has Secure by default. No file, network, or environment access, unless explicitly enabled, execution of other scripts. Its nice to see developer looking arround security, and deno strictly follows the CISA guidelines on the rules of the Principle of Least Privilege, than this allows Deno to be a favorite tool for future big companies.</p><blockquote>Definition 13–1. The Principle of Least Privilege states that a subject should be given only those privileges needed for it to complete its task. — <a href="https://www.dhs.gov/cisa/about-cisa">Cybersecurity and Infrastructure Security Agency (CISA)</a>.</blockquote><h4>SANDBOX</h4><p>The developer <a href="https://www.google.com/search?q=ryan+dahl&amp;stick=H4sIAAAAAAAAAOPgE-LVT9c3NMwwzE42zjCwUOLSz9U3yEiPrzBM09LIKLfST87PyUlNLsnMz9Mvzk8rKU8sSrXKL8pMz8xLzFFILC3JyC8qXsTKWVSZmKeQkpiRs4OVEQBNVqMsVgAAAA&amp;sa=X&amp;ved=2ahUKEwje_tX1_4HqAhWdIrkGHW2OAYMQmxMoATAdegQIBhAD">Ryan Dahl</a> used the same architecture from browsers ”sandbox” where any operation that is called in the context of the sandbox must be authorized. About this topic, the developer can be easy build micro services architecture with security mind, where each piece of software only has permission what it need. Example: access some specific path of environment or just network access.</p><h4>Node.js weakless sec.</h4><p>Otherwise Node.js for default come with full permission, and makes node.js unsafe in some ways, for example: When a developer install a lib and someone modified the lib to send information about password and user from server node.js after the first execution of code. With a White list from Deno this scenario cant occur because of his strong policy of security(Principle of Least Privilege).</p><h4>it&#39;s too early to say</h4><p>Yeah, it’s early to say, Deno will be a tool used by companies, but lets wait what way the community of developers will choose.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c24fa4420222" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transfer data from SQL database to another SQL database —  DBLINK]]></title>
            <link>https://gabrielhribeiro.medium.com/transfer-data-from-sql-database-to-another-sql-database-dblink-9328e18b6711?source=rss-cd385a51120f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9328e18b6711</guid>
            <category><![CDATA[transfers]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[dblink]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Gabriel Ribeiro]]></dc:creator>
            <pubDate>Fri, 01 Feb 2019 21:04:00 GMT</pubDate>
            <atom:updated>2020-01-15T20:48:29.163Z</atom:updated>
            <content:encoded><![CDATA[<h4>Transfer and verify data between two Micrososft SQL Database with dblink.</h4><p>In this article, I will show how you can do this easyly with DBLINK and some keys to transfer/insert and validate if that value exists in two SQL databases. Normally this kind of process is named by ETL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35VjEpkVEUX8KC6jKyCFbw.png" /></figure><h4><strong>Problem</strong></h4><p>In this week I had a problem. Basically I’ve needed to migrate a service that where a python script wich consumes data from a text file and insert the content into a SQL database. Now the data that was previoulsy migrated to a SQL database was needed to be inserted in another database. But this time without help of a python script.</p><h4><strong>Resolution</strong></h4><p>For this example, i used some values to validate if the data already exists in the database.<br> <br>In the example bellow I’ve used the argument [DATA_BASE_1]. This argument can be used as an unique ID. The existence of this ID column is necessary in the both tables.</p><p>First I will declare the following values to insert into the database:</p><p>INSERT INTO [IP].[DATABASE_1].[dbo].[TABLE_x]<br>( [DATA_1] ,[DATA_2] ,[DATA_3] ,[DATA_4])</p><p>Next step is the select from the original database:</p><p>SELECT [DATA_ORIG_1] ,[DATA_ORIG_2] ,[DATA_ORIG_3],[DATA_ORIG_4]<br>from [IP].[DATABASE_2].[dbo].[TABLE_y] as FR</p><p>the third part i made a validate if the values exists in table and the clause where i used the ID from the original data to validate the ID in the destination database:</p><p>and not exists (<br>SELECT [DATA_1],[DATA_2],[DATA_3] ,[DATA_4]<br>FROM [IP].[DATABASE_1].[dbo].[TABLE_x] as IS <br>where <br> [FR].[DATA_ORIG_1] = [IS].[DATA_1])</p><p><strong>Example of all code:</strong></p><p>INSERT INTO [IP].[DATABASE_1].[dbo].[TABLE_x]<br>( [DATA_1] ,[DATA_2] ,[DATA_3] ,[DATA_4])</p><p>SELECT [DATA_ORIG_1] ,[DATA_ORIG_2] ,[DATA_ORIG_3],[DATA_ORIG_4]<br>from [IP].[DATABASE_2].[dbo].[TABLE_y] as FR<br>and not exists (<br>SELECT [DATA_1],[DATA_2],[DATA_3] ,[DATA_4]<br>FROM [IP].[DATABASE_1].[dbo].[TABLE_x] as IS <br>where <br> [FR].[DATA_ORIG_1] = [IS].[DATA_1])</p><p><strong>NEXT STEPS</strong></p><p>After you write the insert with the clause, if you need to repeat this sql clause all day, all time or all minutes, you can create a Store procedure and schedule this SP in a JOB.</p><h3>Make automation easy</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9328e18b6711" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>