<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gigi Dattaradon on Medium]]></title>
        <description><![CDATA[Stories by Gigi Dattaradon on Medium]]></description>
        <link>https://medium.com/@gigi.dattaradon?source=rss-7c011fc22710------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-HZQLnPwyYic2fR2sioYNQ.png</url>
            <title>Stories by Gigi Dattaradon on Medium</title>
            <link>https://medium.com/@gigi.dattaradon?source=rss-7c011fc22710------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:52:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gigi.dattaradon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unveiling the Power of Recursive Feature Elimination: Enhancing Predictive Modeling Through Feature…]]></title>
            <link>https://medium.com/@gigi.dattaradon/unveiling-the-power-of-recursive-feature-elimination-enhancing-predictive-modeling-through-feature-17741400e212?source=rss-7c011fc22710------2</link>
            <guid isPermaLink="false">https://medium.com/p/17741400e212</guid>
            <category><![CDATA[predictive-modeling]]></category>
            <category><![CDATA[feature-selection]]></category>
            <category><![CDATA[model-optimization]]></category>
            <category><![CDATA[feature-importance]]></category>
            <category><![CDATA[feature-extraction]]></category>
            <dc:creator><![CDATA[Gigi Dattaradon]]></dc:creator>
            <pubDate>Thu, 04 Apr 2024 04:17:58 GMT</pubDate>
            <atom:updated>2024-04-04T04:17:58.734Z</atom:updated>
            <content:encoded><![CDATA[<h3>Unveiling the Power of Recursive Feature Elimination: Enhancing Predictive Modeling Through Feature Selection</h3><h3>Introduction</h3><p>In the realm of model creation and data preparation for predicting target variables, numerous features are typically inputted into models, ranging from a handful to over a hundred. Amidst this abundance, understanding and extracting the most impactful features becomes crucial. This article delves into one such technique for this purpose — Recursive Feature Elimination (RFE).</p><h3>Definition</h3><p>Recursive Feature Elimination is a method for feature selection that systematically removes the least significant features until reaching the desired number. The process commences with training a machine learning model, identifying the important features, discarding less important ones, and retraining the model with the remaining features. This iterative process continues until the desired feature count is attained.</p><p><strong>Advantages:</strong></p><ul><li>Utilizes a filtering method to extract the most meaningful features based on coefficients or feature importance attributes from the model.</li><li>Simple and straightforward to implement with any supervised models.</li></ul><p><strong>Limitations:</strong></p><ul><li>The process of selecting and iterating over all features can incur computational expenses when dealing with a large number of features.</li><li>This method overlooks correlations and interactions between features.</li></ul><h3>Sample</h3><p>Let’s examine this code, which conducts feature selection and model training utilizing Recursive Feature Elimination (RFE) with cross-validation to forecast average daily energy demand.</p><pre>## Read dataframe<br>usage = pd.read_csv(&#39;weather_n_energy_data.csv&#39;, encoding=&#39;utf-8&#39;)<br>print(usage.head())<br>usage.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*TI_t7nAROxUe69EJtOa6ww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*li_VLNBi1YZdear3ve-Ulg.png" /></figure><pre>## Find the best number of features<br>## Split data into feature and target variable<br>X = usage.drop(columns=[&#39;date&#39;,&#39;average_daily_demand&#39;])<br>y = usage[&#39;average_daily_demand&#39;]<br><br>## Split data into train and test set<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=42)<br><br>## define the model<br>estimator = RandomForestRegressor(random_state = 42)</pre><p>Initially, it reads weather and energy demand data. Subsequently, the data is divided into feature variables (X) and the target variable (y), with average daily demand serving as the target variable. The feature data (X) is then partitioned into training and testing sets using the ‘train_test_split’ function. In this instance, the RandomForestRegressor model is employed to predict the target variable.</p><pre>## Use RFE with Cross validation to find the best number of feature<br>selector = RFECV(estimator, step=1, cv=5, <br>                 scoring=&#39;neg_mean_squared_error&#39;)<br><br>selector.fit(X_train, y_train)<br>print(&quot;No of selected features:&quot;, selector.n_features_)<br><br># Draw the best number of features selected<br>visualizer = RFECV(estimator, step=1, cv=5, <br>                   scoring=&#39;neg_mean_squared_error&#39;)<br><br>visualizer.fit(X_train, y_train)<br><br>plt.figure(figsize=[5.6, 4])<br>visualizer.show()</pre><p>Recursive Feature Elimination with Cross-Validation (RFECV) is employed to identify the ideal number of features. This iterative procedure removes less significant features until the desired number is attained, with cross-validation serving to mitigate overfitting. The ‘selector’ object is then fitted to the training data to ascertain the optimal number of features, followed by printing the count of selected features. Additionally, a visual depiction of the selected features is generated using the ‘visualizer’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*J5j01bLEWG259pGHHc2nBw.png" /></figure><p>Once the optimal number of features has been determined, we proceed to train the model and extract the most valuable features. Below is the code snippet for feature selection using Recursive Feature Elimination (RFE) and model training.</p><pre>## Feature selection with Recursive Feature Elimination (RFE)<br>model = RandomForestRegressor(random_state = 42)<br><br>number_features = 5<br><br>rfe = RFE(estimator = model, n_features_to_select=number_features)<br>rfe.fit(X_train, y_train)<br><br># Get the ranking of each feature<br>feature_ranking = rfe.ranking_<br><br># Get the selected features<br>selected_features = X_train.columns[rfe.support_]<br><br>print(&#39;Selected Features:&#39; , selected_features)<br><br>model.fit(X_train[selected_features], y_train)<br><br>y_pred = model.predict(X_test[selected_features])<br>mse = mean_squared_error(y_test, y_pred)<br>mape = mean_absolute_percentage_error(y_test, y_pred) * 100<br>r_squared = r2_score(y_test, y_pred)<br>print(f&quot;RandomForest Regressor - Recursive Feature Elimination (RFE)&quot;)<br>print(f&quot;Mean Squared Error = {mse}&quot;)<br>print(f&quot;Mean Absolute Percentage Error = {mape: .2f}%&quot;)<br>print(f&quot;R-squared = {r_squared:.2f}&quot;)<br>plt.figure(figsize=(4,4))<br>plt.scatter(y_test, y_pred, alpha=0.5)<br>plt.xlabel(&#39;Actual&#39;)<br>plt.ylabel(&#39;Predicted&#39;)<br>p1 = max(max(y_pred), max(y_test))<br>p2 = min(min(y_pred), min(y_test))<br><br>plt.plot([p1, p2], [p1, p2], color = &#39;r&#39;, linestyle = &#39;--&#39;, lw=2)<br>plt.grid(True)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*xWBjFlS1dv7OSN1txdY8NQ.png" /></figure><h3>Conclusion</h3><p>In conclusion, the journey through feature selection and model training using Recursive Feature Elimination (RFE) has provided valuable insights into the significance of feature engineering in predictive modeling. By systematically identifying and leveraging the most impactful features, we can enhance model performance and gain deeper insights into complex datasets. The iterative nature of RFE, coupled with cross-validation techniques like RFECV, underscores the importance of robust methodologies in data science workflows. Moving forward, integrating such techniques into predictive modeling endeavors will undoubtedly contribute to more accurate predictions and informed decision-making in various domains.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17741400e212" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting Electricity Demand with Weather Data: A Machine Learning Approach]]></title>
            <link>https://medium.com/@gigi.dattaradon/forecasting-electricity-demand-with-weather-data-a-machine-learning-approach-80f7270bfd38?source=rss-7c011fc22710------2</link>
            <guid isPermaLink="false">https://medium.com/p/80f7270bfd38</guid>
            <category><![CDATA[predictive-modeling]]></category>
            <category><![CDATA[electricity]]></category>
            <category><![CDATA[regression-analysis]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Gigi Dattaradon]]></dc:creator>
            <pubDate>Fri, 08 Mar 2024 12:00:55 GMT</pubDate>
            <atom:updated>2024-03-08T12:08:52.232Z</atom:updated>
            <content:encoded><![CDATA[<p>Electricity demand forecasting is crucial for energy providers to efficiently manage resources and ensure reliable service to consumers. In this article, we’ll explore how weather data can be leveraged to predict average daily electricity using machine learning techniques. We’ll walk through the process of data preparation, model training, evaluation, and interpretation of results.</p><h3>Introduction</h3><p>Electricity demand fluctuates based on various factors, including weather conditions. By integrating weather data into predictive models, we can improve the accuracy of demand forecasts. In this analysis, we’ll use historical weather data alongside electricity demand data on 2021 in New South Wales, Australia, to train machine learning models for predicting average daily electricity demand.</p><h3>Data Preparation</h3><p>We begin by collecting two primary datasets: weather data and electricity demand data. The weather data includes parameters such as maximum temperature, minimum temperature, humidity, and solar radiation from 5 stations: Sydney Airport, Sydney Observatory Hill, Yamba Pilot, Broken Hill and Newcastle Nobbys Signal Station</p><pre>import pandas as pd<br>import numpy as np<br>import os<br>import datetime<br>import seaborn as sns<br>import matplotlib.pyplot as plt<br>from sklearn.model_selection import train_test_split<br>from sklearn.linear_model import LinearRegression<br>from sklearn.ensemble import RandomForestRegressor<br>from xgboost import XGBRegressor<br>from sklearn.metrics import mean_squared_error, mean_absolute_percentage_error, r2_score<br>pd.set_option(&#39;display.max.columns&#39;, 100)<br><br>## set the period of forcast in 2021<br>start_mth = datetime.datetime(2021, 1, 1)<br>end_mth = datetime.datetime(2021, 12, 31)<br><br>## Read Weather Data<br>weather = pd.read_csv(&#39;weather_data.csv&#39;, encoding=&#39;utf-8&#39;)<br>print(weather.head())<br><br># Pivot the weather dataframe<br>weather = weather.pivot(index=&#39;date&#39;, columns=&#39;station_name&#39;, values=[&#39;ev_transpiration&#39;,&#39;max_temp&#39;,&#39;min_temp&#39;,&#39;max_humid&#39;,&#39;min_humid&#39;,&#39;solar&#39;])<br><br># Flatten multi-index columns<br>weather.columns = [&#39;_&#39;.join(col).strip() for col in weather.columns.values]<br><br># Reset index<br>weather = weather.reset_index()<br><br># Set date column as date type<br>weather.date = pd.to_datetime(weather.date, format=&#39;%d/%m/%Y&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*NiqMtIcqPs2lGu0xMTcHyQ.png" /></figure><p>The electricity demand data consists of the total electricity demand recorded at specific intervals.</p><pre>## Read Eletricity Demand Data<br>usage = pd.read_csv(&#39;NSW_Price_Demand.csv&#39;, sep=&#39;,&#39;)<br>usage.columns = usage.columns.str.lower()<br>print(usage.head())<br><br>## Filter only date and total demand columns<br>selected_col = [&#39;settlementdate&#39;,&#39;totaldemand&#39;]<br>usage = usage[selected_col]<br><br>## convert column type<br>usage[&#39;settlementdate&#39;] = pd.to_datetime(usage[&#39;settlementdate&#39;])<br>usage[&#39;totaldemand&#39;] = usage[&#39;totaldemand&#39;].astype(float)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*U0GloPQXDzJbyKCq2iB32Q.png" /></figure><p>After loading the data, we preprocess it by filtering for the desired time period and aggregating the electricity demand to obtain average daily values. We then merge the weather and electricity demand datasets based on the date. This merged dataset forms the basis for our predictive modeling.</p><pre>## Filter usage within start and end month<br>usage = usage[( usage[&#39;settlementdate&#39;]&gt;=start_mth) &amp; (usage[&#39;settlementdate&#39;] &lt;= end_mth)]<br><br>## sum average daily demand and prepare data<br>usage.index = usage.settlementdate<br>usage_d = usage.resample(&#39;D&#39;).mean()<br>usage_d.settlementdate = pd.to_datetime(usage_d.settlementdate).dt.date<br>usage_d_column = [&#39;date&#39;, &#39;average_daily_demand&#39;]<br>usage_d.columns = usage_d_column<br>usage_d = usage_d.reset_index(drop=True)<br>usage_d.date = pd.to_datetime(usage_d.date)<br><br>## add month and day of week columns<br>usage_d[&#39;month&#39;] = pd.to_datetime(usage_d.date).dt.month<br>usage_d[&#39;day_of_week&#39;] = pd.to_datetime(usage_d.date).dt.dayofweek<br><br>### merge usage and weather dataframe together<br>merge = pd.merge(weather, usage_d, how = &#39;inner&#39;, on = &#39;date&#39; )<br>merge.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aB82TMyT5PHe3K4tfT1o2Q.png" /></figure><h3>Exploratory Data Analysis (EDA)</h3><p>Before diving into modeling, we conduct exploratory data analysis to understand the relationships between different variables. One way to visualize these relationships is through a correlation heatmap, which shows the correlation coefficients between pairs of variables. This helps us identify potential predictors for our models.</p><pre>## correlation heat map<br>merge_wo_date = merge.drop(columns=&#39;date&#39;)<br>corr_matrix = merge_wo_date.corr()<br><br>##plot the heat map<br>plt.figure(figsize=(15,8))<br>sns.heatmap(corr_matrix, annot = True, cmap = &#39;coolwarm&#39;, fmt = &quot;.2f&quot;, linewidths=.5, annot_kws={&#39;size&#39;:7})<br>plt.xticks(fontsize=8)<br>plt.yticks(fontsize=8)<br>plt.title(&#39;Correlation Heatmap&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oSJwh0C270-5HNpuXUfQAA.png" /></figure><h3>Model Selection and Training</h3><p>We experiment with three machine learning models for regression: Linear Regression, Random Forest Regressor, and XGBoost Regressor. After splitting the data into training and testing sets, we train each model using the training data and evaluate their performance on the test data.</p><pre>## Split data into feature and target variable<br>X = merge.drop(columns=[&#39;date&#39;,&#39;average_daily_demand&#39;])<br>y = merge[&#39;average_daily_demand&#39;]<br><br>## Split data into train and test set<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=42)<br><br>## Set the model dictionary<br>models = {<br>&#39;Linear Regression&#39;: LinearRegression()<br>, &#39;Random Forest Regressor&#39;: RandomForestRegressor(random_state = 42)<br>, &#39;XGBoost Regressor&#39; : XGBRegressor(random_state = 42)<br>}</pre><p><strong>Linear Regression:</strong></p><p>Linear regression is a fundamental statistical method used to establish the relationship between a dependent variable and one or more independent variables. It operates under the assumption of a linear relationship between the input features and the target variable. Linear Regression offers simplicity and interpretability, providing insights into how each feature contributes to the prediction. However, its linear nature may limit its ability to capture complex nonlinear relationships inherent in the data.</p><p><strong>Random Forest Regressor:</strong></p><p>Random Forest Regressor, a popular ensemble learning technique, constructs multiple decision trees during training. Each tree is built using a random subset of the training data and features. By aggregating predictions from these individual trees, Random Forest Regressor effectively handles nonlinear relationships and mitigates overfitting. Its robustness and efficiency make it well-suited for datasets with high dimensionality and varying complexities.</p><p><strong>XGBoost Regressor:</strong></p><p>XGBoost Regressor, an implementation of gradient boosting algorithms, sequentially builds a series of decision trees. Each subsequent tree aims to correct the errors of its predecessors, optimizing a specified loss function using a gradient descent algorithm. XGBoost Regressor excels in predictive performance, automatically handling missing values and outliers. Though computationally intensive, it often outperforms other algorithms in terms of accuracy and robustness.</p><h3>Results and Evaluation</h3><p>For each model, we assess performance metrics such as Mean Squared Error (MSE), Mean Absolute Percentage Error (MAPE), and R-squared (R²). These metrics provide insights into how well the models are able to predict electricity demand based on weather variables. Additionally, we visualize the predicted versus actual demand to further interpret model performance.</p><pre>## Predict and Evaluation<br>for model_name, model in models.items():<br>model.fit(X_train, y_train)<br>y_pred = model.predict(X_test)<br>mse = mean_squared_error(y_test, y_pred)<br>mape = mean_absolute_percentage_error(y_test, y_pred) * 100<br>r_squared = r2_score(y_test, y_pred)<br><br>## print model accuracy<br>print(f&quot;{model_name}&quot;)<br>print(f&quot;Mean Squared Error = {mse}&quot;)<br>print(f&quot;Mean Absolute Percentage Error = {mape: .2f}%&quot;)<br>print(f&quot;R-squared = {r_squared:.2f}&quot;)<br><br>## plot scatter, and line comparing test data and prediction<br>plt.figure(figsize=(6,6))<br>plt.scatter(y_test, y_pred, alpha=0.5)<br>plt.xlabel(&#39;Actual&#39;)<br>plt.ylabel(&#39;Predicted&#39;)<br>p1 = max(max(y_pred), max(y_test))<br>p2 = min(min(y_pred), min(y_test))<br>plt.plot([p1, p2], [p1, p2], color = &#39;r&#39;, linestyle = &#39; - &#39;, lw=2)<br>plt.grid(True)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*75nE3v7mWFHzjYW47UdSIA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*ti6R02A7mAonG_S9gcY5AA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*8fruApzhCQV1NrRoAAroJw.png" /></figure><h3>Conclusion</h3><p>In this analysis, we demonstrated the utility of weather data in predicting average daily electricity demand. By employing machine learning techniques, we developed models capable of forecasting demand with reasonable accuracy. These models can aid energy providers in making informed decisions regarding resource allocation and operational planning.</p><p>Looking ahead, further refinement of the models and incorporation of additional features could enhance predictive performance. Additionally, ongoing monitoring and evaluation will be essential to ensure the reliability and effectiveness of the forecasting system in real-world scenarios.</p><p><strong>Data Source</strong>:</p><p>The electricity data originates directly from <a href="https://aemo.com.au/energy-systems/electricity/national-electricity-market-nem/data-nem/aggregated-data">Australian Energy Market Operator</a> (AEMO) website, while the weather data is obtained from <a href="http://www.bom.gov.au/climate/data/index.shtml?bookmark=201">Australian Bureau of Meteorology</a> (BOM) website</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80f7270bfd38" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Accessing Electricity Demand and Price Data from Australian Energy Market Operator (AEMO)]]></title>
            <link>https://medium.com/@gigi.dattaradon/accessing-electricity-demand-and-price-data-from-australian-energy-market-operator-aemo-e42c70f0a008?source=rss-7c011fc22710------2</link>
            <guid isPermaLink="false">https://medium.com/p/e42c70f0a008</guid>
            <category><![CDATA[electricity-usage]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[aemo]]></category>
            <category><![CDATA[australian-energy]]></category>
            <dc:creator><![CDATA[Gigi Dattaradon]]></dc:creator>
            <pubDate>Sat, 02 Mar 2024 01:40:36 GMT</pubDate>
            <atom:updated>2024-03-02T01:40:36.441Z</atom:updated>
            <content:encoded><![CDATA[<p>As we delve into the intricate world of energy markets, understanding the dynamics of electricity demand and pricing becomes paramount. The <a href="https://aemo.com.au/">Australian Energy Market Operator</a> (AEMO) serves as a crucial source of real-time data, offering insights into the consumption patterns and pricing structures across various regions within Australia. In this article, we’ll explore how we can automate the retrieval of this vital information from AEMO’s repository for analysis and decision-making purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*PYJieQPP0weeyxkXmDqRug.png" /></figure><h3><strong>Introduction to AEMO Data</strong></h3><p>AEMO maintains a comprehensive dataset that encapsulates the demand and price of electricity utilized in each state of Australia. This dataset spans from the inception of the National Electricity Market (NEM) in 1998 to the most recent monthly observations. Each monthly file contains relevant details such as the region name, date and time stamps, total demand, regional reference price (RRP), and period type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*iTz5D77icCpLDgGxi1bp5Q.png" /></figure><h3><strong>Automating Data Retrieval: The Code</strong></h3><p>Let’s dissect the Python code snippet provided to automate the downloading of these monthly datasets:</p><pre>import numpy as np<br> import pandas as pd<br> import datetime<br> import requests<br> import os<br> import time<br> <br> # Define the start and end date range for data retrieval<br> startdate = &#39;2013–01–01&#39;<br> enddate = &#39;2023–12–31&#39;<br> daterange = pd.date_range(startdate, enddate, freq=&#39;MS&#39;).strftime(&#39;%Y%m&#39;).tolist()<br> <br> # Specify the directory to store downloaded files<br> directory = &#39;C:\\Users\\Windows\\Documents\\&#39; ## replace with our local directory<br> <br> # Define the root URL for dataset retrieval<br> root_url = &#39;https://aemo.com.au/aemo/data/nem/priceanddemand/PRICE_AND_DEMAND_&#39;<br> <br> # Generate a list of URLs to download data for each state and month<br> urls = [root_url + i + &#39;_NSW1.csv&#39; for i in daterange] ## replace with the states we want to download files<br> <br> # Define user-agent headers to mimic browser behavior<br> headers = {&#39;User-Agent&#39;: &#39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3&#39;}<br> <br> # Iterate through each URL and download the corresponding file<br> for url in urls[:]:<br> # Extract the filename from the URL<br> filename = os.path.join(directory, os.path.basename(url))<br> <br> # Send a GET request to the URL<br> response = requests.get(url, headers=headers)<br> <br> # Check if the request was successful (status code 200)<br> if response.status_code == 200:<br> # Open a file and write the content of the response to it<br> with open(filename, &#39;wb&#39;) as f:<br> f.write(response.content)<br> print(f&quot;File downloaded successfully: {filename}&quot;)<br> time.sleep(5) # Introduce a delay to be considerate of server load<br> else:<br> print(f&quot;Failed to download the file {filename}. Status code: {response.status_code}&quot;)</pre><h3><strong>Understanding the Code</strong></h3><ol><li>Date Range Generation: We define a start and end date to specify the time period for which we want to retrieve data. Using pd.date_range, we generate a list of month-year strings (YYYYMM) within this range.</li><li>URL Construction: We construct URLs for each month and state using the root URL provided by AEMO and appending the respective month-year and state code (NSW1 in this case).</li><li>Downloading Files: We iterate through each URL, sending a GET request to download the corresponding CSV file. If the request is successful (status code 200), we save the file to the specified directory.</li><li>User-Agent Headers: We include user-agent headers to mimic browser behavior, ensuring smooth retrieval without triggering any security measures on the server.</li><li>Delay: To be considerate of server load, we introduce a delay of 5 seconds between consecutive requests using time.sleep.</li></ol><h3><strong>Conclusion</strong></h3><p>Automating the retrieval of electricity demand and price data from AEMO’s repository facilitates efficient analysis and decision-making processes. By leveraging Python scripting, we can streamline the process of acquiring these vital insights, enabling stakeholders to make informed choices in the dynamic landscape of energy markets. As we continue to harness the power of data-driven approaches, initiatives like these underscore the importance of automation in modern-day energy management and policy formulation.</p><p><strong>Data Source</strong>: Access this valuable data source directly from AEMO’s website <a href="https://aemo.com.au/energy-systems/electricity/national-electricity-market-nem/data-nem/aggregated-data">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e42c70f0a008" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Forecasting: Anticipating Electricity Demand Patterns in New South Wales, Australia]]></title>
            <link>https://medium.com/@gigi.dattaradon/time-series-forecasting-anticipating-electricity-demand-patterns-in-new-south-wales-australia-1d1453d5e81c?source=rss-7c011fc22710------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d1453d5e81c</guid>
            <category><![CDATA[sarimax]]></category>
            <category><![CDATA[holt-winters]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[fbprophet]]></category>
            <category><![CDATA[electricity-usage]]></category>
            <dc:creator><![CDATA[Gigi Dattaradon]]></dc:creator>
            <pubDate>Thu, 29 Feb 2024 00:31:37 GMT</pubDate>
            <atom:updated>2024-03-21T00:43:53.700Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>Understanding and predicting electricity demand is crucial for ensuring the stability and efficiency of power systems. In this article, we delve into the analysis of electricity usage data from New South Wales (NSW), Australia, spanning from 2015 to 2023. We employ various time series forecasting models to anticipate future electricity demand and evaluate their performance.</p><h3><strong>Data Overview and Preprocessing</strong></h3><p>The dataset comprises electricity usage records including the date of measurement, total demand, and the corresponding regional reference price (RRP). After loading the data and converting relevant columns to appropriate data types, we focus on electricity demand for analysis.</p><pre>import pandas as pd<br>import numpy as np<br>import os<br>import datetime <br>import warnings <br>warnings.filterwarnings(&#39;ignore&#39;)<br>import matplotlib.pyplot as plt<br><br>from statsmodels.tsa.api import ExponentialSmoothing<br>from statsmodels.tsa.stattools import adfuller<br>from statsmodels.tsa.seasonal import seasonal_decompose<br>from statsmodels.tsa.stattools import acf, pacf<br>from pmdarima import auto_arima<br>from statsmodels.tsa.statespace.sarimax import SARIMAX<br>from fbprophet import Prophet<br><br># Set the period of forecast from 2016 to 2023<br>start_mth = datetime.datetime(2016, 1, 1)<br>end_mth = datetime.datetime(2023, 12, 1)<br><br># Read file<br>usage = pd.read_csv(&#39;NSW_Price_Demand.csv&#39;, sep=&#39;,&#39;)<br>usage.columns = usage.columns.str.lower()<br><br># Convert column type<br>selected_col = [&#39;settlementdate&#39;,&#39;totaldemand&#39;,&#39;rrp&#39;]<br>usage = usage[selected_col]<br><br>usage[&#39;settlementdate&#39;] = pd.to_datetime(usage[&#39;settlementdate&#39;])<br>usage[&#39;day&#39;] = usage[&#39;settlementdate&#39;].dt.day<br>usage[&#39;month&#39;] = usage[&#39;settlementdate&#39;].dt.month<br>usage[&#39;year&#39;] = usage[&#39;settlementdate&#39;].dt.year<br><br>usage[&#39;totaldemand&#39;] = usage[&#39;totaldemand&#39;].astype(float)<br>usage[&#39;rrp&#39;] = usage[&#39;rrp&#39;].astype(float)<br><br>usage.index = usage.settlementdate<br>print(usage.info())<br>print(usage.head(2))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*vCU18YUtCC6DRaPYk00wMA.png" /></figure><h3><strong>Exploratory Data Analysis</strong></h3><p>We begin by visualizing the monthly total demand over the years. A line plot depicting the mean usage per month illustrates any underlying trends or seasonal patterns. This analysis provides insights into the overall consumption behavior over the years.</p><pre>## data preparation<br>## Filter usage within start and end month<br>usage = usage[( usage[&#39;settlementdate&#39;]&gt;=start_mth) &amp; (usage[&#39;settlementdate&#39;] &lt;= end_mth)]<br><br>## sum average daily demand by month<br>usage.index = usage.settlementdate<br>usage_mth = usage.resample(&#39;M&#39;).mean()<br><br># Plot data - mean usage per month<br>pivot_df = usage_mth.pivot_table(index=&#39;month&#39;, columns=&#39;year&#39;, values=&#39;totaldemand&#39;)<br><br># Plot the line chart<br>pivot_df.plot(kind=&#39;line&#39;, marker=&#39;o&#39;, figsize=(13, 5))<br>plt.title(&#39;Total Demand by Month (Year over Year)&#39;)<br>plt.xlabel(&#39;Month&#39;)<br>plt.ylabel(&#39;Total Demand&#39;)<br>plt.xticks(range(1, 13))  <br>plt.grid(True)<br>plt.legend(title=&#39;Year&#39;, loc=&#39;best&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g-M-nrL8Am687sK57itsdA.png" /></figure><h3><strong>Time Series Analysis</strong></h3><p>To ensure the suitability of our data for time series analysis, we conduct the Dickey-Fuller test to check for stationarity. Additionally, we decompose the time series into its trend, seasonality, and residual components to gain a deeper understanding of its structure.</p><pre># Check if time series data is stationary<br>def df_test(time_series):<br>    print(&#39;Results of Dickey-Fuller Test:&#39;)<br>    dftest = adfuller(time_series, autolag = &#39;AIC&#39;)<br>    dfoutput = pd.Series(dftest[0:4], index=[&#39;Test Statistic&#39;, &#39;p-value&#39;, &#39;#Lags Used&#39;, &#39;Number of Observations Used&#39;])<br>    for key, value in dftest[4].items():<br>        dfoutput[&#39;Critical Value (%s)&#39; %key] = value<br>    print (dfoutput)<br>    <br># Apply the function to the dataset<br>df_test(usage_mth.totaldemand)<br><br># Check the time series trend, seasonality and residuals (noise component) and analyze.<br>decomp = seasonal_decompose(usage_mth.totaldemand)<br>fig = decomp.plot()<br>fig.set_size_inches(10,6)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*4H-X_qToZSfbmBDi4o_VGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ViIe_iI1OLS3qabykyTk8w.png" /></figure><p>The results of the Dickey-Fuller Test are crucial in determining the stationarity of the time series data. A high p-value of 0.889186 suggests that the null hypothesis, which assumes the presence of a unit root (indicating non-stationarity), cannot be rejected at conventional significance levels. This indicates that the time series data for electricity demand in New South Wales is likely non-stationary, meaning it exhibits trends or seasonality that may require further analysis and adjustment before applying certain time series forecasting models. Given the non-stationarity of the time series data, we’ll bypass the Dickey-Fuller Test result for now and proceed directly to utilizing the data for forecasting.</p><h3><strong>Forecasting Models</strong></h3><p>We employ three distinct forecasting models:</p><h4><strong>Holt Winter Model</strong></h4><p>The Holt Winter model utilizes Exponential Smoothing to capture both trend and seasonality in the data. By training on historical data, it learns patterns and is capable of forecasting future electricity demand accurately.</p><h4><strong>SARIMAX Model</strong></h4><p>The SARIMAX model combines Auto Regressive Integrated Moving Average (ARIMA) with exogenous variables, allowing it to capture both the temporal dependencies within the data and external factors that influence electricity demand. By identifying optimal parameters through techniques like auto_arima, it provides forecasts with seasonal adjustments.</p><h4><strong>Prophet Model</strong></h4><p>The Prophet model, developed by Facebook, is adept at handling irregularities such as holidays and outliers in time series data. By automatically detecting and incorporating seasonality, trends, and holiday effects, it offers robust predictions for electricity demand, making it a powerful tool for forecasting.</p><pre>## data processing for model<br>forecast_mths = 12 ## set the forecast next 12 months<br>usage_mth[&#39;ds&#39;] = usage_mth.index<br><br>train = usage_mth[ : -forecast_mths]<br>test = usage_mth[-forecast_mths :]<br><br>y_train = train[&#39;totaldemand&#39;]<br>y_test = test[&#39;totaldemand&#39;]<br><br>print(&#39;Train size: &#39; + str(train.shape[0]) +   &#39; , Test size: &#39; + str(test.shape[0]))<br><br>## forecast the monthly usage<br>## 1. Holt Winter Forecast<br>hw_model = ExponentialSmoothing(y_train, seasonal_periods = 12, trend =&#39;mul&#39;, seasonal = &#39;mul&#39;)<br>hw_fit = hw_model.fit()<br>hw_pred = hw_fit.forecast(len(y_test))<br><br>## 2. ARIMAX Forecast <br>## Find the best arimax<br>sarimax_model = auto_arima(y_train, start_p = 0, start_q = 0<br>                                ,max_p = 3, max_q = 3<br>                                , m = 12 #12 is the frequncy of the cycle<br>                                , test = &#39;adf&#39;, seasonal = True #set to seasonal<br>                                , d=1, D=1, trace = True, error_action = &#39;ignore&#39;<br>                                , suppress_warnings = True, stepwise = True)<br><br>model_order = sarimax_model.get_params()[&#39;order&#39;]<br>model_seasonal_order = sarimax_model.get_params()[&#39;seasonal_order&#39;]<br><br>sr_model = SARIMAX(y_train, order = model_order, seasonal_order = model_seasonal_order)<br>sr_model_fit = sr_model.fit()<br># print(sarimax_model_fit.summary()) ## uncomment this line to review model summary<br><br>sr_pred = sr_model_fit.forecast(len(y_test))<br><br># # 3. Phophet Model<br># ## data processing for model<br>p_train = train[[&#39;ds&#39;, &#39;totaldemand&#39;]]<br>p_train.columns = [&#39;ds&#39;,&#39;y&#39;]<br><br>## Prophet model<br>pp_model = Prophet()<br>pp_model.fit(p_train)<br>pp_pred = pp_model.make_future_dataframe(periods=len(y_test), freq=&#39;M&#39;)<br>pp_pred = pp_model.predict(pp_pred)<br>pp_pred.index = pp_pred.ds<br>pp_pred = pp_pred.iloc[-len(y_test):, ][&#39;yhat&#39;].values<br><br>## 4.Calculate Mean Absolute Percentage Error and Root Mean Square Error<br>def accuracy(y_test, y_pred):<br>    mape = np.mean(np.abs(( y_test - y_pred) / y_test)*100)<br>    rmse = np.sqrt(np.mean((y_test - y_pred)**2 ))<br>    return(mape, rmse)<br><br>## 5.Review the result from model<br>models = {<br>    &#39;Holt Winter&#39;: hw_pred<br>    ,&#39;ARIMAX&#39;: sr_pred<br>    ,&#39;Prophet&#39;: pp_pred<br>}<br><br>plt.figure(figsize=(12,5))<br><br>for name, y_pred in models.items():<br>    mape, rmse = accuracy(y_test, y_pred)<br>    print(&#39;================================================================&#39;)<br>    print(name)<br>    print(&#39;Mean Absolute Percentage Error (MAPE) : {:.2f}%&#39;.format(mape))<br>    print(&#39;Root Mean Square Error : {}&#39;.format(rmse))<br>    <br>    # Plot forecast<br>    plt.plot(test.index, y_pred, &#39;x-&#39;, label=name)  # Predictions<br><br>plt.plot(train.index, y_train, &#39;b.-&#39;, label=&#39;Train&#39;)  # Train data<br>plt.plot(test.index, y_test, &#39;r.-&#39;, label=&#39;Test&#39;)  # Test data<br>plt.xlabel(&#39;Time&#39;, fontsize=10)<br>plt.ylabel(&#39;Electricity Demand&#39;, fontsize=10)<br>plt.legend(loc=&#39;best&#39;)<br>plt.title(&#39;Electricity Demand Forecast&#39;, fontsize=12)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*VXSjyN_j61rN_ZIXi-I3sg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GF7c2fx2aq2ALyWUuaGR7g.png" /></figure><h3><strong>Model Evaluation</strong></h3><p>After generating forecasts, we evaluate the performance of each model using Mean Absolute Percentage Error (MAPE) and Root Mean Square Error (RMSE). These metrics quantify the accuracy of predictions and help us determine the most effective model for our dataset.</p><h3><strong>Results and Conclusion</strong></h3><p>Upon evaluating the models, we observe their respective strengths and weaknesses in forecasting electricity demand. While each model offers unique insights, the choice of the most suitable model depends on the specific requirements and characteristics of the dataset.</p><p>In conclusion, forecasting electricity usage is a complex yet essential task for energy management and planning. By leveraging advanced time series forecasting techniques like ARIMAX and Prophet, we can anticipate future demand with greater accuracy, enabling stakeholders to make informed decisions and ensure the efficient operation of power systems in NSW, Australia, and beyond.</p><p><strong>Data Source</strong>: The dataset used in this analysis was obtained from the <a href="https://aemo.com.au/energy-systems/electricity/national-electricity-market-nem/data-nem/aggregated-data">Australian Energy Market Operator (AEMO)</a>, providing valuable insights into electricity usage dynamics in the New South Wales region.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d1453d5e81c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advancing Weather Prediction : Comprehensive Classification Models Using Python]]></title>
            <link>https://medium.com/@gigi.dattaradon/weather-prediction-with-python-a-comprehensive-analysis-of-classification-models-f7149d39995d?source=rss-7c011fc22710------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7149d39995d</guid>
            <category><![CDATA[weather-forecasts]]></category>
            <category><![CDATA[comprehensive-analysis]]></category>
            <category><![CDATA[rainforest]]></category>
            <category><![CDATA[classification-models]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Gigi Dattaradon]]></dc:creator>
            <pubDate>Mon, 26 Feb 2024 02:17:30 GMT</pubDate>
            <atom:updated>2024-02-26T03:39:00.812Z</atom:updated>
            <content:encoded><![CDATA[<p>In the previous article, “<a href="https://medium.com/@gigi.dattaradon/automating-weather-prediction-with-python-a-data-science-approach-using-logistic-regression-2b66c9d044cb">Automating Weather Prediction with Python: A Data Science Approach Using Logistic Regression</a>”, we discussed the foundational steps of automating weather prediction using Python, focusing on building a simple weather prediction model using logistic regression. Building upon that foundation, this article delves deeper into the topic by exploring various classification models to predict rainfall based on meteorological data. These models include K-Nearest Neighbors, Decision Tree, Random Forest, Gradient Boosting, and Deep Learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*6AQYXHnFMumxqO2FgRNqQQ.png" /></figure><p><strong>First Stage: Data Preparation</strong></p><p>First and foremost, the process of automating weather prediction begins with loading the necessary files containing the meteorological data. Subsequently, the data undergoes cleaning and preprocessing procedures to ensure consistency and reliability in the analysis. This includes converting specific columns to numeric data types, parsing dates into a standardized format, and sorting the data based on station name and date for organizational purposes. Additionally, missing values within numeric columns are addressed by replacing them with the respective column means to maintain data integrity. These crucial steps lay the groundwork for further analysis and model development in the automation of weather prediction.</p><pre>import pandas as pd<br>import numpy as np<br>import os<br>from sklearn.preprocessing import MinMaxScaler<br>from sklearn.model_selection import train_test_split<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.neighbors import KNeighborsClassifier<br>from sklearn.tree import DecisionTreeClassifier<br>from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier<br>import tensorflow as tf<br>from tensorflow import keras<br>from sklearn.metrics import accuracy_score, classification_report, confusion_matrix<br>from sklearn.exceptions import ConvergenceWarning<br>import warnings<br><br># Ignore convergence warnings<br>warnings.filterwarnings(&#39;ignore&#39;, category=ConvergenceWarning)<br><br># Read the file<br>filename = &#39;NSW_weather_data_12mth.csv&#39;<br>observe = pd.read_csv(filename, sep=&#39;,&#39;, index_col=0)<br><br># Data cleaning and preprocessing<br>int_col = [&#39;ev_transpiration&#39;, &#39;rain&#39;, &#39;pan_ev&#39;, &#39;max_temp&#39;, &#39;min_temp&#39;, &#39;max_humid&#39;, &#39;min_humid&#39;, &#39;wind&#39;, &#39;solar&#39;]<br>for col in int_col:<br>    observe[col] = pd.to_numeric(observe[col], errors=&#39;coerce&#39;)<br>    observe[col] = observe[col].astype(float)<br><br>observe[&#39;date&#39;] = pd.to_datetime(observe[&#39;date&#39;], format=&#39;%d/%m/%Y&#39;)<br><br>observe = observe.sort_values([&#39;station_name&#39;, &#39;date&#39;])<br><br># Handling missing values<br>for col in int_col:<br>    observe[col] = observe[col].fillna(observe[col].mean())<br><br># Feature engineering<br>observe[&#39;rain_today&#39;] = np.where(observe[&#39;rain&#39;] &gt; 0, 1, 0)<br>observe[&#39;rain_yesterday&#39;] = (observe[&#39;rain_today&#39;].shift(1) == 1).astype(int).fillna(0)<br>observe[&#39;rain_tomorrow&#39;] = (observe[&#39;rain_today&#39;].shift(-1) == 1).astype(int).fillna(0)<br><br>observe[&#39;diff_temp&#39;] = observe[&#39;max_temp&#39;] - observe[&#39;min_temp&#39;]<br>observe[&#39;diff_humid&#39;] = observe[&#39;max_humid&#39;] - observe[&#39;min_humid&#39;]<br><br>observe[&#39;month_number&#39;] = observe[&#39;date&#39;].dt.month<br><br># Prepare dataset and split into train and validation sets<br>x_col = [&#39;ev_transpiration&#39;, &#39;max_temp&#39;, &#39;min_temp&#39;, &#39;max_humid&#39;, &#39;min_humid&#39;, &#39;wind&#39;, &#39;solar&#39;, &#39;month_number&#39;]<br>X = observe[x_col]<br>y = observe[&#39;rain_tomorrow&#39;]</pre><p><strong>Model Preparation</strong></p><p>After preprocessing the data and engineering relevant features, we split the dataset into training and validation sets. This ensures that our models are trained on a portion of the data and evaluated on unseen data, facilitating unbiased performance assessment.</p><pre># Split dataset into train and validation sets<br>X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=10)</pre><p><strong>Model Development and Evaluation</strong><br>We proceed to develop and evaluate multiple classification models using the prepared dataset. The models include Logistic Regression, K-Nearest Neighbors, Decision Tree, Random Forest, Gradient Boosting, and a Deep Learning model built using TensorFlow and Keras.</p><pre># Model Development<br><br># Logistic Regression<br>log_reg_model = LogisticRegression()<br>log_reg_model.fit(X_train, y_train)<br><br># K-Nearest Neighbors<br>knn_model = KNeighborsClassifier()<br>knn_model.fit(X_train, y_train)<br><br># Decision Tree<br>tree_model = DecisionTreeClassifier()<br>tree_model.fit(X_train, y_train)<br><br># Random Forest<br>forest_model = RandomForestClassifier()<br>forest_model.fit(X_train, y_train)<br><br># Gradient Boosting<br>gb_model = GradientBoostingClassifier()<br>gb_model.fit(X_train, y_train)<br><br># Deep Learning Model (Sequential)<br>k_model = keras.Sequential()<br>k_model.add(keras.layers.Dense(128, input_dim=X_train.shape[1], activation=&#39;relu&#39;))<br>k_model.add(keras.layers.Dense(128, activation=&#39;relu&#39;))<br>k_model.add(keras.layers.Dense(64, activation=&#39;relu&#39;))<br>k_model.add(keras.layers.Dense(32, activation=&#39;relu&#39;))<br>k_model.add(keras.layers.Dense(8, activation=&#39;relu&#39;))<br>k_model.add(keras.layers.Dense(1, activation=&#39;sigmoid&#39;))<br><br>k_model.compile(loss=&#39;binary_crossentropy&#39;, optimizer=&#39;adam&#39;, metrics=[&#39;accuracy&#39;])<br><br>k_model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=10, batch_size=100)<br><br># Model Evaluation<br><br>models = {<br>    &#39;Logistic Regression&#39;: log_reg_model,<br>    &#39;K-Nearest Neighbors&#39;: knn_model,<br>    &#39;Decision Tree&#39;: tree_model,<br>    &#39;Random Forest&#39;: forest_model,<br>    &#39;Gradient Boosting&#39;: gb_model,<br>    &#39;Deep Learning Model&#39;: k_model<br>}<br><br>for name, model in models.items():<br>    y_pred = model.predict(X_val)<br>    y_pred_binary = [1 if pred &gt;= 0.5 else 0 for pred in y_pred]<br><br>    accuracy = accuracy_score(y_val, y_pred_binary)<br>    accuracy = round(accuracy*100 , 2)<br>    print(&#39;================================================================&#39;)<br>    print(name, &quot; : &quot;, accuracy)<br><br>    print(&quot;Classification Report:&quot;)<br>    print(classification_report(y_val, y_pred_binary))<br><br>    # print(&quot;Confusion Matrix:&quot;) <br>    # print(confusion_matrix(y_val, y_pred_binary)) ## uncomment this line to review confusion matrix<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*YgFZBzpWnVeptGddfPHFQg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*GkoYpGLnMO0GAwJszoMQYQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*_lo8WoQxIr2MKNvRA5GRcA.png" /></figure><p><strong>Conclusion</strong><br>Continuing from our exploration in the previous article, we’ve expanded our analysis to include a range of classification models for weather prediction. Through this comprehensive approach, we aim to provide insights into the effectiveness of different machine learning algorithms in predicting rainfall. Stay tuned for further advancements and insights into leveraging Python for weather prediction tasks.</p><p>In case you missed the previous article, you can find it <a href="https://medium.com/@gigi.dattaradon/accessing-australian-bureau-of-meteorology-bom-weather-data-with-pythons-ftp-library-e35ee7d4e856">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7149d39995d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Weather Prediction with Python: A Data Science Approach Using Logistic Regression]]></title>
            <link>https://medium.com/@gigi.dattaradon/automating-weather-prediction-with-python-a-data-science-approach-using-logistic-regression-2b66c9d044cb?source=rss-7c011fc22710------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b66c9d044cb</guid>
            <category><![CDATA[weather-forecasts]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <dc:creator><![CDATA[Gigi Dattaradon]]></dc:creator>
            <pubDate>Sat, 24 Feb 2024 00:30:00 GMT</pubDate>
            <atom:updated>2024-02-24T00:50:37.651Z</atom:updated>
            <content:encoded><![CDATA[<p>Forecasting the weather accurately is essential for planning various activities and making informed decisions. Leveraging Python programming and basic machine learning techniques, such as logistic regression, can significantly improve weather prediction accuracy. In this article, we’ll focus on building a simple weather prediction model using logistic regression, laying the groundwork for more advanced techniques. But first, let’s understand the data source and context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*964kVXaMuoubpC-f5CEDQw.png" /></figure><p><strong>Introduction</strong><br>The data used in this analysis is sourced from the Australian Bureau of Meteorology (BOM), accessed through Python’s FTP library. It comprises daily weather observations in New South Wales (NSW) spanning twelve months from February 2023 to January 2024. This dataset encapsulates observations from 79 weather stations across the region. You can follow the steps outlined in <a href="https://medium.com/@gigi.dattaradon/accessing-australian-bureau-of-meteorology-bom-weather-data-with-pythons-ftp-library-e35ee7d4e856">this article</a> to capture the weather data.</p><figure><img alt="NSW Weather Data from Australian Bureau of Meteorology (BOM)" src="https://cdn-images-1.medium.com/max/1024/1*ieCGQU7aCAzlV-YDagYdDQ.png" /></figure><p><strong>Setting the Stage: Data Preparation</strong></p><p>The initial step in automating weather prediction involves gathering and preprocessing the data. The following Python script snippet demonstrates this process:</p><pre>import os<br>import pandas as pd<br>import numpy as np<br>from sklearn.preprocessing import MinMaxScaler<br>from sklearn.model_selection import train_test_split<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.metrics import confusion_matrix, classification_report<br>import seaborn as sns<br>import matplotlib.pyplot as plt<br><br># Read the file<br>filename = &#39;NSW_weather_data_12mth.csv&#39;<br>observe = pd.read_csv(filename, sep =&#39;,&#39;,index_col = 0)<br><br># Adjust column types<br>int_col = [&#39;ev_transpiration&#39;,&#39;rain&#39;,&#39;pan_ev&#39;,&#39;max_temp&#39;,&#39;min_temp&#39;,&#39;max_humid&#39;,&#39;min_humid&#39;,&#39;wind&#39;,&#39;solar&#39;]<br>for col in int_col:<br>    observe[col] = pd.to_numeric(observe[col], errors=&#39;coerce&#39;)<br>    observe[col] = observe[col].astype(float)<br><br>observe[&#39;date&#39;] = pd.to_datetime(observe[&#39;date&#39;], format=&#39;%d/%m/%Y&#39;)<br><br># Sort values<br>observe = observe.sort_values([&#39;station_name&#39;,&#39;date&#39;])<br><br>## fill nan value with mean<br>for col in int_col:<br>    observe[col] = observe[col].fillna(observe[col].mean())</pre><p>This script snippet demonstrates the initial steps of data gathering, cleaning, and feature engineering, which are essential for building accurate predictive models.</p><p><strong>Data Preprocessing</strong></p><p>Once the missing values are handled, we proceed to augment the dataset with additional features and perform normalization:</p><pre># Add rain today, rain yesterday column<br>observe[&#39;rain_today&#39;] = np.where(observe[&#39;rain&#39;] &gt; 0, 1, 0)<br>observe[&#39;rain_yesterday&#39;] = (observe[&#39;rain_today&#39;].shift(1) == 1).astype(int).fillna(0)<br>observe[&#39;rain_tomorrow&#39;] = (observe[&#39;rain_today&#39;].shift(-1) == 1).astype(int).fillna(0)<br><br># Add difference temp, humid<br>observe[&#39;diff_temp&#39;] = observe[&#39;max_temp&#39;] - observe[&#39;min_temp&#39;]<br>observe[&#39;diff_humid&#39;] = observe[&#39;max_humid&#39;] - observe[&#39;min_humid&#39;]<br><br># Get month number from datetime<br>observe[&#39;month_number&#39;] = observe[&#39;date&#39;].dt.month<br><br># Normalize columns<br>norm_col = [&#39;ev_transpiration&#39;,&#39;rain&#39;,&#39;pan_ev&#39;,&#39;max_temp&#39;,&#39;min_temp&#39;,&#39;max_humid&#39;,&#39;min_humid&#39;,&#39;wind&#39;,&#39;solar&#39;, &#39;diff_temp&#39;, &#39;diff_humid&#39;]<br>scaler = MinMaxScaler()<br>data_normalize = observe[norm_col]<br>observer_normalize = scaler.fit_transform(data_normalize)<br>observe_normalize = pd.DataFrame(observer_normalize, columns=norm_col)<br>observe[norm_col] = observe_normalize</pre><p>These steps ensure that the dataset is appropriately prepared and features are standardized for further analysis.</p><p><strong>Exploratory Data Analysis (EDA)</strong></p><p>Understanding the relationships between weather variables is crucial. We use simple visualizations, such as correlation heatmaps, to explore these relationships:</p><pre># Correlation heatmap<br>corr_col =  [&#39;rain_tomorrow&#39;,&#39;rain_yesterday&#39;, &#39;rain_today&#39;, &#39;rain&#39;, &#39;ev_transpiration&#39;,&#39;pan_ev&#39;,&#39;max_temp&#39;,&#39;min_temp&#39;,&#39;max_humid&#39;,&#39;min_humid&#39;,&#39;wind&#39;,&#39;solar&#39;, &#39;diff_temp&#39;, &#39;diff_humid&#39;]<br>sns.heatmap(observe[corr_col].corr(), annot=True, cmap=&quot;coolwarm&quot;, annot_kws={&quot;size&quot;: 6})<br>plt.figure(figsize=(14, 14))<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*NSO9tpFi79Kv_lBj5nb1-g.png" /></figure><p>This visualization helps identify which weather features have the most significant influence on rain prediction.</p><p><strong>Model Development: Logistic Regression</strong></p><p>Our focus is on building a logistic regression model, a simple yet powerful classification technique. The following script snippet demonstrates training the model:</p><pre>## prepare data set, then split train , test set<br>x_col = [&#39;ev_transpiration&#39;,&#39;max_temp&#39;,&#39;min_temp&#39;,&#39;max_humid&#39;,&#39;min_humid&#39;,&#39;wind&#39;,&#39;solar&#39;,&#39;month_number&#39;]<br>X = observe[x_col]<br>y = observe[&#39;rain_tomorrow&#39;]<br><br>X_train, X_val, y_train, y_val = train_test_split(X, y , test_size = 0.2, random_state = 10)<br>print(X_train.shape, X_val.shape, y_train.shape, y_val.shape)<br><br>##  Logistic Regression<br>## model<br>log_reg_model = LogisticRegression()<br>log_reg_model.fit(X_train, y_train)<br><br>## predict<br># log_reg_model.predict(X_val)<br><br>## model score, model accuracy<br>log_reg_model_score = log_reg_model.score(X_val, y_val)<br>log_reg_model_accuracy = round(log_reg_model_score * 100, 2)<br><br>print(&quot;The classification accuracy of Logistic Regression model is &quot; + str(log_reg_model_accuracy) + &quot;%&quot;)<br><br>## confusion matrix<br>y_pred = log_reg_model.predict(X_val)<br>cm = confusion_matrix(y_val, y_pred)<br><br>## plot heatmap<br>plt.figure(figsize=(4, 3))<br>plt.imshow(cm, cmap=&#39;Reds&#39;)<br><br># Add text annotations<br>for i in range(len(cm)):<br>    for j in range(len(cm[i])):<br>        plt.text(j, i, str(cm[i][j]), ha=&#39;center&#39;, va=&#39;center&#39;, color=&#39;black&#39;, fontsize=12)<br><br><br># Set labels and title<br>plt.xlabel(&#39;Predicted&#39;, fontsize=10)<br>plt.ylabel(&#39;Actual&#39;, fontsize=10)<br>plt.title(&#39;Confusion Matrix&#39;, fontsize=10, pad=10)<br><br># Set class labels<br>class_labels = [&#39;Not Rain&#39;, &#39;Rain&#39;]<br>plt.xticks(range(len(class_labels)), class_labels, rotation=45)<br>plt.yticks(range(len(class_labels)), class_labels, rotation=0)<br><br>plt.colorbar()<br>plt.show()<br><br>## classification report<br>print(&#39;Logistic Regression Classification Report&#39;)<br>print(&#39;=========================================&#39;)<br><br>print(classification_report(y_val, y_pred, target_names = class_labels))</pre><figure><img alt="Logistic Regression-Confusion Matrix" src="https://cdn-images-1.medium.com/max/670/1*t5OqF9RwwkkHJPibE5qX4Q.png" /></figure><figure><img alt="Logistic Regression — Classification Report" src="https://cdn-images-1.medium.com/max/553/1*sHrwgpjzVeLAZ90CREGTWA.png" /></figure><p>This basic model serves as a foundation for predicting whether it will rain tomorrow based on current weather conditions.</p><p><strong>Conclusion</strong></p><p>By automating weather prediction using Python and machine learning techniques, we can enhance the accuracy and reliability of forecasts, enabling better decision-making and planning in various domains. In the upcoming articles, we will explore more advanced machine learning models for weather prediction, building upon the simple logistic regression discussed here. Stay tuned for deeper insights into enhancing weather forecasting capabilities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b66c9d044cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Accessing Australian Bureau of Meteorology (BOM) Weather Data with Python’s FTP Library]]></title>
            <link>https://medium.com/@gigi.dattaradon/accessing-australian-bureau-of-meteorology-bom-weather-data-with-pythons-ftp-library-e35ee7d4e856?source=rss-7c011fc22710------2</link>
            <guid isPermaLink="false">https://medium.com/p/e35ee7d4e856</guid>
            <category><![CDATA[historical-weather-data]]></category>
            <category><![CDATA[australian-weather-data]]></category>
            <category><![CDATA[python-ftp-library]]></category>
            <category><![CDATA[ftplib]]></category>
            <category><![CDATA[bom]]></category>
            <dc:creator><![CDATA[Gigi Dattaradon]]></dc:creator>
            <pubDate>Tue, 20 Feb 2024 05:17:19 GMT</pubDate>
            <atom:updated>2024-02-20T05:17:19.038Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction:</strong></p><p>The Australian Bureau of Meteorology (BOM) provides a goldmine of real-time forecast weather, warnings, observations, and analyses. Among these, the observation weather dataset stands out, offering daily evaporation, rainfall, temperature extremes, humidity, wind speed, and solar radiation data. Accessing this invaluable resource for research or analysis is made possible through BOM’s FTP services. This article will guide you through the process of efficiently downloading weather data from BOM’s FTP using Python’s ftplib library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*43GeufO29XYE3WH2arZknw.png" /></figure><p><strong>Accessing BOM’s FTP:</strong></p><p>BOM’s FTP service is hosted at ftp.bom.gov.au and allows anonymous access. Simply use ‘anonymous’ as the username and your email address as the password to securely log in.</p><figure><img alt="FTP BOM Log In Setting Up" src="https://cdn-images-1.medium.com/max/1024/1*ozoWyhlh-R8mGQCcibfJVw.png" /></figure><p><strong>Navigating the Directory Structure:</strong><br>Upon logging in, users encounter a directory structure with various folders. To access national weather observations across Australia, navigate to the “/anon/gen/clim_data/IDCKWCDEA0/tables” directory. Here, the state folders contain daily weather observations collected from locations within each state. From here, users can select a particular location and observation time and manually download the file directly from FTP server.</p><figure><img alt="BOM FTP Service — Folder Directory" src="https://cdn-images-1.medium.com/max/1024/1*XATlW1P_pyNf3UFKBfSb8w.png" /></figure><p><strong>Automating the Download Process:</strong></p><p>To streamline the process, I’ve crafted an automated python script:</p><ol><li>Authenticate with the username ‘anonymous’ and your email address as the password.</li></ol><pre>### import library<br>import datetime<br>import time<br>from ftplib import FTP<br>import pandas as pd<br>import io<br>import os<br><br>username = &#39;anonymous&#39;<br>password = &#39;useremailaddress&#39; </pre><p>2. Set start and end times to filter observations from the last 12 months and generate the list of date.</p><pre>## Set start and end time, to filter last 12 months files<br>today = datetime.datetime.now()<br><br>starttime  = today.replace(day=1) - datetime.timedelta(days=365) ### last 12 month<br>endtime = today.replace(day=1) - datetime.timedelta(days=1) ### last month<br><br># Generate list of YYYY-MM dates<br>date_list = []<br>current_datetime = starttime<br>while current_datetime &lt;= endtime:<br>    date_list.append(current_datetime.strftime(&#39;%Y%m&#39;))<br>    current_datetime += datetime.timedelta(days=31)  # Add one month<br><br>print(&quot;List of YYYY-MM dates between start and end time:&quot;)<br>print(date_list)</pre><p>3. Create a function to retrieve a list of observation locations from the FTP server.</p><pre># Connect to the FTP server to get list of location in NSW<br>def get_observation_location(ftp_server, ftp_directory ):<br>    with FTP(ftp_server) as ftp:<br>        ftp.login(username, password)<br>        ftp.cwd(ftp_directory)<br>        <br>        # Get list of directories in the current directory<br>        observe_locations_list = ftp.nlst()<br>        return observe_locations_list</pre><p>4. Implement a function to download files from the FTP server.</p><pre># Function to download files from FTP server<br>def download_files_from_ftp(ftp_server, ftp_directory, local_directory, observe_location, date_list):<br>      with FTP(ftp_server) as ftp:<br><br>        ftp.login(username, password)<br>        ftp_directory_location = ftp_directory +  observe_location + &#39;/&#39;<br>        ftp.cwd(ftp_directory_location)<br>        filenames = [f&quot;{observe_location}-{date}&quot; for date in date_list ]<br>                    <br>        for filename in filenames:<br>            remote_filepath = f&quot;{filename}.csv&quot;<br>            local_filepath = f&quot;{local_directory}/{filename}.csv&quot;<br><br>            try:<br>                with open(local_filepath, &quot;wb&quot;) as local_file:<br>                    ftp.retrbinary(f&quot;RETR {remote_filepath}&quot;, local_file.write)   <br>                    print(f&quot;File &#39;{remote_filepath}&#39; downloaded to &#39;{local_filepath}&#39;&quot;)<br><br>            except Exception as e:<br>                print(f&quot;Error downloading file &#39;{remote_filepath}&#39;: {e}&quot;)</pre><p>5. Define the local directory for storing downloaded files and initiate the download process.</p><pre># Define FTP server details<br>ftp_server = &quot;ftp.bom.gov.au&quot;<br>ftp_directory = &quot;/anon/gen/clim_data/IDCKWCDEA0/tables/nsw/&quot;<br><br># Define local directory to save files<br>local_directory = &quot;C:/Users/Windows/Documents/2.Project/5.Weather/1.Data Set&quot;<br><br># Get observation locations from FTP server<br>observe_locations = get_observation_location(ftp_server, ftp_directory)<br><br># Download files from FTP server<br>for observe_location in observe_locations:<br><br>    with FTP(ftp_server) as ftp:<br>        ftp.login()<br>        download_files_from_ftp(ftp_server, ftp_directory, local_directory, observe_location, date_list)<br>        time.sleep(20)</pre><p><strong>Managing Concurrent Downloads:</strong></p><p>During file downloads, it’s essential to consider BOM’s FTP limitations, restricting concurrent downloads. To prevent connection interruptions and download failures, incorporate a 20-second wait time between each download.</p><p><strong>Conclusion:</strong></p><p>Accessing and downloading weather data from the Australian Bureau of Meteorology’s FTP server using Python’s ftplib library offers researchers and analysts a wealth of information for exploration and analysis. By automating the retrieval process with the provided Python script, users can efficiently access historical weather observations for various locations across Australia, enabling deeper insights into weather patterns and trends. The script can be found on <a href="https://github.com/okay2528/weather_data_nsw">GitHub link</a>, allowing everyone to implement it in their projects and contribute to advancements in meteorological science.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e35ee7d4e856" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Mental Health Services Utilization Patterns in Australia: A Visual Analysis]]></title>
            <link>https://medium.com/@gigi.dattaradon/exploring-mental-health-services-utilization-patterns-in-australia-a-visual-analysis-8285efd9a2f2?source=rss-7c011fc22710------2</link>
            <guid isPermaLink="false">https://medium.com/p/8285efd9a2f2</guid>
            <category><![CDATA[emergency-services]]></category>
            <category><![CDATA[mental-health-services]]></category>
            <category><![CDATA[healthcare-data-analytics]]></category>
            <category><![CDATA[australia-open-data-set]]></category>
            <category><![CDATA[healthcare-visualization]]></category>
            <dc:creator><![CDATA[Gigi Dattaradon]]></dc:creator>
            <pubDate>Sat, 17 Feb 2024 07:32:35 GMT</pubDate>
            <atom:updated>2024-02-17T07:32:35.761Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction:</strong></p><p>Mental health is a critical aspect of overall well-being, and access to mental health services is essential for individuals experiencing psychological distress. Analyzing data on mental health service utilization can provide valuable insights into trends, patterns, and disparities in access to care. This article explores two datasets sourced from the Australian Government Open Data platform (<a href="https://data.gov.au/search">data.gov.au</a>), provided by the Australian Institute of Health and Welfare (AIHW), regarding mental health service utilization in Australia, focusing on Emergency Department (ED) presentations and overnight admitted mental health separations.</p><p><strong>Data Sources:</strong></p><p>The first dataset covers <a href="https://data.gov.au/dataset/ds-aurin-613e987f-8c5a-464f-922b-a36ecf8d3f64/details?q=">Emergency Department presentations by demographics across Statistical Area Level 3 (SA3)</a> regions from 2014 to 2018, with a focus on the years 2015 and 2016. The second dataset focuses on <a href="https://data.gov.au/dataset/ds-aurin-052411d0-6d22-4ba8-95bb-70c5fcd767c4/details?q=">overnight admitted mental health separations in SA3 regions</a> for the years 2015 to 2016.</p><p><strong>Emergency Presentations:</strong></p><p>The national emergency presentation rate for mental health concerns stands at 1,749 per 10,000 population. Interestingly, the Northern Territory exhibits the highest rate at 3,584 per 10,000, primarily driven by the Barkly region.</p><figure><img alt="Northern Territory and Barkly Region Emergency Presentations Rate" src="https://cdn-images-1.medium.com/max/1024/1*NfC0B9rU77XAxugS9juOFQ.png" /></figure><p>Conversely, the Australian Capital Territory (ACT) demonstrates the lowest rate at 222 per 10,000. Notably, mental health-related presentations constitute 3.8% of all emergency presentations nationally. South Australia records the highest proportion at 5%, with Adelaide City being a significant contributor.</p><figure><img alt="South Australia and Adelaide City Mental Health Related Presentations" src="https://cdn-images-1.medium.com/max/1024/1*6VdPdgLY8mPnwlxqKr6MQw.png" /></figure><p><strong>Hospitalizations:</strong></p><p>On the other hand, the national hospitalization rate for mental health issues is 103 per 10,000 population. South Australia emerges with a higher rate of 122 per 10,000, particularly concentrated in Adelaide City.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vn3oMRkaRFpb_1JqMlccrw.png" /></figure><p>In New South Wales, regions such as Sydney Eastern, Sydney Parramatta, Sydney Ryde, and Newcastle require more bed days per 10,000 population compared to other areas. Similarly, Toowoomba in Queensland.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IkDyY424tsyxVnQ5HuSaTg.png" /></figure><p><strong>Comparative Analysis:</strong></p><p>The disparity between emergency presentation rates and hospitalization rates underscores the importance of early intervention and community-based mental health services. While some regions experience high rates of emergency presentations, the corresponding hospitalization rates may not necessarily be commensurate. This suggests a potential gap in access to community-based support services or a need for improved crisis intervention strategies.</p><p><strong>Conclusion:</strong></p><p>Analyzing regional trends in emergency presentations and hospitalizations for mental health provides valuable insights into patterns of service utilization and points of intervention. By addressing disparities in access to care and tailoring services to the specific needs of each region, we can work towards building a more responsive and equitable mental health care system for all Australians. New South Wales and Queensland stands out for requiring more bed days than other regions within the state.</p><p>Please note: For a thorough analysis and review of the Power BI visualization file, it has been shared through this <a href="https://github.com/okay2528/emergency_presentation_au_2015-16">GitHub link</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8285efd9a2f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Design Fabric Workspaces for Medallion Framework]]></title>
            <link>https://medium.com/@gigi.dattaradon/choosing-data-workspaces-for-medallion-framework-779a86b60ee6?source=rss-7c011fc22710------2</link>
            <guid isPermaLink="false">https://medium.com/p/779a86b60ee6</guid>
            <category><![CDATA[lakehouse-architecture]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <category><![CDATA[medallion-architecture]]></category>
            <category><![CDATA[onelake]]></category>
            <category><![CDATA[workspace-design]]></category>
            <dc:creator><![CDATA[Gigi Dattaradon]]></dc:creator>
            <pubDate>Sun, 14 Jan 2024 23:58:42 GMT</pubDate>
            <atom:updated>2024-01-29T09:28:54.733Z</atom:updated>
            <content:encoded><![CDATA[<p>The medallion architecture is a data design pattern that organizes your data into three layers: bronze, silver, and gold. Each layer represents a different level of data structure, quality, and value.</p><p>One of the key decisions you need to make when designing your medallion architecture is how many workspaces to use for each layer. A workspace is a logical container for your data, pipelines, dataflows, and other resources. You can use one workspace for each layer, one workspace for all layers, or a hybrid solution that combines multiple workspaces for some layers and a single workspace for other layers.</p><p>Each option has its own advantages and disadvantages, depending on your specific requirements and trade-offs. In this guideline, we will help you evaluate the pros and cons of each option and choose the best fit for your scenario.</p><h4>Unified Workspace for Consistency</h4><p>In this scenario, a single workspace accommodates all layers of the medallion architecture (bronze, silver, and gold). It centralizes shared data, pipelines, dataflows, and resources across all layers. Maintaining uniform settings and configurations simplifies data movement and synchronization between the layers. This setup enables seamless tracking and auditing of data transformations and quality checks from bronze to gold. Additionally, it leverages Microsoft Fabric’s metadata and schema management features for enhanced data integrity and reliability.</p><p>However, it’s crucial to strike a balance between trade-offs related to security, performance, cost, and collaboration for each layer. Customizing data models, formats, partitions, and indexes for specific use cases may be limited. Furthermore, you might not fully exploit Microsoft Fabric’s distributed and parallel processing capabilities for handling diverse data volumes across the layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*koHNC_QG6sIg2M4GkgdUlA.png" /><figcaption><strong>Unify One Workspace for Consistency</strong></figcaption></figure><h4>Multiple Workspaces for Flexibility</h4><p>In this approach, each layer within your medallion architecture (bronze, silver, and gold) gets its own dedicated workspace. Each workspace houses data, pipelines, dataflows, and resources tailored to the unique requirements of that layer. This offers the flexibility to configure distinct settings and configurations for each workspace, catering to the specific needs of security, performance, cost, and collaboration for each layer.</p><p>Nonetheless, this approach presents certain challenges. Managing data movement and synchronization between these workspaces may involve additional steps and tools. Ensuring data consistency and quality across separate workspaces might require extra validation and monitoring efforts. Furthermore, effectively coordinating the deployment and orchestration of pipelines and dataflows between these workspaces could involve additional tools and processes.</p><h4>Hybrid Approach for Security</h4><p>In this strategy, we aim to limit access to the Bronze and Silver layers to a select group of users, while allowing broader access to the Gold layer.</p><p>For the Bronze and Silver layers, individuals responsible for their management are assigned to Member or Contributor roles, granting them full data management privileges within these environments, as they require write access. Users needing access to specific data within these layers can be assigned the Viewer role, allowing access through SQL endpoints. Data science teams seeking OneLake access to Bronze and Silver data can either have these items shared directly or be assigned the Viewer role along with ReadAll permissions.</p><p>For the Gold layer, access can be divided among multiple smaller workspaces, each scoped to a specific business domain or user group. Within each workspace, end users are assigned the Viewer role, while data engineers managing the Gold layer can be granted Contributor or Member roles for write access. For environments needing stricter access controls, individual warehouses or Lakehouses can define object-level security through SQL endpoints, allowing select tables to be shared with users while keeping others hidden. Lastly, the sharing and permissions management options provide fine-grained control over data access in Spark and OneLake for advanced users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*yKO3HFTrZYI-u788vEP1Pg.png" /><figcaption><strong>Hybrid for Data Security</strong></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=779a86b60ee6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>