As data engineers, we constantly juggle performance, flexibility, and cost. We’re building pipelines that must be robust enough for production, agile enough to evolve with changing schemas, and optimized enough to avoid blowing through compute budgets.

If you’re still relying on Hive tables or managing partitions manually in S3, it’s time to meet Apache Iceberg — a modern table format purpose-built for the data lake era.

In this post, we’ll focus on why Iceberg matters for data engineers, especially around:

• Reducing data scanned in queries
• Supporting time travel for reproducibility and recovery
• Enabling safe schema evolution
• Cutting query costs without sacrificing flexibility

In addition, we will build an example iceberg table and compare with normal external table on Athena.

DataSet: https://cseweb.ucsd.edu/~jmcauley/datasets.html#multi_aspect

Iceberg in a Nutshell

Apache Iceberg brings database-like table management to cloud object storage. It gives you ACID transactions, scalable metadata, schema tracking, and full compatibility with engines like Spark, Flink, Trino, Presto, Athena, and Snowflake (read-only).

Unlike Hive tables, Iceberg separates physical storage layout from logical table structure, and stores rich metadata that enables optimization at read time.

The metadata layer in Apache Iceberg is the core engine that enables its powerful features like time travel, schema evolution, and efficient query pruning. Instead of scanning the entire dataset, Iceberg maintains a hierarchy of metadata files: table metadata, snapshots, manifests, and manifest lists. These files track what data files exist, which columns and partitions they cover, and statistical summaries like min/max values. This allows query engines to skip irrelevant files based on filters, reducing I/O and cost. The metadata is stored in JSON and Avro formats, making it compact, versioned, and fully accessible for auditing or rollback.

Query Less, Pay Less

we know that in platforms like Athena or BigQuery, costs scale linearly with data scanned. Iceberg fights this in three major ways:

Metadata Filtering (Manifest Pruning)

Iceberg stores per-file metadata — like partition values and column-level stats (min/max) — in compact manifest files. This allows query engines to skip irrelevant files before reading data.

-- Only scans files where order_date >= '2024-01-01'
SELECT * FROM orders WHERE order_date >= '2024-01-01';

Without Iceberg: full table scan.
With Iceberg: manifest filtering → file skipping → less I/O → lower cost.

Partition Pruning Without Manual Partition Columns

Forget dt='2024-01-01' in your WHERE clause. Iceberg understands hidden partitions and can still prune data even when you don’t explicitly filter on the partition column.

Column Pruning

Iceberg works seamlessly with Parquet to read only the necessary columns, reducing disk I/O and memory pressure.

Let’s dig in

We first need to load the data into S3 bucket and split the data with partitioned by review_year.

Let’s create the external table and the iceberg table, then we can compare the difference.

Next, we need to load data into the iceberg table, which will generate the data into the data layer under the S3 folder.

Now, we have data in both table, and we can compare the data scanned.

From the sample, we can actually see how powerful Apache Iceberg is.

Cost Analysis

• Iceberg will scan 40–80% less data when querying.
• Iceberg will add ~10% extra storage in S3 bucket to store the metadata.

Time Travel for Free: Debug, Audit, and Reproduce

Another important feature about Iceberg is the time travel mechanism.

Iceberg gives you time travel capabilities out of the box by retaining metadata snapshots. As a data engineer, this means:

• You can roll back tables after a bad write.
• You can let analysts query consistent historical views.
• You can audit changes without relying on complex logging or CDC layers.

SELECT * FROM beer_review_iceberg TIMESTAMP AS OF '2025-04-01T10:00:00';

Schema Evolution

Forget brittle Hive schemas. Iceberg supports real, production-grade schema evolution, with field-level tracking.

Supported changes:

• Add/drop columns
• Rename columns (no breaking changes)
• Reorder columns
• Evolve types (if compatible)

Internally, Iceberg assigns stable field IDs, so column name changes don’t corrupt the table or break downstream tools.

This makes it easier to:

• Avoid data rewrites after minor schema updates
• Decouple producer and consumer teams
• Evolve your model safely over time

Apache Iceberg is more than just a table format — it’s a foundational upgrade for any modern data platform. For data engineers, it brings the best of both data lakes and warehouses: reliable schema evolution, safe time travel, and intelligent query pruning that directly translates to lower cloud costs. Whether you’re building batch pipelines, streaming ingestion, or self-serve analytics layers, Iceberg offers the flexibility, scalability, and performance you need without locking you into a specific vendor or engine. If you’re serious about building a future-proof, cost-efficient data stack, Iceberg deserves a top spot in your toolkit.

Thank you for reading, have a great day!!

In the ever-evolving landscape of book discovery, traditional recommendation systems often fall short. Large language models (LLMs) offer a promising new approach. By leveraging their ability to process vast amounts of text data, LLMs can delve into the intricacies of different genres, writing styles, and reader preferences. This newfound depth holds the potential to revolutionize book recommendations, leading readers not just to familiar tropes, but to truly personalized literary journeys.

One of the exciting advancements in LLM-powered book recommendation systems is the integration of Retrieval-Augmented Generation (RAG). RAG functions as a sophisticated information retrieval tool for the LLM. By efficiently searching vast datasets of book information, RAG identifies titles with similar content and stylistic elements. This retrieved data empowers the LLM to move beyond simple similarity-based recommendations. RAG allows the LLM to grasp the underlying themes and narrative approaches that resonated with the user, enabling it to generate highly personalized suggestions that cater to the user’s specific literary preferences.

In this post, we are going to demonstrate how to build a simple vector store and retrieve the documents that are semantically relevant.

Data Set

We will use a dataset from Kaggle

Source : https://www.kaggle.com/datasets/ishikajohari/best-books-10k-multi-genre-data/

Setup

Vector Store: FAISS

Embedding model: bert-base-uncased from Hugging face — here

Github Code

Link: https://github.com/chiang9/Medium_blog/tree/main/llm_bookreco

Let’s Get Started

You will need to first get the API key from Hugging face, which you can find here.

Next we can start creating the vector embedding from the csv file we just created.

The embedding process might take a while. In this example, we will only sample the top 1000 documents.

CPU times: total: 58min 47s
Wall time: 6min 3s

Langchain provides various of different abstraction for data loader and vector store loader. Documentation can be found here. With Langchain, we will be able to save, load, merge the vector embedding.

Book: Eleanor & Park
Genres: ['Young Adult', 'Romance', 'Contemporary', 'Fiction', 'Realistic Fiction', 'Audiobook', 'Teen']

Book: Are You There God? It's Me, Margaret
Genres: ['Young Adult', 'Fiction', 'Childrens', 'Classics', 'Middle Grade', 'Coming Of Age', 'Realistic Fiction']

Book: The Paper Bag Princess
Genres: ['Picture Books', 'Childrens', 'Fantasy', 'Fiction', 'Dragons', 'Classics', 'Fairy Tales']

Book: The Velveteen Rabbit
Genres: ['Classics', 'Childrens', 'Fiction', 'Picture Books', 'Fantasy', 'Animals', 'Young Adult']

CPU times: total: 578 ms
Wall time: 69 ms

NLP Data Preprocessing

Performing NLP data cleaning for texts when creating a vector store is generally recommended, but the extent of cleaning depends on your specific use case.

• Improved Vector Representation: Raw text with inconsistencies like punctuation, special characters, and typos can lead to noisy and inaccurate vector representations. Cleaning helps the model focus on the core meaning of the text.
• Enhanced Similarity Search: When searching for similar vectors in your store, a cleaner representation allows for more accurate comparisons. Imagine searching for documents about “cooking” — you wouldn’t want results dominated by entries with typos like “coking”.
• Reduced Storage Requirements: Removing unnecessary elements like stop words (common words like “the” or “a”) can help reduce the size of your vector representations, leading to more efficient storage.

In this session, we are going to perform some Natural Language Processing (NLP) data preprocessing to the description part, and we can compare the result.

Average length of description before and after data cleaning
956.1368537740602, 915.5401592260405
Average length of description before and after preprocessing
915.5401592260405, 900.0581477375794

Next, we perform similar step as what we previously have for the original data to generate the docsearch. We noticed that the embedding processing time is shorter.

CPU times: total: 44min 29s
Wall time: 4min 35s

Book: Inkheart (Inkworld, #1)
Genres: ['Fantasy', 'Young Adult', 'Fiction', 'Middle Grade', 'Childrens', 'Adventure', 'Magic']

Book: Artemis Fowl (Artemis Fowl, #1)
Genres: ['Fantasy', 'Young Adult', 'Fiction', 'Middle Grade', 'Childrens', 'Science Fiction', 'Adventure']

Book: Alexander and the Terrible, Horrible, No Good, Very Bad Day
Genres: ['Picture Books', 'Childrens', 'Fiction', 'Classics', 'Realistic Fiction', 'Humor', 'Kids']

Book: Are You There God? It's Me, Margaret
Genres: ['Young Adult', 'Fiction', 'Childrens', 'Classics', 'Middle Grade', 'Coming Of Age', 'Realistic Fiction']

CPU times: total: 469 ms
Wall time: 43 ms

Let’s compare the two embedding created from original description and the preprocessed description.

original docsearch: 
Book: Eleanor & Park
Genres: ['Young Adult', 'Romance', 'Contemporary', 'Fiction', 'Realistic Fiction', 'Audiobook', 'Teen']
score = 63.062381744384766


Book: Are You There God? It's Me, Margaret
Genres: ['Young Adult', 'Fiction', 'Childrens', 'Classics', 'Middle Grade', 'Coming Of Age', 'Realistic Fiction']
score = 66.79731750488281


Book: The Paper Bag Princess
Genres: ['Picture Books', 'Childrens', 'Fantasy', 'Fiction', 'Dragons', 'Classics', 'Fairy Tales']
score = 67.06227111816406


Book: Looking for Alaska
Genres: ['Young Adult', 'Fiction', 'Contemporary', 'Romance', 'Realistic Fiction', 'Coming Of Age', 'Teen']
score = 67.72935485839844


----------------------
preprocessed docsearch: 
Book: Inkheart (Inkworld, #1)
Genres: ['Fantasy', 'Young Adult', 'Fiction', 'Middle Grade', 'Childrens', 'Adventure', 'Magic']
score = 65.75355529785156


Book: Charlie and the Chocolate Factory (Charlie Bucket, #1)
Genres: ['Childrens', 'Fiction', 'Fantasy', 'Classics', 'Young Adult', 'Middle Grade', 'Humor']
score = 68.92379760742188


Book: The Sea of Monsters (Percy Jackson and the Olympians, #2)
Genres: ['Fantasy', 'Young Adult', 'Mythology', 'Fiction', 'Middle Grade', 'Adventure', 'Greek Mythology']
score = 69.14126586914062


Book: The Magician's Nephew (Chronicles of Narnia, #6)
Genres: ['Fantasy', 'Classics', 'Fiction', 'Young Adult', 'Childrens', 'Middle Grade', 'Adventure']
score = 70.03553771972656

Conclusion

The document retrieval is an essential component of RAG, and vector store provides the functionality to do the semantic search efficiently. Langchain is a very powerful abstraction tool in helping us build the vector store, retrieve the relevant document and also provides the functionality to communicate with the LLMs.

Different embedding model and similarity calculation method will give different result. We need to examine the business need to tune the embedding model to find the relevant documents to the given query.

Next we will communicate with the LLMs to complete the RAG book recommendation.

Thank you for reading, and have a great day!

Have you ever found yourself repeatedly listening to the same song, only to discover later that other music you might enjoy exists? Recommending music that aligns with individual preferences can be a complex task.
The current state-of-the-art music recommendation systems primarily focus on the user’s listening history, recommending similar artists and songs based on past preferences. However, this approach can lead to repetitive suggestions and hinder the discovery of truly novel and exciting music.

This post is to review a recommendation system paper from 2024 on repeating recommendations — Integrating Repeat Listening Patterns for Enhanced Music Recommendation. We’ll examine the research methodology, key findings, and the potential implications for the future of music recommendation systems.

Reference:
R. Shigetomi, H. Nishida, K. -i. Sawai and T. Ushiama, “Integrating Repeat Listening Patterns for Enhanced Music Recommendation,” 2024 18th International Conference on Ubiquitous Information Management and Communication (IMCOM), Kuala Lumpur, Malaysia, 2024, pp. 1–7, doi: 10.1109/IMCOM60618.2024.10418302

Background

Existing research suggests a consistent cognitive response to repeated stimuli, including music. This response is characterized by an inverted U-shaped curve in hedonic value, or the level of pleasure derived from the experience. As depicted in Figure 1, the initial exposure to a stimulus typically elicits a positive response that intensifies with subsequent exposures, reaching a peak of enjoyment. However, continued repetition can lead to a gradual decline in pleasure.

People tends to listen to the same music, read the same genre of book, or watch the same tv show over and over again. The repeating pattern can be a key factor in building the recommendation models.

The authors proposed Repeat-aware Self-Attention Networks (ReSANs), which is similar to a model we previously discuss Self-Attention based Sequential Recommendation model (SASRec). Both models employed the self attention mechanism but ReSANs further modify and attention layer and emphasize on the repeating feature.

SASRec Review: https://medium.com/analytics-vidhya/self-attention-on-recommendation-system-self-attentive-sequential-recommendation-review-c94796dde001

Methodology

Similar to the sequential recommendation from SASRec, the model takes a user’s recent listening history as input, represented by a sequence of tracks (i_1, i_2, …, i_t) spanning one week. It then attempts to predict the next song (i_t+1) that the user is likely to listen to based on their previous listening patterns within that timeframe.

While the overall architecture of ReSANs bears a strong resemblance to the existing SASRec model [2], it introduces key modifications in the embedding and self-attention layers, the heart of the model. Unlike SASRec, ReSANs place a specific emphasis on capturing repetition patterns in listening behavior. To achieve this, they introduce a repetition layer (RE) and positional embedding layer (PE) that enriches the representation of user and music items during the process of combining query and key elements within the self-attention mechanism. These modifications aim to improve the model’s ability to understand and leverage the nuances of user listening patterns, specifically those related to repeated listening behavior.

Data Preprocessing

The researchers used a dataset containing music listening history from Last.fm between January 1st, 2020 and March 20th, 2020. They filtered the data to ensure user and track diversity by removing outliers and only keeping users who interacted with a significant number of tracks and tracks with a decent number of listener interactions. Additionally, the authors defined listening sessions based on time intervals and excluded sessions with too few or too many songs. Finally, the data is aggregated into weekly listening sets for each user to be used as input for their model.

Embedding Layer

The study utilized an embedding layer similar to SASRec. This layer processes the user’s one-week listening history, capturing the inherent features and patterns. However, their approach introduced key advancement:

• Repeat embedding: This captures the frequency of each track was listened to within the week.
• Positional embedding: This considers the order in which tracks were listened to within the week.

These embeddings are then combined to create a comprehensive repeat embedding that considers both how many times a track was listened to and the order in which it was listened to. This enriched representation is then used as input for the next layer of the model.

Repeat-aware Self-Attention Layer

The Repeat-aware Self-Attention Layer is the key advancement of the proposed model.

Unlike traditional methods, this layer considers both the content of the tracks (from IE) and the repetition count (from RE_hat) when calculating attention scores. These scores determine how relevant one track is to another within the sequence. The key idea is that the model considers how many times a user has listened to a track (i) when evaluating its connection to another track (j).

Imagine two songs, i and j. Their connection is calculated differently based on how many times song i was listened to: a single listen might lead to a stronger connection with j compared to ten listens. This dynamic approach ensures recommendations that are more responsive to user listening habits. Additionally, the model doesn’t solely prioritize songs with the strongest contextual fit. It considers factors like a user frequently listening to a particular song (i) and might even recommend a different song (j) instead, showcasing a more nuanced understanding of user preferences.

Feed Forward Network and Stacking Self-Attention Blocks

The authors proposed to stacked the self-attention blocks and add the feed forward network to output to the next stage. This layer adds non-linearity to the model, and followed by the normalization and dropout layer to regularize the output.

Performance

To evaluate the effectiveness of their proposed model, the authors compared ReSANs with three other models. All four models utilized the self-attention mechanism to recommend music tracks based on user listening sequences. This allows for a direct comparison of how each model leverages this technique for recommendation purposes.

• SASRec: This is the foundation for many sequential recommendation systems, including ReSANs. It excels at capturing the order in which users listen to music.
• SASRec+R: This builds upon SASRec by incorporating repeat information. Similar to how position matters, this model considers how many times a user listens to a specific track, potentially leading to more accurate recommendations.
• LightSANs: This model also builds on SASRec but uses a reduced dimensional space to represent user interests. This allows the model to learn item-interest interactions more efficiently while maintaining good performance.

The results presented in Table 1 reveal a significant improvement in music recommendation performance achieved by ReSANs. Compared to the other models, ReSANs demonstrated a remarkable performance increase of 30–40%, highlighting its effectiveness in capturing the nuances of user listening behavior through its innovative consideration of repeat listening patterns.

Final Words

We see that ReSANs outperformed a lot of previous sequential recommendation models on music recommendation. This success suggests that incorporating repetition patterns into recommender systems has broader potential. Similar models could be developed for other applications, like book recommendations, taking into account our tendency to revisit preferred items. This opens doors for building more personalized and engaging recommender systems across various domains.

Thank you for reading, and have a nice day!

Reference

1. R. Shigetomi, H. Nishida, K. -i. Sawai and T. Ushiama, “Integrating Repeat Listening Patterns for Enhanced Music Recommendation,” 2024 18th International Conference on Ubiquitous Information Management and Communication (IMCOM), Kuala Lumpur, Malaysia, 2024, pp. 1–7, doi: 10.1109/IMCOM60618.2024.10418302
2. Kang, Wang-Cheng and McAuley, Julian, “Self-Attentive Sequential Recommendation”, in arXiv, 2018, https://doi.org/10.48550/arxiv.1808.09781

Repeating Pattern Enhanced Music Recommendation was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

Recommendation systems are powerful tools used in many industries, analyzing how people interact with items to predict their preferences. One common method, Collaborative Filtering, looks at user-item interactions to score items. However, the order in which users rate items matters! Understanding this hidden pattern helps us build better recommendation systems that deliver personalized suggestions across different fields.

This post is to review of “Self-Attentive Sequential Recommendation”. Even though this is a article from 2018, directly applying attention mechanism on the user context in recommendation system is an interesting approach.

Reference:
Kang, Wang-Cheng and McAuley, Julian, “Self-Attentive Sequential Recommendation”, in arXiv, 2018, https://doi.org/10.48550/arxiv.1808.09781

Background

Recommendation systems usually rely on users’ recent activities as a basis for prediction. However, traditional collaborative filtering models struggle to capture intricate action patterns effectively. Unveiling these patterns holds the key to enhancing our recommendation systems. Similar to natural language processing (NLP) and image processing, every detail within the data influences the subsequent prediction or data points. Markov Chain and Recurrent Neural Network methodologies have emerged as effective tools in processing such data, enabling us to extract valuable insights and refine our recommendation algorithms for better user experiences.

In the study, the author proposed the Self-Attention based Sequential Recommendation model (SASRec), where the self-attention block tends to capture sequential information where the user made in each time step as shown in figure 1.

Methodology

Data Preprocessing

In order to process the sequential user-item interaction data, the study is transforming the data into a time series like data, e.g.

where S is the historical interaction sequence for a user u.

The model will use series from 1 to t-1 to predict the shifted data of 2 to t during the training process.

Embedding Layer

In the dataset, users vary in the length of their item interaction history, however, the data input dimension need to be uniform before passing in the model. The embedding layer in SASRec addresses this by ensuring consistency in the length of user-item interaction sequences. It achieves this by padding zeros to adjust dimensions, facilitating seamless processing within the model.

The author also attempted to incorporate a learnable positional embedding by concatenating it with this embedding layer, but the performance didn’t go as expected.

Attention Mechanisms

Self-attention is a powerful mechanism used in deep learning to process sequential data, such as sentences or time-series data, by considering the relationship between different elements within the sequence. Unlike traditional attention mechanisms that focus on aligning different parts of the input sequence, self-attention allows each element to attend to all other elements, capturing long-range dependencies effectively. By assigning different weights to different elements based on their relevance to each other, self-attention enables the model to dynamically weigh the importance of each element in generating context-aware representations. This flexible and adaptive mechanism has gained widespread popularity in natural language processing tasks, where understanding the contextual relationships between words is crucial for tasks like translation and summarization.

where
Q refers to the queries
K refers to the keys
V refers to the values
square root d is used to scale and avoid having large value from the inner product

the attention layer computes a weighted total of all values, in which the weight for a given query i and the value j corresponds to the connection between query i and key j.

Point-Wise Feed-Forward Network

The attention output from the multi-head layer is passed through this network, transforming it into a higher-dimensional representation. This layer adds non-linearity to the model, and followed by the normalization and dropout layer to regularize the output.

Prediction Layer

The prediction layer in the SASRec model acts as a dual-purpose unit. During training, it calculates the loss function, guiding the model’s learning by comparing the predicted relevance (dot product with sequence representation) of positive and negative items to their actual relevance. This difference is fed into a loss function like binary cross-entropy, prompting the model to adjust its parameters and improve its ability to distinguish relevant items. During testing, the prediction layer takes the test item’s embedding, calculates its dot product with the sequence representation, and uses the resulting score to assess its relevance to the user’s preferences. This score directly influences the recommendation output, suggesting items with the highest scores as the most likely to interest the user.

Performance

The author presents a comparison of the SASRec model with 8 other recommendation models, evaluating their performance on 4 datasets. The results suggest potential advantages of the SASRec approach.

Final words

Prioritizing its publication year and impact, the SASRec model is acknowledged for its pioneering use of self-attention in recommendation systems. With over 1500 citations, it serves as a prominent reference point for subsequent research.

Thank you for reading, and have a nice day!

Reference

1. Kang, Wang-Cheng and McAuley, Julian, “Self-Attentive Sequential Recommendation”, in arXiv, 2018, https://doi.org/10.48550/arxiv.1808.09781

Self Attention on Recommendation System — Self-Attentive Sequential Recommendation Review was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the ever-expanding realm of digital information, recommendation systems have become the unsung heroes that guide us through the vast seas of content. Whether it’s suggesting our next favorite movie, helping us discover a new book, or recommending products tailored to our preferences, these systems play a pivotal role in shaping our online experiences.

This post is to share a summarization of a literature review I recently read on the recommender system. The author strictly select the recent research from 2019 to 2023 which focusing on novel approaches, evaluation techniques and common problems we might face in building the recommender system.

Reference:
I. Saifudin and T. Widiyaningtyas, “Systematic Literature Review on Recommender System: Approach, Problem, Evaluation Techniques, Datasets,” in IEEE Access, doi: 10.1109/ACCESS.2024.3359274.

Background

The paper select 72 researches which focus on different approaches, where Collaborative Filtering (CF) was the major approach and used in 46 studies, Content-Based Filtering (CBF) and Hybrid Filtering (HBF) were used in 11 and 15 studies respectively.

The following four research questions served as the foundational framework for the entire literature review (1):

• What approach is suitable for solving problems in recommender systems?
• What problems arise in solving problems in recommender systems?
• What evaluation techniques are suitable for solving problems in recommender systems?
• What datasets are currently popular for research in recommender system?

Approach

The study separate the recommender system built by the 72 researches into 3 major category — Content Based Filtering (CBF), Collaborative Filtering (CF) and Hybrid Filtering (HBF). The author emphasized three associated studies for each approach, and I will only delve into one from each category.

Content Based Filtering

The Content-Based Filtering (CBF) technique leverages product descriptions and user profiles, conducting product characteristics analysis to generate recommendations. User profiles are shaped by both interaction history and individual preferences. CBF research spans diverse domains like movies and tourist attractions, utilizing a similarity metric in vector space to calculate item similarity for precise recommendations. While CBF stands out for its flexibility, not relying on additional user information, it demands domain expertise for optimal results.

CBF’s reliance on product features and user profiles, coupled with its independence from changing user information, underscores its flexibility compared to other methods. However, this approach comes with the caveat of requiring domain-specific skills to achieve optimal performance, showcasing the dual nature of advantages and challenges within the realm of CBF research.

In the study “Movie Popularity and Target Audience Prediction Using the Content-Based Recommender System”, S. Sahu et al. (2) proposed a multiclass CNN movie recommendation model. There are three major components in the proposed model.

1. Content-based recommendation model
2. Movie hit prediction model
3. Audience targeting model

The first module will preprocess the data and create a multi-dimensional KNN cluster based on the movie meta data and cosine similarity. The result will be passed into the second module to predict movie rating using 1D-CNN module. The audience targeting model will generate recommendations based on the user age group and biography.

Utilizing a multiclass classification model, researchers have attained an impressive accuracy of 96.8%, surpassing benchmarks and emphasizing the capabilities of predictive and prescriptive data analysis in bolstering industrial decisions within information systems. Nonetheless, the literature reviewer acknowledges certain limitations of the model, particularly the necessity for an expert system to predict a movie’s probability of success with reasonable accuracy, a requirement recognized by both moviemakers and researchers in the field.

Reference:
S. Sahu, R. Kumar, M. S. Pathan, J. Shafi, Y. Kumar, and M. F. Ijaz, “Movie Popularity and Target Audience Prediction Using the Content-Based Recommender System,” IEEE Access, vol. 10, pp. 42030–42046, 2022, doi: 10.1109/ACCESS.2022.3168161.

Collaborative Filtering

Different from Content based filtering approach, Collaborative Filtering (CF) approach utilize the utility matrix based on the user to item interaction to estimate the user preference, and the item profile is necessary required. Within collaborative filtering, the collective input of all users is considered, identifying individuals with comparable tastes to suggest new and tailored products to the main customer. This approach is assuming people with similar item interaction will have similar taste.

CF is a powerful approach in recommender system, however, this approach often suffer from the challenge of execution time and scalability. Matrix decomposition algorithm becomes essential in this method. T. Widiyaningtyas et al. (3) is able to process the MovieLens 100k dataset within 13.502 second by utilizing the SVD-WPR approach they proposed.

The study integrated the Singular Value Decomposition (SVD) method to predict unrated items from the user-item utility matrix. To optimize the time complexity, the rating aggregation method, Weighted Point Rating (WPR), is utilized to obtain the recommendation items.

WPR process consists of four steps:

1. Calculating the equal rating items
2. Calculating item points
3. Calculating weight points
4. Calculating weight points rating

The rating aggregation method is brilliant and very effective, however, we are not going into the math formula detail at the moment.

Reference:
T. Widiyaningtyas, M. I. Ardiansyah, and T. B. Adji, “Recommendation Algorithm Using SVD and Weight Point Rank (SVD-WPR),” Big Data and Cognitive Computing, vol. 6, no. 4, 2022, doi: 10.3390/bdcc6040121.

Hybrid Filtering

Unlike singular methods such as Content-Based or Collaborative Filtering, Hybrid Filtering leverages a combination of approaches, often incorporating both user-item interactions and item characteristics. H. Tahmasebi et al. (4) implemented the recommender system based on autoencoder neural network.

In the research, deep learning models in recommender systems are categorized into two main types: neural network models and deep hybrid models. The former includes diverse architectures such as multilayer perceptron (MLP), autoencoder (AE), convolutional neural network (CNN), recurrent neural network (RNN), restricted Boltzmann machine (RBM), neural autoregressive distribution estimation (NADEs), attentional model (AM), adversarial network (AN), and deep reinforcement learning (DRL). On the other hand, hybrid models combine two or more deep learning techniques, utilizing the flexibility of deep neural networks to create a more robust recommendation system. The range of options showcases the adaptability and versatility of deploying deep learning in recommender systems.

The study indicates that the autoencoder network recommender system is able to maintain high accuracy and labeled data is not required. These networks excel at automatically extracting valuable features from the data they process. By reconstructing data and diminishing data dimensions through learned relations, autoencoders prove to be efficient in uncovering meaningful patterns and representations without the need for explicit labeling.

The author proposed an autoencoder neural network approach — SRDNet (Social Recommender Deep Autoencoder Network), which consists of two recommender engine, collaborative filtering and content based filtering. The model gathers data from MovieTweetings, Twitter and IMDB to built social influence item profile based on movie detail, number of tweets, number of retweets and number of likes.

The user-movie utility matrix is constructed using the social influence score. Therefore, user ratings and interests with a higher social influence result in a more substantial impact on the training process of the deep autoencoder network. The autoencoder is mainly used in the CF module and the final recommendation result is calculated based on addition of different ratio of the collaborative filtering and content based filtering modules.

Reference:
H. Tahmasebi, R. Ravanmehr, and R. Mohamadrezaei, “Social movie recommender system based on deep autoencoder network using Twitter data,” Neural Comput Appl, vol. 33, no. 5, pp. 1607–1623, 2021, doi: 10.1007/s00521–020–05085–1.

Problem

The author addressed 5 major problems which the researches faced.

• Accuracy
• Sparsity
• Cold Start
• Running Time
• Scalability

The findings from the research highlight several crucial challenges in building recommendation systems. The Cold Start Problem represents a significant hurdle, where the system lacks sufficient information about a new user or item, degrading overall performance. The Sparsity Problem arises due to limited user assessments, leading to sparse matrices, difficulty finding resemblances, and weak recommendations. Scalability becomes a concern as regular computing grows linearly with users and items, impacting the system’s ability to produce satisfactory recommendations with an increased dataset. To address this, dimension reduction techniques like Singular Value Decomposition (SVD) are proposed. The Running Time Problem emphasizes the need for algorithmic improvements to enhance the speed of recommendations, and finally, the Accuracy Problem is a recurring challenge. The majority of issues lie in Collaborative Filtering, followed by Content-Based Filtering and Hybrid Filtering, with ongoing efforts to enhance accuracy through diverse research methods. The study underscores the dynamic nature of recommendation system research, where addressing these challenges is an ongoing pursuit for perfection.

Evaluation Method

The recommender system is a complex system which can involve a lot of different approaches, dataset and contextual information for the user’s profile or item details, however, the recommendation result is based on the user preference and there is no absolute answer in practice. Evaluation methods become an essential question to all recommendation researches.

Based on the research, the majority of the researches are using RMSE and MAE.

Probability theory offers a logical resolution to the choice between RMSE and MAE, with one of these metrics proving optimal when applied correctly, although practical scenarios may necessitate the use of multiple metrics. Enhancements in model refinement, data transformation, the adoption of robust statistics, or the construction of improved probabilities can enhance results in specific cases, with the latter being the most versatile option, albeit pragmatic considerations may favor other approaches. Additionally, it is highlighted that RMSE is optimal for normally distributed errors, countering the misconception that MAE is exclusive to uniformly distributed errors. Despite MAE’s greater power, there are superior alternatives, and both metrics lack theoretical justification and fail to introduce the extensive literature on the subject.

Dataset

Recommendation systems are often tailored to specific problems, for example, book, music or even stock recommendation. Various datasets sometimes contain distinct contextual information, the recommendation system is not generic to all problems most of the time.

Among 72 researches in the literature review, 44 of the studies are using the Movielens dataset.

Final words

The author highlighted a lot of recommender system studies, many of them proposed novel approaches, however, accuracy is still the major concern across the recommender system researches.

Thank you for reading, and have a nice day!
Please let me know if there is acknowledgement concern in the post.

Reference

1. I. Saifudin and T. Widiyaningtyas, “Systematic Literature Review on Recommender System: Approach, Problem, Evaluation Techniques, Datasets,” in IEEE Access, doi: 10.1109/ACCESS.2024.3359274.
2. S. Sahu, R. Kumar, M. S. Pathan, J. Shafi, Y. Kumar, and M. F. Ijaz, “Movie Popularity and Target Audience Prediction Using the Content-Based Recommender System,” IEEE Access, vol. 10, pp. 42030–42046, 2022, doi: 10.1109/ACCESS.2022.3168161.
3. T. Widiyaningtyas, M. I. Ardiansyah, and T. B. Adji, “Recommendation Algorithm Using SVD and Weight Point Rank (SVD-WPR),” Big Data and Cognitive Computing, vol. 6, no. 4, 2022, doi: 10.3390/bdcc6040121.
4. H. Tahmasebi, R. Ravanmehr, and R. Mohamadrezaei, “Social movie recommender system based on deep autoencoder network using Twitter data,” Neural Comput Appl, vol. 33, no. 5, pp. 1607–1623, 2021, doi: 10.1007/s00521–020–05085–1.

Recommender System Literature Review 2019–2023 was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

In today’s data-driven world, organizations are continuously grappling with massive amounts of information scattered across multiple systems. Extracting valuable insights from such diverse data sources can be a daunting task. This is where ETL (Extract, Transform, Load) comes to the rescue. ETL serves as the backbone of data integration, offering a robust framework for extracting, transforming, and loading data into a consolidated and actionable format. In this blog post, we delve into the world of ETL and how can we integrate the Google Cloud Platform services to do the data ingestion.

So, What is ETL?

ETL stands for Extract, Transform, Load. It refers to a process in data management and data integration that involves extracting data from various sources, transforming it into a desired format or structure, and loading it into a target system or data repository for analysis, reporting, or other purposes.

• Extract: Data is extracted from multiple sources, such as databases, files, APIs, or external systems. The extraction process involves connecting to the source systems, identifying the data to be extracted, and retrieving it. The extracted data can be from structured, semi-structured, or unstructured sources.
• Transform: Once the data is extracted, it undergoes a series of transformations to convert it into a consistent format, standardize values, clean and validate the data, and apply any necessary calculations or business rules. Transformations can include filtering, aggregating, sorting, joining, or applying complex algorithms to the data. The goal is to prepare the data for analysis or loading into the target system.
• Load: Transformed data is loaded into a target system or data repository, such as a data warehouse, data mart, or database. The loading process involves mapping the transformed data to the target schema and structure, applying any necessary data mappings or conversions, and inserting or updating the data in the target system.

ETL vs ELT

In ETL, the transformation step occurs after data extraction and before loading into the target system. This means that the extracted data is transformed and cleaned in a separate processing engine or environment before being loaded into the target system. The transformed data is typically loaded into a structured format, such as a data warehouse, for subsequent analysis and reporting.

ELT, on the other hand, is an alternative approach to data integration and processing. Data is extracted from source systems and loaded into the target system in its raw or near-raw form, often without significant transformations. The loading process takes place first, directly into a scalable storage system, such as a data lake or a cloud-based storage service. Once the data is loaded, the transformations are performed directly within the target system using its built-in processing capabilities or specialized tools. This approach leverages the power of modern cloud-based data processing engines that can handle large volumes of data and perform transformations at scale.

How can we utilize GCP service to build a data ingestion pipeline?

Google Cloud Platform (GCP) provides a range of services and tools that can greatly assist in building ETL pipelines. Here are some key GCP services and features that can be leveraged for ETL:

1. Cloud Storage: GCP’s Cloud Storage offers a highly scalable and durable storage solution for storing raw data, intermediate files, and transformed data. It can be used as a landing zone for ingesting and staging data before processing.
2. BigQuery: BigQuery is a fully managed, serverless data warehouse service on GCP. It provides powerful querying and analysis capabilities, making it an excellent choice for loading and analyzing transformed data. BigQuery supports batch and streaming ingestion methods and can handle large-scale data processing.
3. Dataflow: Google Dataflow is a managed service for building scalable data pipelines. It offers a unified programming model for both batch and stream processing and integrates well with other GCP services. Dataflow simplifies the development and execution of ETL workflows, supporting data transformations and parallel processing.
4. Cloud Pub/Sub: Pub/Sub is a messaging service that allows you to decouple data producers and consumers. It can be used to ingest real-time data streams, providing reliable messaging between different components of the ETL pipeline.
5. Cloud Functions/Cloud Run: GCP’s serverless compute platforms, such as Cloud Functions and Cloud Run, can be used to perform lightweight data transformations or execute specific tasks within the ETL pipeline. They enable scalable and event-driven execution of code without worrying about infrastructure management.

These services can be combined to build a robust and scalable ETL pipeline on GCP. The choice of services depends on your specific requirements, data sources, transformation logic, and target systems. GCP’s integrated ecosystem and managed services make it a powerful platform for designing, deploying, and maintaining ETL workflows in a cloud environment.

Conclusion

In conclusion, ETL is a critical process for organizations seeking to leverage their data for strategic insights and decision-making. By extracting, transforming, and loading data into a target system, businesses can derive meaningful information and drive actionable outcomes. Google Cloud Platform (GCP) offers a powerful suite of services and tools that greatly facilitate the development and implementation of ETL pipelines. With GCP’s robust data storage, processing, and analytics capabilities, organizations can streamline their ETL workflows, improve data quality and consistency, and achieve scalable and efficient data processing. Whether it’s the cloud storage and messaging capabilities of Cloud Storage and Pub/Sub, the data transformation capabilities of Dataflow, or the analytical power of BigQuery, GCP provides a comprehensive ecosystem for building end-to-end ETL pipelines. By harnessing the potential of ETL and GCP, organizations can unlock valuable insights, make data-driven decisions, and stay ahead in today’s data-driven business landscape.

I plan to write a series of post about ETL by utilizing GCP. we will mainly focus on building a robust, reliable data ingestion pipeline via python and terraform to build on the cloud service.

Thank you for reading, and have a great day!!

Golang is an open source programming language supported by Google with a growing user community. Golang is easy to learn and fast to get started with. To compile the program smoothly without problem, it is good to start the Golang project structure. A structured project will make our life easier and reduce the maintenance cost.

In this guide, we will talk about the following topics, feel free to jump to the topic you are looking for.

• Simple Project
• Multi-Package Project
• Multi-Main Project
• Multi-Project Workspace (Golang > 1.18)
• Vendor? Go Mod?

You can find all the code on Github. Please clone or fork the project, and try it locally. You will have a better picture of the structure.

https://github.com/chiang9/golang_project_structure

Simple Project

To begin, we will start with a simple project with a main file which imports some external functions.

simple_project/
├── function1.go
├── go.mod
├── go.sum
└── hello.go

In hello.go, we import the function from function1.go and a third party module, golang.org/x/example. The third party module will be automatically downloaded when running

go mod tidy

> go run simple_project
Hello, world
Hello, world From function1.go - package = main
egakcap ytrap dr3 morf dlrow ,olleH

Note: we are able to use the function from the same module without importing even it is from a different file

Multi-Package Project

In this example, we will showcase a project with multiple packages. It is possible to use various packages in the project.

multi_module_project/
├── cmd
│   ├── mod2
│   │   └── mod2.go
│   └── mod3
│       ├── mod3.go
│       └── mod3_sub
│           └── mod3_sub.go
├── go.mod
├── main.go
└── mod1
    └── mod1.go

main.go:

> go run main.go
Hello, world
Hello, world From mod1.go - package = mod1
Hello, world From mod2.go - package = cmd/mod2
Hello, world From mod3.go - package = cmd/mod3
Hello, world From mod3_sub.go - package = cmd/mod3/mod3_sub

Multi-Main Project

Most of the time, we will need multiple functions for a project. We are also able to have multiple main functions under different modules and we can further build the functions into executables.

multi_main_project/
├── cmd
│   ├── func1
│   │   └── func1.go
│   ├── mod1
│   │   ├── mod1_func
│   │   │   └── mod1.go
│   │   └── mod1_main.go
│   └── mod2
│       └── mod2_main.go
└── go.mod

mod1_main.go:

> go run multi_main_project/cmd/mod1
Hello, world from mod1
Hello, world From func1.go - package = cmd/func1
Hello, world From mod1.go - package = cmd/mod1/mod1_func

> go run multi_main_project/cmd/mod2
Hello, world from mod2
Hello, world From func1.go - package = cmd/func1
Hello, world From mod1.go - package = cmd/mod1/mod1_func

In this project, we have to specify the module we want to run. Simply running go run . will return error.

Note: all the files under the same folder must be in the same package, otherwise it will return error. For example, all files in cmd/mod1 must be the same package, but the package can be different in cmd/mod1/mod1_func

Multi-Project Workspace

Starting from Golang 1.18, the workspace mode has been added. The workspace mode gives us the power to develop the Go project across multiple projects.

Get familiar with workspaces

In this example, we will showcase a basic usage of go work. To begin, we can use the following code to generate the work file.

go work init

After the projects are ready, the project can be importable by running

go work use <project_name>

workspace_project/
├── go.work
├── project1
│   ├── go.mod
│   ├── main.go
│   └── mod1
│       └── mod1.go
└── project2
    ├── go.mod
    ├── main.go
    └── mod1
        └── mod1.go

go.work:

project2/main.go:

> go run project2
Hello, world
Hello, world From mod1.go - package = mod1, project = project1
Hello, world From mod1.go - package = mod1, project = project2

Vendor? Go Mod?

Sometimes we will see a vendor folder in a Golang project, so what exactly is the vendor folder and why do we have it?

When Go language was first published, go get was prebuild to download the third party package, however, the tool was not able to specify the version of the package to download, it will always download the latest version. As you can imagine, this will definitely cause issue when the author of the package makes some major changes to the function, our program will no longer work.

The solution to this issue was the Vendor. The community creates several package management tools, for example, gvt, govendor, godep, glide……etc. The tool will specify the version of the package and download the source code, which will ensure the stability of your project.

Since Golang 1.13, Go starts supporting the vendor feature, and we can start using the vendor feature which is supported officially.

Since Golang 1.15, Go starts supporting the Module feature, which allow us to specify the version when installing the third party package.

Go Modules

So, do we still need the vendor folder since we are able to specify the package version?

Golang is still able to work with the vendor feature, and people are still using it, but we should look forward to deprecate the feature unless there are special needs for the vendor folder.

Thank you for reading, and have a great day!!

Note: this is not an official standard defined structure by the core Go dev team.

Reference:

GitHub - golang-standards/project-layout: Standard Go Project Layout

Golang Project Structure Beginner Guide 2022 was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

Time Series data is a sequence of data points indexed in time order. The most common example of time series data is the daily closing price of the stock market. Time series analysis can help organizations understand the underlying causes of trends or systemic patterns over time.

In this post, we will showcase the Time_Series_Transform package with practical Machine Learning Model on the Stock Market. Time-Series-Transform can help us to reformat the time series data, and simplify the feature engineering process.

Time-Series-Transform

You can find more details about the package from the previous post, please take a look!!

A New Data Approach in Time Series Analysis

Installation

pip install time-series-transform

Note: Make sure tensorflow and plotly are installed on your machine

Let’s get started

Machine Learning is known to be one of the branch of artificial intelligence and one of the most common methods to provide mathematics models to forecast the data. Feature Engineering is an essential steps of transforming the raw data into desired features for models. We are able to generate new features from the raw data in order to simplify, speeding up, and enhance the model accuracy.

This demo is going to train a machine learning model using sklearn, Stock_Transformer, and Time_Series_Transformer to predict the momentum of stock on daily basis. Subsequently, we will use the momentum to generate buy/sell signals and make a plot using Time_Series_Transformer.

Note: This tutorial is aim to demonstrate how to use time_series_transformer, instead of showing how to invest in stock

By using Nasdaq and Gold, we can enrich the features. we expand the Nasdaq and Gold into feature columns. Next, we use make_lead features combining with the up_down_transform function to generate the pricing moment for the next time period.

By separating the data into train and test set, we are ready to build our machine learning pipeline. In this example, we will only build a simple model.

                 precision    recall  f1-score   support

        down       0.40      0.12      0.19        33
    unchange       0.00      0.00      0.00         1
          up       0.40      0.77      0.53        26

    accuracy                           0.40        60
   macro avg       0.27      0.30      0.24        60
weighted avg       0.39      0.40      0.33        60

[[ 4  0 29]
 [ 0  0  1]
 [ 6  0 20]]

We generate the prediction based on our machine learning model. Time_Series_Transform can also give us a hand on visualization.

Before we create the plot, we need to process our result to generate some buy/sell signal. To generate the signals, we use the following rules:

1. the first period should be buy.
2. buy and then hold until sell
3. buy when the predict value is up
4. sell when the predict value is down

Even though the model is not giving a very promising outcome, but the package helps us in processing the data and to visualize the result easily.

What’s Next…

In this example, we build the machine learning model by utilizing the time_series_transform package. This package can also help us in building the Neural Network model features with the power of tensorflow. We are able to transform the data into tfRecord in one line of code.

You can find more detail in the documentation.

Thank you for reading, have a great day!!

Mlearning.ai Submission Suggestions

Time Series data is a sequence of data points indexed in time order. The most common example of time series data is the daily closing price of the stock market. Beside the stock market, we do encounter a lot of different time series data, for instance, the climate changes across time or the sales revenue of a company. Time series analysis helps organizations understand the underlying causes of trends or systemic patterns over time.

Today, we want to dive into a fancy and powerful time series data structure engine.

Time-Series-Transformer

The package not only provides functionalities to process the time series data, but it shines when we need to process multi-dimensional time series data. Moreover, the submodule, Stock_Transformer, is able to extract the stock data and calculate the technical indicators in just a few lines of code.

Installation

pip install time-series-transform

Note: Make sure tensorflow and plotly are installed

Data

In this example, we will use the climate time series data in Delhi from Kaggle

https://www.kaggle.com/datasets/sumanthvrao/daily-climate-time-series-data?resource=download

Let’s get started

When we are processing the time series data, we always need to take extra attention on the time order, and it is not easy to do so with solely numpy or even pandas. The Time_Series_Transform makes the task easier than ever.

There are two main modules in the package.

• Time_Series_Transformer: the core module and it is compatible with user defined time series data
• Stock_Transformer: helps us to extract stock data and provide operation to embed the technical indicator calculation library (pandas-ta)

Time Series Transformer Core

Time_Series_Transformer can helps us to format the time series data and we are able to manipulate the time series data easily.

From the dataset, we can see there are 4 different features, and we can load the data into our package from pandas.dataFrame. The timeSeriesCol will be the column of the time index, and the mainCategory field allows us to handle multi-dimensional data, we will talk about it later.

ime_Series_Transformer data
data column
-----------
date
meantemp
humidity
wind_speed
meanpressure
time length: 1462
category: None


We can slice the data from the time_series_data as below
{'date': array(['2013-01-01', '2013-01-02', '2013-01-03'], dtype=object),
 'meantemp': array([10.        ,  7.4       ,  7.16666667]),
 'humidity': array([84.5, 92. , 87. ]),
 'wind_speed': array([0.        , 2.98      , 4.63333333]),
 'meanpressure': array([1015.66666667, 1017.8, 1018.66666667])}

Data Manipulation

Most of the time, we need to preprocess the data before entering the model. In time series analysis, making lag or lead data (shifting data) is one of the most common methods. The package also provide the function to shift data.

• make_lead
• make_lag
• make_lead_sequence
• make_lag_sequence

make_lead and make_lag shift the data

make_lead_sequence and make_lag_sequence shift the data and return a window list data as feature. It is useful in producing Deep learning feature

{'date': array(['2013-01-01', '2013-01-02', '2013-01-03'], dtype=object),
 'meantemp': array([10.        ,  7.4       ,  7.16666667]),
 'humidity': array([84.5, 92. , 87. ]),
 'wind_speed': array([0.        , 2.98      , 4.63333333]),
 'meanpressure': array([1015.66666667, 1017.8, 1018.66666667]),
 'meantemp_lead_3': array([8.66666667, 6.        , 7.        ]),
 'wind_speed_lead_3': array([1.23333333, 3.7       , 1.48      ]),
 'meantemp_lead_seq_3': array([[7.4       , 7.16666667, 8.66666667],
        [7.16666667, 8.66666667, 6.        ],
        [8.66666667, 6.        , 7.        ]]),
 'meanpressure_lead_seq_3': array([[1017.8, 1018.6667, 1017.1667],
        [1018.66666667, 1017.16666667, 1016.5       ],
        [1017.16666667, 1016.5       , 1018.        ]])}

User defined function

It is also possible to use the user defined function. We can use the transform function to customize the data process.

Restriction of the customize function:

• the function must take an array
• output must be an array with the same size as the original data length

Note: time_series_transform.transform_core_api.util provides some general functions as we imported at the beginning

data column
-----------
date
meantemp
humidity
wind_speed
meanpressure
meantemp_lead_3
wind_speed_lead_3
meantemp_lead_seq_3
meanpressure_lead_seq_3
wind_cust_10
time length: 1462
category: None

Plot

The package also provides function to draw plot and it supports several functions customize the plot.

• add_line
• remove_line
• update_layout
• add_marker

Multi-Dimensional Time Series Data

Sometimes when we are doing Exploratory Data Analysis or preprocessing the time series data before entering the model, we might want to separate the data by their features. Handling multi-dimensional time series data is the strong point of the package.

Time_Series_Transformer can specify the mainCategoryCol parameter to point out the main category. This class only provide one columns for main category because multiple dimensions can be aggregated into a new column as main category.

From the documentation. With the mainCategoryCol, we are able to separate the data by the category and perform same data operations as we did previously.

Next we can further apply the functionality to perform advance preprocessing. To demonstrate the capability of the Time_Series_Transformer, we can separate the data by their month and investigate the behavior of the data.

Moreover, we can also use the plot function on the multi-dimension data, and it will generate several lines and we can select which month we want.

Stock Transformer

Speaking of the time series analysis, the stock market must be the interest of most people. Stock_Transformer makes our life easy and simple. Extraction of the stock data can be done in just one line of code, and we can also use the power of the data manipulation like how we did previously.

Stock extraction methods:

• from_stock_engine_date
• from_stock_engine_period
• from_stock_engine_intraday

To calculate the standard technical indicators, we can use the get_technial_indicator with pandas-ta.

Date       Open       High        Low      Close     Volume  \
25  2019-02-07  41.725962  42.098690  41.227381  41.372601  126966800   
26  2019-02-08  41.076051  41.481974  40.937501  41.421207   95280000   
27  2019-02-11  41.576775  41.615667  41.139252  41.183002   83973600   
28  2019-02-12  41.345854  41.564614  41.248625  41.537876   89134000   
29  2019-02-13  41.659413  41.924356  41.302103  41.365299   89960800   

    Dividends  Stock Splits  MACD_12_26_9  MACDh_12_26_9  MACDs_12_26_9  \
25     0.0000           0.0      1.952193            NaN            NaN   
26     0.1825           0.0      1.915505            NaN            NaN   
27     0.0000           0.0      1.845929            NaN            NaN   
28     0.0000           0.0      1.798691            NaN            NaN   
29     0.0000           0.0      1.727417            NaN            NaN   

       EMA_10        BBL        BBM        BBU  Bandwidth   Percent  
25  40.383502  34.842427  38.655327  42.468227  19.727684  0.856326  
26  40.572176  34.930402  38.865177  42.799952  20.248332  0.824800  
27  40.683235  35.136170  39.081391  43.026611  20.189764  0.766349  
28  40.838624  35.475825  39.343060  43.210295  19.659045  0.783771  
29  40.934383  35.746722  39.558949  43.371175  19.273650  0.736915

Reference

Time Series Transformer:

https://allen-chiang.github.io/Time-Series-Transformer/introduction.html

https://github.com/allen-chiang/Time-Series-Transformer

Thank you for reading, have a great day!!

A New Data Approach in Time Series Analysis was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Apache Spark and Apache Hadoop are both open-source frameworks for big data processing. Spark can be 100x faster than Hadoop for large scale data processing, however, Hadoop has distributed file system (HDFS) while Spark is built on resilient distributed datasets (RDDs). Usually, we use Spark for data computation on top of Hadoop.

In this article, we will focus on the data manipulation of RDD.

All code can be found on github.

Dataset:

We will be using the movielens dataset (ml-100k) for demo.

Datacamp also provides a clear cheat sheet for pyspark.

More Information: https://www.datacamp.com/community/blog/pyspark-cheat-sheet-python

1. Initializing Spark Environment

Whenever we want to use the Spark Engine, we need to first initialize the Spark environment.

SparkContext

The driver program will use the SparkContext to connect and communicate with the cluster and it coordinate the jobs with the resource management system.

SparkSession

Starting from Spark 2.0, SparkSession built the gateway to different data sources, such as SQL or Hive. In Spark 1.x, we need to define the SQLContext or HiveContext to communicate with the corresponding sources.

In short, we need to define the SparkSession if we want to use the Spark Dataframe or Dataset, otherwise SparkContext can execute the RDDs

2. Loading Data

We are able to create the rdd with iterable data or external data sources. We will first load the movielen datafile as text string, and do the manipulation later.

Note: SparkSession need to be defined before accessing the rdd.toDF() function, otherwise exception will be thrown.

+------+---+
|    _1| _2|
+------+---+
| apple| 10|
|orange| 20|
| peach|100|
+------+---+

3. Data Retrieval

By using the following commands, we are able to retrieve information of the data in the rdd.

• getNumPartitions()
• count(), countByKey(), countByValue()
• collect(), collectAsMap(), take(num), top(num)
• max(), min(), mean(), stdev(), variance(), histogram(num of bin), stats()

rdd_movie.take(3)

>> ['196\t242\t3\t881250949', '186\t302\t3\t891717742', '22\t377\t1\t878887116']

4. Data Processing

Processing data in PySpark might reminds you about the Pandas Dataframe. Similar to Pandas, PySpark also provide the functionality to group, aggregate, sort, and reduce. However, there are some functions which are similar but perform differently.

map vs flatMap

map and flatMap looks similar but they are different, and worth extra attention.

The map function is able to maintain the list structure of the original data shape, and the flatMap will unpack the list structure and form a large list data structure.

In the following cells, we will visualize the difference by separating the rdd_movie rdd with proper delimiter.

groupBy vs groupByKey

groupBy function will group the data based on the result from the function in the input, groupByKey will group the data based on the key of the original rdd.

groupby result:

userid last digit = 6
['196', '242', '3', '881250949']
['186', '302', '3', '891717742']
['166', '346', '1', '886397596']
['6', '86', '3', '883603013']

groupByKey result:

rating = 1
['22', '377', '1', '878887116']
['166', '346', '1', '886397596']
['181', '1081', '1', '878962623']
['276', '796', '1', '874791932']

reduce vs reduceByKey

Before we dive into the code, we need to realize that reduce is an action and reduceByKey is transformation.

Definition on the original document: https://spark.apache.org/docs/2.2.0/rdd-programming-guide.html

reduce: Aggregate the elements of the dataset using a function func (which takes two arguments and returns one). The function should be commutative and associative so that it can be computed correctly in parallel.

reduceByKey: When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function func, which must be of type (V,V) => V. Like in groupByKey, the number of reduce tasks is configurable through an optional second argument.

In this section, we will demonstrate some examples of the functions.

5. Sample usage on MovieLen dataset

In this section, we will use the PySpark RDD to process the data of the movie, and retrieve the corresponding genre.

[['1', 'Toy Story (1995)', ['Animation', "Children's", 'Comedy']],
 ['2', 'GoldenEye (1995)', ['Action', 'Adventure', 'Thriller']]]

Thank you for reading, Happy new year.

PySpark CheatSheet and More was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.