Our goal is to develop a contextually relevant search system for the Khmer language using text embeddings generated by a Large Language Model (LLM). Instead of relying on traditional keyword matching, this approach enables the search engine to understand the meaning and context behind words — returning results that are semantically related.

To achieve this, we experimented with several embedding models and found the Language-agnostic BERT Sentence Embedding (LaBSE) model [4] to perform best for Khmer. The embeddings produced by LaBSE are stored and queried through a vector database, enabling powerful semantic retrieval.

In this post, we’ll walk through the end-to-end implementation using Postgres DB, and share how the results revealed some surprising cross-lingual connections between Khmer and English terms.

Previous Works

Khmer Word Search (Buoy et al.)

One notable approach comes from Buoy et al. (2021) in “Khmer Word Search: Challenges, Solutions, and Semantic-Aware Search” [2112.08918]. This study focuses on solving practical problems with Khmer script that make word search difficult, such as words that look the same but are written differently, spelling variations, and stacked versus unstacked syllables.

To address these issues, the authors created several processing steps: character normalization to make spellings consistent, spell-checking at both the letter and sound levels (using a G2P model and the SymSpell algorithm), and a FastText word-embedding model to find semantically query words.

Word-embedding model was trained on Khmer text from websites and news articles to create embedding vectors for each term. When a user enters a search term, the system uses these embedding to find several related terms that have similar embedding from FastText. The searches then use these terms as OR query. This approach improves recall by finding more relevant documents even when words are written differently or different words completely.

Buoy’s method improves Khmer search by fixing spelling and pronunciation differences and expanding queries with related terms. However, it works mainly on individual words and does not fully capture the broader context like phrase or sentences.

As a side notes, if you need a start on Khmer corpora, we have some available on github (https://github.com/phylypo/khmer-text-data, segmentation-crf-khmer) that includes Wikipedia, Oscar and 10,000 Khmer news articles that we crawled in 2019.

Khmer Semantic Search Engine (KSE)

Another approach is by Nimol Thuon (2024) in “Khmer Semantic Search Engine (KSE): Digital Information Access and Document Retrieval” [2406.09320].

The paper presents KSE, a system designed to improve Khmer search results using three parts: keyword search, ontology-based meaning expansion, and weighted ranking.

The system first identifies important words in each document using TF-IDF and basic text processing. It then uses a tourism ontology — a structured map of related places and terms — to better interpret user queries. Finally, it ranks results by giving more weight to keywords found in titles than in the body text.

KSE was tested on about 1,150 tourism articles, with human reviewers providing relevance scores. The system showed that combining keyword search with ontology relationships can improve accuracy in domain-specific Khmer search

KSE is essentially an ontology-enhanced keyword search engine. It enhances traditional keyword search by using domain relationships to expand queries.

Both early approaches expand the input query to more terms and perform keyword search on the document.

Introduction to Embeddings

Word embeddings are a foundational concept in natural language processing (NLP). They represent words as vectors of decimal numbers, capturing semantic meaning in a numerical form that computers can process.

One of the earliest and most influential models in this area was Word2Vec [3], which learns these representations by analyzing large text corpora and identifying words that frequently appear together. Through this process, Word2Vec captures semantic relationships between words — how they relate in meaning rather than just in spelling or frequency.

A well-known example of this capability is the analogy:

King - Man + Woman ≈ Queen

This simple vector operation illustrates how Word2Vec encodes relationships like gender and royalty directly within the embedding space. The idea that meaning can be represented and manipulated mathematically became a cornerstone for future advances in NLP.

Today, this foundational concept has evolved into modern text embedding models, such as OpenAI’s text-embedding-3. These newer models operate in high-dimensional vector spaces, capturing deeper contextual and semantic nuances across entire sentences or paragraphs. As a result, they significantly enhance tasks like semantic search.

Text Embeddings for Context-Aware

Our goal is to enhance Khmer language search by moving beyond traditional keyword matching and leveraging text embeddings. Earlier approaches — such as Word2Vec or FastText — work well for representing individual words but often fail to capture the contextual meaning of phrases or sentences. For example, the word “bank” can refer to an “investment bank” or a “river bank,” depending on context. Text embeddings address this limitation by representing the semantic meaning of entire phrases and sentences, enabling more contextually relevant search results.

This approach aligns with techniques commonly used in Retrieval-Augmented Generation (RAG) systems, where embeddings are used to retrieve the most relevant documents to assist Large Language Models (LLMs) in generating accurate and contextually aware responses. We recognize the increasing capability of modern LLMs — such as Google Gemini and OpenAI’s GPT-5 — in understanding and representing Khmer text, and we aim to leverage these advances to improve Khmer search performance.

One key advantage of using embeddings is their tolerance to spelling variations and misspellings, as semantically similar phrases often appear close together in the embedding space — provided the model has seen similar examples during training. Since we are using pre-trained models without direct access to their training data, we plan to spot-check and analyze the results to infer their effectiveness for Khmer content.

To identify the best-performing approach, we will evaluate both open-source and commercial embedding models (including OpenAI’s state-of-the-art offerings). Our evaluation focuses on how well these models generate high-quality, context-aware embeddings for Khmer text, which are crucial for building a robust Khmer Semantic Search system.

Evaluate Embeddings Models

We look at some candidates and narrow down to two models from Open Source and commercial. Here are some candidates we looked at:

Open Source Multilingual Embedding Models

LaBSE (Language-agnostic BERT Sentence Embeddings) [4]

• Developed by Google, trained in 109 languages (Khmer included).
• Works well for cross-lingual semantic similarity.
• Widely used baseline for multilingual RAG.

distiluse-base-multilingual-cased-v2 (from sentence-transformers)

• A distilled version of multilingual Universal Sentence Encoder.
• Supports 50+ languages, including Khmer.
• More lightweight than LaBSE, often faster in production.

Commercial API Models

OpenAI text-embedding-3-large or text-embedding-3-small

• Trained on a very wide set of languages (unspecified about Khmer).
• High-quality embeddings with cross-lingual alignment.
• Scales easily for production RAG pipelines.

Cohere Multilingual Embeddings

• Strong cross-lingual semantic search performance.
• Support over 100 languages including Khmer (Cohere’s Embed Models (Details and Application) | Cohere)

We chose two options (one from commercial and one from open source) based on their features description. We chose LaBSE and commercial and OpenAI’s text-embedding-3 models based on Khmer support, performance, and accessibility. Here are some highlights between the two models.

Evaluation Setup

In order to compare the performance of these two models, we performed a series of basic tests to gain insight into the embedding space and the models’ performance on Khmer text. While the raw vector values are not directly interpretable, we compare them using a cosine similarity score (ranging from 0 for unrelated to 1 for perfectly related) to measure the semantic relationship between terms. These preliminary tests are not intended to be a concrete quantitative evaluation score but rather to provide a qualitative sense of each model’s language support for Khmer.

Test Criteria 1: Translation (Khmer — English)

The first evaluation criterion is designed to measure the cross-lingual semantic alignment between English and Khmer terms. Achieving a high score on this test is a strong indication that an embedding model successfully represents equivalent words from both languages within the same shared vector space.

The table below presents the cosine similarity scores for direct translations:

The results, which use Cosine Similarity (where 1.0 is very similar and 0.0 is no similarity), show a stark contrast, with scores decisively favoring LaBSE for cross-lingual translation tasks.

Test Criteria 2: King-queen analogy

This test uses the well-known “King-Queen” analogy to illustrate how the model’s embedding space captures semantic relationships through vector arithmetic. While not a core quantitative metric, it provides an insight into how relative categories are encoded within the vector space.

For the Khmer analogy, a custom test such as “(boy or male) — (man) + (woman) ≈ (girl or female)” yielded a high similarity score of 0.9263 with LaBSE. This, combined with the other results, suggests that the LaBSE model provides clear support for vector arithmetic and successfully encodes semantic relationships like gender and royalty in the Khmer embedding space, again decisively For the Khmer analogy, a custom test such as “(boy or male) — (man) + (woman) ≈ (girl or female)” yielded a high similarity score of 0.9263 with LaBSE.

This, combined with the other results, suggests that the LaBSE model provides clear support for vector arithmetic and successfully encodes semantic relationships like gender and royalty in the Khmer embedding space, again decisively favoring LaBSE over the OpenAI model.over the OpenAI model.

words = ["ប្រុស",  # boy or male
         "បុរស",  # man
         "ស្ត្រី",    # woman 
         "ស្រី"]    # girl or female
# Cosine similarity: 0.9263  : analogy (male - man + woman = female)
# Cosine similarity: 0.6896  : male, female
# Cosine similarity: 0.7360  : man, woman
# Cosine similarity: 0.9448  : male, man
# Cosine similarity: 0.8900  : woman, female

Test Criteria 3: Similarity and Dissimilarity

This test begins with a basic check using keywords to determine how similar words cluster in the embedding space and, conversely, how unrelated words are represented with lower similarity scores.

Testing with the OpenAI large model (text-embedding-3-large) showed that similarity scores for unrelated Khmer terms (0.47 and 0.38) were higher than the score for related “eat” synonyms (0.32). This suggests the OpenAI embedding model does not cluster the embedding space for Khmer text as effectively as LaBSE.

Model Testing Conclusion:

Our initial tests were limited to single terms and short phrases. Despite this, LaBSE consistently delivered better similarity scores in both cross-lingual and vector arithmetic evaluations, showing a strong correlation between Khmer and English terms. Based on this preliminary evidence, LaBSE is selected as the foundational embedding model for our Khmer Semantic Search system.

LaBSE for Khmer Text Embedding

LaBSE is an excellent choice for a search system, especially for resource-limited languages like Khmer, because it creates strong, multilingual sentence embeddings that maintain semantic consistency across languages, which is essential for effective search.

Tokenization

Since Khmer is morphologically complex and has no spaces between words, proper tokenization matters. The use of a Khmer word segmentation before embedding might be a good strategy to improve accuracy since many tokenizers may not know how to segment Khmer words. But if their training process does not run through this process, this can affect the embedding process in a negative way. So next we will test how the tokenization works for this model.

For Khmer text, we use the zero width space “\u200B” character as a word separation for spell check and other purposes. Here is an example Khmer text that uses zero width space between words, not visible.

Khmer text: តាម​ទម្លាប់​នៅ​ក្នុង​សហគមន៍​ខ្មែរ កាលណា​មាន

We are going to test the tokenizer to see how it treats “\u200B” characters.

We are going to test the tokenizer to see how it treats “\u200B” characters.

Input text: តាម​ទម្លាប់​នៅ​ក្នុង​សហគមន៍​ខ្មែរ កាលណា​មាន

Tokens: [‘តាម’, ‘##ទ’, ‘##ម្លាប់’, ‘##នៅ’, ‘##ក្នុង’, ‘##ស’, ‘##ហ’, ‘##គ’, ‘##មន៍’, ‘##ខ្មែរ’, ‘កាល’, ‘##ណា’, ‘##មាន’]

Token IDs: [23416, 118714, 499577, 107664, 113574, 43611, 214354, 200777, 443523, 119830, 36040, 68867, 47903]

The original text has 8 words, the model tokenizes this into 13 tokens. The ‘##’ characters in the token indicate this token part of the previous word. For example: កា​លណា​ as two tokens ‘កាល’ and ‘##ណា’. The later token with ## means that it preceded by the previous token.

Now let’s test without any spaces:

Original text without any zero width spaces: តាមទម្លាប់នៅក្នុងសហគមន៍ខ្មែរ កាលណាមាន

Tokens: [‘តាម’, ‘##ទ’, ‘##ម្លាប់’, ‘##នៅ’, ‘##ក្នុង’, ‘##ស’, ‘##ហ’, ‘##គ’, ‘##មន៍’, ‘##ខ្មែរ’, ‘កាល’, ‘##ណា’, ‘##មាន’]

Token IDs: [23416, 118714, 499577, 107664, 113574, 43611, 214354, 200777, 443523, 119830, 36040, 68867, 47903]

The result is the same 13 tokens as the text with zero width spaces. Now tries by adding zero width space “\u200B” characters to many incorrect position to see how the tokenizer output:

Text: តា​ម​ទម្លាប់​នៅ​ក្នុ​ង​សហគមន៍​ខ្មែរ កា​លណា​មា​ន

Tokens: [‘តាម’, ‘##ទ’, ‘##ម្លាប់’, ‘##នៅ’, ‘##ក្នុង’, ‘##ស’, ‘##ហ’, ‘##គ’, ‘##មន៍’, ‘##ខ្មែរ’, ‘កាល’, ‘##ណា’, ‘##មាន’]

Token IDs: [23416, 118714, 499577, 107664, 113574, 43611, 214354, 200777, 443523, 119830, 36040, 68867, 47903]

The consistent token output across both tests confirms that the LaBSE tokenizer effectively removes the zero-width space (\u200B) character during preprocessing, meaning this character has no impact on tokenization.

However, in both results, many tokens were subword units (indicated by the ## prefix), showing the model was not tokenizing to individual Khmer words. To investigate the potential for improvement, we manually separated the words using a standard space:

Text: តា​ម ​ទម្លាប់​ នៅ​ ក្នុ​ង​ សហគមន៍​ ខ្មែរ កា​លណា​ មា​ន

Tokens: [‘តាម’, ‘ទម្លាប់’, ‘នៅ’, ‘ក្នុង’, ‘សហគមន៍’, ‘ខ្មែរ’, ‘កាល’, ‘##ណា’, ‘មាន’]

Token IDs: [23416, 381475, 16636, 17367, 160164, 43067, 36040, 68867, 16887]

As shown above, explicit spacing did result in the tokenizer recognizing whole words like ‘ទម្លាប់’ and ‘សហគមន៍’ as single tokens. While tokenizing to whole words appears logically sound, the key concern is the model’s training data. If the corpus used to train LaBSE was unsegmented, then explicitly adding spaces now might negatively impact the quality of the resulting embedding vectors. Conversely, if the training text was segmented, then adding spaces would be the correct approach. Given that Khmer uses spaces to separate phrases or clauses rather than individual words, this choice carries significant contextual implications. Therefore, the critical next step is to empirically check the impact on embedding quality by comparing the cross-lingual similarity scores between English and Khmer for texts with and without manual word spacing.

Text: អ្វីទៅជាស្នេហា?

Tokens: [‘អ្វី’, ‘##ទៅ’, ‘##ជា’, ‘##ស’, ‘##្ន’, ‘##េ’, ‘##ហ’, ‘##ា’, ‘?’]

# Cosine similarity: (KH vs EN) អ្វីទៅជាស្នេហា? | What is love? => 0.9713

The result above shows a high similarity (0.971) between the unsegmented Khmer sentence and its English translation. We then tested the effect of manual word segmentation:

Text: អ្វី ទៅ ជា ស្នេហា?

Tokens: [‘អ្វី’, ‘ទៅ’, ‘ជា’, ‘ស្នេហា’, ‘?’]

# Cosine similarity: (KH vs EN) អ្វី ទៅ ជា ស្នេហា? | What is love? => 0.9694

The similarity score for the manually segmented text (0.969) is just slightly lower (a difference of 0.002). This demonstrates the model’s tolerance: even when the unsegmented input resulted in multiple subword tokens (e.g., the word ស្នេហា “love” being split), the LaBSE model was still able to generate highly relevant vectors. We obtained similar results across five other sentences (see the appendix).

Our experiments show that manually introducing spaces for explicit word segmentation does not improve the model’s effectiveness. In fact, sentences with manual segmentation resulted in a slightly lower average cross-lingual similarity score (5.33) compared to the unsegmented input (5.36) across six test examples. This suggests that the text used to train the LaBSE model may not have been pre-segmented using a Khmer word tokenizer. Nonetheless, the model demonstrates robustness, accommodating both segmented and unsegmented text inputs effectively, as the resulting similarity scores are extremely close.

System Overview

The architecture for the Khmer Semantic Search system is primarily implemented using a PostgreSQL vector database, with an alternative solution for Mongo Atlas detailed in the Appendix.

The system consists of the following key components:

1. Embedding API Service: An API service is used to generate the numerical representations (text embeddings) of Khmer input. This service utilizes the LaBSE model and provides the necessary real-time embedding for user queries, as well as offline batch embedding for the main data corpus.
2. Vector Datastore (Embedding Storage): This is a dedicated vector database (e.g., PostgreSQL with the pgvector extension) that stores the pre-computed text embeddings of all searchable Khmer content.
3. Query Processing and Retrieval: This component manages the search workflow to deliver semantically relevant results:
   - It first sends the user’s query to the Embedding API Service to generate a corresponding vector embedding.
   - It then performs a similarity search on the Vector Datastore using the query embedding to find and rank the matched entries with the highest similarity scores.

Implementation Steps

Embedding API Service

In our environment, we use fastAPI to wrap our Python code to generate the embedding.

1. Install a few components such as fastapi and sentence-transformers for LaBSE
2. With our Python code, create an endpoint to process the input and generate the output of the embedding vector.

Minimal code to generate the embeddings:

from sentence_transformers import SentenceTransformer
# Load LaBSE model once at startup
model = SentenceTransformer("sentence-transformers/LaBSE")
def get_embedding(q: str = Query(..., description="Input text to embed")):
    # Encode the text using LaBSE
    embedding = model.encode([q])[0]
    # Convert to list for JSON serialization
    embedding_list = embedding.tolist()
    return {
        "embedding": embedding_list
    }

Generation Embedding in Postgres DB

To prepare your data for semantic search, you must add a vector column to your searchable tables, pre-process the text to generate embeddings, and store these vectors. This requires the pgvector extension for PostgreSQL.

1. Database Setup (SQL)

The following commands enable the necessary extension, add the embedding column (specifying the 768 dimensions from the LaBSE model), and create an index for fast similarity search using ivfflat.

CREATE EXTENSION IF NOT EXISTS vector;
ALTER TABLE dict.proverb 
ADD COLUMN embedding vector(768);
CREATE INDEX ON dict.proverb 
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);

You can create multiple embedding fields if you need to embed different combinations of text columns.

2. Batch Embedding Generation and Storage (Python)

After setting up the database, you must iterate through your existing data, generate the LaBSE embedding for each row’s text content, and store the resulting vector in the new column.

The following Python snippet illustrates how to encode the text and update a specific row in the database:

# Assuming 'model' is the loaded LaBSE 
# and 'cur' is your PostgreSQL cursor
# Generate the embedding vector
embeddings = model.encode([text], convert_to_tensor=False)
embedding = embeddings[0].tolist()
cur.execute("UPDATE dict.proverb SET embedding = %s WHERE id = %s", (embedding, row_id))

In this code:

• text is the string from your document that needs to be embedded (e.g., concatenated text from multiple columns).
• row_id is the primary key of the row being updated.
• The embedding vector is converted to a list for insertion into the database.

Query Processing and Retrieval

When a user submits a search query, the system first calls the API service with the query text to generate a corresponding embedding vector. It then performs a similarity search on the vector index to find and rank document embeddings that are closest to the query embedding.

The ranking is performed on the raw cosine distance in ascending order (using the <=> operator for performance), which implicitly ranks the calculated similarity in descending order (highest similarity first).

Below is a sample function to perform the vector query in PostgreSQL:

def query_emb(query_text):
  # <=> = cosine distance → similarity score = 1 - distance
  sql = """
  SELECT id, proverb, 1 - (embedding <=> %s) AS similarity
  FROM dict.proverb
  WHERE embedding IS NOT NULL
  ORDER BY embedding <=> %s
  LIMIT 5;
  """
  embeddings = model.encode([query_text], convert_to_tensor=False)
  embedding = embeddings[0].tolist() if hasattr(embeddings[0], "tolist") else embeddings[0]
  vec = Vector(embedding)
  cur.execute(sql, (vec, ))
  results = cur.fetchall()

Mongo Atlas DB Approach

For an alternative approach, instructions are provided in the Appendix for a Mongo Atlas implementation. Mongo Atlas facilitates powerful vector search capabilities through its Atlas Search feature. This involves storing your LaBSE embeddings as arrays of numbers within your documents and defining a search index using the dedicated vectorSearch field type for efficient similarity queries.

Sample UI

We build using Khmer proverb data we collected. Here are some search examples on Khmer proverbs. The score is to the right for each entry.

Search string: បាយ (rice)

The result is very similar to Khmer text search, especially the first entry. This shows strong correlation on English/Khmer terms as expected from our embedding. It picked up entries with the word “plate” which can be related to a rice plate. And also return entry with the term ស្រូវ​ (raw rice) which is a different word. But it missed one of the proverbs កក​ក៏​បាយ មេម៉ាយ​ក៏​ស្រី (leftover rice is still rice and widow is still women) proverb. We set a score cut off 0.4 so anything lower is not returned.

Try full phrase: កក​ក៏​បាយ មេម៉ាយ​ក៏​ស្រី

The top two entries have the same proverb entry but give different scores because the second entry was appended to other fields used in generating the embedding. This explains why one has the exact matched score (1.0) and the other is only 0.67 since it has other strings not shown in the UI.

Conclusion

In summary, we evaluated several commercial and open-source text embedding models and concluded that LaBSE (Language-agnostic BERT Sentence Embeddings) is the optimal choice for our Khmer Semantic Search system. We focused on evaluating the model through its output, specifically its similarity score, to infer its effectiveness for our use case.

Our testing revealed key insights into the model’s performance on Khmer text:

• Superior Cross-Lingual Alignment: LaBSE consistently demonstrated significantly stronger similarity scores in cross-lingual tasks (Khmer-English translation) and indicating a robust representation of Khmer semantic space.
• Tokenization Robustness: Inspection of the tokenization process showed that the LaBSE tokenizer removes the zero-width space (\u200B) character and handles unsegmented Khmer text effectively reinforcing the choice to use unsegmented text for simplicity and performance.

We detail the system implementation using a PostgreSQL vector database and demonstrate a working example of the semantic search on Khmer proverbs, which can be viewed here: https://dict.tovnah.com/km/dict/proverb.

Future Considerations: Hybrid Search Approaches

A purely semantic search solution is often insufficient for comprehensive document retrieval, as it can lack the precision needed for exact matches. The superior solution is a hybrid search approach that combines the contextual understanding of semantic search (using embeddings) with the precision of full-text keyword search. While the full implementation is outside the scope of this writing, we explored two methods for integrating the full-text component:

1. Search String without Segmentation (RegEx): This method combines semantic search with a string search using regular expressions to bypass the need for explicit word segmentation. The approach is effective for exact matches of a single keyword or string, although its utility decreases for multi-term queries where terms are scattered throughout a document. This implementation is similar to a LIKE function in a database query but includes Khmer Character Cluster logic to minimize partial term matching (e.g., searching for ក ‘KA’ matches កម្លាំង but not កូន). The hybrid score is calculated by adding a fixed weighting of 0.5 for the keyword search result to the semantic similarity score. This feature can be demonstrated on the proverb search page.
2. Full-Text Search with Segmentation (Postgres `tsvector`): This is an advanced solution that solves the issue of multi-term searches but requires proper Khmer word segmentation on both the search input and the document data. We tested this using PostgreSQL’s tsvector feature, which supports weighted full-text search across multiple fields. For Khmer word segmentation, we use the Khmer CRF word segmentation tool [7]. The hybrid score is calculated by combining the semantic score with the full-text score, where the tsvector component is assigned a value of 1.0 for an exact match. An example of this full-text-search without semantic is available on the Khmer oldies songs page [9].

References

• [1] R. Buoy and N. T. Sovis, “Khmer Word Search: Challenges, Solutions, and Semantic-Aware Search,” arXiv:2112.08918 [cs.CL], Dec. 2021.
• [2] N. Thuon, “Khmer Semantic Search Engine (KSE): Digital Information Access and Document Retrieval,” arXiv:2406.09320 [cs.CL], June 2024.
• [3] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient Estimation of Word Representations in Vector Space,” in Proc. of Workshop on Learning Representations for NLP (NIPS’13), 2013.
• [4] F. Feng et al., “Language-Agnostic BERT Sentence Embedding,” in Proc. of Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2020.
• [5] H. Reimers, “Sentence-Transformers,” GitHub. [Online]. Available: https://github.com/huggingface/sentence-transformers#citing--authors. [Accessed: Nov. 2, 2025].
• [6] Cohere, “Cohere’s Embed Models (Details and Application).” [Online]. Available: https://docs.cohere.com/docs/cohere-embed. [Accessed: Oct. 8, 2025].
• [7] P. Tum, “Khmer CRF Word Segmentation,” GitHub. [Online]. Available: https://github.com/phylypo/segmentation-crf-khmer. [Accessed: Nov. 29, 2025].
• [8] P. Tum, “Khmer Proverbs Search Demo,” Tovnah Dictionary. [Online]. Available: https://dict.tovnah.com/km/dict/proverb. [Accessed: Nov. 29, 2025].
• [9] P. Tum, “Khmer Oldies Songs Search,” Tovnah Dictionary. [Online]. Available: https://dict.tovnah.com/km/dict/song. [Accessed: Nov. 29, 2025].
• [10] NLP Techostartup Center. [Online]. Available: https://nlp.techostartup.center/. [Accessed: Oct. 3, 2025].
• [11] OpenAI, “OpenAI API Documentation,” [Online]. Available: https://openai.com/api/. [Accessed: Oct. 20, 2025].

Appendix

Mongo Atlas DB

Storing Embeddings

Each document in your collection would include a field for the text and another for its corresponding embedding.

{
  "text": "This is a Khmer sentence.",
  "embedding": [0.123, 0.456, 0.789, ..., 0.001]
}

Creating an Atlas Search Index

Define an Atlas Search index on your collection with a vectorSearch field for the embedding.

{
  "mappings": {
    "dynamic": true,
    "fields": {
      "embedding": {
        "type": "vectorSearch",
        "dimensions": 768, // Adjust based on your LaBSE model's output dimension
        "similarity": "cosine"
      }
    }
  }
}

Performing a Cosine Vector Search:

When a user query comes in, generate its embedding using LaBSE and then use the $vectorSearch operator in your aggregation pipeline.

from pymongo import MongoClient

# Assume 'query_embedding' is the LaBSE embedding of the user's query
# and 'collection' is your MongoDB collection

results = collection.aggregate([
    {
        "$vectorSearch": {
            "queryVector": query_embedding,
            "path": "embedding",
            "numCandidates": 100, # Number of nearest neighbors to consider
            "limit": 10,          # Number of results to return
            "index": "default"    # Name of your Atlas Search index
        }
    },
    {
        "$project": {
            "text": 1,
            "score": { "$meta": "vectorSearchScore" }
        }
    }
])

for doc in results:
 print(f"Text: {doc['text']}, Score: {doc['score']}")

PostgreSQL for Cosine Vector Search

PostgreSQL can also be used for vector search, especially with extensions like pgvector. This extension allows you to store vector data and perform efficient similarity searches, including cosine similarity.

Enabling pgvector Extension:
First, ensure the pgvector extension is enabled in your PostgreSQL database.

CREATE EXTENSION vector;

Creating a Table with Vector Column:
Create a table to store your Khmer text and their embeddings.

CREATE TABLE khmer_documents (
    id SERIAL PRIMARY KEY,
    text TEXT,
    embedding VECTOR(768) -- Adjust dimension based on your LaBSE model
);

Inserting Data:
Insert your preprocessed text and their LaBSE embeddings into the table.

INSERT INTO khmer_documents (text, embedding)
VALUES ('This is a Khmer sentence.', '[0.123, 0.456, 0.789, ..., 0.001]');

Performing a Cosine Vector Search:
To find documents similar to a query embedding, use the <=> operator for cosine distance.

import psycopg2
import numpy as np

# Assume 'query_embedding' is the LaBSE embedding as a numpy array
query_embedding_str = '[' + ','.join(map(str, query_embedding)) + ']'

with conn.cursor() as cur:
  cur.execute(f"""
SELECT text, 1 - (embedding <=> '{query_embedding_str}') AS cosine_similarity
FROM khmer_documents
ORDER BY embedding <=> '{query_embedding_str}'
LIMIT 10;
  """)
  results = cur.fetchall()
    for text, similarity in results:
     print(f"Text: {text}, Similarity: {similarity}")

Example scores on text between Khmer and English

# textkh = ["សួស្តីឆ្នាំថ្មី!", "តើអ្នកសុខសប្បាយទេថ្ងៃនេះ?", "តើអ្នកឈ្មោះអ្វី?", "កាលយើងជួបគ្នាលើផ្ទៃអាកាស", "ពិតជាច្រៀងពិរោះណាស់", "អ្វីទៅជាស្នេហា?"]


textkh = ["សួស្តី ឆ្នាំ ថ្មី!", "តើ អ្នក សុខ សប្បាយ ទេ ថ្ងៃ នេះ?", "តើ អ្នក ឈ្មោះ អ្វី?", "កាល យើង ជួប គ្នា លើ ផ្ទៃ អាកាស", "ពិត ជា ច្រៀង ពិរោះ ណាស់", "អ្វី ទៅ ជា ស្នេហា?"]
texten = ["Happy New Year!", "How are you today?", "What is your name?", "When we met in the sky", "It’s really beautifully sung", "What is love?"]
embeddings_kh = model.encode(textkh, convert_to_tensor=False)
embeddings_en = model.encode(texten, convert_to_tensor=False)


# iterate between enbeddings_kh and embeddings_en and compute cosine similarity
for i in range(len(textkh)):
    cos_sim = cosine_similarity([embeddings_kh[i]], [embeddings_en[i]])[0][0]
    print(f"Cosine similarity: (KH vs EN) {textkh[i]}  |  {texten[i]}  => {cos_sim:.4f}")

# Cosine similarity: (KH vs EN) សួស្តីឆ្នាំថ្មី!  |  Happy New Year!  => 0.8991
# Cosine similarity: (KH vs EN) តើអ្នកសុខសប្បាយទេថ្ងៃនេះ?  |  How are you today?  => 0.8775
# Cosine similarity: (KH vs EN) តើអ្នកឈ្មោះអ្វី?  |  What is your name?  => 0.8726
# Cosine similarity: (KH vs EN) កាលយើងជួបគ្នាលើផ្ទៃអាកាស  |  When we met in the sky  => 0.8690
# Cosine similarity: (KH vs EN) ពិតជាច្រៀងពិរោះណាស់  |  It’s really beautifully sung  => 0.8754
# Cosine similarity: (KH vs EN) អ្វីទៅជាស្នេហា?  |  What is love?  => 0.9713


# Cosine similarity: (KH vs EN) សួស្តី ឆ្នាំ ថ្មី!  |  Happy New Year!  => 0.8972
# Cosine similarity: (KH vs EN) តើ អ្នក សុខ សប្បាយ ទេ ថ្ងៃ នេះ?  |  How are you today?  => 0.8117
# Cosine similarity: (KH vs EN) តើ អ្នក ឈ្មោះ អ្វី?  |  What is your name?  => 0.8984
# Cosine similarity: (KH vs EN) កាល យើង ជួប គ្នា លើ ផ្ទៃ អាកាស  |  When we met in the sky  => 0.8782
# Cosine similarity: (KH vs EN) ពិត ជា ច្រៀង ពិរោះ ណាស់  |  It’s really beautifully sung  => 0.8729
# Cosine similarity: (KH vs EN) អ្វី ទៅ ជា ស្នេហា?  |  What is love?  => 0.9694

To understand the details of ChatGPT, let’s first start with a basic understanding of how artificial intelligence (AI) performs natural language processing (NLP) and its evolution. Then we will delve into the GPT language model and its training process. Finally, we will discuss some implications of artificial general intelligence (AGI). The following topics will be covered:

• Introduction to NLP
• Transformer architecture
• Large language models such as BERT and GPT
• Training of large language models, including ChatGPT
• Bing Chat and Google Bard
• Sparks of AGI

Introduction to NLP

Natural Language Processing (NLP) is a powerful tool that allows computers to analyze human language. Unlike formal language, like computer programming languages, natural language has loose rules that can change over time. When creating an algorithm to perform NLP, you don’t have the complete rule to program into the algorithm thus you need a completely different approach.

In NLP, the algorithms can perform tasks such as document classification to identify whether an email is spam, or analyze the sentiment of a given text as positive or negative. This type of program artificial intelligence (AI) program can perform a wide variety of other tasks as well such as summarize text, answer questions, translate between languages, identify entities within a text (name entity recognition or NER), and generate text like ChatGPT.

Bag-of-Words

A simple approach to NLP in the early days is to count the frequency of words in a document in the document. This approach is called bag-of-words algorithm. While it was initially used for scoring and searching documents, it can also be used for tasks like text classification and sentiment analysis. Improvements like Term Frequency-Inverse Document Frequency (TF-IDF) help normalize the count and improve performance of the tasks.

Word Vectors

To obtain more meaningful insights, word vectors are used to represent each word as a list of values. This approach captures both the syntactic and semantic information of words.

The idea is that you can map a word into a vector space. So words that are located in close proximity are similar while others with opposite values in some axis are different in that category.

For example, the gender ‘male’ can be considered as one vector value, so “man” and “boy” would have high values belonging to a close proximity, while “woman” and “girl” would have the opposite values in one vector.

Word vectors are generated through unsupervised learning from text documents, using algorithms like word2vec to pre-train the vectors based on the co-occurring of words.

Additionally, vector arithmetic can be performed on words, such as the well-known example from word2vec paper below demonstrates the representation of these words through their values.

[King] — [Man] + [Woman] = [Queen]

Another approach to generate the word vector is called Global Vectors for Word Representation (GloVe). It has shown better performance by capturing global statistics to improve the learning of word representation.

Both approaches rely on the probabilities of co-occurring words during pre-training, but they are context-independent when used as inputs to algorithms. But word vector or word embedding has become the feature inputs to many NLP algorithms. In the next section, we will explore context-dependent approaches.

Language Model

Although word vectors are generated based on the probabilities of co-occurring words, they are context-independent when used as input for many algorithms. To incorporate context, we need to explore the language models approaches.

A language model is a random process that models word sequences by calculating the probability given a sequence of words. There are two approaches to accomplish this:

1. Statistical language models use traditional statistical techniques like N-Gram to learn the probability distribution of words.
2. Neural language models use neural networks, such as Recurrent Neural Networks (RNN), to model language.

N-Gram

N-Gram is an approach that takes into account some number of words in a sequence, where N is the number of words. For example, a Bigram (N = 2) accounts for two sequential words at a time. Given a sequence of words, the next word’s probability is the probability of the second word given the first word, multiplied by the probability of the third word given the second word, and so forth.

Similarly, a Trigram (N = 3) takes into account three words at a time. As N increases, the accuracy of the model improves, but the amount of computation increases exponentially with the dimension of the vocabulary size to the power of N. This approach has a fairly good performance in many of the NLP tasks without having a very big N. But N-gram models often lack the ability to generalize well beyond the specific phrases or sentences they were trained on.

Neural Language Model

The neural language model uses a neural network to model language. Neural network is a type of machine learning approach that is inspired by how the neurons in the human brain function. It consists of layers of interconnected neurons that are designed to process complex data.

In 2003, Bengio et al proposed “A Neural Probabilistic Language Model” that showed a 10–20% performance improvement over the Trigram algorithm based on perplexity score,, a measure of how well it can predict a sequence of words.

The next phase in the neural network language model was the Recurrent Neural Network (RNN), which has become a standard approach for sequence-to-sequence (seq2seq) tasks using specific implementations like Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). Also known as deep learning, these approaches become the standard NLP and other domains such as computer vision and speech recognition.

Transformer Architecture

The RNN approach has limitations due to its serial computation as it processes each input word. In order to efficiently process the inputs in mass parallel computing infrastructure, a team from Google proposed an architecture in the seminal paper “Attention is all you need” by Vaswani et al. in 2017 known as the Transformer. The Transformer architecture allows parallel computing and uses the self-attention mechanism to capture the relationship between input words/tokens. Additionally, the Transformer uses key-value stores and positional encoding to improve performance. The transformer architecture has achieved state-of-the-art results on a wide range of NLP tasks, particularly in machine translation.

BERT

The Transformer architecture addressed the massive processing issue but it needed another breakthrough for the algorithm to learn from massive text data on the Internet. BERT (Bidirectional Encoder Representations from Transformers) is the next leap in NLP, enabled by the ability to pretrain on massive amounts of text data available on the internet. Similar to pretraining in computer vision, BERT pretrain the text using the masked language model (MLM). MLM is a pretraining approach that involves masking some words/tokens in the input sequence and training the model to predict the original token based on its context. This approach allows the model to learn the representations of words depending on its context.

The language model at this point can predict the next word very well. Then it is fine-tuned for the downstream tasks such as questions and answers. This approach allows BERT to perform better than humans on the Stanford Question Answering Dataset (SQuAD).

BERT was released in October 2018 by Google and other transformer-based models that use the MLM approach have revolutionized NLP by significantly advancing language understanding, text generation, and machine translation. BERT has achieved state-of-the-art results on a wide range of NLP tasks, and subsequent models like GPT-2, Ernie, and T5 have also demonstrated the power of transformer architecture and its pretraining approach.

There are more detail timeline of different models from my survey article here: https://medium.com/@phylypo/a-survey-of-the-state-of-the-art-language-models-up-to-early-2020-aba824302c6.

Large Language Model

To better understand the training of large language models such as ChatGPT, it’s important to first grasp the sheer size of these models as it has a direct impact on the training time. As a neural network trains, it adjusts these connections, or weights. Therefore, the more connections the network has, the more computation it must perform during training and inference.

The term “large language model” or LLM is given to models due to the large number of parameters, or weights in the language model. The number of parameters in these models has increased exponentially in recent years. For instance, GPT had 117 million parameters in 2018, GPT-2 had 1.5 billion parameters in 2019, and the largest model as of January 2020, T-NLG from Microsoft, had 17 billion parameters. In June 2020, GPT-3 was released with a whopping 175 billion parameters, representing a significant increase in size from previous models. The chart below with a logarithmic scale on the y-axis illustrates the trend of exponentially increasing the number of parameters in more recent models up to 2021.

Related to ChatGPT model by Open AI, we can see the following is a timeline of OpenAI’s large language models and their sizes and how its grows exponentially over time.

• GPT (Generative Pretrained Transformer): Released in June 2018, this model had 117 million parameters and was trained on a diverse range of internet text.
• GPT-2: Released in February 2019, this model had 1.5 billion parameters and was trained on an even larger and more diverse dataset than GPT.
• GPT-3: Released in June 2020, this model had a massive 175 billion parameters, making it the largest and most powerful language model at the time.
• GPT-3.5: Released on March 15, 2022, this model was trained on data up to June 2021 and included Codex, a language model designed for generating code.
• ChatGPT: Based on GPT-3.5, ChatGPT was released on November 30, 2022, and is specifically designed for chat and conversational applications.
• GPT-4: Started the limited release on February 7, 2023 by Microsoft under the code name Sydney embedded in Bing Search, this model is more powerful than GPT-3.

Training Large Language Models

To train a language model like ChatGPT, a large quantity of high-quality data is necessary. Although there is a vast amount of text data available on the internet, it must be filtered to ensure that only high-quality data is used for training. The quality of the training data is essential in determining the effectiveness of the model. However, the data collection and preprocessing process can be time-consuming and resource-intensive.

Dataset

The exact dataset used in ChatGPT is unknown but based on GPT-3, we see the main datasets used are:

1. Common Crawl 2016–2019: This is a massive corpus of web pages, totaling 45 terabytes in size of compressed plaintext before filtering and 570GB after filtering, roughly equivalent to 400 billion byte-pair-encoded tokens.
2. WebText2: This is a collection of filtered web pages from the Internet.
3. BooksCorpus: This dataset includes over 11,000 books.
4. Wikipedia: This dataset includes a dump of the English Wikipedia, totaling 2.5 billion words.

Training

The training process of a large language model typically consists of two phases.

1. Pre-training: In this phase, the model is trained on a vast amount of text data. A popular approach, such as BERT’s mask language model, employs semi-supervised learning. The model is fed raw text without specific labels, but certain words are programmatically omitted, and the model learns to predict those missing words.
2. Fine-tuning: In the fine-tuning phase, the model is trained on specific data and formats to enable it to perform particular tasks like sentiment analysis, text summarization, or translation. The model updates its weights as it learns from new data, improving its performance on the given task. Some models, like T5 (Text-to-Text Transfer Transformer), are trained to handle multiple tasks, such as translation, summarization, question answering, and text generation, within a single model.

OpenAI’s large language models (LLMs), such as GPT, follow a similar pre-training approach. However, they employ a different fine-tuning process. GPT uses in-context learning, where it is not fine-tuned for specific tasks. During inference, the model is conditioned with a few task demonstrations as input, but its weights are not updated in this process.

The in-context learning approach yields interesting emergent behavior, as the model learns to generalize by reading input text conditioned on previous words then predict the next words. OpenAI demonstrated that GPT-2 can perform multitask learning, but the learning process is slower compared to explicitly supervised approaches.

The pre-training approach for GPT-3 is similar to that of GPT-2, including the model architecture, data, and training methods. Scaling up from GPT-2 to GPT-3 was relatively straightforward in terms of increasing the model and dataset sizes, diversifying the data, and extending the training duration. Evaluation of the GPT-3 model focuses on its performance with zero-shot learning (no examples), one-shot learning (one example), and few-shot learning (several examples).

GPT-3 has demonstrated the ability to achieve performance levels that are close to state-of-the-art results on CoQA reading comprehension tasks when utilizing a large enough model.

ChatGPT, which utilizes the GPT-3 language model, undergoes a fine-tuning process that involves supervised learning. During this process, conversations generated by human trainers are used to train the model on appropriate responses. Additionally, ChatGPT incorporates reinforcement learning from human feedback (RLHF) to further enhance the quality and format of its generated answers. In RLHF, human trainers provide scores indicating their preference for different responses, and the model learns to generate responses that maximize these scores. This version of the model is known as InstructGPT (or GPT-3.5).

OpenAI has also implemented safety measures in ChatGPT to mitigate the risk of toxic, biased, or harmful responses. This ensures that the system adheres to safety guidelines. ChatGPT is now publicly accessible through the OpenAI website’s web interface. The diagram below illustrates the different models and features involved in the ChatGPT system.

The ChatGPT trained model possesses knowledge about current events up to 2021 since it was trained on data collected up to that time period.

GPT-4 is a similar model to its predecessor, but specific details regarding its training data are not widely known. The model underwent fine-tuning using the Reinforcement Learning from Human Feedback (RLHF) approach, similar to InstructGPT. However, the report regarding GPT-4 did not disclose information about its architecture, such as model size, hardware, training compute, dataset construction, or training method. These details were omitted due to competitive considerations within the technology field.

One notable addition to GPT-4 is the ability to process images. The chart below illustrates the improved performance of GPT-4 compared to GPT-3 in academic and professional exams.

Here is the summary of the key differences in model and dataset size for GPT models.

• GPT was trained on a dataset of 40GB of text, with 117 million parameters. It achieved state-of-the-art results on a variety of language modeling tasks at the time of its release in 2018.
• GPT-2 was trained on a much larger dataset of 40GB of text, with 1.5 billion parameters. It also introduced the “unsupervised task” of generating coherent text from a given prompt.
• GPT-3 was trained on an even larger dataset of 45TB of text, with 175 billion parameters. It introduced a range of new features, including the ability to perform arithmetic, answer questions, and translate languages.
• GPT-3.5/ChatGPT has been trained on a dataset with over 570 terabytes of text data.
• GPT-4: training data size and model size are not disclosed.

Environment Impact

With the introduction of LLM (Large Language Models), researchers raised concerns regarding the environmental impact associated with training these models. Reports emerged with headlines like “Training a single AI model can emit as much carbon as five cars in their lifetimes,” citing a study conducted by the University of Massachusetts Amherst. The charts below depict the estimated environmental footprint of the five largest models at that time.

A subsequent study conducted by Google and the University of California, Berkeley, titled “Carbon Emissions and Large Neural Network Training” (Patterson et al., 2021), highlighted that the carbon emissions associated with training large neural networks can vary depending on factors such as the data center’s efficiency and the type of hardware employed (Nvidia P100 GPU vs TPU v2). However, the overall magnitude of emissions aligns with the findings of a 2019 report, estimating that approximately 200 kWh of energy consumption corresponds to roughly 0.1 metric tons of carbon emissions.

Furthermore, the report demonstrates that the training process of GPT-3 consumed 1,287 MWh of energy, resulting in the emission of 552 metric tons of carbon or approximately 1.2 million pounds. To provide a comparative perspective, this level of carbon emission is equivalent to the emissions generated by 10 cars over the course of their lifetimes.

In addition to training time, the size of the models also impacts their usage by users. According to estimates from the lifearchitect.ai website, the utilization of ChatGPT by 10 million users could cost the company up to $12 million per month. Based on the cloud compute cost from Strubell et al., this monthly cost is four times higher than the cost of training the largest model. So every month, the operation of ChatGPT alone emits an amount of carbon equivalent to that produced by 20 cars over their lifetimes.

As some researchers pointed out the amount of greenhouse gas it emitted, others worried that research communities with limited access to computational resources will be unable to compete with large companies that possess extensive computing capabilities.. However, Sam Altman from OpenAI has reassured that “we are at the end of the era where it’s going to be these, like, giant, giant models”[wired.com].

Bing Chat and Google Bard

Microsoft integrated GPT-4 to its search engine Bing in the chat mode in February 6, 2023. The code name of the model is known as Sydney as a Stanford student created a prompt injection to reveal its instructions.

In February 6, 2023, Microsoft integrated GPT-4 into its search engine Bing in the chat mode. The model was internally referred to as “Sydney” because a Stanford student discovered a way to reveal its instructions via prompt injection.

These language models are trained on a collection of internet data up to a certain point in time. As for ChatGPT, it learned about current events up to 2021. While GPT-4 has similar limitations, Bing Chat allows the model to access the Internet, specifically through Bing Search, for up to 3 searches based on the revealed instructions. This feature enables the model to provide more up-to-date information.

Following the popularity of ChatGPT, Google introduced its own large language model called Bard in March 2023. Bard is powered by a Language Model for Dialogue Applications (LaMDA).

As of early May 2023, the latest version of Bard is powered by another language model called PaLM 2. PaLM, a Pathways Language Model, has 540 billion parameters, nearly five times larger than LaMDA, which has 137 billion parameters. PaLM is trained on a dataset of text and code that is 100 times larger than what LaMDA is trained on. It features a new AI architecture designed to be more efficient and scalable. PaLM can handle various tasks and learn new tasks more effectively, resembling how humans learn.

PaLM 2 has a “significantly smaller” number of parameters than PaLM but performs similarly. It is more heavily trained on multilingual text, spanning more than 100 languages. It has the ability to understand, generate and translate nuanced text across a wide variety of languages. PaLM 2 also passes advanced language proficiency exams at the “mastery” level. It has an improved capabilities in logic, common sense reasoning, and mathematics. In addition, it can generate code and has debugging skills across multiple programming languages.

Sparks of AGI

As AI technology has advanced, we have witnessed impressive achievements, such as IBM’s Deepblue program defeating chess Grandmaster Garry Kasparov in 1997 and DeepMind Alpha Go beating world champion Go player Lee Sedol in 2017. However, these are considered artificial narrow intelligent (ANI) systems, meaning that they are only capable of mastering specific tasks and cannot transfer their skills to similar ones as humans can. In contrast, humans are considered to have general intelligence, which enables us to generalize from one learned task to the next rapidly. For AI to become artificial general intelligent (AGI), it must possess this ability to generalize and master any task. Once it reaches this stage, it may become an artificial super intelligent (ASI) quickly, which can bring significant knowledge and unimaginable impacts to the world.

According to Nick Bostrom, a philosopher and researcher at the University of Oxford, author of the book, “Superintelligence: Paths, Dangers, and Strategies” (2014), AGI is predicted to be achieved by machines around 2045, as per the average estimates of experts surveyed. However, Bostrom also raised concerns about the challenges of controlling AI once it reaches AGI.

One challenge is that we cannot simply program our goals into AI. In a thought experiment called the paperclip maximizer, imagine instructing a smart machine to produce as many paperclips as possible. This could lead the AI to prioritize paperclip production above all else, disregarding human instructions and may prevent humans from turning it off. Bostrom refers to this as the alignment problem.

Additionally, there is another challenge known as the “Control Problem.” Bostrom asks how we can establish initial conditions that ensure safety in the outcome. Suggestions like isolating the AI or limiting its access to the outside world may not be effective, as a highly intelligent machine could find ways to manipulate us or possess advanced technology beyond our awareness. This idea was depicted in the movie “Ex Machina,” where an inventor tries to maintain control over a smart AI confined to an isolated environment but ultimately loses control of it.

As researchers strive to achieve the goal of AGI, in his book “The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World” (2015), Pedro Domingos discusses the diverse approaches taken by different AI disciplines. Domingos categorizes these disciplines into five tribes:

1. Symbolists (logic-based AI with decision tree algorithms),
2. Connectionists (neural network-based AI with a focus on deep learning),
3. Evolutionaries (evolution-based AI utilizing genetic algorithms),
4. Bayesians (probability-based AI with methods like Naive Bayes), and
5. Analogizers (psychology-based AI using kernel machines).

Each tribe advocates for their approach as the one capable of advancing us towards AGI. This is evident in the debate between Gary Marcus, a symbolist, and Yoshua Bengio, a connectionist. The symbolist tribe defends their approach, challenging the popularity of the connectionist tribe, particularly deep learning.

( AI DEBATE : Yoshua Bengio | Gary Marcus — https://www.youtube.com/watch?v=EeqwFjqFvJA)

Domingos suggests that a combination of different AI tribes can create a master algorithm to reach AGI, such as in the context of curing cancer, which requires expertise from multiple fields that no single human alone can master.

Recent trends in AI research include the influential paper “Attention is All You Need” by Vaswani et al. (2017), which introduced transformer models and paved the way for large-scale models like ChatGPT in deep learning.

In the evolutionaries tribe, reinforcement learning has demonstrated remarkable progress. Notably, DeepMind’s AlphaGo, AlphaStar, and AlphaFold are prime examples of reinforcement learning-based systems. These systems have achieved unprecedented success in various games, such as Go and StarCraft 2, and have even made breakthroughs in tackling complex challenges in biology and physics, such as protein folding.

Affirming the potential of the reinforcement learning approach, David Silver’s 2021 paper “Reward Is Enough” supports the notion that learning through behavior rewards can eventually enable the replication of human cognitive capabilities and the achievement of AGI.

In terms of AGI advancements, a Google ex-employee claimed in July 2022 that the chatbot LaMDA possesses sentience. Additionally, Sebastian Bubeck from Microsoft, who had early access to GPT-4, published a paper titled “Sparks of Artificial General Intelligence: Early experiments with GPT-4” in March 2023. Bubeck highlighted GPT-4’s remarkable ability to solve novel and intricate tasks across multiple domains, approaching human-level performance in areas such as mathematics, coding, vision, medicine, law, and psychology. He sees it as a promising early version of an AGI system.

Geoffrey Hinton, considered the godfather of deep learning and the recipient of the 2018 Turing Award, recently left Google to express his concerns about AI as an existential threat. He emphasized that humanity is a transitory phase for evolutionary intelligence and urged caution in our approach to AI development.

Summary

A chatbot is a program designed to accurately predict the next word, creating the impression of human-like conversation or expertise on a particular subject. It functions as a sophisticated statistical machine, though some critics, like Noam Chomsky, argue that chatbots such as ChatGPT as mere impersonators lacking novelty or true understanding.

However, recent advancements in large language models have shown remarkable performance, surpassing humans in various tasks and achieving mastery-level proficiency in advanced language skills. While these chatbots have proven useful in assisting us with many tasks, concerns have arisen regarding their negative impact on society. Issues like the spread of fake news, deep fakes, and other worrisome developments have raised alarms among many individuals.

In comparison, AI recommendation systems are designed to maximize user engagement and generate profits through targeted advertising. By analyzing our preferences and clicks, these systems can gradually manipulate our perspectives, leading us down rabbit holes or influencing us with radical views.

As machines continue to grow more intelligent, whether or not this represents the emergence of AGI, it is only a matter of time before the machine surpasses human intelligence and reaches the level of artificial superintelligence. This raises important questions: Are we adequately prepared for this future? Can we ensure the safety and well-being of humanity in the face of such advancements?

Web References

• https://www.reuters.com/technology/chatgpts-popularity-explodes-us-lawmakers-take-an-interest-2023-02-13/
• https://venturebeat.com/ai/openai-chief-says-age-of-giant-ai-models-is-ending-a-gpu-crisis-could-be-one-reason-why/
• https://www.fierceelectronics.com/sensors/chatgpt-runs-10k-nvidia-training-gpus-potential-thousands-more
• https://en.wikipedia.org/wiki/GPT-3
• https://lambdalabs.com/blog/demystifying-gpt-3
• https://www.technologyreview.com/2019/06/06/239031/training-a-single-ai-model-can-emit-as-much-carbon-as-five-cars-in-their-lifetimes/
• https://arstechnica.com/information-technology/2023/02/ai-powered-bing-chat-spills-its-secrets-via-prompt-injection-attack/
• https://stanford-cs324.github.io/winter2022/lectures/environment/
• https://www.theguardian.com/technology/2022/jul/23/google-fires-software-engineer-who-claims-ai-chatbot-is-sentient
• https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html

Video References

1. The AI revolution: Google’s developers on the future of artificial intelligence | 60 Minutes
   https://www.youtube.com/watch?v=880TBXMuzmk (Apr 16, 2023) ~27mins
2. The A.I. Dilemma — March 9, 2023 (Center for Humane Technology) (Apr 5, 2023) ~107mins
   https://www.youtube.com/watch?v=xoVJKj8lcNQ
3. Sparks of AGI: early experiments with GPT-4 (Sebastien Bubeck) Apr 6, 2023 ~48 mins
   https://www.youtube.com/watch?v=qbIk7-JPB2c
4. Stuart Russell — How Not To Destroy the World With AI (CITRIS) Apr 5, 2023 ~110mins
   https://www.youtube.com/watch?v=ISkAkiAkK7A
5. Yann LeCun and Andrew Ng: Why the 6-month AI Pause is a Bad Idea (DeepLearningAI)Apr 7, 2023 (30 mins)
   https://www.youtube.com/watch?v=BY9KV8uCtj4
6. Possible End of Humanity from AI? Geoffrey Hinton at MIT Technology Review’s
   https://www.youtube.com/watch?v=sitHS6UDMJc

Paper References

1. GPT:
   Radford, Alec, Karthik Narasimhan, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. “Improving language understanding by generative pre-training.” OpenAI Blog 1, no. 8 (2018).
2. GPT2:
   Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. “Language models are unsupervised multitask learners.” arXiv preprint arXiv:1807.03819 (2018).
3. GPT-3:
   Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. “Language models are few-shot learners.” arXiv preprint arXiv:2005.14165 (2020).
4. InstructGPT:
   Ouyang, Long, et al. “Training language models to follow instructions with human feedback.” arXiv preprint arXiv:2203.02155 (2022).
5. GPT-4:
   OpenAI. “GPT-4 Technical Report.” arXiv preprint arXiv:2303.08774 (2023).
6. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. “Attention is all you need.” Advances in Neural Information Processing Systems 30 (2017): 5998–6008.
7. Silver, David, et al. “Reward is enough.” arXiv preprint arXiv:2104.08677 (2021).
8. Bubeck, Sébastien. “Sparks of AGI: Early Experiments with GPT-4.” arXiv preprint arXiv:2303.12712 (2023).
9. Strubell, Emma, Ananya Ganesh, and Andrew McCallum. “Energy and policy considerations for deep learning in NLP.” arXiv preprint arXiv:1906.02243 (2019).
10. Patterson, David A., Joseph Gonzalez, Christopher Ré, Jeffrey Dean, and Matei Zaharia. “Carbon emissions and large neural network training.” arXiv preprint arXiv:2104.10350 (2021).

Book References

1. Bostrom, Nick. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014.
2. Domingos, Pedro. The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World. Basic Books, 2015.

This is the summary of the steps that we will go over in this tutorial.

1. Create Azure Container Registry (Private)
2. Build the Docker container
3. Push the image into ACR registry
4. Create AKS cluster and add a node pool
5. Create a Kubernetes secrete for accessing the registry
6. Deploy the container to AKS from private registry

The code can be found here: https://github.com/phylypo/pycaret-deployment-azure

About the App

This is an ML insurance application that predict the cost for customer. It is built using Pycaret where the model is loaded from a saved pkl file. The web interface is built using Flask framework. The interface looks like this:

Environment Setup

First, sign in to Azure portal (https://portal.azure.com).

From the top icon, click on cloud shell icon highlight in yellow above. You may need to setup your storage account for the first time. The storage would save all your files during your session. Make sure you select Bash not PowerShell.

You should see something like below:

We will do most of the work in this Bash command window.

Create Resource Group

Before creating any Azure resources, we will need a Resource Group to put all of Azure resource that we will create. At the end, we can delete this resource group and all the resources in this Resource Group will be deleted so we don’t incur any future charges.

We will be using a few variables to store values in this tutorial including the resource group name. Run the commands below in the cloud shell prompt.

RESOURCE_GROUP=PYCARET-KUBE-RG
ACR_NAME=pycaretacr
CLUSTER_NAME=PYCARET-AKS

The ACR_NAME sometimes is hardcoded in the commands in later steps for simplicity. Please change accordingly. Here is an example output.

Now, create the Resource Group using the Azure API command ‘az’ as follow. We put this resource in Western US availability zone ‘westus2’. Change this to your prefer zone as needed.

az group create --name $RESOURCE_GROUP --location westus2

Azure Container Registry (ACR)

The Azure Container Registry (ACR) will be our private repository that store our Docker image that we will create. We will begin by creating ACR.

az acr create --resource-group $RESOURCE_GROUP --name $ACR_NAME --sku Premium

Clone the github repository

git clone https://github.com/phylypo/pycaret-deployment-azure.git
cd pycaret-deployment-azure

Build Docker image

az acr build --registry $ACR_NAME --image pycaret-ins-5000:v1 .

This will take several minutes (4 minutes for me) and ignore some of the warning in red. When successful you should see similar to below.

We will verify via the result from the web portal in a bit but first let’s create AKS next.

Create Kubernetes Cluster

# create kubernetes cluster
az aks create \
    --resource-group $RESOURCE_GROUP \
    --name $CLUSTER_NAME \
    --node-count 2 \
    --enable-addons http_application_routing \
    --enable-managed-identity \
    --generate-ssh-keys \
    --node-vm-size Standard_B2s

After about 10 minutes, you should see the output similar to below.

Now add node pool the cluster.

# add node pool
az aks nodepool add \
    --resource-group $RESOURCE_GROUP \
    --cluster-name $CLUSTER_NAME \
    --name mypool \
    --node-count 2 \
    --node-vm-size Standard_B2s

This will take several minutes. The output looks something like this:

Reviewing Created Resources

This will take several minutes so while waiting let’s verify our ACR. From the portal main screen, enter “resource group” in the search string and click on the “Resource Groups” result with the icon.

Now you should see the resource group we created along with AKS resource group MC_<Resource-group>…AKS_… which is part of Kubernetes cluster.

Now, click the resource group PYCARET-KUBE-RG and you should see the resources in the resource group which is the Kubernetes service and the Container registry.

Setup the User and Password in ACR

Let’s click on pycaretacr from above screen. Then in Settings menu on the left click Access Keys, then toggle to Enabled in Admin user. Now you should see the Name and password. Copy the two values these for the next step.

Create Azure Secret for registry credential

We are going to create a Kubernetes secret so the cluster can use to download the image.

But first we need to setup the credential to the AKS with the follow command so it allow to use kubectl command.

az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP

You might be prompt to override if you have connect to AKS before.

Create the Kubernetes secret with the following command. Change the password to the string copy from the portal from previous steps. The username should be the same as ACR name unless you change it. Note the docker-server contain the url with ACR_NAME.azurecr.io, so change if needed.

kubectl create secret docker-registry azure-reg-cred \
    --docker-username=pycaretacr \
    --docker-password=<PASSWORD_COPIED_EARLIER> \
    --docker-server=pycaretacr.azurecr.io

You should see the secret get created. This will not check if the username/password is correct. This is just a variable for the deployment to use in the deployment file below with secret name “azure-reg-cred”.

Deploy Service and Deployment

Now we will run the kubectl command to create deployment and service using the image we upload in ACR and the secret we configured above. Here is a snippet of the code in azure_deployment.yaml.

Run the kubctl apply with this file:

kubectl apply -f azure_deployment.yaml

From the service public ip, browse to the ip

# check pod status -- should see Running
kubectl get pod
# check for service ip -- to get EXTERNAL-IP
kubectl get service

Now you can browse to the EXTERNAL-IP of your app and you should see the screen below and if you enter the form and click Predict, it should output the “Expected Bill” line.

You can check the amount of memory it uses with below command. Mine shows 322 MB for memory usage.

kubectl top pod azure-insurance-bd9c6555c-f52h2 -n default --containers

Clean up

After you done and want to clean up, you can run this command below to delete the resource group. Since we created all the resource in this resource group, it will also delete all the resources including the Kubenetes cluster and the registry. This will take over 15 mins to complete but the command will exit right away. You can verify to make sure all the resource are delete.

az group delete --name $RESOURCE_GROUP --yes --no-wait

Troubleshoot

I will not cover any troubleshooting in detail but it took me many tries to get this working so I just want to show the basic approach. To troubleshoot mainly get the node status in this case ImagePullBackOff error and you check the detail with kubectl describe command below.

kubectl describe pod azure-insurance-bd9c6555c-p68bz

This is an example that I input the wrong username/password in the secret.

Summary

We cover how to create Azure Container Registry (ACR), Azure Kubenetes Service (AKS) and how to build and deploy the image to ACR. Then deploy the image into the AKS and see the app running. We get into Kubernetes secret and many of kubectl command. These are all done through the browser without the need to install any application by Cloud Shell in Azure portal.

I hope you learn something from this. Happy coding.

1. First, we look at how to automate the creation of certificates from your company Certificate Authority (CA) on Windows OS.
2. Then we look at how to use free issuer “Let’s Encrypt” to generate the certificate on Windows and Linux.
3. Now in this tutorial, we will look into setting up a certificate on Kubernetes.

In this step-by-step tutorial, we are going to set up SSL/TLS on the Ingress Controller using Nginx-ingress to handle HTTPS traffic to the Kubernetes service running on Azure Kubernetes Service (AKS). The certificate we are using is issued by Let’s Encrypt, a free service running as an Automated Certificate Management Environment (ACME) by using cert-manager.

The code is mainly based on the tutorial by Marcel Dempers in the reference section but updated for Kubernetes running on Azure cloud or Azure Kubernetes Service (AKS) rather than running locally using miniKube or Kind.

The source code can be found here: https://github.com/phylypo/kubernetes

Let’s Get Started

We assume you have Azure account to create resources like AKS. So first, log in to the Azure portal. We will be using Azure API to run all of our commands by using Cloud Shell. Click on the Cloud Shell icon as highlighted below. If this is the first time running Cloud Shell, you need to set up a storage account first.

After you launched Cloud Shell, you should see the terminal like below.

Now you can clone the code from this project in GitHub that contains the source code so you can follow along.

git clone https://github.com/phylypo/kubernetes.git
cd kubernetes/cert_devops

The directory cert_devops contains the configuration files that are needed for this tutorial. All the commands are in cert_devops.sh file.

Create AKS cluster

First, we need to create an Azure Kubernetes Cluster. We are going to use a new resource group “k8rg” to place all our resources which can be easily cleaned up afterward.

az group create --name k8rg --location westus2
az aks create --resource-group k8rg --name myAKSCluster --node-count 1

The last command can take a few minutes. Once done we want to set up the credential for kubectl. The get-credentials commands will ask to override or merge the configuration if the config exists. Then we can test out the kubectl commands.

az aks get-credentials --resource-group k8rg --name myAKSCluster
# test kubectl
kubectl get pod
kubectl get services

Install Cert-Manager

Now install cert-manager to handle certificate creation. We will test with a self-signed certificate first to ensure cert-manager is working properly. Then proceed with Let’s Encrypt in the later section.

kubectl apply --validate=false -f cert-manager.yaml
kubectl -n cert-manager get all

Create a self-signed certificate to test the cert-manager.

# create a test namespace 
kubectl create ns cert-manager-test

# create a self sign issuer
kubectl apply -f ss_issuer.yaml
# create certificate
kubectl apply -f ss_certificate.yaml

Now verify that the certificate is created.

kubectl describe certificate -n cert-manager-test
kubectl get secrets -n cert-manager-test
# cleanup by deleting the namespace and testing resources
kubectl delete ns cert-manager-test

Create Ingress Controller

We are going to create an Ingress Controller in the namespace “ingress-nginx”.

kubectl create ns ingress-nginx 
kubectl -n ingress-nginx apply -f deploy-ingress-ctl.yaml
# see all the different components created
kubectl -n ingress-nginx get all

Now browse to the public IP to see if the ingress is showing the nginx reponse. From the screen above, it is in the service output under the EXTERNAL-IP which is 20.64.143.87. You should see “404 Not Found” message from nginx since we have not configured the ingress controller.

Setup DNS

Now we can set up our DNS that will be used in the URL to point to this IP address. This Ingress Controller listens to both ports 80 and 443 for HTTPS.

Here is an example for my DNS cert.tovnah.com to point to this IP. This change may take some time to propagate depending on your service provider. Mine took about 15 mins. You can verify by checking nslookup command (nslookup cert.tovnah.com) and you should see the new ip address you updated.

Setup Let’s Encrypt Issuer

Now we will set up the certificate issuer, in this case, we use “Let’s Encrypt” that will be signing our certificate.

kubectl apply -f cert-issuer-nginx-ingress.yaml 
kubectl describe clusterissuer letsencrypt-cluster-issuer

Now we can deploy a test app for the ingress controller to forward the traffic.

kubectl apply -f deployment-app.yaml
kubectl apply -f service-app.yaml
kubectl get pods

Next, we need to point the ingress controller to this app. First edit ingress.yaml and replace “<your.dns.com>” with the correct hostname in my case “cert.tovnah.com”.

code ingress.yaml
kubectl apply -f ingress.yaml

kubectl get servies

Now browse to our new app and we should see the “Hello World!” message using the public IP of the app.

Setup Certificate

Now, we will create a certificate that will be issued by Let’s Encrypt issuer.

# first edit the certificate.yaml by updating the dns name
code certificate.yaml
kubectl apply -f certificate.yaml

Verify that the certificate is created successfully.

kubectl describe certificate example-app

You should see the last entry if a successful message in the Events section.

The certificate has been successfully issued

Check secret and should see the type: kubernetes.io/tls

kubectl get secret

Now test browsing to the DNS name and you should see the lock and valid certificate issued by Let’s Encrypt (R3).

Summary

We have looked at how to create a Kubernetes cluster, install cert-manager that allows us to use “Let’s Encrypt” to issue us a certificate. We deployed an nginx Ingress Controller and an example app in the Kubernetes cluster. Then we create a certificate to bind to the ingress controller that points to the example app. This allowed us to use HTTPS to access our app. We are using Azure API through Cloud Shell which allows us to run all the commands via a web browser.

This concludes our series on setting up server certificates. I hope this gives you a good understanding of setting up a certificate for your app.

To delete all of the resources in Azure that we have set up in this tutorial so that we don’t pay for the unused resources in Azure, we can simply delete the resource group that we initially created.

az group delete --name k8rg --yes --no-wait

If you follow the tutorial from Youtube below by creating your local Kubernetes cluster on your home machine, you need to make sure your router settings is forwarding the traffic on specific ports to your machine.

In the router settings, you need to open both 80 and 443 port forwarding to the machine IP. Notice that port 80 is needed for Let’s Encrypt challenges.

In my Linksys router it looks something like this under Security:

In addition, if you have McAfee running, you will need to disable McAfee Firewall temporarily. If you have the Windows firewall as your primary service, you will need to disable that.

Reference

Youtube Tutorial: https://www.youtube.com/watch?v=hoLUigg4V18

Github: https://github.com/marcel-dempers/docker-development-youtube-series/tree/master/kubernetes/cert-manager

1. First, we look at how to automate the creation of certificates from your company Certificate Authority (CA) on Windows OS.
2. Then we look at how to use free issuer “Let’s Encrypt” to generate the certificate on Windows and Linux.
3. Lastly, we will look into setting up a certificate on Kubernetes.

Let’s Encrypt provides free digital certificate generation in an automated way. In addition, it provides a full life cycle certificate management as Automatic Certificate Management Environment (ACME) that automatically re-new the certificate as needed.

Using Let’s Encrypt for Windows IIS

The step below shows the automation of creating digital certificates without having your own Certificate Authority. Instead, we use “Let’s Encrypt” for free.

First, let see how the HTTPS with invalid sites would look like.

The first step is the create HTTPS site binding with any generic certificate. To do this, edit site binding and choose an existing certificate temporarily so you can proceed with clicking OK.

You will see an invalid certificate when browsing using IE with a similar screenshot below.

Here is a screenshot of an invalid certificate when browsing with Chrome.

Now we will set up a process to generate a certificate from Let’s Encrypt. I follow the instruction from Youtube in the reference below. Here are the steps:

1. Download the win-acme client from: https://www.win-acme.com/
2. Then extract into a permanent directory since the app will create a task scheduler where you run from. Now run the wacs.exe file and choose your site to generate the certificate.
3. Then set up site binding to listen to HTTPS

You can look at the scheduled renewal in Task Schedule to verify that it will get renewal when the certificate is about to expire.

Now you should be able to see the secure website from the browser.

Using Let’s Encrypt for Linux

Whether you use Apache or Nginx on any flavor of Linux, Let’s Encrypt has instructions for setting it up. Just go to https://certbot.eff.org/ and choose your app and OS.

Here is a highlight of using Nginx on Ubuntu.

1.Install the certbot and the related python package for nginx

#sudo  apt install certbot python3-certbot-nginx

2.Run the certbo and specify the nginx

#sudo certbot --nginx

The command will prompt you for email and also determine the DNS for the certificate and output the certificate.
Here is an example output:

It added the SSL portion in the config similar to below:

listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/ml.tovnah.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/ml.tovnah.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

It will also ask if you want to redirect your HTTP to HTTPS. I choose yes. Then the successful output should look like this:

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
Congratulations! You have successfully enabled https://ml.tovnah.comYou should test your configuration at:https://www.ssllabs.com/ssltest/analyze.html?d=yourdomain.com
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

I tried the test URL and it went into detail on the certificate, protocol, key exchange, and cipher strength. See the output report below.

Note that if the server support TLS 1.1 and 1.0 the grade on the protocol will be B. Those protocols are suggested to be disabled.

The next step is to test the renewal with this command with dry-run to ensure the process is working:

# sudo certbot renew — dry-run

You should see the “Congratulations, all renewals succeeded” message.

Lastly, you need to confirm that there is a cron job that kept checking for certificate expiration and renewal as needed. In my case, I can see it with this command:

# systemctl list-timers

You should see an entry for certbot.timer.

NEXT                         LEFT      LAST  PASSED  UNIT
Tue 2022–01–04 11:40:29 UTC  17h left  n/a   n/a     certbot.timer

Next, we will look into setting up a certificate on Kubernetes.

Relate Notes and References

How to Install Lets Encrypt Certificates on IIS with Autorenew: https://www.youtube.com/watch?v=vbk5kUT7GeY

How To Secure Apache with Let's Encrypt on Ubuntu 18.04 | DigitalOcean

1. Here, we look at how to automate the creation of certificates from your company Certificate Authority (CA) on Windows OS.
2. Then we look at how to use free issuer “Let’s Encrypt” to generate the certificate on Windows and Linux.
3. Lastly, we will look into setting up a certificate on Kubernetes.

Introduction

HTTPS is an important part of having secured communication between your site and the users. This Public Key Infrastructure (PKI) framework allows the encryption of the communication via the public and private keys. The server public key which is signed by an authoritative certificate authority (CA) is trusted by the browser. This trust system will ensure that no one else can decrypt the data except two parties that are communicating.

As we set up more and more web services on Windows IIS with HTTPS, automation becomes a time saving for DevOps to automate this process. This page detail the process to automate the digital certificate signing process using your company CA. The automation involves using Ansible to generate a certificate signing request (CSR) and get the CA to sign the certificate then import it into IIS to bind a website as HTTPS.

Architecture and tools

1. Ansible server (we use AWX running on Linux)
2. Windows Server with IIS for web hosting
3. Certificate Authority on Windows server

First, we will look at the manual process to create a certificate request to get it signed and then complete the certificate import and then create a binding to enable HTTPS. After that, we will look at how to automate each process using Ansible.

Assumptions

1. You have your own Certificate Authority server.
2. You have Ansible setup for automation. The ansible code will only cover the portion related to the task.

Manual Process to Setup Https

We will start with a summary of what the process involves.

1. From the webserver, we first need to create a certificate signing request (CSR). This process involves creating a private key and generating CSR file with information about the server such as a common name that the CA can sign to create a certificate. This process allows only the server with the corresponding private key to use the certificate.
2. Take the CSR file generated and use the CA to sign and generate the certificate. The CA will output a CER file.
3. From the web server, we can import the CA file in the complete certificate request function. This will import to the server local store.
4. Now we can set up the website on IIS to bind as HTTPS using this certificate.

Now let us look into the detail of each step.

1. Creating CSR

From the webserver, in Server Management, double click Server Certificates

Then select “Create Certificate Request” on the Actions tab and fill in the information.

Click Next, then select Cryptographic Service Provider such as change Bit length to 2048 and click Next.

Then save the file and click Finish.

Example output file:

-----BEGIN NEW CERTIFICATE REQUEST-----
MIIEOzCCAyMCAQAwVTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMQswCQYDVQQH
...
fbceNKxOOlIyDFYMe36tzf/+EnkjDOFiZUR2kp5KVZquPflXDPIElhhbSgo2jaO8
nlq6Vewcu1HxLlQSLLoGOIN9eXX/qRoNcl+H+HgsOw==
-----END NEW CERTIFICATE REQUEST-----

2. Sign the CSR and Generate a Certificate in CA

In these steps, log in to your Certificate Authority (CA) server and import your CSR file that was created from the previous step.

First, open the CA tool from the Server Management, select the Tools menu, and then select “Certification Authority”.

From the server icon, right-click and select “All Tasks”, then select “Submit new request…”.

Then select the file created from the previous step.

Then from the “Pending Requests” folder, right-click the item, select “All Tasks” then select “Issue”.

Now the request will be moved to “Issued Certificates”. You can double-click to verify the certificate information.

Now export the certificate for IIS by right-clicking on the certificate, then selecting “All Tasks” and “Export Binary Data…”.

Now select “Binary Certificate” from the Column that contains binary data, and select “Save binary data to a file” for Export options.

Then select the directory and file to save.

3. Complete Certificate request in IIS

Now go back to the webserver and complete the certificate request. From the Server Certificate, click on “Complete Certificate Requests…”. Then choose the file from the previous step, and enter the Friendly name that will be shown in the certificate personal store, and click OK.

Now you should see the new certificate, you are ready for the binding with your website.

4. Bind the certificate to HTTPS

From your site, select Binding…, then click “Add…”. On the “Add Site Binding” screen change the Type to “https”, then add hostname. Finally, select the newly created certificate and click OK.

Now you should be able to have the site configured for HTTPS.

DevOps Automation

Now we cover the automation part with Ansible. We will divide into 4 different steps:

1. Automate CSR Request
2. Automate Generation of File and Script for CA
3. Automate Signing and Generating Certificate
4. Automate Importing and Binding the Certificate in IIS

With each of the steps, we want to check the state that if the certificate already exists it does not regenerate. This approach is all based on the existing share drive data that indicate if the CSR or CER file is needed.

Let’s us go into detail.

1. Automate CSR Request

We will be using certreq.exe to generate the CSR. The command takes in a conf file and the name of the output CSR file.

certreq.exe -new reqconfig.inf cert_csr.txt

A simple example of reqconfig.inf:

[NewRequest]
; At least one value must be set in this section
subject = "CN=test.tovnah.com"
Exportable = TRUE
KeyLength = 2048
KeySpec = 1
KeyUsage = 0xf0
MachineKeySet = TRUE

When you run the command you should see the output file.

Example of the output CSR file:

-----BEGIN NEW CERTIFICATE REQUEST-----
MIIDpTCCAo0CAQAwGjEYMBYGA1UEAwwPdGVzdC50b3ZuYWguY29tMIIBIjANBgkq
hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApjsXV8PicJFrbsCgCbNai6DaPmUmwFjm
<cut for brevity>
gzNZn7KBFxwIW8xM+mLr1D7sLNTqodSlo0yMyXP+9BhthwIpblktLutDblh7b745
kdETiEodJa2qk2dIZTpNfgUs7kmB7b6AVw==
-----END NEW CERTIFICATE REQUEST-----

To automate this, in Ansible we use the win_command module in our task. It looks like this:

- name: 'Generate Certificate Signing Request (CSR)'
  win_command: 'certreq -new -f {{cert_req_inf}} {{csr_file}}'
  register: cmdout
  become: yes
  become_method: runas
  become_flags: logon_type=new_credentials logon_flags=netcredentials_only
  vars:
    ansible_become_user: '{{admin_usr}}'
    ansible_become_pass: '{{admin_passwd}}'

Notice that we run runas to become the user that has permission to generate the CSR.

2. Automation Generating File and Script for CA

The request file config and the output CSR file are put in the share directory so it can be accessed from the webserver and the CA server. The folder structure is set up to have one server request per folder.

The cert_req_inf file is generated certificate to match its hostname. To accommodate this we use the win_template module.

- name: Create cert_req_inf file from template
  win_template:
     src: 'cert_req_inf.j2'
     dest: '{{cert_req_inf_file_path}}'
  become: yes
  become_method: runas
  become_flags: logon_type=new_credentials logon_flags=netcredentials_only
  vars:
    ansible_become_user: '{{admin_usr}}'
    ansible_become_pass: '{{admin_passwd}}'

The template file will have the full content just with variables it can substitute to match the specific hostname. The template file will look something like this:

[NewRequest]
Subject = "CN={{ssl_cert_subject}}" ; website hostname
Exportable = TRUE ; TRUE = Private key is exportable
KeyLength = 2048 ; Valid key sizes: 1024, 2048, 4096, 8192, 16384
KeySpec = 1 ; Key Exchange – Required for encryption
KeyUsage = 0xA0 ; Digital Signature, Key Encipherment
MachineKeySet = True
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"
RequestType = PKCS10 ; or CMC.

[EnhancedKeyUsageExtension]
; If you are using an enterprise CA the EnhancedKeyUsageExtension section can be omitted
OID=1.3.6.1.5.5.7.3.1 ; Server Authentication
OID=1.3.6.1.5.5.7.3.2 ; Client Authentication

[Extensions]
; If your client operating system is Windows Server 2008, Windows Server 2008 R2, Windows Vista, or Windows 7
; SANs can be included in the Extensions section by using the following text format. Note 2.5.29.17 is the OID for a SAN extension.
2.5.29.17 = "{text}"
_continue_ = "dns={{ssl_cert_subject}}&"
_continue_ = "dns={{ssl_cert_subject}}.mycompany.com"

[RequestAttributes]
; If your client operating system is Windows Server 2003, Windows Server 2003 R2, or Windows XP
; and you are using a standalone CA, SANs can be included in the RequestAttributes
; section by using the following text format.
"SAN="dns={{ssl_cert_subject}}&dns={{ssl_cert_subject}}.mycompany.com"
; Multiple alternative names must be separated by an ampersand (&).
CertificateTemplate = MyWebServerTemplate ; Modify for your environment by using the LDAP common name of the template.

The same is needed to generate the batch command file that will be executed at the CA server to sign and generate the certificate.

- name: Create sign_req.bat file from template
  win_template:
     src: 'sign_req.bat.j2'
     dest: '{{cert_bat_file_path}}'
  become: yes
  become_method: runas
  <same options as earlier>

The sign_req.bat will be using the same certreq.exe command with submit option. Make sure you get the config option correct, or you will get the prompt that needs to click which will pause and fail the automation. Test your script and ensure you don’t get the prompt.

Here is an example output of the sign_req.bat file:

certreq ^
-config "<ca_cert_server\mycompany-ca>" ^
-attrib "CertificateTemplate:MyWebServerTemplate" ^
-submit "new_ssl_cert_req.csr" "my_cert.cer"

3. Automate Signing and Generating Certificate

After the batch file is generated, we just need to run that file in the CA server. Since the host that we have been running is on the webserver, we need to give WinRm access to all of the web servers. If you need to automate a lot of the web servers, this might pose more security risks. One approach to limit the risk is by giving WinRm access to the CA server to only the ansible server and you can harden the firewall accordingly. To run a remote connection from the Ansible server, we use the delete_to command.

- import_tasks: cert_gen.yml
  delegate_to: localhost

The trick is that the cert_gen task will use delegate_to to connect to the CA server. So from localhost, which is the Ansible server, we delegate_to CA server to a batch file. In that case, the Ansible server will remote connect to CA server using WinRM and execute the batch file. The Ansible code in the cert_gen task looks like this

# tasks/cert_gen.yml  
- name: 'Execute Submit_req.bat to sign and create certificate'
  win_command: '{{cert_bat_file_path}}'
  become: yes
  become_method: runas
  <similar options as earlier>
  delegate_to: '{{ca_cert_server}}'

4. Automate Importing and Binding the Certificate in IIS

We can import the certificate using the win_certificate_store module. The Ansible code is as follow:

- name: 'Importing Certificate:{{cert_cer_file_path}}'
  win_certificate_store:
    path: '{{cert_cer_file_path}}'
    store_location: LocalMachine
    key_storage: machine 
    key_exportable: yes
    state: present
  become: yes
  become_method: runas
  <same options as early>
  register: cert_import

Again, the cert_cer_file_path is the cer file in the share drive that the CA and webserver have access to. The runas option is important since that user has access to the share drive.

Note that we register to cert_import variable. This is important for us to be able to get the thumbprint for the next few steps.

We also need an extra step to repair the certificate from the import process.

- name: 'Repair import cert in case private key is missing'
  win_command: 'certutil -repairstore my "{{cert_import.thumbprints[0]}}"'
  become: yes
  become_method: runas
  <same options as early>
  when: cert_import.changed == true

Next is to bind the HTTPS to this certificate thumbprint with win_iis_webbinding module.

- name: 'Configure https binding'
  win_iis_webbinding:
    name: '<my hostname>'
    protocol: 'https'
    host_header: '<my hostname>'
    port: '443'
    state: 'present'
    certificate_hash: '{{cert_import.thumbprints[0]}}'

Other steps may include the DNS update win_dns_record module as you create a new DNS A-record for the website.

Hope this will be helpful in your process to automate certificate creating and binding. Let me know your feedback below.

Next, see how we can use a free certificate issue “Let’s Encrypt” to generate certificates for us.

Other Format and Tools

We only cover the IIS setup above. A similar process can apply to set up the certificate on different applications and OS. But you need the right format that each application is expecting.

PEM Encoded

Some applications expect two files as input: a public key and a private key in PEM format. PEM or Privacy Enhanced Mail is a Base64-encoded of the binary formatted certificate or DER. To take the certificate in our IIS and convert it to PEM format we can use openssl utility that is available in many OS.

To produce a format that OpenSSL needs, you will have to export the certificate into a pfx (Personal Information Exchange) format that has a public and a private key. From the Windows Certificate plugins in MMC, select the desired certificate and select “All Tasks”, then “Export…”. In the Certificate Export Wizard, select the “Yes, export the private key” option and enter the password when prompted.

This password is needed by the OpenSSL command below when the tool try to open the pfx file.

openssl pkcs12 -in c:\mycert.pfx -clcerts -nokeys -out c:\mycert.pem

openssl pkcs12 -in c:\mycert.pfx -clcerts -nocerts -out c:\mykey.pem -nodes

These commands will ask for the password and out the file mycert.pem which is the public key and mykey.pem which is the private key.

Other applications may require one PEM encoded file that contains a private and a public key. To produce the file from the pfx format above, run:

openssl -in c:\mycert.pfx -out c:\mycert.pem
Enter Import Password:
Enter PEM pass phrase:
Verifying — Enter PEM pass phrase:

This will first prompt for “Import Password” to open the pfx file, then it will prompt for “PEM pass phrase” to be set.

Java Keystore

For applications that use Java such as Tomcat, you may need a different tool to convert your certificate. Java has a keytool that can store the certificate and its key. Some options in Tomcat can import crt and pem format directly.

Here is the keytool command that imports the certificate and the key into a key store.

keytool -importkeystore -deststorepass <DEST_STORE_PASSWORD>
-destkeypass <DEST_KEY_PASSWORD>
-destkeystore <KEYSTORE_FILENAME>
-srckeystore <PKCS12_KEYSTORE_FILENAME> -srcstoretype PKCS12
-srcstorepass <PKCS12_KEYSTORE_PASSWORD> alias tomcat

The KEYSTORE_FILENAME defaulted under your profile directory with the name “.keystore” (ie. c:\Users\me\.keystore). Some use the convention of naming jks extension such as mykeystore.jks.

If you use the keytool to create the CSR request then you can use the keytool command with “-importcert” option to import the certificate signed by your CA. See more options using simple guide here.

In the next post, you see how you can use a free Let’s Encrypt Certificate Authority that has a complete life cycle automation.

References

• How to Generate an IIS Certificate Request with PowerShell (and Bind)
• Install Enterprise Root Certificate Authority

Time series forecasting is an important topic for machine learning to predict future outcomes or extrapolate data such as forecasting sale targets, product inventories, or electricity consumptions. I found many general ML classes that cover specific context in detail but not in a full scope. This post hope to cover a wider scope by highlighting the different approaches to time series forecasting from statistical methods to a more recent state of the arts deep learning algorithms in late 2020.

The topics will be as follow:

1. Statistical technique: ARIMA
2. Exponential smoothing approaches (MA, Holt ES, Double ES, Holt-Winter)
3. ML algorithms with Linear Regression and XGBoost
4. Deep Learn approaches with CNN, RNN, and Transformer
   (LSTM, CNN with LSTM, TCN, LSTNet, N-BEATS, TFT)
5. Commercial applications: Facebook Prophet and Amazon DeepAR

The full working code for each of these algorithms is in the Python notebooks here: https://github.com/phylypo/TimeSeriesPrediction.

In the notebook, we also cover other algorithms not mentioned here like FFT, HMM, and other State Space approaches.

Introduction to Time Series

Time series is a sequence of data that has some order usually with a time component in a set interval. A typical example is the stock daily closing price over time or DNA sequences.

Typical tasks involve times series includes:

• Data Analysis: also known as exploratory data analysis (EDA) of a time series involves analyzing the data for its structure such as seasonality or nature of its temporal process.
• Prediction/Forecasting: Prediction of future values of time series involves using past data to predict the future.
• Classification: You can classify time series into different types such as detection of an ECG heartbeat graph is a normal type or not.
• Anomaly Detection: Another task is detecting values that are not within the expected range in the time series.

We will be mainly focusing on a predicting task but partly cover some analysis in order to better understand the time series dataset.

Types of time series

There are two types of time series:

1. univariate: time series with a single observation per time increments.
2. multivariate: time series that has more than one observation per time increments. These observations or time-dependent variables can capture the dynamic of multiple time series.

We will be focusing on univariate in this article for simplicity. There are some example codes that use multivariate.

Dataset

To model or make prediction, we first need the dataset. These dataset are self explanatory with link to download. There are a few publicly available datasets.

Univariate Dataset:

• Air passengers: number of air passenger per month over 12 years (1949–1960) https://assets.digitalocean.com/articles/eng_python/prophet/AirPassengers.csv
• Sunspot: monthly count of the number of observed sunspots for just over 230 years (1749–1983) https://storage.googleapis.com/laurencemoroney-blog.appspot.com/Sunspots.csv
• Shampoo Sales: monthly sales of shampoo over a 3 year period (https://raw.githubusercontent.com/jbrownlee/Datasets/master/shampoo.csv)
• Milk production: average monthly milk production (in lbs) of cows from Jan/1962 to Dec/1975. (https://raw.githubusercontent.com/plotly/datasets/master/monthly-milk-production-pounds.csv)

Multivariate Dataset:

• Traffic: A collection of 48 months (2015–2016) hourly data from the California Department of Transportation. The data describes the road occupancy rates (between 0 and 1) measured by different sensors on the San Francisco Bay area freeways.
• Solar-Energy: the solar power production records in the year 2006, which is sampled every 10 minutes from 137 PV plants in Alabama State.
• Electricity: The electricity consumption in kWh was recorded every 15 minutes from 2012 to 2014
• Exchange-Rate: the collection of the daily exchange rates of eight foreign countries including Australia, British, Canada from 1990 to 2016.

All three above datasets can be found here: https://github.com/laiguokun/multivariate-time-series-data.

Kaggle Tourism:

• Part one requires competitors to predict 518 tourism-related time series. It contains 518 yearly time series. (https://www.kaggle.com/c/tourism1)
• Part two requires competitors to predict 793 tourism-related time series. It contains 793 different time series. The first 366 columns contain monthly time series. The next 427 time series contain quarterly time series. (https://www.kaggle.com/c/tourism2)

Makridakis Competitions:

• M3: A total of 3003 different time series was used.
• M4: 100,000 real-life series
  https://mofc.unic.ac.cy/the-dataset/, https://github.com/Mcompetitions/M4-methods/tree/master/Dataset

We will be using the AirPassenger dataset throughout this article. Here is what the monthly count of air passengers looks like from 1949 to 1960.

Components of Time Series

The above chart of the AirPassengers data clearly shows a linear trend and some seasonality. These are example of components in time series. Formally, time series consists of:

1. Level: local mean of the series
2. Trend: the change between successive time points
3. Seasonality: the deviation from the local mean due to seasonality
4. Noise: residual components or random variation

𝑦 = 𝑙𝑒𝑣𝑒𝑙 + 𝑡𝑟𝑒𝑛𝑑 + 𝑠𝑒𝑎𝑠𝑜𝑛𝑎𝑙𝑖𝑡𝑦 + 𝑛𝑜𝑖𝑠𝑒
where 𝑦 is the observation.

With the AirPassengers dataset above, it can be decomposed into trend, seasonality, and noise as below:

Predictions and Metrics

The goal of our task is to predict the future values. Then we need to measure how our predictions for different approaches performed in comparison. There are two types of prediction that we need to distinguish before we measure it.

1. One-step ahead prediction: predicting the next one step from our last observation
2. Multi-horizons forecast: predicting multiple steps ahead

In the multi-horizon forecast, we can accomplish this through two approaches:

1. Iterated approaches: utilize one-step-ahead prediction and recursively feeding predictions to future inputs.
2. Direct approach: to explicitly generate predictions for multiple time steps at once.

Before we can measure how well our algorithms predict the future values, we need to discuss how we split our data into training and test sets before running the experiment.

Train and Test Split

In machine learning, we need to test our algorithm on the data that we have not seen. This is done by setting aside a portion of the dataset that will not be used in the training. This set is only used for testing the model. So in our dataset, we want to train on the earlier part of the dataset and leave out the later part of the dataset to evaluate how well the model performs. We don’t want to randomly split the dataset since sequence matters. We can split the data as follow:

• Training set: take the early portion of the data (ie. the first 80%)
• Test set: set aside the latest part of the data (ie. last 20% or the last year of monthly data, or last month of the yearly data)

Metrics

To measure how well the algorithms perform, there are several metrics that we can use:

• MSE: Mean Squared Error
  Pandas Series: np.square(y_hat - y).mean()
• RMSE: Root Mean Squared Error
  Pandas Series: np.sqrt(MSE)
• MAE: mean absolute error
  Pandas Series: abs(y_hat - y).mean()
• MAD: mean absolute deviation
  Pandas Series: abs(y_hat - y.mean()).mean()
• MAPE: mean absolute percentage error
  Pandas Series: (abs(y - y_hat) / y).mean() * 100
• sMAPE: Symmetric mean absolute percentage error (scales the error by the average between the forecast and actual)
  Pandas Series: abs(y - y_hat).sum()/(y + y_hat).sum( ) * 100
• MASE: Mean Absolute Scaled Error — scales by the average error of the naive null model

where
y : the ground truth observation value
y_hat : predicted value

We will be using MAE as our metric to see how well our algorithms are doing.

Naive Approaches

We need a baseline approach that we can compare the performance of our algorithms. There are several naïve approaches :

• Null model: predict the next value as the previous one (persistence model)
• Seasonal Naive: add seasonality to the prediction by using the previous season value
• Mean: take an average of all previous values as the forecast value
• Random Walk: randomly add noise to the previous value as the next prediction

Here is the illustration of the predictions of the two models (null model and seasonal naive).

The MAE for the Null model for this dataset to predict the last 12-month is 49.95 and for the Seasonal Naive model is 45.60. We will use this as our baseline comparison.

Smoothing

The technique of smoothing is using the moving average or exponential smoothing as a way to filter out some of the noises in the data.

Simple Moving Average

Each data point is an average over n data points. The larger n is, the smoother the chart becomes. In pandas, you just call:

pandas.DataFrame.rolling(n, center=False).mean()

The moving average could be in the center and trailing Moving Average by specified the center argument

Here are examples of moving average with 3-month, 5-month moving average, and 3-month exponential moving average that will be explained below.

Exponential Smoothing

Linear Exponential Smoothing (Holt ES)

The Linear Exponential Smoothing (Holt ES) is also known as a simple exponential weighted moving average. Whereas the simple moving average weighted the previous points equally, simple exponential MA gives more weight to the more recent values.

In pandas, we can use:

pandas.DataFrame.ewm(alpha=0.12).mean()

Double Exponential Smoothing

Also known as Holt-Winters Double ES, double exponential smoothing takes into account the trend.

Triple Exponential Smoothing (Holts-Winters ES)

Also known as Holts-Winters ES, Triple Exponential Smoothing take into account the seasonality in addition to the trend.

Using these exponential smoothing’s to predict future value we get the following:

The MAE for these are:

• MAE Simple ES: 50.86
• MAE Double ES: 52.62
• MAE Triple ES: 18.44

Notice that the linear and double exponential smoothing is doing worst than the Null model while the Holt-Winter is doing very well on this dataset.

ARIMA

Also known as the classical Box-Jenkins methodology, ARIMA is the statistical approach created specifically for time series.

ARIMA is part of the state-space model (SSM). SSM also includes the exponential smoothing technique we have seen earlier. It is based on a structural analysis of the problem that can be described by different components like trend, seasonality, cycle. It is suitable for an approach where the structure of the time series is well-understood.

ARIMA has 3 distinct structures as follow:

• AR : Auto Regression — uses the dependent relationship between an observation and some number of lagged observations. The lag 𝑛 value is the previous 𝑛 value of the observation.
• I : Integrated — uses the differencing of raw observations in order to make the time series stationary.
• MA : Moving Average — uses the dependency between an observation and a residual error from a moving average model applied to lag observations.

In the library that we will be using, the notation is ARIMA(p,d,q) where:

• p: (AR) The number of lag observations included in the model, also called the lag order.
• d: (I) The number of times that the raw observations are differenced, also called the degree of differencing.
• q: (MA) The size of the moving average window, also called the order of moving average.

Since we will be applying the seasonality, the API we will be using is SARIMAX. The S means Seasonality where X means eXogenous which allows for the inclusion of other data besides the univariate values.

ARIMA Parameters Search

We can import the following library: sm.tsa.statespace.SARIMAX. Examples of parameter combinations for Seasonal ARIMA will have p,d,q for ARIMA, then for S in seasonality, there are p,d,q, and seasonal value. For example:

SARIMAX: (0, 1, 0) x (0, 1, 1, 12) where 12 is the seaonsal value.

You may analyze for the p, d, q based on the dataset or use grid search with pmdarima library which does the grid search for us more efficiently.

You will need to install the package first then import and run the method as the following:

# required restart of the runtime
#!pip install pmdarima

# Import the library, require install and restart
from pmdarima import auto_arima

stepwise_fit = auto_arima(data[‘AirPassengers’], 
  start_p = 1, start_q = 1,
  max_p = 3, max_q = 3, m = 12,
  start_P = 0, seasonal = True, …)

You can also do a manual grid search yourself. The code is included in the notebook by finding the smallest AIC value.

We use the following hyperparameters as input to the ARIMA algorithm:

ARIMA(1, 1, 1)x(1, 1, 1, 12)12 — AIC:803.3627295936407

We got the following prediction with the MAE of 15.22.

The gray area indicates the confidence level. As we predict further time steps, the range gets wider. Since this is one-step ahead prediction, at each time step we use the prediction value as the input for the next time step prediction.

ML: Classical Supervised Learning Approaches

We’ll explore two basic supervised learning techniques:

1. Linear Regression
2. Decision Tree with XGBoost (Gradient Boosted Tree)

Generating Features

In earlier statistical approaches like ARIMA, the system was built for time series data. We can just feed the data directly. But for these ML approaches, we need to generate features that the algorithms can use to account for time ordering. Some of the features include:

• lags : value at time step t-N
• amplitude: amplitude from the mean value
• max/min value: max or min value at that window
• value > 1 std : count of value great than 1 standard deviation

For our case, using lag values for our features is sufficient. To generate lag features we just use the data and shift by one value per lag. As an example, we want to create 2 lags feature from this data [2, 4, 6, 8, 7, 9], we would have:

lag2  lag1  y
— — — — — — -
nan   nan   2
nan   2     4
2     4     6
4     6     8
6     8     7
8     7     9

Linear Regression

We will start with linear regression. We will fit the data with linear regression with 12 lags values as our features since we knew that our seasonality is 12 months. So the lags f01 to f12 are the features and y is the actual value:

 f12  f11  f10  f09  f08  f07  f06  f05  f04  f03  f02  f01    y
------------------------------------------------------------------
[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, 112],
[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, 112, 118],
[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, 112, 118, 132],
[nan, nan, nan, nan, nan, nan, nan, nan, nan, 112, 118, 132, 129],
[nan, nan, nan, nan, nan, nan, nan, nan, 112, 118, 132, 129, 121],
[nan, nan, nan, nan, nan, nan, nan, 112, 118, 132, 129, 121, 135],
[nan, nan, nan, nan, nan, nan, 112, 118, 132, 129, 121, 135, 148],
[nan, nan, nan, nan, nan, 112, 118, 132, 129, 121, 135, 148, 148],
[nan, nan, nan, nan, 112, 118, 132, 129, 121, 135, 148, 148, 136],
[nan, nan, nan, 112, 118, 132, 129, 121, 135, 148, 148, 136, 119],
[nan, nan, 112, 118, 132, 129, 121, 135, 148, 148, 136, 119, 104],
[nan, 112, 118, 132, 129, 121, 135, 148, 148, 136, 119, 104, 118],
[112, 118, 132, 129, 121, 135, 148, 148, 136, 119, 104, 118, 115],
[118, 132, 129, 121, 135, 148, 148, 136, 119, 104, 118, 115, 126],
[132, 129, 121, 135, 148, 148, 136, 119, 104, 118, 115, 126, 141],

…

Our first input will start with the 13th row that has the complete 12 feature values without nan.

from sklearn import linear_model

model = linear_model.LinearRegression()
model.fit(trainX, trainy)

# make a one-step prediction
yhat = model.predict(asarray([testX]))

We see the result with MAE of 17.69.

XGBoost

With the same technique, we can replace the algorithm with XGBoost.

from xgboost import XGBRegressor
model = XGBRegressor(objective=’reg:squarederror’, n_estimators=500)
model.fit(trainX, trainy)

# make a one-step prediction
yhat = model.predict(asarray([testX]))

We get an MAE of 25.58.

This shows that the linear regression is pretty good with this simple dataset. Although XGBoost is known to do very well in many of Kaggle competition datasets.

Deep Learning with RNN — LSTM

We now look at deep learning approaches with RNN. Instead of using the plain RNN, we will use Long Short-Term Memory (LSTM). LSTM improved over the plain RNN by helping to mitigate the vanishing and exploding gradient issue.

In the preprocessing step, we also added MinMaxScaler to ensure our data is in the same range (0–1). This seems to improve our prediction.

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
scaler.fit(x_train)

x_train = scaler.transform(x_train)
x_valid = scaler.transform(x_valid)

Model:

tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(260,
  activation=’relu’, return_sequences=True)),
  tf.keras.layers.Dropout(0.15),
  tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(248)),
  tf.keras.layers.Dropout(0.05),
  tf.keras.layers.Dense(1), ...)

With Adam optimizer:

optimizer = tf.keras.optimizers.Adam()
model.compile(loss=”mse”, optimizer=optimizer, metrics=[“mae”])
model.fit(dataset,epochs=100,verbose=1)

Epoch 100/100

20/20 [==============================] — 3s 134ms/step — loss: 0.0039 — mae: 0.0507

The latest result in the notebook, our MAE is 47.37.

After tweaking some parameters and architecture, we were able to get the MAE on the test set to be 33.39.

References:

• https://github.com/gianfelton/12-Month-Forecast-With-LSTM/blob/master/12-Month%20Forecast%20With%20LSTM.ipynb
• https://colab.research.google.com/github/lmoroney/dlaicourse/blob/master/TensorFlow%20In%20Practice/Course%204%20-%20S%2BP/S%2BP%20Week%203%20Lesson%204%20-%20LSTM.ipynb

CNN and LSTM

We can add a CNN layer in front of LSTM to help encode the input. This showed an improvement of LSTM alone.[]

For optimizer, we use momentum.

tf.keras.layers.Conv1D(filters=60, kernel_size=5,
  strides=1, padding=”causal”,
  activation=”relu”,
  input_shape=[None, 1]),
  tf.keras.layers.LSTM(200, return_sequences=True), #60
  tf.keras.layers.LSTM(200, return_sequences=True),
  tf.keras.layers.Dropout(0.15),
  tf.keras.layers.Dense(200, activation=”relu”),
  tf.keras.layers.Dense(10, activation=”relu”),
  tf.keras.layers.Dense(1),

Output:

Epoch 150/150

5/5 [==============================] — 0s 53ms/step — loss: 23.7124 — mae: 24.2093

The MAE for the test set is 40.25. The error is larger than LSTM above. We used a scaler to improve the performance in the LSTM approach.

Reference: https://colab.research.google.com/github/lmoroney/dlaicourse/blob/master/TensorFlow%20In%20Practice/Course%204%20-%20S%2BP/S%2BP%20Week%204%20Lesson%205.ipynb

TCN

Temporal Convolutional Networks (TCN) is a variation of Convolutional Neural Networks for sequence modeling tasks, by combining aspects of RNN and CNN architectures. This model performs a multi-horizon prediction.

TCN is based on two principles:

1. the network produces an output of the same length as the input
2. there can be no leakage from the future into the past.

To accomplish the first point, the TCN uses a 1D fully-convolutional network (FCN) architecture.

The paper showed improved performance over LSTM, GRU, and RNN in many of the datasets.

TCN implementations for different ML libraries can be found here:

• Pytorch: http://github.com/locuslab/TCN
• Keras: https://github.com/philipperemy/keras-tcn
• Tensorflow: https://github.com/Baichenjia/Tensorflow-TCN

Our code uses the keras-tcn library as follow:

# installing tcn
#!pip install keras-tcn

from tcn import TCN, tcn_full_summary

i = Input(shape=(lookback_window, 1))
m = TCN()(i)
m = Dense(1, activation='linear')(m)

model = Model(inputs=[i], outputs=[m])
model.summary()
model.compile('adam', 'mae')

model.fit(x_train, y_train, epochs=150, verbose=0)

Output:

Model: "functional_25"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_13 (InputLayer)        [(None, 12, 1)]           0
_________________________________________________________________
tcn_12 (TCN)                 (None, 64)                91136
_________________________________________________________________
dense_12 (Dense)             (None, 1)                 65
=================================================================

Total params: 91,201
Trainable params: 91,201
Non-trainable params: 0

We got the MAE of 15.01 for the AirPassenger dataset.

References:

• “An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling”, Shaojie Bai, J. Zico Kolter, Vladlen Koltun (https://arxiv.org/abs/1803.01271)
• https://towardsdatascience.com/farewell-rnns-welcome-tcns-dd76674707c8
• https://dida.do/blog/temporal-convolutional-networks-for-sequence-modeling

Long- and Short-Term Temporal Network (LSTNet)

Multivariate time series forecasting often faces a major research challenge, that is, how to capture and leverage the dynamics dependencies among multiple variables.

The shortcoming of classical ARIMA:

• ARIMA models are adaptive to various exponential smoothing techniques and flexible enough to subsume other types of time series models. However, ARIMA models, are rarely used in high dimensional multivariate time series forecasting.
• vector autoregression (VAR) is arguably the most widely used models in multivariate time series due to its simplicity. The model capacity of VAR grows linearly over the temporal window size and quadratically over the number of variables.

LSTNet Components:

1. CNN with k-th filter with RELU
2. RNN (GRU) — with RELU
3. RNN Skip Connection
4. Temporal Attention Layer
5. Autoregression with highway

Model consisted of: (CNN, and RNN(GRU) and dropout layer)

class LSTNet(nn.Module):

  def __init__(self, args, data):
    super(LSTNet, self).__init__()
    self.P = args.window;
    self.m = data.m
    self.hidR = args.hidRNN;
    self.hidC = args.hidCNN;
    self.hidS = args.hidSkip;
    self.Ck = args.CNN_kernel;
    …
    self.conv1 = nn.Conv2d(1, self.hidC, kernel_size = (self.Ck, self.m));
    self.GRU1 = nn.GRU(self.hidC, self.hidR);
    self.dropout = nn.Dropout(p = args.dropout);

The MAE is 13.96 for the AirPassengers dataset.

References:

• https://modelzoo.co/model/lstnet
• https://opringle.github.io/2018/01/05/deep_learning_multivariate_ts.html
• “Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks”, Lai et al., SIGIR 2018 (https://arxiv.org/pdf/1703.07015.pdf ) Code: https://github.com/laiguokun/LSTNet

N-BEATS

Neural Basis Expansion Analysis for Interpretable Time Series (N-BEATS) is built for:

• multi-horizons
• multiple series
• interpretabilities: seasonality and trend.

“Pure DL using no time-series specific components outperforms well-established statistical approaches on M3, M4 and TOURISM datasets (on M4, by 11% over statistical benchmark, by 7% over the best statistical entry, and by 3% over the M4 competition winner),” N-BEATS.

Architectural design:

• simple and generic, yet expressive (deep).
• not rely on time series specific feature engineering or input scaling for pure DL architecture in time series forecasting.
• explore interpretability, extendable towards making its outputs human interpretable.

Model types in the notations:

• I : interpretability
• G : general
• G+I : ensembles with I and G

Interpretabilities is provided with graph of trend and seasonality like below:

# install the libraries
#!pip install pytorch_lightning
#!pip install pytorch_forecasting

import pytorch_lightning as pl
from pytorch_lightning.callbacks import EarlyStopping

from pytorch_forecasting import TimeSeriesDataSet, NBeats, Baseline
from pytorch_forecasting.data import NaNLabelEncoder
from pytorch_forecasting.data.examples import generate_ar_data
from pytorch_forecasting.metrics import SMAPE

trainer = pl.Trainer(
  max_epochs=100,
  gpus=0,
  weights_summary="top",
  gradient_clip_val=0.1,
  callbacks=[early_stop_callback],
  limit_train_batches=30,
)

net = NBeats.from_dataset(training, learning_rate=4e-3,
  log_interval=10, log_val_interval=1, weight_decay=1e-2, 
  widths=[32, 512], backcast_loss_ratio=1.0)

trainer.fit(net, train_dataloader=train_dataloader,  
  val_dataloaders=val_dataloader,
)

Performance on AirPassengers dataset,

We have to finetune a lot of hyper parameters. We added monthly hint and show a little improvement.

After some finetune we get the MAE of 26.42.

References:

• Paper: “N-BEATS: Neural Basis Expansion Analysis for Interpretable Time Series Forecasting”, Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, Yoshua Bengio, May 2019, ICLR 2020 (https://arxiv.org/abs/1905.10437)
• https://medium.com/@kshavgupta47/n-beats-neural-basis-expansion-analysis-for-interpretable-time-series-forecasting-91e94c830393
• https://github.com/philipperemy/n-beats

Temporal Fusion Transformers (TFT)

“By interpreting attention patterns, TFT can provide insightful explanations about temporal dynamics, and do so while maintaining state-of-the-art performance on a variety of datasets,” Lim et al.,2019.

For interpretability, TFT uses the attention mechanism as follow:

• Attention mechanisms are used to identify salient portions of input for each instance using the magnitude of attention weights.
• TFT uses separate encoder-decoder attention for static features at each time step on top of the self-attention to determine the contribution time-varying inputs.

Here are the performance cited in the paper.

The paper compares to other models (ARIMA, ETS, TRMF, DeepAR, DSSM, ConvTrans, Seq2Seq, and MQRNN). Here is some information on the different models.

• ETS (Error, Trend, Seasonal) method is an approach method for forecasting time series univariate. (https://otexts.com/fpp2/arima-ets.html)
• DeepAR uses stacked LSTM layers to generate parameters of one-step-ahead Gaussian predictive distributions. This will be described later.
• Deep State-Space Models (DSSM) utilize LSTMs to generate parameters of a predefined linear state-space model with predictive distributions produced via Kalman filtering — with extensions for multivariate time series data.
• ConvTrans: the Transformer-based architecture with the use of convolutional layers for local processing and a sparse attention mechanism to increase the size of the receptive field during forecasting. (https://arxiv.org/abs/1907.00235)
• The Multi-horizon Quantile Recurrent Forecaster (MQRNN) uses LSTM as convolutional encoders to generate context vectors that are fed into multi-layer perceptrons (MLPs) for each horizon.

Here is the main part of the code. See the notebook for a complete working code.

!pip install pytorch_lightning
!pip install pytorch_forecasting

import pytorch_lightning as pl
from pytorch_forecasting import TimeSeriesDataSet, TemporalFusionTransformer, Baseline
from pytorch_forecasting.data import GroupNormalizer
from pytorch_forecasting.metrics import PoissonLoss, QuantileLoss, SMAPE
from pytorch_forecasting.models.temporal_fusion_transformer.tuning import optimize_hyperparameters

…

trainer = pl.Trainer( gpus=0, gradient_clip_val=0.1)

tft = TemporalFusionTransformer.from_dataset(
  training,
  learning_rate=0.03,
  hidden_size=16,  # most important hyperparameter
  attention_head_size=1,
  dropout=0.1,  # between 0.1 and 0.3 are good value
  hidden_continuous_size=8,  # set to <= hidden_size
  output_size=7,  # 7 quantiles by default
  loss=QuantileLoss(),
  # reduce learning rate if no improvement in validation loss
  reduce_on_plateau_patience=4,
)

I tried to use AirPassengers dataset but it didn’t turn out well. There are several parameters that might have not tuned properly.

References:

• “Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting” Bryan Lim, Sercan O. Arık, Nicolas Loeffb, Tomas Pfister, Dec 2019 (Paper)https://arxiv.org/abs/1912.09363.pdf
• https://github.com/google-research/google-research/tree/master/tft
• https://github.com/louisyuzhe/deeplearning_forecast
• https://pytorch-forecasting.readthedocs.io/en/latest/tutorials/stallion.html

Commercial Applications

Facebook Prophet

In 2017, Facebook open sourced a project called Prophet. “Prophet is a modular regression model with interpretable parameters that can be intuitively adjusted by analysts”. It decomposed the series into three main model components:

1. trend
2. seasonality
3. holidays

Use only time as a regressor but possibly several linear and nonlinear functions of time as components.

Has a number of advantages:

• Flexibility: We can easily accommodate seasonality with multiple periods
• Fitting is very fast
• Has easily interpretable parameters

from fbprophet import Prophet

my_model = Prophet(interval_width=0.98, daily_seasonality=False, weekly_seasonality=False, yearly_seasonality=True,)

my_model.fit(train_data)

future_dates = my_model.make_future_dataframe(periods=12, freq=’MS’)

forecast = my_model.predict(future_dates)

With the AirPassenger dataset, we get an MAE of 33.92.

You can see the explainability aspect with these:

References:

• Paper: https://peerj.com/preprints/3190/
• Code: https://github.com/facebookincubator/prophet
• Tutorial: https://www.digitalocean.com/community/tutorials/a-guide-to-time-series-forecasting-with-prophet-in-python-3

Amazon DeepAR

“DeepAR, a forecasting method based on autoregressive recurrent networks, which learns such a global model from historical data of all the time series in the data set,” Salinas et al., 2019. It uses a similar LSTM-based recurrent neural network architecture. This only runs on Amazon Sagemaker but there is an implementation in PyTorch in the link below. We did not get to try out the PyTorch code.

Classical Techniques Problem:

• there is a need for forecasting thousands or millions of related time series
• Examples: forecasting the load for servers in a data center, or forecasting the demand for all products of a large retailer
• related time series can be leveraged for making a forecast for an individual time series.
• allows fitting more complex (and hence potentially more accurate) models without overfitting
• eliminate manual feature engineering and model selection steps

Features:

• the model learns seasonal behavior and dependencies on given covariates across time series
• by learning from similar items, it can provide forecasts for items with little or no history at all

References:

• “DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks”, Salinas et al., 2019. (https://arxiv.org/abs/1704.04110)
• Implementation in PyTorch: https://github.com/zhykoties/TimeSeries
• https://towardsdatascience.com/prophet-vs-deepar-forecasting-food-demand-2fdebfb8d282

Conclusion

Time series fits well with a structural approach such as ARIMA which shows the interpretability with trend and seasonality. Holt-Winters triple exponential smoothing can account for trend and seasonality and give a similar result for the simple dataset we used.

Then in the machine learning approach, it required the input to be formatted with features. The typical linear regression is not that bad for our dataset. We also tried XGBoost which is frequently used by the Kaggle competition winners.

In deep learning, the sequence to sequence approaches like RNN and LSTM does show some promise. New architectures started to emerge just beyond CNN and RNN. TCN and LSTNet showed great performance surpassing ARIMA.

Then TFT and N-BEATS are the latest approaches that are the current state of the arts. While TFT outperformed previous approaches, N-BEATS outperformed previous winner approaches in the M4 competition.

Please see the link to Colab notebooks in Github here:
https://colab.research.google.com/github/phylypo/TimeSeriesPrediction

We will be using Google map API to display the map. The end result would look something like this.

I am going to detail each step that I took to set up and implement this. You can see the full code in an online IDE repl.it at the end.

1. Covid-19 data collection
2. Map data collection
3. Google map API key setup
4. Development tool
5. Code structure with Google map API
6. Implementing in React JS

1. Covid-19 data collection

First, we need the time-series data of Covid-19 for each province in Cambodia. The best place to start is John Hopkin dataset (https://github.com/CSSEGISandData/COVID-19). This dataset contains daily updates of Covid-19 for the whole world by date. Unfortunately, it does not contain a detailed breakdown by provinces in Cambodia.

The next source that I look at is the Cambodia Ministry of Health (https://covid19-map.cdcmoh.gov.kh/). Although the page has a breakdown by region, it does not have the time-series data.

The last source I found is Wikipedia (https://en.wikipedia.org/wiki/COVID-19_pandemic_in_Cambodia) which breaks down by region and detail for each case in a table. The data is not easily formatted for use but this is what we need.

Now we can enter this into our dataset. I manually enter it in Excel first then export it to a database. For our final dataset, we need something like this in Javascript with positive case count day by day that breakdown by provinces.

var dataHeader = ['Date', 'Banteay Meanchey', 'Battambang', 'Kampong Cham', 'Kampong Chhnang', 'Kampot', 'Kandal', 'Kep', 'Koh Kong', 'Phnom Penh', 'Preah Vihear', 'Siem Reap', 'Preah Sihanouk', 'Tboung Khmum'];

var data = [
  ['Jan-26', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
  ['Jan-27', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0],
  ['Mar-13', 0, 0, 3, 0, 0, 0, 0, 0, 2, 0, 1, 1, 0],
  ['Mar-15', 0, 0, 3, 0, 0, 0, 0, 0, 5, 2, 1, 1, 0],
  ['Mar-17', 2, 4, 3, 3, 0, 0, 4, 1, 9, 2, 2, 1, 2],
  ['Mar-19', 2, 6, 9, 3, 2, 1, 4, 1, 10, 2, 2, 1, 4],
  ['Mar-20', 2, 8, 10, 3, 2, 1, 4, 1, 11, 2, 2, 1, 4],
  …
]

Now that we have Covid-19 data, we will look at the map data next.

2. Map data collection: Provinces in Cambodia

Although Google map API provides the tool to display maps, it does not provide coordinate data to draw. So we need coordinates to tell the API to show a specific location and to draw borders by province. Openstreetmap (https://www.openstreetmap.org/) provides the region data for free.

To search for a province, just provide the name and specify the output format like ‘json’ like the following:

https://nominatim.openstreetmap.org/search.php?q=ខេត្តមណ្ឌលគិរី&polygon_geojson=1&format=json

You will the output like below:

This has the “lat” and “lon” which is the coordinate for the center of the region that we can use. The main data we need is the “geojson” of “Polygon” which has the list of coordinates. This is what we need to draw the region for each province.

I save each of the provinces into a data file that can be loaded by the script. We don’t want to hit the website for each request. It is also faster to read from the local file system too.

So I would have a js file that would look like this for all the regions I needed.

var coordMondulkiri = [[106.3385777, 13.1342029],…]
var coordBonteayMeanchey = [[102.3338282, 13.537473],…]

This is a pretty big data file. Some provinces have over 5K of coordinates. This may affect our development tool as we will look in the later section.

3. Google Map API Key setup

Before we can start coding, we need to create a Google map API key. You can follow the instruction here: https://developers.google.com/maps/documentation/javascript/get-api-key

After you create the key, you need to enable your billing. Or else you get this error below:

“Google Maps JavaScript API error: ApiNotActivatedMapError

https://developers.google.com/maps/documentation/javascript/error-messages#api-not-activated-map-error"

The Maps JavaScript API is not activated on your API project yet after creation. You need to enable the billing for your account in the Google Cloud Platform Console. Google allows you to use $200/month of usage before charging you. See pricing for detail (https://cloud.google.com/maps-platform/pricing).

Security

The key can be seen by anyone who inspects your code. So you need to restrict the use of the key just on your site. To set the restriction, go to Google Cloud Platform, then Credentials, select your API project, and then under Application restrictions, enter url with wildcard if needed similar to below.

If you test your map and see a similar screen like below, that means some setting is not correct. In Chrome, go to the Developer tool and see the detail of the error. If this is related to the URL restriction, the error will mention the url that you can add. Don’t forget to click on “Save” on the bottom of the screen in your Google Cloud Platform screen.

4. Development Tool

I want to quickly try out the Google Map API before I design for React JS used on our website.

I started the development in codepen.io just to get the map up. Then as I added the coordinate data to test the drawing the border, I started seeing issues with big files and unresponsiveness for long lines. So I started looking into other online JS editing tools. Here is a summary of what I found.

1. https://codepen.io/: can’t save after 1mb, more responsive than some
2. JSFiddle: froze after long text — ok for big file than 1MB
3. Jsbin: seem responsive https://jsbin.com/?html,js,output
   Can’t save too large file — smaller than codepen
4. https://codesandbox.io/ — online IDE, has nodejs and hosting
5. https://repl.it/@phylypo/ — node support, provide hosting

I chose repl.it as the online IDE development tool. The basic code structure is as follow:

HTML: index.htm

<html>
<head>
  <link href=”style.css” rel=”stylesheet” type=”text/css” />
  <script
src=”https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&callback=initMap&libraries=&v=weekly" defer> 
  </script>
  <script src=”my-map.js”></script>
</head>
<body>
  <div id=”map”></div>
</body>
</html>

Change “YOUR_API_KEY” to the value of your Google Map API key. The callback=initMap is the method that will be called when the API is ready. The div id=”map” is where the map goes. You want to style the map using the css similar to below.

CSS: style.css

#map {
  height: 90%;
  position: relative;
  z-index: 1;
}

JS: my-map.js

var map;

function initMap() {
  map = new google.maps.Map(document.getElementById(“map”), {
    zoom: 8,
    center: { lat: 12.467471, lng: 105.12 }
  });
}

In your Javascript, you can call Google map as shown. Here set the zoom level of the map and center the map to a latitude and longitude.

5. Code structure with native API

I won’t go over the full code since you can see yourself in the link below but I want to highlight a few features.

For the GUI, I added a button for the start/stop the time series. There is a date and total positive cases up to that date. In addition, I added the input of type range so the user can slide to a specific date at their own speed.

The interface allows the user to click on the boundary map to see an information windows with display the number of positive cases for that province.

Create UI Elements

We can create a DIV and add to map.controls as below.

startStopDiv = document.createElement(“div”);
startStopDiv.index = 1;
displayStartStop(startStopDiv, “Start”)
map.controls[google.maps.ControlPosition.BOTTOM_CENTER].push(
  startStopDiv
);

The displayStartStop function sets the style for the DIV and adds an event listener.

function displayStartStop(ccdiv, str){
  ccdiv.style.backgroundColor = “#FFDC00”;
  ccdiv.style.border = “2px solid #fff”;
  ...
  ccdiv.innerHTML = str;
  ccdiv.addEventListener(“click”, animationClick);
}

We will go over animationClick function below.

Create Info Window

The Info window popup is provided by the API. The InfoWindow object is initialized once in the initMap method and then add to the maps through the “click” event listener as below. The “content” is the html text to be displayed. So as you click on the bMap drawing, the InfoWindow will pop up at the bMap.center location. We will go over bMap object next.

var infoWindow;
infoWindow = new google.maps.InfoWindow();
var content = getInfoBubbleStr(name, infected, date)
google.maps.event.addListener(
  bMap,’click’, (function(bMap,content,infoWindow) {
    return function() {
      infoWindow.setContent(content);
      infoWindow.setPosition(bMap.center);
      infoWindow.open(map, bMap);
    };
  })(bMap, content, infoWindow));

Draw the border

To draw the border, we use the Polygon object. This object needs the coordinates to draw. Each coordinate has the latitude and longitude numbers. The polygon object has a stroke color for the border and a fill color to color the area of the polygon.

var bMap = new google.maps.Polygon({
  paths: coords,
  strokeColor: color,
  strokeOpacity: 0.65,
  strokeWeight: 3,
  fillColor: color,
  fillOpacity: 0.5,
  center: pcenter,
});
bMap.setMap(map);

The variable “map” is google.maps.Map object that we initialized in the initMap method.

Animation

To animate, we use the method animationClick which calls beginAnimate using the window.setInterval or window.clearInterval to stop. Each step we just call createBorders which draws the border at a specific date. The create border draw Polygon as showed earlier.

function animationClick() {
  if (startAnimation) {
    startAnimation = false;
    beginAnimate(200);
  } else {
    startAnimation = true;
    window.clearInterval(animation);
  }
};

function beginAnimate(timeout) {
  animation = window.setInterval(function () {
    createBorders(curStep);
    curStep++;
    if (curStep >= maxStep) {
      window.clearInterval(animation);
    }
  }, timeout);
}

Get Live Data from DB

The basic dataset is a list of dates followed by positive cases per province which can be seen in the timeseries.js. The timeseries.js output was statically entered for testing. But this file is not used at the end since we want to get up to date data from the database.

I enhanced to call from our Graphql server which outputs JSON data from our database. The JSON output is the record of each patient. The sample data can be seen in domnung_ts.js. The actual fetch method does not work due to cross-site scripting. This will only work when I deploy to our website. So I hardcoded the output from the Graphql as an example. Then I need to process the raw data and output to a format similar to timeseries.js. I won’t go into that detail. You can see the full code here: https://repl.it/@phylypo/cambodia-covid-19-boundary-map#index.html

6. Debugging

With the Chrome development tool, it is very convenient to debug by adding breakpoints and see the output in addition to inspecting and testing code in the console.

From the console tab, you can click on the file to the right of the log and see the code.

Then you can add a breakpoint and click refresh to stop on the breakpoint.

Now you can mouse over totalcaseVal to see the value or type in the console. Now you can step through each line of code if needed.

Once I’m happy with the code, I will continue to React JS next.

7. Implementing in React JS

The next part is to move the code to React JS. With the React JS component, there are quite a few packages to choose from for Google map API.

React JS components for Google Map API:

1. google-map-react : has a small set of components (4.6K stars)
2. React-google-map : a component to for maps and markers. (23 stars)
3. Google-maps-react : use of “lazy” loading of dependencies (1.3K stars)
   Simple and clean set up with straight forward api
4. react-google-maps : large number of react components (4.4K stars)
   Lots of components to draw, complicated setup, do most of the drawing, hard with overlay UI

I started with google-maps-react (#3) which seems pretty straightforward with a detailed tutorial. Then as I started to add more components and drawings, I ran into issues. So tried another one. “React-google-maps” seems to have a lot of components for drawing circle and polygon but run into issues with text UI that display date and total cases.

I decided to use the Google native API to draw and add UI components. So I went back to use Google-maps-react which has a clean setup. At the end of the development, I did not have to change much. I added the react component for the map and then use the native Google map API to do the rest of the drawing as done earlier.

React JS Code

At this step, I assume you already have a React JS running. We will just add a map component to your existing project. First, we start the React JS development by setup the package.json with the package “google-maps-react”. Then you need to install the package with your favorite package tool such as npm. In your module, I show some code structure for the Google map API.

Package: package.json

“google-maps-react”: “².0.6”,

Module: MapCovid.js

import { Map, GoogleApiWrapper } from 'google-maps-react';

export class MapCovid extends Component {
  // callback when getting timeseries data is completed
  startMapCallback = (totaldata) => {
    this.createBubbles(data.length-1);
    this.createNavButton(map);
  }

  onMapReady = (mapProps, map) => {
    this.map = map;
    loadJsonApi(this.startMapCallback);
  }

  render() {
    return (
      <div>
        <Map
          google={this.props.google}
          onReady={this.onMapReady}
          zoom={7}
          style={mapStyles}
          initialCenter={phnompenh}
        >
        </Map>
      </div>
    );
  }
}

export default GoogleApiWrapper({
  apiKey: 'YOUR_GOOGLE_API_KEY'
})(MapCovid);

Loading Process

When the map is ready, it calls the onMapReady method which sets the map context to this.map and calls a method to load the JSON data from the database.

Drawing Components

I used the same code from the earlier JS code with minor syntax changes. For example:

To access google map use window.google:

var bMap = new window.google.maps.Polygon({…})

I can’t add an event handler the same way as in html for inner div for nav object in the map. Normally we can do this:

<input id=”play-range” type=”range” oninput=”this.playRange(this.value)”>

Here we need to iterate to object to add an event listener. Also we can’t pass in the parameters directly. We have to use one of the following approaches:

var rangeinput = ccdiv.firstElementChild.firstElementChild;
rangeinput.addEventListener(“input”, this.playRange, false); 
…

playRange = (evt) => {
  var val = evt.currentTarget.value
  …
}

You can see the final integration with React JS here:

• https://domnung.com/t/covid-19-map (circle map)
• https://domnung.com/t/covid-19-map-boundary (region map)

Conclusion

Google map API provides a range of tools to draw on the map. To use it you need to enable the Google map API with billing enabled. You will need to restrict the usage to specific sites to avoid unrelated charges.

This post also covers the JS online editing IDE that can be used for free. Converting the code to be used in React JS was a challenge at first but I was able to use the native Google map API to complete the integration.

We accomplished our goal of visualizing the spread of Covid-19 in Cambodia at the province level through Google Map API and data from Wikipedia and OpenStreetMap.

You can see the full code here:
https://repl.it/@phylypo/cambodia-covid-19-boundary-map#index.html

Note that this does not have the React JS version since I didn’t set up React JS on repl.it.

I hope this gives you a high-level implementation with some specific detail that can get you started. Happy coding.

I have been spending some time experimenting with machine learning with Khmer languages and started to play with one of the latest advances in artificial intelligence called BERT. In this article, I write about the history and an introduction related to natural language processing (NLP) to give you an idea of what the algorithm is but more importantly I detailed the process that it involves in its learning of the language.​ You can read more detailed technical approaches about the different algorithms here.

This article is also available in Khmer.

Introduction

Computer scientists have been trying to create an artificial intelligence (AI) program that can understand human language. We had seen progress in which a machine challenged humans in a game of Jeopardy in 2011. Watson, a program that was made by IBM, won Jeopardy games against the two human champions. This signified how the machine can learn the nuance of the English language well enough to play the game. The creators of this system took many years to train and refine the algorithms to specifically play this game.

BERT

One of the recent progress in this area is a generalized approach that an algorithm can learn a language without explicitly program-specific rules of the language. Advancement in this approach is known as BERT created by Google in 2018 that can perform questions and answers tasks better than humans. BERT, in this case, is not a Sesame Street character, but a machine learning (ML) algorithm that stands for “Bidirectional Encoder Representation for Transformers” which uses an underlying algorithm called “transformer”. This approach can train on a very large amount of text from Wikipedia and books. We just feed all these texts into the algorithm. With the data, the machine is able to learn the lexical (vocabulary), syntactic (structure of the text), and semantic (content or meaning of the text) of the language without us telling it what it is.

We won’t go into detail about how the algorithm works, but mention the structure of the algorithm. This type of machine learning approach is called “deep learning” which refers to an architecture that mimics how we think our brain works at the neuron level. A set of neurons in our brain take in some information and trigger other neurons downstream in multiple layers. Known as the father of deep learning, Jeffrey Hinton, a professor from the University of Toronto, started to popularize this type of machine learning in 2012 in computer vision with AlexNet algorithm that was able to identify objects in an image more accurately than before on the ImageNet competition. This competition regularly took place to highlight the improvement in new algorithms.

A popular approach used in computer vision is to first train the model on a large set of images. These images are already labeled such as ImageNet which has 14 million images with 20,000 different classes (different objects in the images). Then use this partly algorithm to train on images that you want to analyze. This increases the accuracy of just training on the final images of data. This approach is called transfer learning.

BERT uses this same transfer learning approach by pre-train the model on a large amount of text. The text is easy to get from the Internet such as Wikipedia without having to do any work to label the data as we see in the computer vision approach where a human has to identify what the object is in the image.

With this large amount of text (800 million words from books and 2.5 billion words from English Wikipedia), BERT creates its own task by feeding each sentence into the algorithm by eliminating some words. These missing words are called “masking” words. Then it tries to predict the missing words. By masking some input words and learning how to predict them, the algorithm learned the structure of the sentence and some form of the word’s meaning.

The advancement in this approach is due to faster computers that are able to run on larger amounts of text. In addition, the underlying algorithm “transformer” allows the algorithm to parallelize the processing better than earlier algorithms. With these combinations including the deep architecture that uses many layers with over 300 million parameters, this algorithm became the state of the art during its time.

This BERT architecture can apply to many languages besides English as long as you have enough text to train. In fact, Google has pre-trained models of about 100 languages. These languages are the top 100 languages with the largest Wikipedias. Khmer is not one of them since the number of articles written in Khmer in Wikipedia was ranked 155th of all the languages.

SQuAD

BERT was shown to be able to perform a question and answer tasks based on a given text better than humans. The dataset that we used is called Stanford Question Answering Dataset (SQuAD) with a score of 87% accuracy versus humans with 82% accuracy for exact match answer.

The SQuAD is a dataset created by Stanford University for the AI community to test how well a machine can understand the text. Based on a given text, there are several questions that are asked. To be able to answer correctly, the human or program has to understand the structure of the text.

For example: (Text passage on left, questions and the correct answers are on the right).

In this example of the second question, the answer is not in the same sentence as the question. So the computer has to figure out the answer from the previous sentence.

BERT on the Khmer Language

Now, we will train this algorithm on the Khmer language. We are going to evaluate whether the computer understands the text using a task called document classification. Given a document, the computer has to identify which class or category of a document​ belongs to. A popular example is for the algorithm to predict if an email is a spam or not.

Earlier, we created a Khmer news portal website that takes headlines from many Khmer sites and showed them on one page. In the process, we realized that there are so many traffic accident-related articles that overwhelm other important headlines. As a result, we created an ML classification task to identify each of the Khmer news articles they are traffic accidents. Then group them into a section so it will overwhelm the front page. So we will use the same task to see if the Khmer BERT algorithm can classify those articles well or not.

We have 820 articles that we chose and label each of them as ‘1’ if it is about a traffic accident or ‘0’ if it is not. This will be our training dataset.

Example text and its label as ‘accident’ or not.

In this process, we have 2 steps. In step one, we feed a lot of Khmer text into the algorithm BERT so that it can learn the Khmer language structure. This process is called pre-train the language model. In step two, we feed the news articles that we chose with the labels that tell the algorithm to learn to distinguish between class 0 or class 1. This step is called finetune for a classification task.

Step 1: Pre-train the Language Model

In step 1, we feed a large corpus of Khmer text into the algorithm. The text must allow the algorithm to split the text into words. Since Khmer does not use spaces to separate words, we need an extra step to process the text that helps the computer to determine which series of letters to form a word. This is called “word segmentation”. We will use an algorithm called “CRF” that has shown to perform well for the Khmer language. See my previous article for detail.

At this point, all of our text has spaces between words. The algorithm then generates a list of all the different words​ that are seen in the articles. Each of the words will be assigned to a unique number that the algorithm will use.

For example:

Input Text: “ឃាត់​ ⁣ជនសង្ស័យ ⁣ម្នាក់ ⁣បន្ទាប់ពី”
Tokenized: [‘ឃាត់’, ‘ជនសង្ស័យ’, ‘ម្នាក់’, ‘បន្ទាប់ពី’]
Token IDs: [1585, 1438, 1062, 1482]

In the example above, we first split the text into 4 words by our segmentation process. Then it turns into series numbers with each word corresponding to the word in the dictionary we generated previously. These series of numbers become the inputs to the algorithm.

In this pre-train phase, it first chooses 15% of the word and indicates it as a “mask”. Then it learns to predict the mask word. As it compares the prediction to the actual word, it goes back and adjusts its weights or parameters (values in its connections) so that the prediction would be correct. It tries the next input and readjusts its weights again. It does this again and again until all the text was processed.

This is the part that takes a long time to run when you have a lot of data. The pre-train BERT model by Google for English runs over 3 billion words which took about 54 hours on Google hardware (TPUs) to complete. In our case, we only feed a small set of data in comparison. It took us only a few minutes for 334 thousand words of 1000 Khmer news articles.

This training process allows the system to learn the contextual structure of the sentence and how the word related to each other.

Step 2: Finetune BERT for Classification Task

In step 2, we will use our label data of 820 news articles we label as traffic accidents or not. Label 1 implies it is traffic accident-related and label 0 implies that it is not. We feed them the article text and its label. We update the model with extra architecture to be able to do the classification tasks.

We do not program anything specific about traffic accidents. The machine does not know anything about what label 1 or 0. It has to infer what are the differences between articles labeled as 0 and those with the label 1.

I can introduce a possible approach that the computer might try to solve this task. It may learn that articles of category 1 have the words “car” or “motorcycle”. But we know that some articles about “car show” or “motorcycles on sale” do not have the labeled 1. So it needs to learn to distinguish that “car show” or “motorcycle sell” related articles that contain the words “car” or “motorcycle” are not labeled as 1. It might find that articles of label 1 contain death or injury, but we know that articles about robbery and murder are labeled as 0. So it has to distinguish those as label 0. The algorithm will try to learn the features in the sentence, not just words as identified above, in order to make the distinction between the categories.

In fact, the approach mentioned of just looking at words is an older approach called “bag of words” where the program uses the different word counts in each article to make the classification. This approach does not perform well since it does not take into account the word orders or sentence structure that can be important in document classification.

To continue with our training​ process, the first step is to separate our articles into a training set and a validation set. The goal of the validation set is to set aside some articles so that the algorithm does use them during training. This can be used at the end to verify how well the algorithm performs. We saved 10% of the articles as a validation set. We use 90% of the total data of 820 documents. So the total number of articles is 738 for training and 82 for validation sets.

Training set: 738
Validation set: 82
Total Documents: 820

In the first step, it takes a set of input data and tries to adjust its weights to make the predictions better. After it goes through all 738 training data, the amount of error (loss) result is about 0.7 or 70% as shown in the blue line below. The process of going through the whole dataset is known as one epoch. With the current state, we test and see how it predicts the validation set. We see that the accuracy is about 50% as indicated as the red line in the chart below. This is not good. This accuracy is the same as if we were to randomly guess the answer.

Now we let it go through the same data again and readjust its weights and got about the same loss in the next 5 epochs. It performs about the same on a validation set of around 50% accuracy.

Then the sixth epoch shows a little better prediction. The validation set shows about 70% accuracy. This is a lot better than before but not great yet. Then in the seventh epoch, the accuracy went up to over 90%. Then in 9 epochs, it goes up to close 99%. As it goes through we see in the 17th epoch, it reaches 100% then 99% on the rest of the epoch. This accuracy beat our previous models.

Conclusion

We have seen that in 2018, the BERT algorithm has surpassed human capability in question and answer tasks in SQuAD 1.1 dataset. We see that we can pre-train BERT with Khmer text so it can learn the structure of the language. Then we finetune the algorithm with news articles data that we label as a traffic accident-related. Our result shows that the model can predict whether it is a traffic-related accident article with very high accuracy.

We may not know the exact approach that the algorithm used to distinguish whether an article is a traffic accident or not. But from our results, it was able to distinguish with high accuracy of 99%. It only misclassified about 1 for every 100 articles.

Does this mean that the algorithm understands the Khmer text? That depends on how you look at it. All we feed into the algorithm are just numbers that reference the Khmer word in our dictionary list. You can say it does not know anything about any of the Khmer words. But as we feed a lot of these numbers that signify Khmer words and its order. The algorithm knows some structure of those numbers that corresponded to our Khmer words. It knows enough about the structure to be able to distinguish if an article is a traffic accident-related or not.

You can see the full code which includes how we train Khmer language model from scratch:

https://github.com/phylypo/khmer-text-data/tree/master/bert-pretrain-from-scratch

You can find other datasets that you can use here: https://github.com/phylypo/khmer-text-data.

Other Notes

It would be a better illustration to have the algorithm generate Khmer text to illustrate its understanding of the language. Unfortunately, BERT is not meant for that. Other another algorithm ULMFiT that I trained uses a similar dataset and is able to generate text. Give the algorithm some words, it will predict the next few words that will make sense and even grammatically correct. See this site that I made from another algorithm here: http://ml.tovnah.com/khmer-ulmfit/.

ខ្ញុំ​បាន​ចំណាយ​ពេល​ខ្លះ​ដើម្បី​ពិសោធន៍​ ⁣”របៀប​ដែល⁣​ម៉ាស៊ីន​រៀន” ⁣⁣ជាមួយ​និង​ភាសាខ្មែរ​ ⁣ហើយ​ចាប់​ផ្តើម​សាកល្បង​ជាមួយ​ការ​ជឿនលឿន​ចុង​ក្រោយ​នៃ​បញ្ញា​សិប្បនិម្មិត ដែល​គេ​ហៅ​ថា បឺត ⁣(BERT) ។ នៅ​ក្នុង​អត្ថបទ​នេះ ខ្ញុំ​សរសេរ​អំពី​ប្រវត្តិ​ខ្លះៗ និង​ការណែនាំ​ ទាក់ទង​ទៅ​នឹង ដំណើរ​ការ​ភាសា​ធម្មជាតិ (NLP) ដើម្បី​ផ្តល់​ឱ្យ​អ្នក​យល់​ពី​ក្បួន​ដោះស្រាយ ប៉ុន្តែ​សំខាន់​ជាង​នេះ​ទៅទៀត ខ្ញុំ​បាន​រៀបរាប់​លម្អិត​អំពី​ដំណើរការ ដែល​ពាក់ព័ន្ធ​នឹង​របៀប​ម៉ាស៊ីន​រៀន​ភាសា​យើង។​ ⁣អ្នក​អាច​អាន​បន្ថែម វិធីសាស្ត្រ​លម្អិត​ផ្សេង ⁣នៅ​ទីនេះជា​ភាសា​អង្គ្លេស។

សេចក្តីផ្តើម

អ្នក​វិទ្យាសាស្ត្រ​ខាង​កុំព្យូទ័រ​កំពុង​ព្យាយាម​បង្កើត​កម្មវិធី​សិប្បនិម្មិត (AI) ដែល​អាច​យល់​ពី​ ⁣ភាសា​របស់​មនុស្ស។ ⁣យើង​បាន​ឃើញ​វឌ្ឍនភាព​ដែល​ម៉ាស៊ីន ⁣បាន​ប្រកួត​ប្រជែង​ជាមួយ​មនុស្ស ⁣នៅ​ក្នុង​ល្បែង Jeopardy ក្នុងឆ្នាំ២០១១⁣។​ ⁣​ វ៉ាតសុន​ ⁣(Watson) ⁣​ដែល​ជា​កម្មវិធី ⁣ដែល​បង្កើត​ឡើង​ដោយ ⁣ក្រុមហ៊ុន IBM បាន​ឈ្នះ​ល្បែង Jeopardy ប្រឆាំង​នឹង​អ្នក​ជើងឯក​ពីរ​រូប។ នេះ​ជា​សញ្ញា​បង្ហាញ​ពី​របៀប​ ⁣ដែល​ម៉ាស៊ីន​អាច​រៀន​ពី​ភាសា​អង់គ្លេស⁣​បាន​ល្អ​គ្រប់គ្រាន់​ ⁣ដើម្បី​នឹង​លេងល្បែង​នេះ។ អ្នកបង្កើត ⁣ប្រព័ន្ធ​នេះ ⁣បាន​ចំណាយ​ពេល​ជា​ច្រើន​ឆ្នាំ ⁣ដើម្បី​បណ្តុះបណ្តាល​និង​កែ​លម្អ​ក្បួន​ដោះស្រាយ ⁣ដើម្បី​លេង​ល្បែង​នេះ។

បឺត​ ⁣(BERT)

ការ​រីក​ចម្រើន​ថ្មី​មួយ​នៅ​ក្នុង​ផ្នែក​នេះ ⁣គឺ​ជា​វិធីសាស្ត្រ​មួយ​ ⁣ដែល​ម៉ាស៊ីន​អាច​រៀន​ភាសា​បាន ⁣ដោយ​មិន​ចាំបាច់​ ⁣សរសេរ⁣​ច្បាប់​ជាក់លាក់ ⁣របស់​ភាសា​នោះ។ ការ​ជឿនលឿន​នៃ​វិធីសាស្ត្រ​នេះ ⁣ត្រូវ​បាន​គេ​ស្គាល់​ថា BERT បង្កើត​ដោយ​ហ្គូហ្គល ⁣(Google) ⁣ក្នុងឆ្នាំ២០១៨ ដែល​អាច​ ⁣ឆ្លើយ​សំណួរ​បាន ⁣ប្រសើរ​ជាង​មនុស្ស។ ក្នុង​ករណី​នេះ ⁣BERT មិនមែន​ជា​តួអង្គ Sesame Street ទេ ⁣តែ​ជា​ឈ្មោះ​នៃ​កម្មវិធី​របស់​ម៉ាស៊ីន​ដែល​រៀន (ML) ⁣ដែល​មាន​នៃ​ថា ⁣“Bidirectional Encoder for Transformers” ។ ⁣គេ​អាច​បណ្តុះបណ្តាល​វិធីសាស្ត្រ​នេះ​ដោយ​ប្រើ ​អត្ថបទ​ជា​ច្រើន​ពី​វីគីភីឌៀ​ ⁣(Wikipedia) ⁣និង​សៀវភៅ។ យើង​គ្រាន់តែ​បញ្ចូល​អត្ថបទ​ទាំងអស់​នេះ ⁣ទៅ​ក្នុង​ក្បួន​កម្មវិធី​⁣ប៉ុណ្ណោះ ⁣ហើយ​វា​និង​រៀន​ដោយ​ខ្លួន​វា។ ជា​មួយ​អត្ថបទ​ជា​ច្រើន​នោះ ⁣ម៉ាស៊ីន​អាច​រៀន​ពី​វាក្យស័ព្ទ​ ⁣ រចនាសម្ព័ន្ធ​នៃ​អត្ថបទ និង​ខ្លឹមសារ ⁣ឬ​អត្ថន័យ​នៃ​អត្ថបទ ⁣ដោយ​មិន​ប្រាប់​ពី​ក្បួន​ជាក់លាក់​អ្វី​ទេ។

យើង​នឹង​មិន​និយាយ​លម្អិត​អំពី​របៀប​ដែល​កម្មវិធី​ដំណើរ​ការ​នោះ​ទេ ⁣ប៉ុន្តែ​យើង​និយាយ​អំពី​រចនាសម្ព័ន្ធ​នៃ​វិធីសាស្ត្រ​នេះ។ ⁣ម៉ាស៊ីន​ដែល​រៀន​របៀប​នេះ ⁣ត្រូវ​បាន​គេ​ហៅ​ថា ⁣“ការ​រៀន​ស៊ី​ជម្រៅ”​ ⁣(deep learning) ដែល​សំដៅ​ទៅលើ ⁣ស្ថាបត្យកម្ម​ដែល​ធ្វើ​តាម​របៀប​ដែល​ខួរក្បាល​មនុស្ស ⁣ តាម​កម្រិត​ណឺរ៉ូន ⁣(neuron) ⁣ដំណើរការ។ ពេល​ដែល​សំណុំ​នៃ​ណឺរ៉ូន ⁣⁣នៅ​ក្នុង​ខួរក្បាល​របស់​យើង​ទទួល​ព័ត៌មាន ⁣ពី​ភ្នែក​ឬ​ត្រចៀក វា​ផ្ញើ​បន្ត ⁣​ទៅ​ណឺរ៉ូន​ផ្សេង​ទៀត ជា​ច្រើន​សន្ទាប់​ជា​បន្តបន្ទាប់។ លោក Jeffrey Hinton ជា​សាស្ត្រាចារ្យ មកពី ⁣សាកលវិទ្យាល័យ​តូរ៉ុនតូ ⁣(Toronto) ⁣បាន​ចាប់ផ្តើម ⁣ធ្វើ​ឲ្យ​មាន​⁣ប្រជាប្រិយភាព ⁣លើ​ម៉ាស៊ីន​ដែល​រៀន ⁣ប្រភេទ​នេះ​ ⁣ក្នុង​ឆ្នាំ ២០១២ នៅ​ក្នុង​ចក្ខុវិស័យ​នៃ​កុំព្យូទ័រ​ ⁣(computer vision) ⁣ជាមួយ​កម្មវិធី ⁣អាឡិចណិត​ ⁣(AlexNet) ⁣ដែល​ឈ្នះ​ការប្រកួតប្រជែង របៀប​ដែល​ម៉ាស៊ីន​អាច​មើល​រូបភាព ⁣បាន​ល្អ​ប្រសើរ​ជាង​មុន ។ ការប្រកួតប្រជែង ⁣ImageNet Competition នេះ ⁣បាន​ធ្វើឡើង​ជា​ទៀងទាត់ ⁣ដើម្បី​បង្ហាញ​ពី​ភាព​ប្រសើរ​ឡើង​នៃ​កម្មវិធី​ថ្មីៗ។

វិធីសាស្ត្រ​ប្រជាប្រិយ​មួយ​⁣ដែល​ត្រូវ​បាន​ប្រើ​នៅក្នុង​ចក្ខុវិស័យ​កុំព្យូទ័រ ⁣គឺ​ដំបូង​ត្រូវ​បណ្តុះបណ្តាល​កម្មវិធី​ដោយ​ប្រើ​រូបភាព​យ៉ាង​ច្រើន។ ⁣ការ​ធ្វើ​បែប​នេះ ⁣អនុញ្ញាត​ឲ្យ​​ ⁣កម្មវិធី​រៀន​ពី​ទ្រង់ទ្រាយ​នៃ​រូបភាព។ ⁣ ⁣រូបភាព​ទាំងនេះ​ត្រូវ​បាន​ដាក់​ស្លាក​រួច​ហើយ ⁣នូវ​អ្វី​ដែល​ជា​វត្ថុ​ ⁣ក្នុង​រូបភាព​នោះ ⁣ដូចជា ImageNet ដែល​មាន​រូបភាព​ចំនួន ១៤ លាន​រូប ⁣និង ២០,០០០ ថ្នាក់ (វត្ថុ​ផ្សេងៗ​ក្នុង​រូបភាព) ។ បន្ទាប់មក គេ​យក​កម្មវិធី​នេះ​ ⁣ទៅ​ហ្វឹកហាត់​លើ​រូបភាព​ដែល​គេ​ចង់​វិភាគ។ របៀប​នេះ​បង្កើន​លទ្ធផល​ត្រឹមត្រូវ​ច្រើន​ជាង​ការ​បណ្តុះបណ្តាល​ត្រឹម​តែ​លើ​រូបភាព​ដែល​អ្នក​ចង់​វិភាគ។ វិធីសាស្ត្រ​នេះ​ត្រូវ​បាន​គេ​ហៅ​ថា​ការ​រៀន​ផ្ទេរ ⁣(transfer learning)។

BERT ប្រើ​វិធីសាស្ត្រ​រៀន​ផ្ទេរ​ដូចគ្នា ⁣ដោយ​បណ្តុះបណ្តាល​លើ​ចំនួន​អត្ថបទ​ដ៏​ច្រើន។ អត្ថបទ​ងាយ​រក​បាន​ពី​អ៊ីនធឺណិត (Internet) ⁣ដូច​ជា​ទំព័រ​វីគីភីឌៀ ⁣(Wikipedia) ⁣ដោយ​មិន​ចាំបាច់​ធ្វើការ​ដាក់​ស្លាក ⁣ដូច​ជា​វិធីសាស្ត្រ​ចក្ខុវិស័យ​កុំព្យូទ័រ ⁣ដែល​មនុស្ស​ ⁣ត្រូវ​កំណត់​ថា​នៅ​ក្នុង​រូបភាព​នោះ​មាន​វត្ថុ​អ្វី។

ជា​មួយ​អត្ថបទ​ដ៏​ច្រើន​នេះ (៨០០ លាន​ពាក្យ​ពី​សៀវភៅ​និង ២,៥ ពាន់​លាន​ពាក្យ​ពី​វីគីភីឌៀ​អង់គ្លេស) បឺត​មាន​ភារកិច្ច​ផ្ទាល់​ខ្លួន​ដោយ​បញ្ចូល​ប្រយោគ​នីមួយៗ​ទៅ​ក្នុង​ក្បួន​ដោះស្រាយ ⁣ដោយ​លុប​បំបាត់​ពាក្យ​ខ្លះ។ ពាក្យ​ដែល​បាត់​ទាំងនោះ​ត្រូវ​បាន​គេ​ហៅ​ថា ⁣ពាក្យ“លាក់បាំង” (masking)។ បន្ទាប់​មក ⁣វា​ព្យាយាម​ទាយ​ពាក្យ​ដែល​បាត់​នោះ។ ⁣ដោយ​ការ​លាក់បាំង​ពាក្យ​មួយ​ចំនួន ⁣វា​ប្រឹង​រៀន​ពី​របៀប⁣​ទស្សន៍ទាយ​ពាក្យ​ទាំង​នោះ ។​ ⁣⁣ក្បួន​នេះ​បាន​បង្រៀន​វា​ ពី​រចនាសម្ព័ន្ធ​នៃ​ភាសា ⁣ទម្រង់​របស់​អត្ថបទ ⁣និង​អត្ថន័យ​របស់​ពាក្យ។

ការ​ជឿនលឿន​ក្នុង​វិធីសាស្ត្រ​នេះ ⁣គឺ​ដោយសារ​តែ​កុំព្យូទ័រ​មាន​ល្បឿន​លឿន​ជាង​មុន ⁣ដែល​អាច​ដំណើរ​ការ​លើ​អត្ថបទ​បាន​កាន់តែ​ច្រើន។ លើស​ពី​នេះ​ទៀត ⁣ក្បួន​មូលដ្ឋាន ⁣“transformer” អនុញ្ញាត​ឱ្យ​កម្មវិធី​ជា​ច្រើន​ដំណើរការ​ស្រប​ជា​មួយ​គ្នា​យ៉ាង​លឿន​ប្រសើរ​ជាង​ក្បួន​មុនៗ។ ជា​មួយ​គ្នា​នេះ ⁣ស្ថាបត្យកម្ម​ប្រើ​ស្រទាប់​ជា​ច្រើន​ដែល​មាន​ប៉ារ៉ាម៉ែត្រ​ជាង ៣០០ លាន ⁣ធ្វើ​ ⁣ឲ្យ​ក្បួន​នេះ​បាន​ក្លាយ​ជា​ក្បួន​ល្អ​បំផុត​ក្នុង​កំឡុង​ពេល​នោះ​ក្នុង​ឆ្នាំ​២០១៨។

ស្ថាបត្យកម្ម BERT នេះ​អាច​អនុវត្ត​បាន​ចំពោះ​ភាសា​ណា​ក៏​បាន​ដែរ ⁣ដរាប​ណា​ភាសា​នោះ ⁣មាន​អត្ថបទ​គ្រប់គ្រាន់​សម្រាប់​ហ្វឹកហាត់។ តាម​ពិត​ហ្គូហ្គល​បាន​ហ្វឹកហាត់​ភាសា​គំរូ​មុន​ប្រមាណ ១០០ ភាសា។ ភាសា​ទាំងនេះ​គឺ​ជា​ភាសា១០០ ដែល​មាន​ចំនួន​អត្ថបទ​នៅ​វីគីភីឌៀ​ច្រើន​ជាង​គេ។ ភាសា​ខ្មែរ​មិន​ត្រូវ​បាន⁣​ហ្គូហ្គល​​ហ្វឹកហាត់​ទេ ⁣ព្រោះ​ចំនួន​អត្ថបទ ⁣ដែល​សរសេរ​ជា​ភាសា​ខ្មែរ ⁣នៅ​ក្នុង​វីគីភីឌៀ ⁣បាន​ជាប់​ចំណាត់​ថ្នាក់​ទី ១៥៥ នៃ​ភាសា​ទាំងអស់។ ⁣នេះ​ជាមូល​ហេតុ​ដ៏​សំខាន់​មួយ​​ ⁣ដែល​យើង​ត្រូវ​បញ្ចូល​អត្ថបទ​ ⁣ជា​ភាសា​ខ្មែរ​ឲ្យ​បាន​ច្រើន​នៅ​ទំព័រ​វីគីភីឌៀ។

ស្គ្វ៉ដ (SQuAD)

គេ​បាន​បង្ហាញ​ថាBERT អាច​ឆ្លើយ​សំណួរ​តាម​អត្ថបទ​បាន​ប្រសើរ​ជាង​មនុស្ស។ សំណុំ​ទិន្នន័យ​ដែល​បាន​ប្រើ​មាន​ឈ្មោះថា ស្គ្វ៉ដ (SQuAD) ⁣មាន​ឈ្មោះ​សម្រាយ​គឺ ⁣⁣Stanford Question Answering Dataset ដែល ⁣BERT ⁣បាន​ពិន្ទុ ៨៧% ទល់​នឹង​មនុស្ស​ដែល​បាន​ពិន្ទុ ៨២% សំរាប់​ចម្លើយ​ត្រូវ​គ្រប់​ពាក្យ។

SQuAD ⁣គឺជាសំណុំទិន្នន័យដែលបង្កើតឡើងដោយ ⁣សាកលវិទ្យាល័យស្ទែនហ្វត​ ⁣(Stanford University) ⁣សម្រាប់សហគមន៍បញ្ញាសិប្បនិម្មិត ដើម្បីសាកល្បងថាតើ ម៉ាស៊ីនអាចយល់អត្ថបទ បានល្អប៉ុណ្ណា។ ផ្អែកលើអត្ថបទដែលបានផ្តល់ឱ្យ វាមានសំណួរជាច្រើនដែលត្រូវសួរ។ ដើម្បីអាចឆ្លើយបាន កុំព្យូទ័រត្រូវតែយល់ពីរចនាសម្ព័ន្ធរបស់អត្ថបទខ្លះ។

ឧទាហរណ៍ៈ (អត្ថបទនៅខាងឆ្វេង ⁣សំណួរនិងចម្លើយត្រឹមត្រូវគឺនៅខាងស្តាំ) ។

ដើម្បីបកប្រែអត្ថបទនេះ៖

កាលីហ្វ័រញ៉ាភាគខាងត្បូង ជាញឹកញាប់ហៅកាត់ថា SoCal គឺជាតំបន់ភូមិសាស្ត្រនិងវប្បធម៌ ដែលជាទូទៅមានស្រុកចំនួន ១០ នៅខាងត្បូងបំផុតរបស់រដ្ឋ California ។ តំបន់នេះ ត្រូវបានពិពណ៌នាថាជា “ស្រុក ៨” ដោយផ្អែកលើប្រជាសាស្ត្រ និងទំនាក់ទំនង សេដ្ឋកិច្ច៖ Imperial, Los Angeles, Orange, Riverside, San Bernardino, San Diego, Santa Barbara, និង Ventura ។ និយមន័យនៃតំបន់ចំនួន ១០ ដែលទូលំទូលាយជាងនេះរួមមានរដ្ឋ Kern និង San Luis Obispo ក៏ត្រូវបានប្រើដោយផ្អែកលើការបែងចែកនយោបាយ ជាប្រវត្តិសាស្ត្រ ផងដែរ។ កាលីហ្វ័រញ៉ាខាងត្បូង គឺជាមជ្ឈមណ្ឌលសេដ្ឋកិច្ចដ៏សំខាន់មួយ សម្រាប់រដ្ឋកាលីហ្វ័រញ៉ា និងសហរដ្ឋអាមេរិក។

សំនួរ​:ទោះបីជាត្រូវបានគេពិពណ៌នាជាប្រពៃណីថាជា“ ស្រុក ៨” តើតំបន់នេះមានស្រុកប៉ុន្មានដែរ?

ចំលើយ: ១០

ក្នុងឧទាហរណ៍នេះ ⁣ដូចដែលអ្នកបានឃើញហើយ ⁣ចម្លើយមិនមាន ⁣នៅក្នុងប្រយោគ ⁣ជាមួយនឹងសំណួរនោះទេ។ ដូច្នេះកុំព្យូទ័រត្រូវរកចម្លើយពីប្រយោគមុន។

ប្រើភាសាខ្មែរក្នុងBERT ⁣

ឥឡូវយើងនឹងបណ្តុះបណ្តាលកម្មវិធីនេះដោយ​បញ្ចូលភាសាខ្មែរ។ យើងនឹងវាយតំលៃថាតើកម្មវិធីនេះយល់ពីអត្ថបទខ្មែរឬទេ ⁣ដោយឲ្យវាបែងចែកឯកសារ ⁣(document classification)។ កុំព្យូទ័រ នឹងពិនិត្យមើលថា ⁣តើ⁣ឯកសារនិមួយៗ ត្រូវជាប្រភេទឯកសារ​ណាមួយ។ ⁣ឧទាហរណ៏ដែលគេនិយមប្រើគឺឲ្យកម្មវិធីសម្រាច់ថា ⁣តើអ៊ីម៉ែលនិមួយៗ ⁣គឺជាអ៊ីម៉ែលឥតបានការ ⁣(spam) ⁣ឬអត់។

មុននេះយើងបានបង្កើតគេហទំព័រព័ត៌មានខ្មែរ ដែលយកចំណងជើងពីគេហទំព័រខ្មែរជាច្រើន ហើយបង្ហាញរួមលើទំព័រមួយ។ នៅក្នុងដំណើរការនេះ ⁣យើងបានឃើញថា​ ⁣មានអត្ថបទ ទាក់ទងនឹងគ្រោះថ្នាក់ចរាចរណ៍ច្រើន ⁣ដែលនាំឲ្យពិបាកមើល​ ⁣ចំណងជើងសំខាន់ៗផ្សេងទៀត។ ជាលទ្ធផល ⁣យើងបានបង្កើតកម្មវិធី AI មួយ ⁣ដើម្បីរកអត្ថបទដែលមានគ្រោះថ្នាក់ចរាចរណ៍ ⁣ហើយ​ទុកដើយឡែក ⁣ពី​អត្ថបទដ៏ទៃ ⁣ដើម្បីកុំឲ្យនៅជាន់ ⁣លើអត្ថបទនៅទំព័រដើម។ ដូច្នេះយើងនឹងប្រើកិច្ចការដូចគ្នានេះ ⁣ដើម្បីមើលថាតើកម្មវិធី ⁣BERT នេះអាចបែងចែកអត្ថបទទាំងនោះបានល្អឬទេ។

យើងមានអត្ថបទ ៨២០ មាត្រាដែលយើងបានជ្រើសរើសនិងដាក់ស្លាកនីមួយៗថា​ ⁣“១” ប្រសិនបើវានិយាយអំពីគ្រោះថ្នាក់ចរាចរណ៍ ⁣ឬ “០” ប្រសិនបើវា ⁣មិនមែនជាអត្ថបទ ⁣គ្រោះថ្នាក់ចរាចរណ៍ទេ។ នេះជាសំណុំអត្ថបទ ⁣ដែល​យើង​និង​ប្រើ​​ឲ្យ⁣កម្មវិធីបឹតបែងចែក​អត្ថបទ។​

នៅក្នុងដំណើរការនេះ យើងមាន ២ ជំហាន។ នៅជំហានទី ១ យើងបញ្ចូលអត្ថបទជាភាសាខ្មែរជាច្រើនទៅក្នុងក្បួនដោះស្រាយ BERT ដើម្បី​ឲ្យ​វា⁣អាចរៀន ⁣រចនាសម្ព័ន្ធភាសាខ្មែរ។ ដំណើរការនេះត្រូវបានគេហៅថាការហ្វឹកហាត់ជាមុន ⁣(pre-train) ⁣នៃគំរូភាសា។ ក្នុងជំហ៊ានទី ២ យើងបញ្ចូលអត្ថបទ ⁣ដែលយើងបានជ្រើសរើសជាមួយស្លាក ⁣ដែលប្រាប់ពីក្បួនដោះស្រាយ ⁣ដើម្បីរៀនបែងចែករវាងថ្នាក់ ០ ឬថ្នាក់ទី ១ ។ ជំហ៊ាននេះ ⁣ត្រូវបានគេហៅថា ការកែរតំរូវ ⁣(finetune) សំរាប់កិច្ចការបែងចែក​អត្ថបទ។

ជំហានទី ១: ⁣ការហ្វឹកហាត់ភាសាគំរូជាមុន

ក្នុងជំហានទី ១ យើងបញ្ចូលអត្ថបទជាភាសាខ្មែរទៅក្នុងក្បួនដោះស្រាយ។ អត្ថបទត្រូវតែអនុញ្ញាតឱ្យ ក្បួនដោះស្រាយ បំបែកអត្ថបទទៅជាពាក្យ។ ដោយសារខ្មែរមិនប្រើ អក្សរដកឃ្លារវាងពាក្យ ⁣យើងត្រូវធ្វើកិច្ច​ការមួយដែល​ត្រូវបានគេហៅថា ⁣“ការបែងចែកពាក្យ” (word segmentation)។ យើងនឹងប្រើក្បួនដែលហៅថា CRF ដែលមានដំណើរការល្អ​ ⁣សម្រាប់ភាសាខ្មែរ។ សូមមើលអត្ថបទមុនរបស់ខ្ញុំសម្រាប់ព័ត៌មានលម្អិត។

ដល់ចំណុចនេះ យើងបាន​ដាក់ចន្លោះរវាងពាក្យដល់​គ្រប់អត្ថបទរួចហើយ។ បន្ទាប់មក ⁣ក្បួនដោះស្រាយ⁣ត្រូវបង្កើតបញ្ជីពាក្យទាំងអស់ ⁣ដែល​មាន​នៅគ្រប់​អត្ថបទ⁣។​ ពាក្យនីមួយៗនឹងត្រូវមានតំណាងលេខមួយៗ ⁣ដែលក្បួនដោះស្រាយអាចយក​ទៅប្រើ។

Input Text: “ឃាត់​ ⁣ជនសង្ស័យ ⁣ម្នាក់ ⁣បន្ទាប់ពី”
Tokenized: [‘ឃាត់’, ‘ជនសង្ស័យ’, ‘ម្នាក់’, ‘បន្ទាប់ពី’]
Token IDs: [1585, 1438, 1062, 1482]

ក្នុងឧទាហរណ៍ខាងលើ ⁣ដំបូងយើងបែងចែកអត្ថបទជា ៤ ពាក្យដោយ ⁣កម្មវិធី​បែងចែកពាក្យ ⁣របស់យើង។ បន្ទាប់មកវាប្រែទៅជាលេខចំពោះពាក្យ​និមួយៗ ​ដែលត្រូវនឹងពាក្យនៅក្នុងវចនានុក្រម ⁣ដែលយើងបានបង្កើតពីមុននេះ។ លេខទាំងនេះនឹង​ត្រូវបញ្ចូលទៅក្នុងក្បួនដោះស្រាយ។

នៅក្នុងតំណាក់កាល pretrain នេះ ⁣ដំបូងវាជ្រើសរើស 15% នៃពាក្យហើយ​ដាក់ជាពាក្យ ⁣“របាំង” ។ បន្ទាប់មកវារៀនទាយពាក្យរបាំងនេះ។ នៅពេលវាប្រៀបធៀបការទាយទៅនឹងពាក្យរបាំង ⁣វាកែសម្រួលទម្ងន់ឬប៉ារ៉ាម៉ែត្ររបស់វា ដើម្បីឲ្យការទាយ​នោះបានប្រសើលើក​ក្រោយ។ វានឹងបញ្ចូលប្រយោគបន្ទាប់ហើយកែទំងន់របស់វាម្តងទៀត។ វាធ្វើម្តងហើយម្តងទៀត ⁣រហូតទាល់តែអត្ថបទទាំងអស់ត្រូវបានដំណើរការ។

នេះគឺជាផ្នែកមួយ ⁣ដែលចំណាយពេលយូរក្នុងដំណើរការ ⁣នៅពេលមានអត្ថបទច្រើន។ ហ្គូហ្គោល ⁣បង្រៀន BERT សម្រាប់ភាសាអង់គ្លេសដោយ​ប្រើអត្ថបទដែល​មានជាង ៣ ពាន់លានពាក្យ ⁣ហើយចំណាយពេលប្រហែល ៥៤ ម៉ោងដោយ​ប្រើ​កុំព្យុទ័រហ្គូហ្គល TPUs ។ ក្នុងករណីរបស់យើង ⁣យើងមានអត្ថបទតិចោះ​ ⁣ចូច្នេះយើងចំណាយពេលតែពីរបីនាទីប៉ុណ្ណោះ ⁣សម្រាប់អត្ថបទ⁣ព័ត៌មានខ្មែរ​ចំនួន ១០០០ អត្ថបទ ⁣ដែល​មានពាក្យចំនួន ៣៣៤ ពាន់ពាក្យ។

ដំណើរការបណ្តុះបណ្តាលនេះ ⁣អនុញ្ញាតឲ្យ⁣កម្មវិធីយើងរៀនពីរចនាសម្ព័ន្ធនៃប្រយោគ ⁣និងវិធីដែលពាក្យទាក់ទងគ្នា។

ជំហានទី ២៖ កែរតំរូវ ⁣BERT សម្រាប់ការបែងចែកប្រភេទឯកសារ

ក្នុងជំហ៊ានទី ២ យើងនឹងប្រើអត្ថបទរបស់យើងដែលមានចំនួន ៨២០ ដែលយើងដាក់ស្លាក ⁣ថាជាគ្រោះថ្នាក់ចរាចរណ៍ឬអត់។ ស្លាកលេខ ១ បញ្ជាក់ថាវាជាគ្រោះថ្នាក់ចរាចរណ៍ ⁣ហើយស្លាកលេខ ០ បញ្ជាក់ថាមិនមែនទេ។ យើងបញ្ចូលអត្ថបទនិងស្លាករបស់វាទៅក្នុង​ក្បួនដោះស្រាយ ⁣ដែលមានស្ថាបត្យកម្មបន្ថែមដើម្បីឲ្យវាអាចធ្វើការបែងចែកប្រភេទឯកសារ។

យើងមិនបានដាក់កម្មវិធីអ្វីជាក់លាក់អ្វីអំពីគ្រោះថ្នាក់ចរាចរណ៍ទៅ​ក្នុងក្បួនដោះស្រាយយើង​ទេ។ ម៉ាស៊ីនមិនដឹងថាអ្វីជាស្លាកលេខ ១ ឬ ០ទេ​ក្រៅពីអត្ថបទ​ដែល​យើង​ពញ្ចូល។ វាត្រូវតែសន្និដ្ឋាន ⁣អំពីអ្វីដែលជាភាពខុសគ្នា ⁣រវាងអត្ថបទដែលមានស្លាកលេខ ០ និងអត្ថបទដែលមានស្លាកលេខ ១ ។

ខ្ញុំអាចណែនាំវិធីសាស្រ្តដែលកុំព្យូទ័រអាចដោះស្រាយបញ្ហានេះ។ វាអាចរកឃើញថាអត្ថបទនៃជំពូកទី ១ មានពាក្យ ⁣“ឡាន” ឬ ⁣“ម៉ូតូ” ។ ប៉ុន្តែយើងដឹងថាអត្ថបទមួយចំនួនអំពី ⁣“ការបង្ហាញរថយន្ត” ឬ ⁣“ម៉ូតូសម្រាប់លក់” មិនមែនជាស្លាក ⁣១ទេ។ ដូច្នេះវាចាំបាច់ត្រូវរៀនសម្គាល់ថាអត្ថបទអំពី ⁣“ការបង្ហាញរថយន្ត” ឬ“ ការលក់ម៉ូតូ” ដែល​មាន​ពាក្យ​ ⁣ឡាន ⁣ឬ​ ⁣ម៉ូតូ ⁣មិនមែនជាជំពូកទី ១ ទេ។ វាអាចរកឃើញថាអត្ថបទរបស់ ស្លាកទី ១ មាន​ពាក្យ ⁣“⁣ការស្លាប់” ⁣ឬ “⁣របួស” ⁣ប៉ុន្តែយើងដឹងថា អត្ថបទអំពីអំពើចោរប្លន់ ⁣និងឃាតកម្ម ⁣ជាស្លាកថា ០ ដែល​មិន​ទាក់​ទង​នឹង​ចរាចរណ៍ទេ។ ដូច្នេះវាត្រូវបែងចែកចំណុចទាំងនោះ ⁣ជាស្លាកលេខ ០។ កម្មវិធីក្បួនដោះស្រាយយើង នឹងព្យាយាមរៀនពីលក្ខណៈពិសេស ⁣នៅក្នុងប្រយោគ ⁣មិនគ្រាន់តែមើលពាក្យ ⁣ដូចដែលបាននិយាយខាងលើទេ ⁣ ក្នុងគោលបំណង ⁣ដើម្បី​អាចរក ភាពខុសគ្នា រវាងអត្ថបទនិមួយៗ។

តាមពិតវិធីសាស្រ្តដែលបានលើកឡើង ដោយគ្រាន់តែមើលពាក្យ គឺជាវិធីសាស្រ្តចាស់មួយ ដែលមានឈ្មោះថា “ថង់ពាក្យ” (bag of words) ដែលកម្មវិធីប្រើ ចំនួនពាក្យនៅក្នុងអត្ថបទនីមួយៗ ដើម្បីធ្វើការបែងចែកប្រភេទឯកសារ។ វិធីសាស្រ្តនេះមិនមានដំណើរការល្អទេ ⁣ព្រោះវាមិនគិតពីលំដាប់លំដោយរបស់ពាក្យ ⁣ឬរចនាសម្ព័ន្ធរបស់​ប្រយោគ ⁣ដែលអាចមានសារៈសំខាន់ ⁣ក្នុងការបែងចែកប្រភេទឯកសារ។

ដើម្បីបន្តដំណើរការបណ្តុះបណ្តាលរបស់យើង ជំហានដំបូង គឺត្រូវបំបែកអត្ថបទរបស់យើង ទៅជាសំណុំបណ្តុះបណ្តាល (training set) និងសំណុំសុពលភាព (validation set)។ គោលដៅនៃសំណុំសុពលភាព គឺត្រូវទុកអត្ថបទខ្លះៗដើម្បីកុំឲ្យក្បួនដោះស្រាយដឹង ⁣នៅពេលបណ្តុះបណ្តាល។ សំណុំនេះនឹងត្រូវបានប្រើនៅចុងបញ្ចប់ ⁣ដើម្បីផ្ទៀងផ្ទាត់ ⁣ថាតើក្បួនដោះស្រាយអាចបែងចែកប្រភេទឯកសារបាន​ល្អប៉ុណ្ណា។ យើងបានរក្សាទុកអត្ថបទ ១០ ភាគរយជាសំណុំសុពលភាព។ យើងប្រើអត្ថបទ ៩០% សម្រាប់ការបណ្តុះបណ្តាល ⁣ឯកសារក្នុងចំនួនសរុប ៨២០ អត្ថបទ។ ដូច្នេះចំនួនអត្ថបទសំរាប់ ⁣ការបណ្តុះបណ្តាលមានចំនួន ៧៣៨ និងសំណុំសុពលភាពមានចំនួន ៨២ ។

សំណុំបណ្តុះបណ្តាល:738
សំណុំសុពលភាព: 82
ឯកសារសរុប: 820

នៅជំហានដំបូង កម្មវិធី​យកអត្ថបទ​មកបញ្ចូលហើយព្យាយាម​កែសម្រួល​ទម្ងន់របស់វាដើម្បីធ្វើឱ្យការទាយកាន់តែប្រសើរឡើង។ បន្ទាប់ពីវាបានឆ្លងកាត់គ្រប់ទិន្នន័យបណ្តុះបណ្តាលទាំងអស់ម​ក​ដែល​មានចំនួន ៧៣៨ ចំនួនកំហុស(ការបាត់បង់) លទ្ធផលគឺប្រហែល ០,៧ ឬ ៧០% ដូចដែលបានបង្ហាញក្នុងខ្សែពណ៌ខៀវខាងក្រោម។ ដំណើរការនៃការឆ្លងកាត់សំណុំទិន្នន័យទាំងមូលត្រូវបានគេស្គាល់ថាជាអ៊ីប៉ក់ ⁣(epoch) ⁣មួយ។ ក្នុងពេលនេះ ⁣⁣យើងសាកទាយសំណុំសុពលភាពម្ដង​ថា​តើវាមានភាពត្រឹមត្រូវប៉ុណ្ណា។ ⁣យើង​ឃើញថា​វា​បានត្រូវប្រហែល ⁣50% ⁣ដូចតាមបន្ទាត់ក្រហមនៅក្នុងតារាងខាងក្រោម។ លទ្ធិផលនេះ​គី​មិនល្អ​ទេ។ លទ្ធិផលនេះគឺដូចនឹងយើងត្រូវទាយចម្លើយដោយចៃដន្យ។

ឥឡូវយើងអនុញ្ញាតឱ្យវាឆ្លងកាត់ទិន្នន័យដូចគ្នាម្តងទៀតនិងកែទំងន់របស់វាហើយទទួលបានការបាត់បង់ដូចគ្នាក្នុងរវាង៥ អ៊ីប៉ក់ខាងមុខទៀត។ វាបានទាយ​ត្រូវ ⁣លើសំណុំសុពលភាពប្រហែល 50% ។

បន្ទាប់មក ⁣ដល់អ៊ីប៉ក់ទី ៦ យើងឃើញការទាយបានល្អជាងមុនបន្តិច។ សំណុំសុពលភាព ⁣បង្ហាញអំពីភាពត្រឹមត្រូវប្រហែល 70% ។ នេះល្អប្រសើរជាងមុន ⁣តែមិនទាន់អស្ចារ្យទេ។ បន្ទាប់មកនៅក្នុងអ៊ីប៉ក់ទី៧ ⁣ភាពត្រឹមត្រូវបានកើនឡើងដល់ជាង 90% ។ បន្ទាប់មក ⁣ក្នុងអ៊ីប៉ក់ទាំង ៩ វាឡើងដល់ ៩៩% ។ ⁣មក ⁣ដល់អ៊ីប៉ក់ទី ១៧ វាឈានដល់ ១០០% បន្ទាប់មក ៩៩% នៅលើអ៊ីប៉ក់ក្រោយៗ។ លទ្ធិផលនេះបានល្អ​ជាងកម្មវិធីចាស់ៗរបស់យើង។

សេចក្ដីបញ្ចប់

នៅក្នុងឆ្នាំ ២០១៨ ក្បួនដោះស្រាយរបស់ BERT បានកើន​លើសសមត្ថភាពរបស់មនុស្ស ⁣ក្នុងបញ្ហាសំណួរនិងចម្លើយ ⁣ក្នុងសំណុំទិន្នន័យ SQuAD១.១ ។ យើងឃើញថា ⁣យើងអាចបង្រៀន​ ⁣BERT ⁣ទុកជាមុន ដោយប្រើអត្ថបទជាភាសាខ្មែរ ⁣ដើម្បីឲ្យវារៀនពី ⁣រចនាសម្ព័ន្ធនៃភាសា។ បន្ទាប់មក ⁣យើងកែរកម្មវិធីដោះស្រាយ ⁣ជាមួយអត្ថបទព័ត៌មាន ⁣ដែលយើងដាក់ស្លាកជាគ្រោះថ្នាក់ចរាចរណ៍ឬអត់។ លទ្ធផលរបស់យើងបង្ហាញថា ⁣កម្មវិធីអាចទាយយ៉ាង⁣ត្រឹមត្រូវ ⁣ថាតើអត្ថបទនេះមានទាក់ទងនឹងគ្រោះថ្នាក់ចរាចរណ៍ឬអត់ ។

យើងប្រហែលជាមិនដឹងជាក់លាក់ ⁣ពី​របៀបដែលក្បួនដោះស្រាយ ក្នុងការបែងចែក​⁣​អត្ថបទនោះទេ។ ប៉ុន្តែតាមលទ្ធផល យើង​ដឹងថា⁣⁣វាអាចបែងចែកយ៉ាងត្រឹមត្រូវល្អ ⁣បានដល់ ៩៩% ។ វាទាយខុសតែ ១ ឯកសាប៉ុណ្ណោះ ក្នុងចំនួន ១០០ ឯកសា។

តើនេះមានន័យថាBERT ⁣បានយល់ពីភាសាខ្មែរឬ? ⁣នោះអាស្រ័យលើរបៀប ⁣ដែលអ្នកគិតមើល។ អ្វីដែលយើងបញ្ចូលទៅក្នុងក្បួនដោះស្រាយនេះ ⁣គឺគ្រាន់តែជាលេខយោង ⁣របស់ពាក្យដែលមានក្នុងបញ្ជីវចនានុក្រមរបស់យើងប៉ុណ្ណោះ។ អ្នកអាចនិយាយថា ⁣វាមិនដឹងអ្វីទាំងអស់ពីពាក្យខ្មែរ។ ប៉ុន្តែនៅពេលយើងចិញ្ចូលលេខទាំងនោះជាច្រើន ⁣វាអានរៀនពីទ្រង់ទ្រាយពាក្យខ្មែរនិងលំដាប់របស់វា។ BERT ដឹងពីរចនាសម្ព័ន្ធមួយចំនួន​⁣នៃលេខទាំងនោះ ⁣ដែលត្រូវនឹងពាក្យខ្មែរយើង។ វាដឹងគ្រប់គ្រាន់ ⁣អំពីរចនាសម្ព័នរបស់ពាក្យខ្មែរ ដែលអាចបែងចែកបាន⁣ថាអត្ថបទណាមួយដែលទាក់ទងនិងគ្រោះថ្នាក់ចរាចរណ៍។

អ្នកអាចមើលកូដនៃវិធីដែលយើងបណ្តុះបណ្តាលក្បួនដោះស្រាយនៅទីនេះជាមួយសំណុំទិន្នន័យផ្សេងទៀតដែលអ្នកអាចពិសោធន៍បាន: https://github.com/phylypo/khmer-text-data.

កំណត់ចំណាំផ្សេងទៀត

យើង​អាចបង្ហាញប្រសើរជាងនេះ ⁣បើក្បួនដោះស្រាយនេះអាចសរសេរអត្ថបទជាភាសាខ្មែរ ⁣ដើម្បីបង្ហាញការយល់ដឹងអំពីភាសា។ ប៉ុន្តែ BERT មិនធ្វើ​សម្រាប់សរសេរអត្ថបទទេ។ ខ្ញុំបានបណ្តុះបណ្តាល ⁣កម្មវិធីៈULMFiT ផ្សេងទៀតដែល ដោយប្រើសំណុំទិន្នន័យ ⁣ស្រដៀងគ្នា ⁣និងអាចសរសេរពាក្យខ្លះបាន។ ដោយផ្តល់⁣ពាក្យមួយចំនួន ⁣វានឹងសរសេរពាក្យពីរបីបន្ទាប់ ⁣ដែលមាន​អត្ថន័យសមស្រប ⁣និងអាច​ត្រឹមត្រូវ​តាមវេយ្យាករណ៍ខ្មែរ ⁣ ដើម្បីបង្ហាញថាវាយល់អំពីភាសា។ សូមមើល​ទំព័រដែលខ្ញុំបានធ្វើពីវិធីនេះ៖ http://ml.tovnah.com/khmer-ulmfit/ ។