What is Clustering in Node.js?

Node.js runs in a single-threaded environment, which means by default, only one CPU core is used. Modern machines often have multiple cores, and clustering allows us to utilize all of them.

The cluster module in Node.js enables us to spawn multiple worker processes, all running the same code. Each worker handles incoming HTTP requests independently, and the OS or master process distributes requests among them.

const cluster = require('cluster');
const os = require('os');
const express = require('express');

if (cluster.isMaster) {
  const numCPUs = os.cpus().length;
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }
} else {
  const app = express();
  app.get('/', (req, res) => res.send(`Handled by PID ${process.pid}`));
  app.listen(3000);
}

Each forked worker is an independent process, meaning they do not share memory or data.

The Concept of Workers

When you run your Node.js application in cluster mode:

• The master process forks multiple worker processes
• Each worker runs the same server code and listens to the same port
• The master balances incoming requests across these workers

Example: “Cluster mode with 4 workers” means 4 independent Node.js processes are handling requests.

Real-World Problem: Session Management

Consider you are building an Express application using express-session for authentication. By default, sessions are stored in memory.

Problem:

1. User logs in, and their session is stored in Worker 1’s memory.
2. Next request from the same user goes to Worker 3 (due to load balancing).
3. Worker 3 doesn’t have the session info — resulting in the user being logged out.

This happens because memory is not shared across workers.

Solution: Use a Shared Session Store (Redis)

To resolve this, store sessions in an external, shared location like Redis.

Setup with Redis:

npm install express-session connect-redis ioredis

Code Example:

const session = require('express-session');
const RedisStore = require('connect-redis')(session);
const Redis = require('ioredis');

const redisClient = new Redis();

app.use(session({
  store: new RedisStore({ client: redisClient }),
  secret: 'my-secret',
  resave: false,
  saveUninitialized: false,
  cookie: { secure: false } // Set true if using HTTPS
}));

Now, all workers access session data from Redis, ensuring consistency.

How the OS Handles Clustering

When a request comes:

• The master process accepts it
• It uses round-robin or OS-level scheduling to forward it to a worker
• The chosen worker handles the request completely

If the worker is doing async tasks (e.g., DB calls), it doesn’t block the process — thanks to Node’s event loop.

Use Cases That Require Shared Storage

Use Case Shared Store Used Session management Redis / DB Rate limiting (IP-based) Redis Real-time user analytics Redis / PubSub WebSocket sticky sessions Redis + Load Balancer Cart/session persistence Redis / DB File upload progress tracking Redis

FAQs

1. Is clustering multithreading?

No, it’s multi-process. Each worker is a separate single-threaded process.

2. Do workers share memory?

No. Use Redis or DB for shared data.

3. Can I use native variables to share data?

No. They’ll be isolated to the process they are created in.

4. Can workers communicate?

Yes, via inter-process communication (IPC), but it’s limited. Prefer external stores.

5. Does PM2 support clustering?

Yes! PM2 makes clustering production-ready using pm2 start app.js -i max

6. Is clustering helpful for async apps?

Yes — although Node.js handles async well, clustering allows usage of all CPU cores.

Advanced Considerations for Production Systems

Worker Crash Handling

Workers can crash due to unhandled exceptions. A production-grade cluster should monitor worker health:

cluster.on('exit', (worker, code, signal) => {
  console.log(`Worker ${worker.process.pid} died. Restarting...`);
  cluster.fork();
});

Sticky Sessions for WebSocket Support

WebSocket connections must be consistently handled by the same worker. Solutions:

• Use a sticky-session proxy (e.g., Nginx or HAProxy)
• Use socket.io with Redis adapter to sync events across workers

Horizontal Scaling (Multiple Machines)

Clustering scales vertically on a single machine. For horizontal scaling:

• Deploy to multiple machines
• Use a load balancer (e.g., Nginx, AWS ELB)
• Combine with Redis or DB for shared state

Performance Monitoring

Use tools like:

• PM2 for cluster management & monitoring
• New Relic or Datadog for application metrics
• Redis INFO command to monitor session store performance

Conclusion

Node.js clustering boosts performance by running your app across multiple processes. But, for shared data like sessions, you must use an external store like Redis. Understanding how to properly manage session data in a clustered environment is crucial for building scalable, consistent, and reliable applications.

As your application grows, consider advanced topics like crash recovery, WebSocket handling, and horizontal scaling. Combined, these practices help ensure your Node.js app is both resilient and scalable.

Happy scaling! 🚀

Are you looking to build a web application in Go? This blog post will walk you through the creation of a simple Book Hub application using Go. By the end of this guide, you’ll have a solid understanding of Go’s powerful features, including structs, pointers, and interfaces, and how to leverage them to build a functional web application.

Project Overview

Our Book Hub application will allow users to perform CRUD (Create, Read, Update, Delete) operations on a collection of books. We’ll be using the Gorilla Mux package for routing and GORM (Go Object-Relational Mapping) for interacting with a MySQL database.

Folder Structure

Here’s a quick overview of the project structure:

BookHub/
├── cmd/
│ └── main/
│ └── main.go
├── pkg/
│ ├── config/
│ │ └── app.go
│ ├── controllers/
│ │ └── controller.go
│ ├── models/
│ │ └── models.go
│ ├── routes/
│ │ └── routes.go
│ └── utils/
│ └── utils.go
└── .gitignore

File Descriptions

- `cmd/main/main.go`: Entry point for the application. Contains the `main` function.
- `pkg/controllers/controller.go`: Handles the business logic for our routes.
- `pkg/config/app.go`: Manages the application configuration, including database connections.
- `pkg/routes/routes.go`: Defines the application’s routes.
- `pkg/models/models.go`: Defines the data models and handles database operations.
- `pkg/utils/utils.go`: Contains utility functions, such as parsing request bodies.

Detailed Explanation of Each Component

main.go

The `main.go` file is the entry point of our application. It sets up the HTTP server and registers the routes.

package main
import (
 "log"
 "net/http"
 "github.com/gorilla/mux"
 _ "github.com/jinzhu/gorm/dialects/mysql"
 "github.com/Ashwin202/go-bookhub/pkg/routes"
)
func main() {
 r := mux.NewRouter()
 routes.RegisterBookStoreRoutes(r)
 http.Handle("/", r)
 log.Print("Listening on port 9000")
 log.Fatal(http.ListenAndServe("localhost:9000", r))
}

routes.go

In the `routes.go` file, we define the routes for our application.

package routes
import (
 "github.com/gorilla/mux"
 "github.com/Ashwin202/go-bookhub/pkg/controllers"
)
var RegisterBookStoreRoutes = func(router *mux.Router) {
 router.HandleFunc("/book/", controllers.CreateBook).Methods("POST")
 router.HandleFunc("/book/", controllers.GetAllBooks).Methods("GET")
 router.HandleFunc("/book/{bookId}", controllers.GetBookById).Methods("GET")
 router.HandleFunc("/book/{bookId}", controllers.UpdateBookById).Methods("PUT")
 router.HandleFunc("/book/{bookId}", controllers.DeleteBookById).Methods("DELETE")
}

controller.go

The `controller.go` file contains the logic for handling HTTP requests and interacting with the models.

package controllers
import (
 "encoding/json"
 "net/http"
 "github.com/gorilla/mux"
 "github.com/Ashwin202/go-bookhub/pkg/models"
 "github.com/Ashwin202/go-bookhub/pkg/utils"
 "strconv"
)
var CreateBook = func(w http.ResponseWriter, r *http.Request) {
 var book models.Book
 utils.ParseBody(r, &book)
 b := book.CreateBook()
 marshalledBook, _ := json.Marshal(b)
 w.WriteHeader(http.StatusCreated)
 w.Write(marshalledBook)
}
var GetAllBooks = func(w http.ResponseWriter, r *http.Request) {
 allBooks := models.GetAllBooks()
 marshalledBooks, _ := json.Marshal(allBooks)
 w.WriteHeader(http.StatusOK)
 w.Write(marshalledBooks)
}
var GetBookById = func(w http.ResponseWriter, r *http.Request) {
 vars := mux.Vars(r)
 bookID := vars["bookId"]
 ID, err := strconv.ParseInt(bookID, 0, 0)
 if err != nil {
 w.WriteHeader(http.StatusBadRequest)
 w.Write([]byte("please send correct book id"))
 return
 }
 bookDetails, _ := models.GetBookById(ID)
 marshalledBook, _ := json.Marshal(bookDetails)
 w.WriteHeader(http.StatusOK)
 w.Write(marshalledBook)
}
var UpdateBookById = func(w http.ResponseWriter, r *http.Request) {
 vars := mux.Vars(r)
 bookID := vars["bookId"]
 var newBookDetails models.Book
 utils.ParseBody(r, &newBookDetails)
 ID, err := strconv.ParseInt(bookID, 0, 0)
 if err != nil {
 w.WriteHeader(http.StatusBadRequest)
 w.Write([]byte("please send correct book id"))
 return
 }
 bookDetails, db := models.GetBookById(ID)
 bookDetails.Name = newBookDetails.Name
 bookDetails.Author = newBookDetails.Author
 bookDetails.Publication = newBookDetails.Publication
db.Save(&bookDetails)
 marshalledBook, _ := json.Marshal(bookDetails)
 w.WriteHeader(http.StatusOK)
 w.Write(marshalledBook)
}
var DeleteBookById = func(w http.ResponseWriter, r *http.Request) {
 vars := mux.Vars(r)
 bookID := vars["bookId"]
 ID, err := strconv.ParseInt(bookID, 0, 0)
 if err != nil {
 w.WriteHeader(http.StatusBadRequest)
 w.Write([]byte("please send correct book id"))
 return
 }
 bookDetails := models.DeleteBookById(ID)
 marshalledBook, _ := json.Marshal(bookDetails)
 w.WriteHeader(http.StatusOK)
 w.Write(marshalledBook)
}

models.go

In the `models.go` file, we define the data models and the functions to interact with the database.

package models
import (
 "github.com/jinzhu/gorm"
 "github.com/Ashwin202/go-bookhub/pkg/config"
)
var db *gorm.DB
type Book struct {
 gorm.Model
 Name string `gorm:"" json:"name"`
 Author string `json:"author"`
 Publication string `json:"publication"`
}
func init() {
 config.Connect()
 db = config.DB
 db.AutoMigrate(&Book{})
}
func (b *Book) CreateBook() *Book {
 db.NewRecord(b)
 db.Create(&b)
 return b
}
func GetAllBooks() []Book {
 var books []Book
 db.Find(&books)
 return books
}
func GetBookById(Id int64) (*Book, *gorm.DB) {
 var getbook Book
 db := db.Where("id = ?", Id).Find(&getbook)
 return &getbook, db
}
func DeleteBookById(Id int64) Book {
 var book Book
 db.Where("id = ?", Id).Delete(&book)
 return book
}

Understanding Pointers and Structs

Why Are Models Defined in This Way?
- Models are defined using structs to represent data in a structured format.
- The `gorm.Model` struct is embedded to provide fields like ID, CreatedAt, UpdatedAt, and DeletedAt.

Why Use Pointers for Parameters and Returns?
- Pointers are used to reference memory locations, allowing functions to modify the original data without copying it.
- Using pointers for parameters and returns helps manage memory efficiently, especially with large data structures.

What Does `func init()` Do?
- The `init()` function initializes the package. It’s called automatically when the package is imported.
- In our model, `init()` connects to the database and performs auto-migration.

Why Use Structs?
- Structs group together variables under a single name, defining a new data type.
- They provide a way to model real-world entities and their attributes.

utils.go

The `utils.go` file contains utility functions, such as parsing request bodies.

package utils
import (
 "encoding/json"
 "net/http"
 "io/ioutil"
)
func ParseBody(r *http.Request, x interface{}) {
 if body, err := ioutil.ReadAll(r.Body); err == nil {
 if err := json.Unmarshal([]byte(body), x); err != nil {
 return
 }
 }
}

app.go

The `app.go` file manages the application configuration, including the database connection.

package config
import (
 "os"
 "fmt"
 "github.com/jinzhu/gorm"
 _ "github.com/jinzhu/gorm/dialects/mysql"
 "github.com/joho/godotenv"
)
var DB *gorm.DB
func Connect() {
 godotenv.Load()
 databaseURI := fmt.Sprintf("%s:%s@tcp(%s)/%s?charset=utf8&parseTime=True&loc=Local",
 os.Getenv("DATABASE_USER"),
 os.Getenv("DATABASE_PASSWORD"),
 os.Getenv("DATABASE_URL"),
 os.Getenv("DATABASE_NAME"))
 d, err := gorm.Open("mysql", databaseURI)
 if err != nil {
 panic(err)
 }
 DB = d
}
func GetDB() *gorm.DB {
 return DB
}

Conclusion

By following this guide, you’ve learned how to build a simple Book Hub application in Go. We’ve covered:

• Setting up the project structure and understanding the purpose of each file.
• Defining and using structs to model data.
• Utilizing pointers for efficient memory management and function interactions.
• Implementing CRUD operations using GORM.
• Setting up routing with Gorilla Mux.
• Parsing request bodies and managing database connections with utility functions.

This project showcases the power and simplicity of Go in building web applications. With these concepts and techniques, you’re well-equipped to expand this project further or create your own web applications in Go. Happy coding!

Lets begin with a story,
Once upon a time in the big world of the internet, there were two kinds of digital places: websites and apps. Websites were like big playgrounds everyone could visit, but sometimes they didn’t work very well on different devices. Apps, on the other hand, worked really well on phones and tablets, but only on certain ones.

Then, something special came along called a Progressive Web App (PWA). It was like a magical mix of a website and an app! PWAs could do cool things like work even when there wasn’t any internet and look nice on all kinds of devices, just like magic.

People liked PWAs a lot because they were easy to use and worked for everyone, no matter what device they had. And so, PWAs became a favorite for people who wanted the best of both worlds in one neat package.

What is Progressive Web App?

A Progressive Web App (PWA) is a type of web application that utilizes modern web technologies to provide users with an app-like experience directly through their web browser. PWAs offer features typically associated with native mobile apps, such as offline functionality, push notifications, and the ability to be installed on a device’s home screen, while maintaining the accessibility and universality of traditional websites. They are designed to work across various platforms and devices, delivering a consistent and engaging user experience regardless of the device or network conditions.

The Superpowers of Progressive Web Apps (PWAs)

PWAs vs. Native Apps:

• PWAs work on all devices like magic, while native apps need different versions for each device.
• You can get PWAs straight from the internet, but native apps need to go through special stores.

PWAs vs. Traditional Websites:

• PWAs feel like cool apps but live on the internet, just like regular websites.
• They can work even when the internet is acting funny, unlike regular websites.

Why PWAs Are Awesome:

• PWAs give you a special app feeling, but you can share them easily like a website.
• They can do neat things even without the internet, like sending you messages.
• PWAs are super fast and don’t make you wait, just like turning on a light switch!

To Get Started

Understanding of Web Technologies:

• Familiarize yourself with HTML, CSS, and JavaScript, as these are the building blocks of web development.
• Learn about modern JavaScript frameworks and libraries such as React, Angular, or Vue.js, which can streamline PWA development.

Knowledge of PWA Principles:

• Understand the core principles of PWAs, including service workers, web app manifests, and responsive design.
• Explore the capabilities and features that differentiate PWAs from traditional web applications.

Development Environment:

• Set up a development environment on your computer.
• Choose a code editor like Visual Studio Code, Sublime Text, or Atom.
• Install Node.js and npm (Node Package Manager) for managing project dependencies.

Learn Service Workers:

• Service workers are a crucial component of PWAs responsible for caching resources and enabling offline functionality.
• Learn how to implement and configure service workers in your web application.

Example:

// Service Worker Code
self.addEventListener('install', event => {
  event.waitUntil(
    caches.open('my-cache-v1').then(cache => {
      return cache.addAll([
        '/index.html',
        '/styles.css',
        '/script.js',
        '/images/logo.png'
      ]);
    })
  );
});

// Fetch Event
self.addEventListener('fetch', event => {
  event.respondWith(
    caches.match(event.request).then(response => {
      return response || fetch(event.request);
    })
  );
});js

Manifest File:

• Understand the importance of the web app manifest file in defining the appearance and behavior of your PWA when installed on a device’s home screen.
• Learn how to create and configure the manifest file to provide a native app-like experience.

Example

{
  "name": "My Progressive Web App",
  "short_name": "MyPWA",
  "start_url": "/index.html",
  "display": "standalone",
  "background_color": "#ffffff",
  "theme_color": "#2196f3",
  "icons": [
    {
      "src": "/images/icon-192x192.png",
      "sizes": "192x192",
      "type": "image/png"
    },
    {
      "src": "/images/icon-512x512.png",
      "sizes": "512x512",
      "type": "image/png"
    }
  ]
}

HTTPS Setup:

• PWAs require a secure HTTPS connection to ensure the integrity and security of data transmission.
• Set up HTTPS for your development environment and deploy your PWA on a secure hosting platform.

Testing and Debugging:

• Learn how to test and debug PWAs across different devices and browsers.
• Utilize browser developer tools and testing frameworks to identify and fix issues during development.

Performance Optimization:

• Explore techniques for optimizing the performance of your PWA, such as lazy loading, code splitting, and image optimization.
• Ensure that your PWA loads quickly and provides a smooth user experience, even on low-end devices and slow networks.

Accessibility and SEO:

• Consider accessibility guidelines and best practices to ensure that your PWA is usable by everyone, including users with disabilities.
• Implement SEO (Search Engine Optimization) techniques to improve the discoverability and visibility of your PWA in search engine results.

In the ever-evolving realm of web development, Progressive Web Apps stand out as a revolutionary blend, seamlessly integrating the best of traditional websites and native applications. Embarking on a captivating journey through the realms of coding ingenuity and creative innovation, we delve into the enchanting synergy of service workers and web app manifests. Together, let’s unlock the boundless potential of PWAs, charting a course towards redefined user experiences and a transformed digital landscape.

In part 3 of the series, I explained the basics of Graph Neural Networks.

If you haven’t gone through the same please check out Graph Neural Networks- Part 3

In this part, I’ll explain what is Graph Convolutional Networks.

Before getting into it have you ever thought that how you get the recommendations on your social media like Instagram, Twitter, etc?

Persons who are coming recommendations will have some sort of connection with us for example he/she may be our second connection. At this point, everyone will doubt how the data is been shared?

This sharing of data or feature vectors can be explained using Graph Convolutional Networks.

Message Passing Layers

The graph information including node features and structural properties and fed through the message passing layers. These layers are made up of node embeddings that contain the knowledge of other nodes and edges in a compressed format. This is done by gathering the current node information of the neighbor nodes combining it in a certain way to get a new embedding updating it to form new embeddings. This is also called Graph convolution.

Let's look into the working of Message passing layers:

Let's assume this is our input graph. Here you can see a yellow node with number 1 with a yellow-colored feature vector. This is the nod that we are focused on the following. To update the node state we collect the information of the direct neighbors which means we perform message passing. To update the current node state we take the information about the direct neighbors and the information about the current node (which is represented as h). We perform aggregation in the neighboring state to combine their information. Then we combine the current state information with the combined state information to get the new state or embedding in the layer k+1.

Here you can see the node features of the blue nodes are been added to the new embedding and then we update the annotations in the graph and it will become:

Here you can see the blue nodes feature vectors are also been changed as some of them are directly connected green nodes. In short, there will be a new embedding for every node in our graph. The size of the new embedding is a hyperparameter and depends on the graph data we use. As you can see the node 5 has only known about the blue nodes and does not know about the yellow node with the number but this would change when we pass it through a message-passing layer. We can do several message-passing steps which correspond to the number of nodes in GNN.

Again we collect the information about the nodes and aggregate it in some way to form a new embedding.

So if you look into the yellow node some information about the green node has also been added to it. This means that node 1 knows something about node 5. In short, all nodes have known something about other nodes in one way or another, so there will be some sort of relation between them even though they are not connected directly. This knowledge is stored in each embedding and contains feature-based as well as structural information about the nodes. These embeddings can be used for prediction as they contain all the information about the graph that we need.

But if we pass the same graph through many message passing layers, over smoothing may occur. The message passing system makes use of the two core functions Aggregate and Update, which take feature vectors from neighbors and combine them with the node’s features to update the node’s representation. This operation works in such a way that the representations of interacting nodes (inside this process) are quite similar.

That’s the end of the Graph Neural Networks blog series. Do feel free to connect to share your feedback.

In part 2 of the series, I explained the conversion of theoretical knowledge of graphs to a practical form and explain the applications of Graphs.

If you haven’t gone through the same please check out Graph Neural Networks- Part 2

In this part, I’ll explain what is Graph Neural Networks.

From the previous blogs we know the main objective of graphs, we understand that graphs have an enormous expressive capacity, and as a result, they are getting a lot of interest in the field of machine learning. Every node is coupled with an embedding, which specifies the node in the data space.

GNNs are neural networks that can be directly applied to graphs, and provide an easy way to do node-level, edge-level, and graph-level prediction tasks.

Before getting to know what is Graph Neural Networks, we should know:

Why do Convolutional Neural Networks fail on graphs?

The main reason for that is unfixed node ordering. In graphs, due to permutation invariance in each turn, the ordering of the node can be changed. So we cant give the adjacency matrix itself as the input to predict the output. The second reason is that graphs have non-Euclidean spaces. The other reason is that graphs are size-independent. For a better understanding go through the image given below:

In the first picture of a cat there, we can see five nodes but in the next picture, you can see there are six nodes. From this, we could understand that the number of nodes will not be the only criteria to confirm it as a cat as it is size-independent.

From the next picture, we could understand that even if we flipped the image the node order is changed, so the adjacency matrix itself can’t be given as the input.

All images have fixed pixels and that can’t be interchanged(Euclidean data) while graphs are not. These are reasons for not using CNN in graphs.

Graph Neural Networks

Graph Neural Networks are neural networks that can be applied directly to graphs. The limitations that are explained above can be overcome by using this Graph Neural Networks. These models optimize GNNs for use on larger graphs and apply them to domains such as social networks, citation networks, and online communities.

In the past few years, different variants of Graph Neural Networks are being developed with Graph Convolutional Networks (GCN) being one of them. GCNs are also considered as one of the basic Graph Neural Networks variants.

That’s it for part 3 of the Graph Neural Network series. In part 4, I will explain what is GCN and it’s working.

Thank you for reading and please do connect for any suggestions or feedback.

In part 1 of the series, I explained the basic introduction to graphs which include the types of labels and graphs.

If you haven't gone through the same please check out Graph Neural Networks- Introduction.

In this part, I’ll explain the conversion of theoretical knowledge of graphs to a practical form and explain the applications of Graphs.

How can we convert the graph to a format that the computer can understand?

In order to convert these graphs into a format that computer we just convert them into the matrix.

The first matrix is called the Incidence matrix. This is an N x M matrix where N represents the number of nodes and M represents the number of edges.

The next is the Adjacent matrix. It is a connection square matrix having 0’s and 1’s. These 0 or 1 depending on whether nodes of the graph are adjacent or not.

The next is the Degree matrix. It is a diagonal matrix that contains information about the degree of each node.

Why we are using Graphs?

Graphs are the ultimate abstraction for many real-world problems and today, technology exists that can treat them as such.

We all are using social media in our day-to-day life. There all we are using is Graph data structures in Facebook Graph API’s Recommendation systems, Web analytics etc. The reason is that all are dynamic in nature so in order to handle dynamic data’s Graphs are the best.

More complex problems can be solved using graphs by reducing them into simpler representations or by transforming them into representations from various perspectives.

So Graphs are more effective at representing abstract notions such as linkages and interactions. They also provide a visually intuitive manner of thinking about these issues. Graphs are also an excellent tool for understanding social relationships.

Some Practical Applications of Graphs.

1. Social Graphs
   In social graphs, we can consider all the nodes as a person or a place, and edges are drawn between them which resembles the sort of relation between them. Facebook is one of the best examples of the application of graphs to real-life problems.
2. Recommendation Engine
   Netflix, YouTube, Tinder, and Amazon are all examples of recommender systems in use. They can be considered as predicting the link between the nodes.
3. Path Optimization Algorithms
   Finding the best connection that fits the predefined criteria is done in this algorithm. Either Breadth First Search (BFS) or Depth First Search (DFS) can be used to find a path between two vertices.
4. Knowledge Graphs
   A knowledge graph is organized as a graph, which is not always true of knowledge bases. For a better understanding go through the video

That’s it for part 2 of the Graph Neural Network series. In part 3, I will explain what is GNN.

Thank you for reading and please do connect for any suggestions or feedback.

Have you ever heard about a machine discovering an antibiotic that can treat several bacterial strains, including multi-drug-resistant bacteria?

Yes, its Halicin was discovered by a pioneering machine learning approach called Graph Neural Network (GNN). So GNN has been taken into consideration as an appealing modeling technique for molecular assets prediction, and several research has proven that GNN ought to yield extra promising outcomes than conventional.

Before getting into the topic let's understand what is a graph.

Graph

A Graph is a structured data type that has nodes and edges. Nodes are also called vertices. Graphs are a type of Non-Euclidean data, which means they exist in 3D, unlike other data types like images, text, and audio. Mathematically we can represent a graph as a set of vertex/nodes (V) and edges (E)

Diagrammatically a graph can be represented as

Each node and vertices can have labels. According to the labeling pattern they can be classified as Numerical labels, Textual labels, Unique labels. Let's learn more about it.

Labels seen on the graphs are not always being a number as they can be text too.

If the nodes and edges have numeric data then it's called Numeric labels.

If the labels are not numbers but texts then it's called Textual labels.

If multiple nodes are there they are given the same labels called Unique labels.

A node is a person, a node’s label is a person’s name, and the node’s features are the person’s characteristics.

Graphs are of two types directed graph and undirected graph.

Directed graph

In this type, the direction between the nodes is specified. Consider you are using Twitter and the one who you follow may not not be following you. So there is only one direction. So in case of this, we use a directed graph.

Undirected graph

Here direction between the nodes is not present. Consider a Facebook friend request, if you receive a request from your friend and you accept it, you both will become friends and can see the content of each and this is a typical example of an undirected graph.

That's it for part 1 of the Graph Neural Network series. In part 2, I will explain how we can convert graphs in a format computers can understand.

Thank you for reading and please do connect for any suggestions or feedback.

OCR or Optical Character Recognition is the use of technology to distinguish printed or handwritten text characters inside digital images of physical documents. In the simplest terms, OCR also referred to as text recognition, involves examining the text of a document and translating the characters into code that can be used for data processing.

Malayalam is a South Indian language — which is the principal language of the State of Kerala, spoken by about 36 million people in the world. The Malayalam script is a Brahmic script used commonly to write the Malayalam language. Like many other Indic scripts, Malayalam follows a writing system that is partially alphabetic and partially syllable-based. The Malayalam script uses both old and new scripts for depicting characters. Due to the complexity of the Malayalam character set, an efficient method for the recognition of handwritten characters has not been proposed till now.

The main objective is to develop a handwritten Malayalam word recognition system.

The two phases identified are:

i) To recognize Handwritten Malayalam character

ii) To develop a Malayalam word recognition system to achieve a better accuracy rate in Malayalam handwritten character recognition system.

Working

Here we use Open source computer vision (OpenCV) for reading the image and for doing some image processing techniques. For finding text in an image the first process is to do color conversion, For this, we have to store the image to a variable and by using cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) the RGB image is converted into a grayscale image. As we have binarized the image, binarized image pixel values from 0 to 255. By doing binarization there is a chance of generating noise in the image. This noise won’t help our OCR to detect the text correctly. To avoid that we must add some blur. This can be attained by using the command cv2.GaussianBlur(src, ksize, sigmaX[, dst[, sigmaY[,borderType=BORDER_DEFAULT]]] ) .

Normally the kernel size is given as a 5x5 matrix. For edge detection, canny edge detection is used to measure edges of the horizontal and vertical axis present around the text region in a grayscale image.

This edge detection helps us to find the information on shapes that can be figured out from the image. While detecting the edges we will give an upper and lower threshold. Anything above the lower threshold and upper threshold will be accepted and others are rejected.

Contours are defined as the line joining all the points along the boundary of an image that are having the same intensity. After the edge detection, the copy of this image is given as the source for finding the contour and cv2.RETR_EXTERNAL is used to find the outer contour and CHAIN_APPROX_SIMPLE will return all the endpoints required for drawing the contour. Then the contours are stored as tuples and sorted using imutils.grab_contours, sort_contours respectively.

Then we have to filter out too large and too small areas. For this, we have to give some conditions. Then by using cv2.threshold we can find the text. Considering all the text is dark we give a lower threshold to 0 and output threshold to 255 which means anything greater than 0 pixels will be converted to 255 and this is done by cv2.THRESH_BINARY_INV. Finding out the dimensions of the bounding box and then we have grown the bounding box outside each character.

Conclusion:

Optical character recognition is a necessary first step for all applications that consider typewritten or handwritten manuscripts as input. Recognition of printed text gives good results. Almost all the data read was correct. Only a few recognized fields contained mistakes, but they have been unreadable or damaged during the scanning process.

We conclude that Malayalam letters can’t be figured out by using openCV to create complex shapes and similarities between their letters. The accuracy that is produced by our system is less than 30%

Any criticism, suggestion or addition, corresponding to the project are welcome.

Introduction

Climatology and prognostication are vital since it helps verify future climate expectations. Meteorology focuses on current climate equivalent to humidity, air pressure, and temperatures and forecasting the short-run weather conditions to come. Weather warnings are significant as they could predict any possible danger to life or property. Weather is the single most event that affects human life in each dimension, starting from food to fly. In contrast, on the opposite hand, it’s the original unfortunate phenomenon. Therefore, the prediction of weather phenomena is of significant interest for human society to avoid or minimize the destruction of weather hazards. Weather prediction is advanced because of noise and missing values dataset. Varied efforts were created to create weather predictions as correct as possible. However, still, the complexities of noise are touching accuracy.

WEATHERWORKS

Weatherworks is an online platform that aims at weather detection by classifying sky images. The multi class application distinguishes the weather into rainy, cloudy, shiny, and sunrise, allowing suitable precautions that the user could undertake.

From the perspective of agriculture, the applications help forecast the weather for the day, which helps avoid unnecessary chaos.

We would like to thank Kaggle for providing the dataset for the completion of the project.

Working

Here we are using the PyTorch framework.

To install torch: pip install torch

PyTorch is an open-source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing, primarily developed by Facebook’s AI Research lab.

The training images and test images are preprocessed using the opencv and made into 32x32 px. These preprocessed images are randomly selected and split into test and train datasets, and moved into a directory “Preprocessed_image.” Taking the train directory as a whole is converted into tensor using torchvision.datasets.ImageFolder. Using torch.utils.data.DataLoader, the training dataset is divided into a batch size of 50.

Then we define the network. Our model name is Weatherclass.

Here we use Convolution neural networks. The neural network input was [3,32,32], which represents the number of channels, 32x32 the size of the image. Since it is an RGB image, the number of channels is 3. After defining the model, we call the fit function, which helps us get the validation dataset’s accuracy and training loss.

The model’s accuracy was about 86.00%, and then we load the weights and bias to “Weather_model.pt” file for inference.

Defining the model and loading the “Weather_model.pt” is done on another python file is that “Loading model.py.”

The image of dimension [3,32,32] is passed through the convolution layers and between each convolution payer there is an ReLU activation function. After passing through these layers the output flattens in such a manner that we get 4 classes as outputs. This was done in a sequential order.

To Accept and give out the prediction in a web application, we have used the Flask API of Python, a micro web framework.

Our application name is app

app=Flask(__name__)

Our index page has both GET and POST methods. The GET method is used to request data from the server that is our image URL. The POST method is used to submit our prediction to the server.

To create an application using a flask, we use two folders — templates and static.

Templates are used to render HTML, which will display in the user’s browser.

Static folder contains assets used by the templates, including CSS files, JavaScript files, and images.

INFERENCE

Weatherworks is a basic model that classifies images into 4 classes.

The current accuracy is 86 with the existing dataset of around 170 in each class. The model is able to differentiate the images into cloudy, rainy, shine and sunrise.

Although, it is to be inferred that the number of classes as well as the dataset corresponding to each class is to be increased to raise the accuracy as well as to make its usability significantly efficient. As we can infer, the presence of classes such as windy, hurricane, winter, autumn etc, can make the application more efficient for the expected consumers, essentially the farmers.

Any criticism, suggestion or addition, corresponding to the project are welcome.

GitHub reference : https://github.com/Ashwin202/Weatherworks.git