Porn is now closer than almost anything else you might want. It’s one tap from a locked phone, one autocomplete from an address bar, one algorithmic slip inside an app you opened for a completely different reason. For a lot of people this stopped being a moral question a while ago and became a logistics question: the friction is gone, so the habit runs on rails and it’s the easiest sneakiest addiction you will ever see. So if you want to protect yourself, your kids, your family your friend this doc is for you. Now if this is for you then think of this as the Odysseus rule is if you can’t control yourself then prevent yourself

“a choice you make now to lock in a safe decision and stop future temptations. It comes from the The Odyssey myth where Odysseus tied himself to a ship mast to safely hear the deadly Sirens.”

Fyi, If you don’t care to read the entire thing just Cmd+A then Paste it into Claude code and tell it to do it for you. But for your phone you have to do it manually and It is NOT THAT hard ~ Just do it.

And when you go looking for help, here’s what the market offers you:

• Accountability apps that monitor your screen activity and report it to another person. Some capture screenshots at intervals. To work at all, they need deep, persistent access to everything you look at.
• Filtering apps that install a VPN or device-management profile, then route your entire internet connection through their servers so they can inspect it.
• Subscriptions, usually $10–$20 a month, forever.

I want to be fair to these tools. For some people — particularly people who genuinely want a human accountability partner — they’re the right call, and the trade is one they’ve made with their eyes open.

But it is a trade. You are handing a company continuous visibility into your browsing, your screen, or both, and paying for the privilege, to solve a problem your operating system can already solve for free. For a lot of people that’s not a trade worth making — and worse, the privacy cost becomes the reason they never set anything up at all.

There’s another path. Everything below runs entirely on your own machine. Nothing is sent anywhere. Nothing is monitored. There’s no account, no subscription, and no third party who can see what you do.

The only catch is that it involves the terminal — which is exactly where Claude Code comes in.

The shortcut: let Claude Code do it

If you’re not technical, this section is the whole article. You do not need to understand any of the commands that follow.

Claude Code is Anthropic’s coding assistant that runs in your terminal. It reads files, writes files, and runs commands on your machine — with your approval at each step. Which means you can hand it a technical guide and say “do this,” and it will.

Setup, once

Install it — on macOS, open Terminal (Cmd+Space, type “Terminal”) and paste:

curl -fsSL https://claude.ai/install.sh | bash

Then start it:

claude

The first run opens your browser to log in. You’ll need a Claude Pro, Max, Team, or Enterprise plan — the free tier doesn’t include Claude Code. (Homebrew users: brew install --cask claude-code works too. Full install docs: code.claude.com/docs/en/setup.)

Then just ask

It will read the guide, write the script, ask your permission before touching anything in /etc, install it, apply the block, and check its own work. When it needs sudo, it'll tell you and wait.

That last one matters. Ask it. You should always understand what’s been changed on a machine you own, and Claude Code will walk you through it in plain language.

A note on trust: Claude Code runs locally and asks before it does anything consequential. But you’re still giving an AI assistant permission to modify system files — read what it proposes, and don’t blanket-approve. The whole point of this approach is staying in control of your own machine.

What you’re actually building, in plain English

Most blocking guides hand you one technique, which is why so many blocks quietly fail. A web request passes through several independent stages, and a block at one stage does nothing about the others.

Here’s the path a request actually takes:

Stage What can block here The app or browser Screen Time, browser policy. Native apps ignore web filters entirely. The browser’s own DNS Nothing — this skips the next two stages when enabled. So does a browser VPN. The OS resolver /etc/hosts — covers every account and every app on the machine. Your DNS server A filtering resolver — whole categories, maintained by someone else. Your router Covers every device on the Wi-Fi. Phones, laptops, TVs, guests.

Read that top to bottom and the strategy writes itself:

• A hosts file is precise but local and manual — good for specific named sites.
• A filtering resolver is broad and continuously maintained — good for a whole category you could never enumerate by hand.
• Neither survives a browser VPN.

So you use more than one. That’s the entire idea.

The mistake that makes a block look like it works

This is the part that costs people an afternoon, and it’s the reason the guide below exists at all.

Your first attempt sinks each domain to a dead IPv4 address:

0.0.0.0	reddit.com

You apply it. The file looks right. And the site still loads.

So you blame the browser. You clear caches. You try a different browser. It still loads.

The browser was never the problem. Here’s what the system resolver actually returned with a correct-looking, IPv4-only block in place:

Domain Resolved to Result www.reddit.com 0.0.0.0 blocked reddit.com 2a04:4e42:400::396 still reachable redgifs.com 2a06:98c1:310a::6812:247b still reachable i.redd.it 2a04:4e42:3d::396 still reachable pornhub.com 0.0.0.0 blocked

An entry pointing at 0.0.0.0 answers the IPv4 question only. When a domain also publishes an IPv6 (AAAA) record, macOS goes out to real DNS for the IPv6 address, gets a live one, and connects over IPv6 — which it prefers when available.

Your block is bypassed by the operating system itself, silently.

Notice how convincing the failure is. Domains without IPv6 block perfectly, so the file appears to work. Whether any given site leaks depends on whether it publishes an AAAA record — which is why half your list can look fine while the half you actually care about doesn’t.

The fix is to sink both address families. Every domain needs two lines:

0.0.0.0	reddit.com
::	reddit.com
0.0.0.0	www.reddit.com
::	www.reddit.com

:: is the IPv6 equivalent of 0.0.0.0. The script in the appendix emits both automatically, which is why it writes four lines per domain you add.

This single detail is why a hand-rolled block fails and a commercial app appears to succeed. It has nothing to do with the app being better. It has to do with two lines instead of one.

Never verify a block in a browser

Testing by visiting the site is how the IPv6 leak hides for so long, because a browser tells you that something loaded, never why. Ask the resolver directly instead:

dscacheutil -q host -a name reddit.com

You want 0.0.0.0 and nothing else. Any ipv6_address line in that output is a leak, no matter how correct your hosts file looks.

Loop over your whole list and check every entry. If you take one habit from this article, take that one. (Or just tell Claude Code: “verify every domain on my blocklist at the resolver level and flag any IPv6 leaks.”)

Two more limits nobody mentions

There are no wildcards. You cannot write *.reddit.com. Every subdomain must be named, and any you miss stays open — including the media subdomains that serve the actual images.

Apps use different domains than websites. Reddit’s mobile and desktop apps talk to oauth.reddit.com and gateway.reddit.com. Block only reddit.com and the website goes dark while the app keeps working perfectly.

A list that actually holds looks more like this:

sudo blocksites add \
  reddit.com old.reddit.com new.reddit.com np.reddit.com \
  sh.reddit.com amp.reddit.com m.reddit.com out.reddit.com \
  oauth.reddit.com gateway.reddit.com api.reddit.com \
  redd.it i.redd.it v.redd.it preview.redd.it \
  external-preview.redd.it redditmedia.com redditstatic.com

Working that out for each site is tedious and error-prone. It’s also exactly the kind of thing an AI assistant is good at: “figure out every domain the X app and website use, and block all of them.”

The category problem, and why a hand-written list can’t solve it

Here’s where you have to be realistic with yourself.

A hosts file is exact. That’s its strength for a handful of named sites and its fatal weakness for a category. If your goal is “block adult sites” rather than “block these four sites,” a hand-written list is the wrong instrument entirely. There are millions of those domains, the large ones run dozens of mirror domains each, and new ones appear continuously.

A static list gives you the feeling of a block while leaving the door open.

The fix is a filtering DNS resolver — you point your machine at a DNS server that already refuses to resolve entire categories, maintained by someone else, updated constantly. Two free options:

Resolver Addresses Adult domains Everything else Cloudflare for Families 1.1.1.3 / 1.0.0.3 0.0.0.0 resolves normally CleanBrowsing Adult 185.228.168.10 NXDOMAIN resolves normally

Set it on every network service at once — and note the IPv6 addresses, for exactly the same reason they mattered above:

sudo bash -c 'networksetup -listallnetworkservices | tail -n +2 | sed "s/^\*//" |
while IFS= read -r s; do
  networksetup -setdnsservers "$s" \
    1.1.1.3 1.0.0.3 2606:4700:4700::1113 2606:4700:4700::1003 && echo "set: $s"
done'

This survives reboots and applies to every account on the machine.

Yes, you’re now using a third party — but only as a resolver, the same role your ISP already plays. Cloudflare’s family resolver sees DNS queries, which your ISP’s resolver sees anyway. Nobody is screenshotting your desktop.

The division of labor: the resolver handles the category; the hosts file handles your specific named sites. Reddit resolves normally through both resolvers above, because it isn’t classified as adult. That’s not a bug — it’s why you do both layers.

The bypasses you need to know about

A guide that doesn’t tell you about these is selling you false confidence.

Browsers with a built-in VPN. Opera and Opera GX ship a free one. Two clicks and browser traffic tunnels out through Opera’s servers — hosts file and filtering DNS both bypassed completely, nothing to install. No system-level configuration closes this. If the machine is meant to be genuinely restricted, that browser has to go.

DNS-over-HTTPS. Chrome, Edge and Firefox can resolve domains themselves over HTTPS, skipping the OS resolver entirely — meaning both your hosts file and your DNS setting. Anyone can point Secure DNS at a non-filtering provider in settings and walk around the whole setup. For Chrome and Edge, the reliable fix is a managed policy setting DnsOverHttpsMode to off. (Firefox has a canary-domain escape hatch, but Cloudflare's family resolver doesn't trigger it — don't count on it.)

Chrome holds connections open. A mundane one that wastes real debugging time: Chrome keeps its own DNS cache and reuses live sockets. It can keep loading a site well after the block is correctly in place. Quit it fully — Cmd+Q, not just closing the window — before concluding anything is broken.

The iPhone, where none of this transfers

iOS has no hosts file and no root access. The equivalent lives in Screen Time.

Settings → Screen Time → Content & Privacy Restrictions → Content Restrictions → Web Content → Limit Adult Websites.

That gives you two things at once: Apple’s maintained category filter, and a Never Allow list underneath it for specific domains you add by hand — the same division of labor as on the Mac. It filters at the WebKit level, so it covers every iOS browser. There’s no browser-VPN-shaped hole.

Lock it, or it’s decorative. Set a Screen Time passcode under Lock Screen Time Settings, and make it different from your device unlock passcode. Without that, the restrictions are two taps from being switched off by anyone holding the phone. Add a recovery Apple ID when prompted — otherwise forgetting this passcode means wiping the device.

The gap that will bite you: web content filters do not touch native apps. The Reddit app talks to its API domains directly and sails past Web Content restrictions entirely. Delete the app, then set iTunes & App Store Purchases → Installing Apps → Don’t Allow so it can’t just be reinstalled.

DNS on iOS: setting DNS manually in Wi-Fi settings applies to that one network only and does nothing on cellular. Install a DNS configuration profile instead — Cloudflare publishes one for 1.1.1.1 for Families that covers Wi-Fi and cellular together.

Doing it for a whole household

If the goal is a family rather than one computer, the router is by far the best value for effort. Set its DNS servers to 1.1.1.3 and 1.0.0.3 in the admin page — usually at 192.168.1.1 — and every device that joins the Wi-Fi inherits the filter. Phones, laptops, tablets, TVs, guests. Nothing to install per device.

Its limit is precisely its scope: it stops at your front door. Cellular data and any other Wi-Fi network are unaffected.

What this can and can’t do

The failure mode of a blocking guide is that someone believes it more than they should. So:

It stops casual and habitual access, which is usually the actual goal. Removing the two-second path to a site is most of the benefit for most people, and this setup does that thoroughly. Habits run on friction. Put friction back.

It does not stop a motivated person with admin rights. Everything here is reversible by the same admin account that applied it — one command lifts the hosts block, one setting changes DNS back. If you’re blocking yourself, that’s fine; friction is the point, and the ten minutes it takes to undo is usually enough for the urge to pass. If you’re blocking someone else who has the admin password, you are not really blocking them.

A hand-written list ages badly. Mirror domains and new sites appear constantly. That’s exactly what the filtering resolver is for, and why it’s worth doing both layers rather than picking one.

If you need more, the next steps are a separate non-admin account for daily use, a configuration profile for browser policy, and a LaunchDaemon that reapplies the blocklist on a schedule. Worth it only if your situation warrants it — and all three are things you can ask Claude Code to build.

And the thing worth saying plainly: if this is compulsive rather than habitual — if it’s costing you sleep, work, or a relationship, and stopping feels genuinely outside your control — a technical block is a support, not a treatment. It buys you time and space. It doesn’t do the harder work. A therapist who works with compulsive behaviour will do more for you than any script, and there’s nothing weak about going and getting one.

The point

The reason most people never set any of this up isn’t that they don’t want to. It’s that the free path required knowing about /etc/hosts, IPv6 AAAA records, dscacheutil, and networksetup — and the paid path required handing a stranger a live feed of your screen.

Claude Code collapses the first path down to a paragraph of plain English. The technical knowledge is still required — it’s just no longer required of you.

Your machine. Your files. Nothing leaves. No subscription. Fifteen minutes.

Appendix: the script

Save this as blocksites. It keeps your domain list in /etc/blocked-sites.txt and regenerates a clearly delimited block inside /etc/hosts, so adding and removing sites never involves opening a critical system file by hand.

(You can skip this entirely and let Claude Code write it — but here it is, because you should be able to see exactly what’s running on your machine.)

#!/bin/bash
#
# blocksites — system-wide website blocklist via /etc/hosts
#
# Affects every user account and every app on this Mac.
# Domain list lives in /etc/blocked-sites.txt (root-owned).
#
# Usage:
#   blocksites add <domain> [domain...]   add domains and apply
#   blocksites rm  <domain> [domain...]   remove domains and apply
#   blocksites list                       show the current blocklist
#   blocksites apply                      rebuild /etc/hosts from the list
#   blocksites off                        remove the block (list is kept)
#   blocksites status                     show whether the block is active

set -euo pipefail

LIST=${BLOCKSITES_LIST:-/etc/blocked-sites.txt}
HOSTS=${BLOCKSITES_HOSTS:-/etc/hosts}
BEGIN='# BEGIN blocksites — managed block, edit /etc/blocked-sites.txt instead'
END='# END blocksites'
SINK='0.0.0.0'
SINK6='::'   # without an IPv6 sink, AAAA lookups fall through to real DNS and the block leaks

if [[ $EUID -ne 0 && -z ${BLOCKSITES_HOSTS:-} ]]; then
  echo "blocksites: must run as root (use sudo)" >&2
  exit 1
fi

secure_list() {
  chown root:wheel "$LIST" 2>/dev/null || true
  chmod 644 "$LIST" 2>/dev/null || true
}

[[ -f $LIST ]] || { : > "$LIST"; secure_list; }

# Strip scheme, path, port, leading www. and lowercase — so pasting a full URL works.
normalize() {
  printf '%s\n' "$1" \
    | tr '[:upper:]' '[:lower:]' \
    | sed -E 's#^[a-z]+://##; s#/.*$##; s#:[0-9]+$##; s#^www\.##' \
    | sed -E 's#^\.+##; s#\.+$##'
}

flush_dns() {
  dscacheutil -flushcache 2>/dev/null || true
  killall -HUP mDNSResponder 2>/dev/null || true
}

# Rewrite $HOSTS: everything outside the markers is preserved verbatim.
apply() {
  local tmp
  tmp=$(mktemp /tmp/blocksites.XXXXXX)

  awk -v b="$BEGIN" -v e="$END" '
    $0 == b { skip = 1 }
    !skip   { print }
    $0 == e { skip = 0 }
  ' "$HOSTS" > "$tmp"

  # Drop trailing blank lines so repeated applies do not accumulate whitespace.
  while [[ -s $tmp && -z $(tail -n1 "$tmp") ]]; do
    sed -i '' -e '$d' "$tmp"
  done

  local count=0
  if [[ -s $LIST ]]; then
    {
      printf '\n%s\n' "$BEGIN"
      while IFS= read -r domain; do
        [[ -z $domain || $domain == \#* ]] && continue
        printf '%s\t%s\n'     "$SINK"  "$domain"
        printf '%s\t%s\n'     "$SINK6" "$domain"
        printf '%s\twww.%s\n' "$SINK"  "$domain"
        printf '%s\twww.%s\n' "$SINK6" "$domain"
        count=$((count + 1))
      done < "$LIST"
      printf '%s\n' "$END"
    } >> "$tmp"
    count=$(grep -cvE '^\s*(#|$)' "$LIST" || true)
  fi

  # Validate before installing: we must never leave localhost unresolvable.
  grep -qE '^\s*127\.0\.0\.1\s+localhost' "$tmp" || {
    echo "blocksites: refusing to write — localhost entry missing" >&2
    rm -f "$tmp"; exit 1
  }

  cat "$tmp" > "$HOSTS"          # preserves existing inode/ownership/mode
  rm -f "$tmp"
  chown root:wheel "$HOSTS" 2>/dev/null || true
  chmod 644 "$HOSTS" 2>/dev/null || true
  flush_dns
  echo "blocksites: $count domain(s) blocked (plus their www. forms). DNS cache flushed."
}

cmd=${1:-status}
shift || true

case "$cmd" in
  add)
    [[ $# -gt 0 ]] || { echo "usage: blocksites add <domain>..." >&2; exit 1; }
    for raw in "$@"; do
      d=$(normalize "$raw")
      [[ -n $d ]] || continue
      if grep -qxF "$d" "$LIST" 2>/dev/null; then
        echo "  already blocked: $d"
      else
        printf '%s\n' "$d" >> "$LIST"
        echo "  + $d"
      fi
    done
    sort -u -o "$LIST" "$LIST"
    apply
    ;;
  rm|remove)
    [[ $# -gt 0 ]] || { echo "usage: blocksites rm <domain>..." >&2; exit 1; }
    for raw in "$@"; do
      d=$(normalize "$raw")
      [[ -n $d ]] || continue
      if grep -qxF "$d" "$LIST" 2>/dev/null; then
        grep -vxF "$d" "$LIST" > "$LIST.tmp" || true
        mv "$LIST.tmp" "$LIST"
        secure_list
        echo "  - $d"
      else
        echo "  not in list: $d"
      fi
    done
    apply
    ;;
  list)
    if [[ -s $LIST ]]; then cat "$LIST"; else echo "(blocklist is empty)"; fi
    ;;
  apply)
    apply
    ;;
  off)
    local_tmp=$(mktemp /tmp/blocksites.XXXXXX)
    awk -v b="$BEGIN" -v e="$END" '
      $0 == b { skip = 1 }
      !skip   { print }
      $0 == e { skip = 0 }
    ' "$HOSTS" > "$local_tmp"
    while [[ -s $local_tmp && -z $(tail -n1 "$local_tmp") ]]; do
      sed -i '' -e '$d' "$local_tmp"
    done
    cat "$local_tmp" > "$HOSTS"
    rm -f "$local_tmp"
    flush_dns
    echo "blocksites: block removed from $HOSTS (list preserved in $LIST — 'blocksites apply' re-enables)."
    ;;
  status)
    if grep -qF "$BEGIN" "$HOSTS"; then
      n=$(grep -cvE '^\s*(#|$)' "$LIST" 2>/dev/null || true)
      echo "blocksites: ACTIVE — $n domain(s) blocked"
    else
      echo "blocksites: inactive"
    fi
    ;;
  *)
    sed -n '3,14p' "$0" | sed 's/^# \{0,1\}//'
    exit 1
    ;;
esac

Install it:

sudo install -o root -g wheel -m 755 blocksites /usr/local/bin/blocksites

Everyday use:

sudo blocksites add reddit.com redgifs.com   # add and apply
sudo blocksites rm reddit.com                # remove and apply
sudo blocksites list                         # what's on the list
sudo blocksites status                       # is the block live
sudo blocksites off                          # lift it, keep the list
sudo blocksites apply                        # put it back

A few design decisions worth calling out, because they’re the difference between a script you trust with /etc/hosts and one you don't:

• Marked region. Everything outside the BEGIN/END markers is copied through verbatim. Local dev entries, VPN entries — all survive untouched.
• It refuses to write a broken file. Before installing, it checks the 127.0.0.1 localhost line is still present. A hosts file without it breaks things in confusing ways.
• Input normalization. Paste a full URL with capitals, a path and a port; it reduces to a bare domain.
• It flushes DNS on every change, so blocks take effect immediately.
• Idempotent. Run apply a hundred times and the file is identical each time.

Cheat sheet

Goal Command or setting Block named sites, all accounts sudo blocksites add example.com Check a block really works dscacheutil -q host -a name example.com See the live block sudo blocksites status Lift it temporarily sudo blocksites off Category filtering, one Mac networksetup -setdnsservers … 1.1.1.3 Category filtering, whole house Router DNS → 1.1.1.3 / 1.0.0.3 Category filtering, one iPhone Screen Time → Limit Adult Websites Named sites, one iPhone Screen Time → Never Allow Flush macOS DNS cache sudo dscacheutil -flushcache

Commands tested on macOS 26.5. Resolver behaviour verified against 1.1.1.3 and 185.228.168.10 at time of writing; filtering services change their categorisation over time, so re-verify with dscacheutil rather than assuming.

Now- Are you Ready ?!!.

The Epic Journey of Scaling: From Garage Startup to Tech Giant

Picture this: You’re in a garage, hunched over a laptop, furiously coding the next big thing. Your web app is alive, and your first user (probably your mom) just signed up. Congratulations! But what happens when your brilliant idea catches fire, and suddenly, you’re not juggling one user, but millions?

Welcome to the wild ride of scaling a web application. Buckle up, future tech mogul, because we’re about to embark on an adventure from zero to millions of users!

Chapter 1: The Humble Beginnings

Remember that laptop in the garage? That’s your entire tech stack right now. Web server, database, cache — it’s all crammed into one machine. It’s like having your entire office in a studio apartment. Cozy, but not for long.

Your users (still mostly family and that one weirdly supportive ex) type in your fancy domain name, and voila! They’re connected to your digital brainchild. Simple, right? Oh, sweet summer child, if only it stayed that way.

Chapter 2: Growing Pains and Database Dilemmas

As word spreads about your app (turns out your ex has a surprisingly large social media following), you realize your trusty laptop is sweating. It’s time for your first big decision: SQL or NoSQL?

Choosing a database is like choosing a life partner. SQL databases are the dependable type — great with structure and relationships. NoSQL? That’s your free-spirited option, flexible and ready for anything. Choose wisely; this relationship will last a while.

SQL vs. NoSQL: Know Your Options

• SQL Databases (Relational Databases): Think of them as well-organized spreadsheets. They’re great for applications where the data is structured, and relationships between data entities are crucial (like financial records or inventory management).
• NoSQL Databases: Perfect for when your data is unstructured or when you need to handle massive amounts of data across distributed systems (like social media feeds or real-time analytics).

Chapter 3: The Great Server Split

Remember when you thought you’d be the next Zuckerberg with just one server? Cute. It’s time to separate your web server from your database. Think of it as your app moving out of your parents’ house. It’s a bit scary, but necessary for growth.

By splitting your server, you’re now able to scale each component independently. Need more power for your web server? Just upgrade or add another one. Is your database getting bogged down with too many read requests? Time to optimize or add more database servers.

Chapter 4: Scaling Up or Scaling Out? The Eternal Question

Now you’re faced with another choice: do you beef up your servers (vertical scaling) or add more of them (horizontal scaling)?

• Vertical Scaling: Adding more resources (CPU, RAM) to your existing servers. It’s like upgrading your studio apartment to a penthouse but in the same building. This approach works up to a point, but eventually, you’ll hit hardware limitations (and your wallet might feel it too).
• Horizontal Scaling: Adding more servers to your infrastructure. This is like building more apartments to house your growing population. Each server handles a slice of the workload, allowing you to scale out almost indefinitely. It’s more complex to manage but provides greater flexibility and redundancy.

Chapter 5: Load Balancers — The Traffic Cops of the Internet

With multiple servers, you need something to direct traffic. Enter load balancers, the unsung heroes of the web. They’re like incredibly efficient traffic cops, making sure no server gets overwhelmed while others twiddle their digital thumbs.

Why Load Balancers Are Essential:

• Distribute Traffic Evenly: Ensure no single server is overloaded while others are idle.
• Enhance Reliability: If a server goes down, the load balancer redirects traffic to healthy servers, maintaining uptime.
• Improve Scalability: Easily add or remove servers as needed without disrupting the user experience.

Chapter 6: Database Drama — The Replication Saga

Your database is feeling the strain of fame. Time for some database replication! Think of it as creating a boy band. You have one lead singer (the master database) and several backup dancers (slave databases). The lead handles the hard stuff (writes), while the backups support with the easier tasks (reads).

Benefits of Database Replication:

• Improved Performance: Spread read requests across multiple replicas, reducing the load on the master database.
• Increased Reliability: If the master database fails, promote a replica to take over, ensuring continuity.

Chapter 7: The Cache Money Crew

By now, your app is cooking! But users are impatient creatures. Enter caching, your new best friend. It’s like having a super-smart assistant who remembers everything, so you don’t have to keep asking the same questions.

And don’t forget its cool cousin, CDN (Content Delivery Network), who makes sure your static content is always ready for its global audience.

Now, you might be wondering, “What exactly is a CDN, and why do I need one?”

Imagine you’re running a global pizza delivery service. Instead of baking all the pizzas in one central kitchen and shipping them worldwide (which would result in a lot of cold, sad pizzas), you set up kitchens all around the world. Each kitchen stores pre-made popular pizzas (your static content) and quickly heats them up when ordered. That’s essentially what a CDN does!

Why You Should Care About CDNs:

• Speed and Performance: CDNs store copies of your static content (images, CSS, JavaScript files) on servers distributed globally. When a user accesses your site, the content is served from the nearest server, reducing load times and improving the user experience.
• Reduced Load on Your Servers: By offloading the delivery of static content to the CDN, your servers are free to handle more important tasks, like processing dynamic requests and running business logic.
• Improved Reliability: CDNs can provide redundancy. If one server goes down, another can take over, ensuring that your site remains available to users around the world.
• Scalability: As your traffic grows, CDNs can handle increased loads without requiring you to add more infrastructure, making them a cost-effective solution for scaling your web application.

Chapter 8: Stateless Architecture — The Art of Forgetting

Going stateless is like embracing zen philosophy for your servers. No attachments, no memories. Each request is a brand-new day. It’s liberating for your servers and a nightmare for session management. But hey, that’s what JSON Web Tokens (JWTs) are for!

You might ask, “What’s so special about JWTs, and how do they help in a stateless architecture?”

JWTs are like digital passports for your users. When a user logs in, your server issues a JWT that contains encoded information about the user, such as their ID and roles. This token is then stored on the client side (usually in local storage or a cookie) and sent with each subsequent request.

How JWTs Make Stateless Work:

• Self-Contained Tokens: JWTs carry all the information needed to identify a user, meaning the server doesn’t need to store session data. Each server can validate the token and extract user details, making it easy to scale horizontally.
• Security and Efficiency: JWTs are signed to ensure their integrity, preventing tampering. They can also be encrypted to protect sensitive information. This approach reduces the need for database lookups or server-side session storage, speeding up authentication.
• Flexibility: JWTs are versatile and can be used for authentication in both web and mobile applications. They’re also great for scenarios where single sign-on (SSO) is required across multiple services or domains.
• Expiration and Refresh: JWTs can include an expiration time, ensuring that tokens are valid only for a limited period. When a token expires, the user must re-authenticate or use a refresh token to obtain a new JWT, balancing security and user convenience.

Chapter 9: Going Global — The Multi-Data Center Tango

You’re worldwide, baby! But with great reach comes great responsibility. Multiple data centers ensure your app is fast everywhere. It’s like opening franchise stores, but way cooler and with more blinking lights.

“Why would I need multiple data centers?” you might wonder.

Think of data centers like strategically placed warehouses. If you only have one warehouse in New York, it’s going to take a while to deliver a package to someone in Tokyo. But if you have a warehouse in Tokyo too, that delivery is a lot quicker. The same principle applies to data centers:

Benefits of a Multi-Data Center Strategy:

• Reduced Latency: Users are directed to the nearest data center, minimizing the distance data must travel and speeding up response times.
• Disaster Recovery: If one data center suffers an outage (natural disasters, power failures), your application can automatically redirect traffic to another, minimizing downtime and maintaining availability.
• Load Balancing Across Regions: Distributing traffic across multiple data centers prevents any single location from becoming a bottleneck, ensuring smooth operation even under high loads.
• Regulatory Compliance: For some industries, it’s crucial to store data in specific regions to comply with local laws. Multiple data centers make it easier to adhere to these regulations by localizing data storage.

Chapter 10: The Decoupling Dance — Embracing Microservices

As your app grows more complex, it’s time to break it down. Microservices architecture is like turning your monolithic app into a bunch of specialized food trucks instead of one big restaurant. Each service does one thing, but does it well.

“Why go through the hassle of microservices?” you might ask.

Imagine your monolithic app as a giant Swiss Army knife. It’s got everything, but it’s cumbersome and hard to update. Microservices are like having separate, specialized tools that you can easily upgrade or replace without affecting the others.

Advantages of Microservices:

• Independent Development and Deployment: Each service can be developed, deployed, and scaled independently, allowing your teams to work faster and more efficiently.
• Resilience and Fault Isolation: If one service fails, it doesn’t necessarily bring down the entire application. This isolation improves overall reliability and uptime.
• Technology Flexibility: Different services can be written in different languages or use different databases, giving you the flexibility to choose the best tools for each job.
• Better Resource Utilization: Scale each microservice according to its specific needs. For example, your authentication service might require less computing power than your image processing service, allowing you to optimize costs and resources.

Chapter 11: Automate All the Things!

Finally, as your empire grows, you can’t possibly keep an eye on everything. Automation and monitoring tools become your new best friends. They’re like having a really attentive robot butler who tells you when things are going wrong before you even notice.

But wait, what should you automate, and why?

The Essentials of Automation and Monitoring:

• Continuous Integration/Continuous Deployment (CI/CD): Automate the process of integrating code changes and deploying them to production. This reduces human error, speeds up development cycles, and ensures consistent quality.
• Infrastructure as Code (IaC): Use tools like Terraform or AWS CloudFormation to automate the provisioning and management of your infrastructure. This makes scaling up or down as easy as running a script.
• Automated Testing: Implement automated tests to catch bugs and issues before they reach production, ensuring a smoother user experience.
• Monitoring and Alerting: Set up monitoring tools (like Prometheus or Datadog) to keep track of application performance, server health, and user metrics. Configure alerts to notify you of potential problems, allowing you to proactively address issues before they escalate.

The Never-Ending Story

And there you have it! You’ve scaled from a single user to millions. But remember, in the world of tech, there’s no “happily ever after.” It’s more like “happily keep iterating forever after.”

Your journey of scaling is never truly over. There’s always a new challenge, a new technology, a new paradigm. But isn’t that why we got into this crazy world of web development in the first place?

So, future tech titan, are you ready to start your scaling adventure? Remember, every tech giant started with a single server and a dream. Your empire awaits!

Frequently Anticipated Questions

As you embark on your scaling journey, you might find yourself pondering these questions:

1. Q: Do I need to implement all these scaling strategies at once? A: Not at all! Scaling is a gradual process. Start with the basics like separating your web and data tiers, then implement other strategies as your user base grows and you identify specific needs.
2. Q: How do I know when it’s time to scale? A: Keep an eye on key metrics like response times, server load, and user complaints. If you notice slowdowns or your current setup struggling to handle traffic, it’s probably time to consider scaling.
3. Q: Is horizontal scaling always better than vertical scaling? A: Not necessarily. For smaller applications, vertical scaling (upgrading your existing servers) can be simpler and more cost-effective. However, as you grow, horizontal scaling often provides more flexibility and fault tolerance.
4. Q: How do microservices compare to a monolithic architecture in terms of development speed? A: Initially, a monolithic architecture can be faster to develop. However, as your application grows, microservices can speed up development by allowing teams to work independently on different services.
5. Q: Are there any downsides to using a CDN? A: While CDNs offer many benefits, they can add complexity to your setup and may have associated costs. Also, if your content changes frequently, you need to ensure your CDN is updated promptly to avoid serving stale content.
6. Q: How do I handle data consistency across multiple data centers? A: This is a complex issue known as the CAP theorem problem. Common strategies include eventual consistency models, conflict resolution algorithms, and distributed consensus protocols like Paxos or Raft.
7. Q: Is it possible to over-optimize or over-scale my application? A: Yes! It’s important to scale based on actual needs rather than anticipated ones. Premature optimization can lead to unnecessary complexity and costs. Always measure and analyze before scaling.
8. Q: How do I ensure security when scaling, especially with microservices? A: Security in a scaled environment involves many aspects: use HTTPS everywhere, implement proper authentication and authorization (like OAuth 2.0), use security groups and firewalls, and regularly audit your systems. For microservices, consider using an API gateway to handle security concerns centrally.

Remember, scaling is as much an art as it is a science. These questions are just the beginning of your journey. Keep learning, stay curious, and don’t be afraid to experiment!

Rules of Thumb for Back-of-the-Envelope Estimation in System Design

When you’re faced with a system design problem or need to estimate capacity for a new project, follow these rules of thumb to guide your back-of-the-envelope calculations:

1. Start with the Scale

Rule: Always begin by estimating the scale of your system.

How to do it:

• Determine your total user base (e.g., monthly active users)
• Estimate daily active users (usually 10–25% of total users)
• Calculate requests per second:
• Start with daily operations
• Divide by seconds in a day (86,400)
• Add a peak factor (usually 2–3x average)

Example:

• 1 million total users
• 200,000 daily active users (20%)
• 2 million daily operations
• Average QPS = 2,000,000 / 86,400 ≈ 23
• Peak QPS = 23 * 3 = 69

2. Estimate Storage

Rule: Calculate storage needs based on data creation and retention policies.

How to do it:

• Determine the size of each data item
• Calculate daily data generation
• Multiply by your data retention period

Example:

• Each user profile is 200 KB
• 1,000 new users per day
• Data retained for 5 years
• Daily storage increase = 1,000 * 200 KB = 200 MB
• 5-year storage = 200 MB * 365 * 5 ≈ 365 GB

3. Assess Bandwidth

Rule: Estimate both incoming and outgoing bandwidth.

How to do it:

• Calculate the size of each request and response
• Multiply by your peak QPS

Example:

• Average request/response size is 100 KB
• Peak QPS is 69
• Bandwidth = 100 KB * 69 = 6.9 MB/s ≈ 55 Mbps

4. Compute CPU Needs

Rule: Estimate based on CPU-intensive operations per request.

How to do it:

• Determine CPU-intensive operations per request
• Estimate CPU cores needed per operation
• Multiply by peak QPS

Example:

• Each request requires 0.1 CPU cores for 100ms
• Peak QPS is 69
• CPU cores needed = 0.1 * 69 * 0.1 ≈ 0.69 cores

5. Calculate Memory Requirements

Rule: Estimate based on active dataset and caching needs.

How to do it:

• Determine the size of your active dataset
• Add overhead for OS and applications (usually 20–30%)
• Consider caching requirements (often 20–30% of active data)

Example:

• Active dataset is 500 GB
• OS and application overhead: 100 GB
• Cache: 100 GB
• Total memory needed: 700 GB

6. Plan for Redundancy and Growth

Rule: Always add capacity for redundancy and future growth.

How to do it:

• Multiply your estimates by 2–3 for redundancy
• Plan for 2–3 years of growth (often 20–50% year-over-year)

Example:

• If you need 10 servers based on current estimates:
• 20–30 servers for redundancy
• 30–45 servers accounting for 2 years of 20% annual growth

7. Consider Availability Requirements

Rule: Higher availability requires more resources and complexity.

How to do it:

• Determine required uptime (e.g., 99.99% vs 99.999%)
• Plan for multi-region deployment for higher availability
• Implement robust monitoring and automated failover

Example:

• For 99.999% availability:
• Deploy in at least 3 regions
• Implement real-time data replication
• Set up automated failover mechanisms

8. Use Powers of 2 and Round Numbers

Rule: Simplify calculations using powers of 2 and round numbers.

How to do it:

• Use 2¹⁰ (1024) as thousand, 2²⁰ as million, etc.
• Round to the nearest power of 10 or 2

Example:

• 1.7 million users becomes 2 million
• 240 MB becomes 256 MB (2⁸)

9. State Your Assumptions Clearly

Rule: Always clearly state the assumptions behind your estimates.

How to do it:

• Write down each assumption as you make it
• Be prepared to adjust calculations if assumptions change

Example: “Assuming 20% of total users are active daily, and each user performs 10 operations per day…”

10. Sanity Check Your Results

Rule: Always perform a sanity check on your final numbers.

How to do it:

• Compare your results to similar known systems
• Verify that the proportions make sense (e.g., storage vs. bandwidth)
• If something seems off, revisit your assumptions and calculations

Example: If your estimation suggests you need more servers than Google, you’ve probably made a mistake somewhere!

Remember, these rules of thumb are starting points. Always be ready to adjust based on the specific requirements of your system and the realities of your business. The goal of back-of-the-envelope estimation is to get a reasonable approximation quickly, not to produce exact figures.

— —

Now, this last part which is the rules of thumbs this is as much for me as it is for you because frankly that was a lot so here it is for me and you to look up.

Rules of Thumb for Back-of-the-Envelope Estimation in System Design

When you’re faced with a system design problem or need to estimate capacity for a new project, follow these rules of thumb to guide your back-of-the-envelope calculations:

1. Start with the Scale

Rule: Always begin by estimating the scale of your system.

How to do it:

• Determine your total user base (e.g., monthly active users)
• Estimate daily active users (usually 10–25% of total users)
• Calculate requests per second:
• Start with daily operations
• Divide by seconds in a day (86,400)
• Add a peak factor (usually 2–3x average)

Example:

• 1 million total users
• 200,000 daily active users (20%)
• 2 million daily operations
• Average QPS = 2,000,000 / 86,400 ≈ 23
• Peak QPS = 23 * 3 = 69

2. Estimate Storage

Rule: Calculate storage needs based on data creation and retention policies.

How to do it:

• Determine the size of each data item
• Calculate daily data generation
• Multiply by your data retention period

Example:

• Each user profile is 200 KB
• 1,000 new users per day
• Data retained for 5 years
• Daily storage increase = 1,000 * 200 KB = 200 MB
• 5-year storage = 200 MB * 365 * 5 ≈ 365 GB

3. Assess Bandwidth

Rule: Estimate both incoming and outgoing bandwidth.

How to do it:

• Calculate the size of each request and response
• Multiply by your peak QPS

Example:

• Average request/response size is 100 KB
• Peak QPS is 69
• Bandwidth = 100 KB * 69 = 6.9 MB/s ≈ 55 Mbps

4. Compute CPU Needs

Rule: Estimate based on CPU-intensive operations per request.

How to do it:

• Determine CPU-intensive operations per request
• Estimate CPU cores needed per operation
• Multiply by peak QPS

Example:

• Each request requires 0.1 CPU cores for 100ms
• Peak QPS is 69
• CPU cores needed = 0.1 * 69 * 0.1 ≈ 0.69 cores

5. Calculate Memory Requirements

Rule: Estimate based on active dataset and caching needs.

How to do it:

• Determine the size of your active dataset
• Add overhead for OS and applications (usually 20–30%)
• Consider caching requirements (often 20–30% of active data)

Example:

• Active dataset is 500 GB
• OS and application overhead: 100 GB
• Cache: 100 GB
• Total memory needed: 700 GB

6. Plan for Redundancy and Growth

Rule: Always add capacity for redundancy and future growth.

How to do it:

• Multiply your estimates by 2–3 for redundancy
• Plan for 2–3 years of growth (often 20–50% year-over-year)

Example:

• If you need 10 servers based on current estimates:
• 20–30 servers for redundancy
• 30–45 servers accounting for 2 years of 20% annual growth

7. Consider Availability Requirements

Rule: Higher availability requires more resources and complexity.

How to do it:

• Determine required uptime (e.g., 99.99% vs 99.999%)
• Plan for multi-region deployment for higher availability
• Implement robust monitoring and automated failover

Example:

• For 99.999% availability:
• Deploy in at least 3 regions
• Implement real-time data replication
• Set up automated failover mechanisms

8. Use Powers of 2 and Round Numbers

Rule: Simplify calculations using powers of 2 and round numbers.

How to do it:

• Use 2¹⁰ (1024) as thousand, 2²⁰ as million, etc.
• Round to the nearest power of 10 or 2

Example:

• 1.7 million users becomes 2 million
• 240 MB becomes 256 MB (2⁸)

9. State Your Assumptions Clearly

Rule: Always clearly state the assumptions behind your estimates.

How to do it:

• Write down each assumption as you make it
• Be prepared to adjust calculations if assumptions change

Example: “Assuming 20% of total users are active daily, and each user performs 10 operations per day…”

10. Sanity Check Your Results

Rule: Always perform a sanity check on your final numbers.

How to do it:

• Compare your results to similar known systems
• Verify that the proportions make sense (e.g., storage vs. bandwidth)
• If something seems off, revisit your assumptions and calculations

Example: If your estimation suggests you need more servers than Google, you’ve probably made a mistake somewhere!

Remember, these rules of thumb are starting points. Always be ready to adjust based on the specific requirements of your system and the realities of your business. The goal of back-of-the-envelope estimation is to get a reasonable approximation quickly, not to produce exact figures.

The rest of the parts are coming soon !

NEXT : The details of the framework !

System Design In a Hurry | Part I was originally published in The Modern Scientist on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

In the world of web development, creating a browser extension can be an exciting way to enhance your skills and create something fun and interactive. Today, I’ll walk you through the process of building a simple yet engaging number game called “1,2” as a browser extension. This project is perfect for beginners looking to dip their toes into extension development or for more experienced developers wanting a quick, enjoyable coding exercise who have not built a browser extension before.

What is the “1,2” Game? The “1,2” game is a two-player number game where players take turns adding either 1 or 2 to a running sum, aiming to reach a predetermined target number. The player who reaches the target number exactly wins the game. It’s simple, yet it involves strategy and can be surprisingly addictive!. Also- I am unbeatable in it.

Keep in mind that browser extension can be very useful in most areas even in AI.

Here’s an example of how the game would work between two players (You and the Opponent) with the target number being 10.

Game Example with Target 10

Initial Setup

• Target Number: 10
• Players: “You” and “Opponent”
• The starting player is determined randomly.

Game Flow

Coin Toss

• Let’s assume the coin toss determines that “You” go first.

Turn 1 (You)

• Choices: 1 or 1,2
• Action: You choose 1,2
• Current Sum: 2
• Commentary: “You chose 1,2. The sum increased from 0 to 2. It’s now Opponent’s turn.”

Turn 2 (Opponent)

• Choices: 3 or 3,4
• Action: Opponent chooses 3,4
• Current Sum: 4
• Commentary: “Opponent chose 3,4. The sum increased from 2 to 4. It’s now Your turn.”

Turn 3 (You)

• Choices: 5 or 5,6
• Action: You choose 5
• Current Sum: 5
• Commentary: “You chose 5. The sum increased from 4 to 5. It’s now Opponent’s turn.”

Turn 4 (Opponent)

• Choices: 6 or 6,7
• Action: Opponent chooses 6,7
• Current Sum: 7
• Commentary: “Opponent chose 6,7. The sum increased from 5 to 7. It’s now Your turn.”

Turn 5 (You)

• Choices: 8 or 8,9
• Action: You choose 8,9
• Current Sum: 9
• Commentary: “You chose 8,9. The sum increased from 7 to 9. It’s now Opponent’s turn.”

Turn 6 (Opponent)

• Choices: 10
• Action: Opponent chooses 10
• Current Sum: 10
• Commentary: “Opponent chose 10. The sum increased from 9 to 10. Opponent wins!”

Game Over

• Winner: Opponent

Notes

• The players take turns choosing either one or two numbers.
• The choices are always consecutive numbers starting from the current sum + 1.
• The game continues until one player reaches the exact target number.
• The player who reaches the exact target number wins the game.

Setting Up the Project

To begin, we’ll need to set up our project structure. Create a new directory for your extension and include the following files:

1. manifest.json (for extension configuration)
2. popup.html (for the game interface)
3. popup.js (for game logic)
4. styles.css (for styling)

The Game Logic: The heart of our game lies in the Game class within popup.js. Let's break down some key components:

class Game {
  constructor(targetNumber, opponent) {
    this.targetNumber = targetNumber;
    this.opponent = opponent;
    this.currentSum = 0;
    this.currentPlayer = Math.random() < 0.5 ? 'You' : 'Opponent';
    this.winner = null;
  }

  // ... other methods ...
}

This constructor initializes our game state, randomly choosing the starting player.

The play method handles each move:

play(choice) {
  const [lower, upper] = choice;
  this.currentSum = upper;

  if (this.currentSum === this.targetNumber) {
    this.winner = this.currentPlayer;
  } else {
    this.currentPlayer = this.currentPlayer === 'You' ? 'Opponent' : 'You';
  }

  return true;
}

The easy opponent chooses randomly, while the impossible opponent uses a strategy to play optimally.

User Interface and Interaction: The updateUI function refreshes the game display after each move:

function updateUI() {
  // ... update display elements ...

  if (game.currentPlayer === 'You' && !game.winner) {
    const choices = game.getChoices();
    choices.forEach(choice => {
      const button = document.createElement('button');
      button.textContent = choice[0] === choice[1] ? choice[0] : `${choice[0]},${choice[1]}`;
      button.addEventListener('click', () => playMove(choice));
      choicesElement.appendChild(button);
    });
  } else if (game.currentPlayer === 'Opponent' && !game.winner) {
    choicesElement.textContent = 'Opponent is thinking...';
  }

  // ... update result display ...
}

This function dynamically creates buttons for the player’s choices or displays a waiting message during the opponent’s turn.

Styling and Animation: To make the game more visually appealing, we’ve added CSS animations for various game events. For example, the win animation:

function showWinAnimation() {
  const winAnimation = document.getElementById('win-animation');
  winAnimation.style.display = 'block';
  setTimeout(() => {
    winAnimation.style.display = 'none';
  }, 3000);
}

This function displays a winning animation for 3 seconds when a player wins.

The whole code is here; try it : https://github.com/OMS1996/12browsergame/blob/main/12/popup.js

But you are here not to see how the method works, are you ?
You are here for two things : The browser extension and to play the game.

SO,

Browser Extension Development: From Code to Chrome

After developing our “1,2” game, the next step is turning it into a browser extension. This process involves a few key components and steps specific to extension development. As we go make sure to have the repo I shared with you on the ready.

Let’s dive into the details:

The Manifest File

At the heart of every Chrome extension is the manifest.json file. This file is crucial as it provides important information about your extension to the browser. Here's a basic manifest.json for our game:

{
  "manifest_version": 2,
  "name": "1,2 Number Game",
  "version": "1.0",
  "description": "A simple number game extension",
  "browser_action": {
    "default_popup": "popup.html"
  },
  "permissions": ["storage"]
}

Key components:

• manifest_version: Specifies which version of the Chrome manifest file format we're using.
• name and description: Metadata about your extension.
• browser_action: Defines what happens when the user clicks the extension icon. In our case, it opens popup.html.
• permissions: Lists any special permissions your extension needs.

Directory Structure

my-extension/
  ├── manifest.json
  ├── popup.html
  ├── popup.js
  └── styles.css

Loading the Extension in Chrome

To test your extension in Chrome:

a. Open Chrome and navigate to chrome://extensions/. b. Enable "Developer mode" by toggling the switch in the top right corner. c. Click "Load unpacked" and select your extension directory.

Your extension should now appear in the list and be available in your browser toolbar.

Debugging

Chrome provides powerful tools for debugging extensions:

• Click “Inspect views: popup.html” on your extension’s card in chrome://extensions/ to open DevTools for your popup.
• Use console.log() statements in your JavaScript to output debug information.
• The “Errors” button on the extension’s card will show any runtime errors.

Browser Storage

If you want to save game states or settings, you can use Chrome’s storage API. Add this permission to your manifest.json:

"permissions": ["storage"]

Then in your JavaScript

chrome.storage.sync.set({key: value}, function() {
  console.log('Value is set to ' + value);
});

chrome.storage.sync.get(['key'], function(result) {
  console.log('Value currently is ' + result.key);
});

Publishing Your Extension

Once your extension is ready:

a. Create a .zip file of your extension directory.

b. Go to the Chrome Developer Dashboard (you'll need a one-time developer account fee Something around 5$).

c. Click "New Item" and upload your .zip file. d. Fill in all required fields and submit for review.

Updating Your Extension

To update your published extension:

a. Increment the “version” in your manifest.json.

b. Create a new .zip file.

c. In the Developer Dashboard, click "Edit" on your extension and upload the new .zip.

Here is a video on the matter :
https://www.youtube.com/watch?v=kKBc2Q5JO9Y&t=8s

Building a Simple Yet Engaging Browser Game Extension: The ‘1,2’ Game was originally published in The Modern Scientist on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’ll begin with a relatable story to set the stage, followed by a detailed exploration of its components but if you are not ready for the story go through the basics section. Remember to be patient and that it is okay to not understand all the lingo at one go, Ready to dive in? Let’s get started, considering your time is valuable.

DISCLAIMER : In this post, my aim is to provide you with a clear and concise breakdown of networking concepts in a practical and that is why I explain TCP/IP as it reflects how real computer networks are implemented making it much more relevant to learn despite OSI’s theoretical vision. B

The OSI model is a theoretical framework that describes a complete range of functions required for network communications.

Basics

The client-server model is all about requests and responses between two components — a client and a server.

A client is an application or computer that makes requests to access resources or services. This could be a web browser, mobile app, etc.

A server is a computer that receives these requests and provides the requested resource or performs a task to fulfill the client’s request. Servers are usually more powerful computers.

For example, when you type a website URL in your web browser (client):

1. The browser sends an HTTP request message to the web server asking it to send the website page.
2. The server receives the request, processes it, and finds or generates the page data.
3. The server then sends an HTTP response back to the browser with the requested website page content (usually HTML, CSS, JS files).
4. The browser displays the pages, images, etc. on your screen.

IP

Here is a simplified explanation of IP addresses:

IP addresses are like home addresses for devices on the internet. They uniquely identify each device so that data can be sent to the correct destination.

When you type a website name like etherealai.io or apple.com into your browser, a DNS service looks up the website’s IP address. This allows your device to make a request to that address to load the site.

both apple & etherealai servers can then send back the website data to your device’s IP address in response. This allows two-way communication over the internet between your device and the website’s servers.

So IP addresses act like postal addresses that ensure internet data reaches the intended recipients. DNS services help map domain names we can remember to numerical IP addresses that devices use to route traffic. Together, they allow seamless data transfer to the right destinations.

Requests

A request is like asking someone a question. It is how a client program initiates communication and demands resources from a server.

For example, when you type youtube.com in the browser (client program) to visit youtube website:

1. The browser constructs an HTTP GET request message for youtube web server, asking — “please give me the homepage for youtube.com”.
2. This request message also contains the address of the browser, so youtube knows where to send back the response.
3. The youtube web server receives this request.

So in essence, a web request is a message from a client asking a web server to provide some resource like a webpage, image, file or data. This initiation of communication by the client is termed a request.

Here is a simple explanation of some common HTTP requests:

GET — Fetches data from a server. For example, getting a web page, image or other file from a website:

“Hey server, can you GET me the homepage file?”

POST — Sends user-generated data to the server. For example, submitting a form on a website:

“Hey server, here’s some form data I want you to POST to your database.”

PUT — Uploads data to update something on the server. For example, uploading a file or updating a database entry:

“Hey server, please PUT this new file in place of the old one.”

DELETE — Deletes data from the server. For example, deleting a file:

“Hey server, please DELETE that document you have stored.”

So in short:

GET fetches data POST submits user data PUT updates data DELETE erases data

These requests allow the client and server to perform basic CRUD (create, read, update, delete) operations on data over the web.

HTTP

HTTP (Hypertext Transfer Protocol) is the communication rules between the client and the server. When Alice types a website URL into her browser (the client):

• The browser uses HTTP to send a GET request message to the server asking for the website content
• The server understands this HTTP request, fetches the website content, and constructs an HTTP response with the content
• The server then transmits the HTTP response back to Alice’s browser with the website files
• Alice’s browser receives this response and displays the website to Alice using the HTML, image files etc. included in it.

Now repeat this 7 times- I mean it. do it.

HTTP/HTTPs is in the application layer

TCP is in the Transport layer

End of Basics

Preface

TCP/IP Model Overview

Before we dive into the story of Alice and Bob, let’s look at the layers of the TCP/IP model that will be referenced, do not worry too much about all of them we will be mainly focusing on some of them. The important thing is that we are using now HTTP from the application layer, TCP from the transport layer and IP in the internet layer.

Each layer of the TCP/IP model has specific protocols and types of data units it handles, which play a crucial role in how data is transmitted across a network. With this model in mind, we will follow the journey of data as it travels from Alice to Bob, encapsulated and transformed at each step of the process. We will be walking through this top-down. From Application layer (HTTP) through Transport layer (where are a lot of the action happens starting with TCP) and so on.

The Story of Alice and Bob in the World of Networking

Client-Server Model: A model of interaction in a network where a client (Alice) requests resources or services, and a server (Bob) provides them.

Beginning the Journey: Alice’s Request

1. Alice’s Intent: Alice wants to access a webpage hosted on Bob’s server.
2. Private IP and MAC Address: Alice’s device has a private IP address (e.g., 192.168.1.101) in her LAN and a unique MAC address (e.g., A1:B2:C3:D4:E5:F6).
3. DNS Query: Alice’s device checks her DNS cache for Bob’s server IP. If it’s not there, a DNS query is sent to resolve Bob’s domain name to a public IP address.
4. DNS Cache Update: Once resolved, the DNS information is stored in Alice’s cache for future use.

Networking Layers in Action

1. Application Layer (HTTP Request): Alice’s browser creates an HTTP GET request.
2. Transport Layer (TCP Segmentation): This request is segmented by TCP. Each segment is labelled with sequence numbers for ordered reassembly and includes control information for reliable delivery.
3. Network Layer (IP Datagram): Each TCP segment is then encapsulated in an IP datagram. This datagram includes Alice’s private IP as the source and Bob’s public IP as the destination.
4. Data Link Layer (Ethernet Frame): The IP datagram is further encapsulated into an Ethernet frame, which includes Alice’s MAC address and her router’s MAC address.

On the Road: NAT and Routing

1. NAT at Alice’s Router: Alice’s router uses Network Address Translation (NAT) to replace Alice’s private IP with its own public IP in the datagrams.
2. Routing Through the Internet: These datagrams traverse the internet, hopping through routers using protocols like BGP. Here, stateless decisions are made solely based on destination IP, ignoring what the packet contains (payload).

Entering Bob’s Domain: VPC and Security

1. Bob’s VPC: Bob’s server is in an Amazon VPC, an isolated network within AWS’s cloud infrastructure. 2. Security Groups: The VPC uses security groups, acting as stateful firewalls, to manage incoming traffic. These groups allow HTTP requests from any public IP, including Alice’s.
2. CGNAT in AWS: Upon reaching AWS, Alice’s public IP-based request is translated to an internal private IP using Carrier-Grade NAT (CGNAT), allowing communication with Bob’s server.

Bob’s Server Processing

1. Data Reception: Bob’s server receives the request, where the Ethernet frame is stripped to reveal the IP datagram, which is then processed to retrieve the TCP segment.
2. TCP Reassembly: The server’s TCP stack reassembles segments using sequence numbers, ensuring the data is in the correct order and complete.
3. HTTP Request Processing: The HTTP request reaches the application layer on Bob’s server, where the requested webpage is prepared.

Response Journey Back to Alice

1. Response Creation: Bob’s server creates an HTTP response.
2. Encapsulation Reversed: The response goes through similar encapsulation — TCP segments, IP datagrams, and finally Ethernet frames.
3. Return Route: These frames travel back through AWS’s infrastructure, public internet, and Alice’s router.

Final Steps: Completing the Cycle

1. NAT at Alice’s Router: Alice’s router translates the public IP back to Alice’s private IP and forwards the Ethernet frame to Alice’s device.
2. Response Delivery: Alice’s device receives the frame, extracts the IP datagram and TCP segment, and finally, the HTTP response is rendered in her web browser.

Networking Concepts in the Alice and Bob Story

1. Private IP and Public IP

• Private IP: Alice’s device has a private IP (192.168.1.101) used within her local network. It’s not routable on the internet.
• Public IP: The router translates Alice’s private IP into a public IP to communicate over the internet.
• Alternative: In some setups, VPNs could be used for secure communication over public networks while maintaining private IP addresses.

2. DNS Query and Cache

• DNS Query: Alice’s device sends a DNS query to resolve Bob’s domain name to an IP address.
• DNS Cache: The resolved IP address is stored locally to speed up future access.
• Alternative: Sometimes, local hosts files might be used for small-scale, manual DNS-like functionality.

3. TCP Segmentation and Reassembly

• Segmentation: TCP breaks down data into manageable segments, each with a sequence number for orderly reassembly.
• Reassembly: At the destination, TCP reassembles these segments in the correct order.

4. IP Datagram and Routing

• IP Datagram: Each TCP segment is encapsulated in an IP datagram, containing source and destination IP addresses.
• Routing: Routers direct the datagram through the network based on the destination IP.
• Alternative: MPLS (Multiprotocol Label Switching) can be used in some networks for more efficient data routing.

5. Ethernet Frame and MAC Address

• Ethernet Frame: The datagram is further encapsulated in an Ethernet frame, containing MAC addresses of source and destination hardware.
• MAC Address: Unique to each device, used for local network communication.

6. NAT (Network Address Translation)

• NAT: Translates Alice’s private IP to a public IP for internet communication, and vice versa for incoming traffic. ( Why ? Here is a Link by cloudflare explaining it)

7. VPC and Security Groups

• VPC: Virtual Private Cloud, providing an isolated network within the cloud.
• Security Groups: Act as firewalls to control inbound and outbound traffic.

8. Handshakes and Connection Establishment (Detailed Story)

• Initiation: Alice’s browser initiates a connection to Bob’s server by sending a TCP segment with a SYN (synchronize) flag set, indicating the start of a new connection.
• Acknowledgment: Bob’s server acknowledges this request by sending back a TCP segment with SYN-ACK (synchronize-acknowledge) flags set.
• Final Agreement: Alice’s device responds with an ACK (acknowledge) flag set, completing the handshake and establishing the connection.
• Purpose: This handshake ensures both parties are ready for communication and agree on parameters like sequence numbers.

Basically translates to :

you good ?

I am good, you good ?

yeah, I am good.

Congestion Control

• TCP Congestion Control: Adjusts the rate of data transmission based on network capacity, preventing congestion.

10. Firewalls and Protocols

• Firewalls: Software or hardware-based systems that enforce security rules.
• Protocols: PPP (Point-to-Point Protocol) is an alternative for direct connections between two nodes, often used in WANs.

10. Firewalls and Protocols

• Firewalls: Software or hardware-based systems that enforce security rules.
• Protocols: PPP (Point-to-Point Protocol) is an alternative for direct connections between two nodes, often used in WANs.

Optional

Here are the networking types in brief:

PAN — Connects small personal devices within a short range LAN — Connects computers within a limited area like a building or office

CAN — Interconnects LANs into campus-wide network MAN — Covers a city area connecting nearby buildings

WAN — Network spans a large geographical region like countries

The types vary by coverage area from smallest to largest:

PAN < LAN < CAN < MAN < WAN

Key difference is the geographic span — from within feet between individual devices in a PAN to thousands of kilometers connecting nations across a WAN.

11. Stateful vs. Stateless

• Stateful (Security Groups in AWS): These remember the state of previous packets, allowing or denying new packets based on established connections.
• Stateless (IP Routing): Makes decisions based purely on the current packet, without memory of past interactions.

12. Fragments and Bytes

• Fragments: Large IP datagrams can be broken into smaller fragments to fit the MTU (Maximum Transmission Unit) of the network.
• Bytes: Each fragment and packet consists of bytes, the basic units of digital data.

13. Header Information

• Metadata and Headers: Both TCP and IP headers contain critical information (like source/destination IP, port numbers, sequence numbers) essential for routing and reassembly of data.

14. Sequence Numbers

• Role in TCP: Used to order segments correctly and ensure data integrity. Also crucial in congestion control and for recovery from packet loss.

15. Ethernet Frames

• PPP Alternative: While Ethernet is dominant in LANs, PPP is used for point-to-point direct connections in WANs. PPP frames differ from Ethernet frames, particularly in how they encapsulate data.

16. Firewalls

• Network Security: Firewalls act as barriers between networks, inspecting incoming and outgoing traffic based on predefined security rules.

17. Congestion Control

• TCP’s Role: TCP dynamically adjusts the rate of data transmission to prevent congestion on the network, ensuring reliable and efficient communication.

18. Handshakes — A Closer Look

• SYN-SYN-ACK-ACK Process: This is the cornerstone of establishing a reliable connection in TCP. It’s a mutual agreement on connection parameters and a check for the readiness of both parties to communicate.
• UDP’s Lack of Handshake: UDP, being stateless and faster, skips this process, which is why it’s preferred for real-time applications where speed is more critical than reliability, like streaming services.

Now, now this is actually very good, you have gotten this far. For the most part you are now more than functional. You want to learn more ?

Indeed, the initial story between Alice and Bob was based on TCP (Transmission Control Protocol). Let’s first explore the key differences between TCP and UDP (User Datagram Protocol), and then reimagine the story using UDP.

TCP vs. UDP

TCP (Transmission Control Protocol)

• Reliability: Ensures data delivery with acknowledgments and retransmissions if necessary.
• Ordered Delivery: Maintains the order of data packets.
• Error Checking: Performs extensive error checking and recovery.
• Congestion Control: Adjusts transmission rate based on network capacity.
• Handshake Protocol: Employs a three-way handshake (SYN, SYN-ACK, ACK) for establishing connections.
• Use Cases: Ideal for applications where reliability is crucial, like web browsing, email, file transfers.

UDP (User Datagram Protocol)

• Speed and Simplicity: Less overhead than TCP, faster transmission.
• No Guarantee of Delivery: No acknowledgments or retransmissions.
• No Ordered Delivery: Packets may arrive out of order.
• No Congestion Control: Does not adjust for network capacity; can overwhelm network with rapid transmission.
• Stateless: No connection establishment or maintenance.
• Use Cases: Suited for real-time applications where speed is prioritized over reliability, such as live video streaming, online gaming, VoIP.

## Comparison of TCP and UDP

Understanding the differences between TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) is crucial for our story. Here’s a quick comparison:

This table sets the stage for the nuances that will come into play in Alice and Bob’s digital communication adventure.

The Story Reimagined with UDP

In this version, Alice is streaming a live video from Bob’s server, a scenario where UDP’s speed and real-time data transfer are more beneficial than the reliability offered by TCP.

Alice’s Request

1. Streaming Intent: Alice wants to watch a live video hosted on Bob’s server.
2. Private IP and MAC Address: Alice’s device still has a private IP and a unique MAC address.

UDP in Action

1. No DNS Cache: Assuming the IP is known or resolved by another means (as UDP does not involve DNS caching).
2. Datagram Creation: Alice’s device sends a UDP datagram to request the stream. Unlike TCP, there is no segmentation or sequence numbers.
3. No Handshake: There’s no SYN-SYN-ACK handshake process. The request is sent directly to Bob’s server.

Data Transmission

UDP Datagram and Routing: The UDP datagram, encapsulated in an IP packet and then an Ethernet frame, is sent through Alice’s router, which

performs NAT (replacing her private IP with its public IP) and sends it over the internet. 2. Speed over Reliability: The datagrams travel to Bob’s server without the need for acknowledgments or retransmissions. This means some packets might be lost or arrive out of order, but the stream continues uninterrupted, favoring real-time viewing over completeness.

No Congestion Control: UDP doesn’t adjust its sending rate based on network conditions. If the network is congested, packet loss may increase, affecting the quality of the stream.

Bob’s Server Processing

1. Handling Datagram: Bob’s server receives the UDP datagrams. Since UDP doesn’t guarantee order, the server is designed to handle packets arriving out of sequence.
2. Stream Delivery: The server starts streaming the video. It sends out UDP datagrams containing the video data back to Alice.

Response Journey Back to Alice

1. Continuous Stream: Unlike TCP, there’s no error checking or retransmission from Bob’s server. Lost packets result in minor glitches in the stream rather than delays.
2. NAT at Alice’s Router: The datagrams are translated back to Alice’s private IP and delivered to her device.

Viewing Experience

1. Real-Time Streaming: Alice experiences the live stream with minimal buffering. However, she might notice occasional glitches due to packet loss.
2. Stateless Communication: The connection is not “maintained” in any way. If Alice pauses or stops the stream, there’s no formal “disconnection” process as in TCP.

In short, The UDP version of the story highlights how UDP’s characteristics are suitable for applications where speed and real-time data transfer are more important than ensuring every single packet is received and in order

. This makes UDP ideal for live streaming, online gaming, and VoIP, where occasional packet loss is preferable to the latency that would be introduced by TCP’s reliability mechanisms.

Key Takeaways in the UDP Story:

1. Efficiency and Speed: UDP’s lack of a connection establishment process, absence of acknowledgments, and simpler header structure contribute to faster data transmission.
2. Loss Tolerance: Applications using UDP are generally designed to tolerate some level of data loss. For instance, in video streaming, losing a few frames might not significantly impact the viewer’s experience.
3. No Congestion Handling: UDP does not slow down transmissions in response to network congestion, which can lead to increased packet loss but ensures continual data flow.
4. Stateless Nature: UDP does not track the state of the connection or the data being transmitted, making it suitable for broadcasting where data is sent to multiple recipients simultaneously.

So, while TCP is about reliability and ordered delivery, UDP prioritizes speed and efficiency, making it the protocol of choice for time-sensitive applications where occasional data loss is an acceptable trade-off.

A Story of Routing: Alice Sends an Email to Bob

The Setup: Alice’s Email

Alice wants to send an email to Bob. Her email is a data packet with the address of Bob’s email server (let’s say bobsemail.com which translates to the IP address 93.184.216.34). Her own device has a private IP address (192.168.1.5), assigned by her router.

Breaking Down the Data: MTU in Play

Her email is a bit large, so her device breaks it down into smaller packets to comply with the MTU limit of her local network, which is 1500 bytes. Each packet is now small enough to travel without getting “stuck” due to size restrictions.

Router’s Role with Addresses

Alice’s router (Router A) has a public IP address (203.0.113.5) that represents Alice's network on the internet. It receives the packets and notes the destination address (93.184.216.34), which includes a route prefix (93.184) that indicates the general location of Bob's server in the network topology.

The First Hop: Leaving the Local Network

Router A sends the packets to the next hop, which is the ISP's router (Router B). Router B reads the route prefix and knows it needs to send the packets towards a specific direction, closer to Bob's server's network.

Address Resolution and Further Hopping

As packets move through the internet, each router along the way (Router C, Router D, etc.) uses the route prefix to forward the packets toward their destination. They don't need the exact address just yet, just like a letter doesn't need the exact house address until it gets to the local post office.

Reaching Bob’s Network

The packets finally arrive at the router (Router X) responsible for Bob's server network. Router X recognizes the full IP address and directs the packets to the specific server (93.184.216.34).

Bob’s Server Response

Bob’s server receives the packets, reassembles the

email, and sends an acknowledgment back to Alice. The acknowledgment packets have Alice’s router’s public IP address (203.0.113.5) as the destination.

The Journey Back

The acknowledgment takes a similar path back. Routers on the return path use the route prefix associated with Alice’s ISP to guide the packets. Each router examines the destination IP, updating the route as needed based on network conditions.

Final Delivery

The packets reach Alice’s router, which translates the public IP back to Alice’s private IP (192.168.1.5) using NAT, and delivers the acknowledgment to her device.

More or less you get it now, But I mean, if you have more time ?

Let’s dive a bit more into the application layer. Instead of HTTP now we will look into other types so, Let’s first briefly explain HTTPS and FTP and then retell Alice and Bob’s story to illustrate these protocols in action.

HTTPS (Hypertext Transfer Protocol Secure)

• What is HTTPS?: HTTPS is the secure version of HTTP, used for secure communication over a computer network, commonly the internet. It encrypts data to keep it confidential, which is crucial for sensitive transactions like online banking or shopping.

When you visit a website with HTTPS, there is an extra protection added through something called SSL (Secure Sockets Layer). This does two main things:

1. Encrypts the data being sent — So if anyone tries to intercept the traffic, they can’t read it because it appears scrambled and meaningless.
2. Verifies website identity — It makes sure you are sending data to and receiving data from exactly the website you tried to visit, and not some imposter.

This prevents hackers from being able stealthily steal data or pretend to be another website.

• Encryption: It uses SSL/TLS protocol to encrypt the data during transit, ensuring that the information sent between the client and server is unreadable to anyone else.

This is sufficient understanding of HTTPs for most cases, but if you want more, feel free to read this otherwise move on to FTP.

HTTPS uses TLS (Transport Layer Security) protocol to encrypt communication and provide authentication.

When you first visit a HTTPS website:

1. Your browser requests the site’s SSL certificate containing its public key and identity.
2. The TLS handshake happens — The website sends its signed SSL certificate, and your browser verifies it matches the domain.
3. Your browser then uses the public key from the certificate to securely transmit a randomly generated symmetric session key to the server.
4. This encrypted exchange of the symmetric key leverages TLS to enable secure transmission of data going forward.
5. All further HTTP communication uses this symmetric session key and the TLS record protocol to encrypt application data flowing in each direction.

So in summary:

• TLS enables authenticated key exchange used in HTTPS and encrypts the initial handshake.
• The SSL certificate provides authentication and ensures the key exchange is with the right server.
• Session keys are used for fast symmetric encryption of bulk application data sent after the handshake.
• TLS provides the underlying authenticated and encrypted channels that SSL/HTTPS security builds on top of.
• TLS is an upgraded version of SSL. It is essentially SSL version 3.1. TLS supports newer, stronger encryption algorithms and keys than SSL such as AES

FTP (File Transfer Protocol)

• What is FTP?: FTP is a standard network protocol used for the transfer of computer files between a client and server on a computer network. It’s often used for uploading or downloading large files from a server.
• Modes of Operation: FTP can operate in active and passive modes, which define how the connection between a client and server is established.

More….. Let’s code then.

Again, if you’re in a hurry- you’re good at this point, but we can still do more if you want ? Perhaps a change of pace… Some code ?.

Using Python, we’ll focus on some of the most critical parts, such as making HTTP requests, handling DNS lookups, and demonstrating TCP connections.

1. Making an HTTP Request (GET)

Python’s requests library is commonly used for making HTTP requests. Here's a simple example of sending an HTTP GET request:

import requests

url = 'http://example.com'
response = requests.get(url)

print(response.text)  # Prints the HTML content of the requested page

2. DNS Lookup

Python’s socket library can be used for DNS lookups to resolve hostnames to IP addresses:

import socket

hostname = 'example.com'
ip_address = socket.gethostbyname(hostname)

print(f"The IP address of {hostname} is {ip_address}")

3. Establishing a TCP Connection

Creating a basic TCP client in Python involves using the socket library. Here's a simple example to establish a TCP connection:

import socket

host = 'example.com'
port = 80  # HTTP port

# Create a socket object with IPv4 addressing and TCP protocol
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Connect to the server
client_socket.connect((host, port))
print(f"Connected to {host} on port {port}")

# Close the connection
client_socket.close()

The decision to use a low-level TCP connection (as shown in point 3) versus an abstracted HTTP request (as in points 1 and 4) depends on your specific needs and the level of control required over the communication process. But then again I almost never use this and mainly use the requests library.

When to Use a Direct TCP Connection:

1. Custom Protocol Implementation: If you are working with a custom protocol that operates over TCP but isn’t HTTP.
2. Lower-Level Network Operations: For educational purposes or when you need to manipulate aspects of the TCP protocol, such as custom timeout settings, packet sizes, or direct streaming of data.
3. Non-HTTP Services: Interacting with services that don’t communicate over HTTP, like SMTP for email, FTP for file transfers, or other TCP-based protocols.

Example Scenario:

• Direct TCP Connection: Suppose you’re developing a client that communicates with a custom TCP server for a real-time data streaming service. Using a direct TCP connection would be more appropriate as it gives you the ability to fine-tune the connection and data handling according to the protocol’s requirements.

4. HTTPS Request

For HTTPS requests, Python’s requests library handles SSL/TLS encryption automatically:

response = requests.get('https://example.com')
print(response.text)  # Prints the secure HTML content of the requested page

5. Using FTP

Python’s ftplib library can be used for FTP operations. Here's an example of connecting to an FTP server:

from ftplib import FTP

ftp_server = 'ftp.example.com'
ftp = FTP(ftp_server)
ftp.login()  # login with anonymous user

ftp.retrlines('LIST')  # List directory contents

ftp.quit()

It’s particularly useful for handling large files or batches of files, and for situations where HTTP might not be as efficient or appropriate. Here are some common use cases for FTP:

1. Website Management: Uploading or downloading web files (like HTML, CSS, images) to and from a web server.
2. Data Transfer: Sharing or backing up large datasets that might be too big for email attachments or other methods.
3. Software Distribution: Distributing software and updates, especially in a corporate environment or for server-to-server transfers.

4. Remote File Management: Managing files on a remote server, including renaming, deleting, or changing file permissions.

For example

from ftplib import FTP

def download_file(ftp_server, username, password, remote_file_path, local_file_path):
    ftp = FTP(ftp_server)
    ftp.login(username, password)  # Replace with actual username and password

    with open(local_file_path, 'wb') as file:
        ftp.retrbinary(f'RETR {remote_file_path}', file.write)

    ftp.quit()
    print(f"Downloaded {remote_file_path} to {local_file_path}")

# Example usage
download_file('ftp.example.com', 'user', 'pass', '/path/to/remote/file.txt', 'local_file.txt')

Here is most of the code that you will probably need

from ftplib import FTP

# Connect to an FTP server
def connect_to_ftp(host, username, password):
    ftp = FTP(host)
    ftp.login(username, password)
    return ftp

# List files in the current directory on the server
def list_files(ftp):
    files = ftp.nlst()
    for file in files:
        print(file)

# Upload a file to the server
def upload_file(ftp, file_path, server_path):
    with open(file_path, 'rb') as file:
        ftp.storbinary(f'STOR {server_path}', file)
    print(f"Uploaded {file_path} to {server_path}")

# Download a file from the server
def download_file(ftp, server_path, local_path):
    with open(local_path, 'wb') as file:
        ftp.retrbinary(f'RETR {server_path}', file.write)
    print(f"Downloaded {server_path} to {local_path}")

# Change directory on the server
def change_directory(ftp, path):
    ftp.cwd(path)
    print(f"Changed directory to {path}")

# Disconnect from the server
def disconnect_from_ftp(ftp):
    ftp.quit()

# Main function to demonstrate FTP operations
def main():
    host = "ftp.example.com"
    username = "user"
    password = "pass"

    ftp = connect_to_ftp(host, username, password)

    print("Files in the root directory:")
    list_files(ftp)

    # Change to a specific directory
    change_directory(ftp, '/path/to/directory')

    # Upload a file
    upload_file(ftp, 'local_file.txt', 'server_file.txt')

    # Download a file
    download_file(ftp, 'server_file.txt', 'downloaded_file.txt')

    # Disconnect
    disconnect_from_ftp(ftp)

if __name__ == "__main__":
    main()

Breakdown of the FTP Example

Host

• Host: The host is the address of the FTP server that you are trying to connect to. It can be a domain name (like ftp.example.com) or an IP address. It's similar to a physical address for a house; it tells your FTP client where to find the FTP server on the internet.

connect_to_ftp Function

• Purpose: This function establishes a connection to the FTP server.
• Parameters:
• host: The address of the FTP server.
• username and password: Credentials for logging into the FTP server. These are required because most FTP servers have some level of access control.

list_files Function

• Purpose: Lists all files and directories in the current directory on the FTP server.
• Implementation: Uses the nlst() method of the FTP object, which sends a command to the server to list directory contents.

upload_file Function

• Purpose: Uploads a file from the local machine to the FTP server.
• Parameters:
• file_path: Path of the file on the local machine.
• server_path: Path where the file will be stored on the server.
• Implementation: Opens the local file in binary mode and uses storbinary to upload it, which reads the local file and sends it to the FTP server.

download_file Function

• Purpose: Downloads a file from the FTP server to the local machine.
• Parameters:
• server_path: Path of the file on the server.
• local_path: Path where the file will be saved on the local machine.
• Implementation: Uses retrbinary to retrieve the file. It writes the data to the local file as it's received from the server.

change_directory Function

• Purpose: Changes the current working directory on the FTP server.
• Parameter:
• path: The directory path on the server to change to.
• Implementation: Uses cwd (change working directory) command to change the directory.

disconnect_from_ftp Function

• Purpose: Closes the connection to the FTP server.
• Implementation: Calls quit() on the FTP object, which properly closes the connection.

Now let us put things together

I want you to think about creating a service for uploading and streaming videos involves several considerations, especially regarding the application layer and transport layer protocols. ….. Done thinking ?

Let’s outline how such a service might be structured, In a simplified scenario, you might use Python for the backend with Flask as a web framework, and an FTP server for file storage. For streaming, HTTP Live Streaming (HLS) can be an effective method.

[ But Omar I don’t know what HLS is and after all of this I would have thunk I would use UDP…. You’ve made it this far patience ]

Below is a conceptual example of how this could be implemented First

1. Flask Web Service for Video Upload

NOTE : In a real-world scenario, you might have multiple videos to upload, and you could implement batch processing or parallel uploads to improve efficiency. However, the core concept of uploading videos remains the same.

I will be using flask for this

• Flask and os: These are Python modules. Flask is a web framework used to create web applications, and os is a module for interacting with the operating system.
• Flask App Instance: app = Flask(__name__) creates an instance of the Flask class. This instance acts as the web application.

Here’s how you might set up a Flask application to handle video uploads:

from flask import Flask, request, jsonify, send_from_directory
import os

app = Flask(__name__)
UPLOAD_FOLDER = '/path/to/uploaded/videos'
app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER

@app.route('/upload', methods=['POST'])
def upload_file():
    if 'file' not in request.files:
        return jsonify({'error': 'No file part'})
    
    file = request.files['file']
    if file.filename == '':
        return jsonify({'error': 'No selected file'})
    
    if file:
        filename = file.filename
        filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename)
        file.save(filepath)
        return jsonify({'success': f'File {filename} uploaded successfully.'})

if __name__ == '__main__':
    app.run(debug=True)

In this setup, the video is uploaded as a binary file through a multi-part form request, which is a standard way to upload files over HTTP. A binary file is a type of computer file that is not a text file. Unlike text files, which store data in a format that is readable by humans (like ASCII characters), binary files store data in binary form, which is typically readable only by computers.

Route Decorator: @app.route('/upload', methods=['POST']) tells Flask that this function (upload_file) should be called when a POST request is made to the /upload URL.

As for the actual process.

• Suppose an uploaded file video.mp4 is to be saved in the directory /uploads. If the UPLOAD_FOLDER in the Flask app is set to /uploads, the filepath would be /uploads/video.mp4.
• Calling file.save('/uploads/video.mp4') would save the uploaded video file to that location on the server.

Networking Protocol for Upload: In the code provided, we are using HTTP for video uploads. HTTP (Hypertext Transfer Protocol) is a widely used protocol for data transfer over the web. It’s often used for file uploads because it’s supported by most web servers and can handle binary data like videos.

We are not using FTP (File Transfer Protocol) in this example for simplicity, but FTP is another protocol specifically designed for file transfers. FTP provides more features and control over file transfers than HTTP, making it a good choice for scenarios where you need fine-grained control, such as transferring large numbers of files or managing directories on a remote server. FTP is not as widely supported in web applications as HTTP, which is why HTTP is often preferred for simple file uploads.

2. File Transfer to Remote Server

After the video is uploaded to the Flask server, it can be transferred to a remote server (e.g., FTP server) for storage:

from ftplib import FTP

def transfer_to_ftp(filepath, filename):
    ftp_server = 'ftp.example.com'
    username = 'user'
    password = 'password'
    
    ftp = FTP(ftp_server)
    ftp.login(username, password)

    with open(filepath, 'rb') as file:
        ftp.storbinary(f'STOR {filename}', file)
    
    ftp.quit()

This function should be called after the file is saved in the upload_file endpoint.

3. Video Streaming Service

To stream videos, you’ll need to convert them into a format suitable for streaming (like HLS) and then serve them via HTTP. This part can be complex and might involve using additional tools like FFmpeg for video processing and a more advanced setup for serving the content.

A simple example to serve video files using Flask:

@app.route('/video/<filename>')
def stream_video(filename):
    return send_from_directory(app.config['UPLOAD_FOLDER'], filename)

This example provides a basic framework for a service that uploads videos and allows them to be streamed. In a real-world scenario, you’d need to consider many more details, such as robust error handling, data encryption, efficient storage and retrieval, video transcoding for different resolutions, and bandwidth optimization. Additionally, the actual streaming of video would likely be done using specialized streaming servers or services that can handle the demands of video streaming better than a simple Flask app.

For a scalable, production-grade system, you would explore cloud-based solutions like AWS S3 for storage, Elastic Transcoder for video processing, and CloudFront for content delivery. These services provide the necessary infrastructure to handle large-scale video streaming applications.

Now back to the choice of HLS over UDP

HLS (HTTP Live Streaming)

• Protocol Layer: HLS operates at the application layer and is built on top of HTTP.
• How It Works: HLS streams multimedia content using a series of small HTTP-based file downloads. It breaks the overall stream into a sequence of small HTTP-based file segments, each containing a short interval of playback time.
• Adaptive Streaming: HLS supports adaptive bitrate streaming, meaning it can adjust the quality of video streams in real time based on the viewer’s available bandwidth and device capabilities. (the Settings button on YouTube)
• Compatibility and Reach: HLS is widely supported on various platforms and devices, including browsers, smartphones, and smart TVs.
• Reliability: Since HLS uses HTTP, it benefits from the reliability of TCP (Transmission Control Protocol), ensuring that all data packets are delivered correctly and in order as opposed to UDP

So to answer the question you never asked, why Not Use UDP for HLS?

• Reliability over Speed: For many streaming applications, particularly those delivered over the internet to a diverse range of devices, the reliability and adaptability of HLS outweigh the speed advantage of UDP. HLS ensures that all viewers receive a continuous and adaptable stream, regardless of their network conditions.
• Network Compatibility: UDP can face challenges with firewalls and network address translation (NAT), as many networks are optimized for TCP-based traffic.
• Error Correction: HLS’s use of TCP allows for error correction, ensuring that the video and audio data are received correctly, which is crucial for providing a smooth viewing experience.
• Browser Support: HLS is widely supported in web browsers and does not require additional plugins or software, unlike some UDP-based streaming solutions.

Besides UDP and FTP (File Transfer Protocol), as well as its secure version SFTP (SSH File Transfer Protocol), still have their use cases, but they are less common in modern media streaming and content delivery scenarios, especially for large-scale streaming services. BUT MAKE NO MISTAKE UDP is Faster so it is up to you to make that judgment call if need be.

Troubleshooting on MAC

Troubleshooting network connectivity issues in the terminal on a Mac can be a useful skill for diagnosing and resolving networking problems. Here are some common troubleshooting steps you can perform using terminal commands:

Check Network Interfaces

• To view a list of available network interfaces, open Terminal and type:
• This command will display information about each network interface, including their IP addresses and status.

ifconfig

Check IP Configuration

• To check the IP configuration of a specific network interface (e.g., Wi-Fi or Ethernet), use the ifconfig command followed by the interface name. For example:

ifconfig en0

Ping Test

• You can use the ping command to test network connectivity to a remote host. For example:

This command sends ICMP echo requests to the specified host (in this case, google.com) and displays the response time.

ping google.com

try -c as an argument, see what happens.

DNS Resolution:

• To troubleshoot DNS resolution issues, you can use the nslookup or dig command to query DNS servers. For example:

dig google.com

Traceroute

traceroute is a network diagnostic tool that helps you trace the route that packets take from your computer to a destination host or IP address. It provides information about each intermediate router or hop along the path and measures the time it takes for packets to reach each hop. This can be useful for diagnosing network latency or routing issues. Here's how to use traceroute in the terminal on a Mac:

traceroute [destination]

for example traceroute etherealai.io

Finally

here’s a table summarizing the commonly used networking protocols and technologies in modern applications, including web services, media streaming, and various other scenarios:

NSlook up

nslookup [domain or IP]

A while ago, I saw this Gif on linkedin

Thank you for reading

Images & References:

• https://en.wikipedia.org/wiki/Client%E2%80%93server_model#/media/File:Client-server-model.svg
• https://www.freecodecamp.org/news/what-is-http/
• https://www.cloudflare.com/learning/ddos/glossary/tcp-ip/
• https://www.cloudflare.com/learning/dns/glossary/what-is-my-ip-address/
• https://eu.dlink.com/uk/en/support/faq/routers/how-do-i-find-my-ip-address-in-windows-vista-or-windows-7

Networking in a Hurry 💨 was originally published in The Modern Scientist on Medium, where people are continuing the conversation by highlighting and responding to this story.

Here are the main concepts:

1. Relational Inductive Biases: An inductive bias is an assumption made by a learning algorithm that guides it towards a specific solution when faced with more than one solution. Relational inductive bias is the assumption that the relationships between entities (i.e., the graph structure) are important for learning.
2. Graph Networks (GNs): The authors introduce a class of models called Graph Networks, which can reason about graph-structured data. GNs can be thought of as a generalization of both Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to graph-structured data. The authors present a general GN block, which takes a graph as input and returns a graph as output. The GN block comprises three stages: edge update, node update, and global update. In each stage, information is aggregated from the relevant elements (edges, nodes, or the whole graph), transformed via a “model” function, and then combined with the existing features.
3. Graph Representations: The authors use a graph representation that includes global attributes (features of the entire graph), node attributes (features of individual nodes), and edge attributes (features of individual edges). This is a more general representation than those used in standard graph theory, which typically includes only nodes and edges.
4. Learning and Generalization: The authors also explore how GNs can be used for learning and generalization. They argue that GNs have strong generalization capabilities due to their relational inductive bias. GNs can be used to learn on one type of graph and generalize to other types of graphs with similar structural properties.
5. Applications: The paper shows how GNs can be applied to a wide range of problems, including physical systems, combinatorial optimization, multi-agent systems, and social and biological networks.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Global features in a Graph Network:

1. The global features, denoted by u in the paper, capture information about the entire graph. These are attributes or characteristics that represent the graph as a whole, not just individual nodes or edges. For example, in a social network graph, a global feature could be the overall density of the graph, the average number of friends per user, or the total number of communities.
2. The calculation of these global features depends on the specific problem and the data you’re dealing with. In many cases, they are computed as a function of node and edge attributes. In the Graph Network (GN) block described in the paper, the global features are updated in the “global update” stage. Here, information from the nodes and edges is aggregated (for example, by summing or averaging), and then combined with the existing global features using a learnable function. This update process allows the global features to capture complex characteristics of the graph structure and the node and edge features.
3. Process of a Graph Network with an example:
4. Let’s take the example of a social network graph, where nodes represent people, edges represent friendships, and we have a global attribute representing the overall density of the network.

• Edge Update: In this stage, the attributes of each edge are updated based on the attributes of the edge itself, the attributes of the nodes it connects (source and target nodes), and the global attribute. For example, if edge attributes represent the strength of friendship, this could be updated based on the individual’s attributes (like common interests), and the overall density of the network.
• Node Update: Here, the attributes of each node are updated based on the node’s own attributes, the updated attributes of the edges connected to it, and the global attribute. For instance, a person’s “influence score” could be updated based on their individual characteristics, the strength of their friendships, and the overall network density.
• Global Update: Finally, the global attributes are updated based on the updated node and edge attributes. In our example, the overall network density could be updated based on the updated friendship strengths and individual influence scores.
• The output of the GN block is a new graph where the nodes, edges, and global attributes have all been updated.

In the first part of this series, we focused on learning low-dimensional embeddings for nodes in a graph using shallow embedding approaches. These approaches optimized individual embedding vectors for each node. However, in this article, we shift our attention to more complex encoder models specifically designed for graph data.

Graph Neural Networks (GNNs) serve as a general framework for defining deep neural networks on graph-structured data. The main idea behind GNNs is to generate node representations that consider both the graph’s structure and any available feature information.

Optional point

One possible approach is to use the adjacency matrix as input for a deep neural network. For example, we can flatten the adjacency matrix A and feed it into a Multi-Layer Perceptron (MLP) to obtain an embedding of the entire graph: z_G = MLP(A[1] ⊕ A[2] ⊕ … ⊕ A[|V|]) Here, A[i] represents a row of the adjacency matrix A, and ⊕ denotes vector concatenation.

z_G = MLP(A[1] ⊕ A[2] ⊕ … ⊕ A[|V|])

z_G: The embedding vector representing the entire graph.

MLP: Multi-Layer Perceptron, a type of neural network architecture.

A[i]: The i-th row of the adjacency matrix A, which represents the connections between nodes in the graph.

⊕: Vector concatenation operator, used to combine the row vectors into a single input vector for the MLP.

|V|: The total number of nodes in the graph.

However, this approach relies on the arbitrary ordering of nodes in the adjacency matrix, making it non-permutation invariant.

To address this limitation, we aim for permutation invariance or equivariance in the model. Permutation invariance means that the function’s output remains the same regardless of the ordering of rows or columns in the adjacency matrix: f(PAP^T) = f(A) (Permutation Invariance)

Equation 5.2: f(PAP^T) = f(A) (Permutation Invariance)

f: The function that takes the adjacency matrix A as input and produces an output.

P: Permutation matrix that represents the reordering of rows/columns in the adjacency matrix.

^T: Transpose operation, used to transpose the adjacency matrix.

Permutation equivariance ensures that the output is consistently permuted when the adjacency matrix is permuted: f(PAP^T) = Pf(A) (Permutation Equivariance)

Here, P is a permutation matrix.

f: The function that takes the adjacency matrix A as input and produces an output.

P: Permutation matrix that represents the reordering of rows/columns in the adjacency matrix.

Pf(A): The output of the function f after applying the permutation P to the output.

By designing models that exhibit permutation equivariance or invariance, we can effectively capture the structure and features of graphs, enabling powerful deep learning techniques for graph data analysis.

Why are GNNs “Hard” ?

Unfortunately you cannot just throw jello on the wall and hope it sticks, that is just not how GNNs work. Developing complex encoders for graph data poses a challenge because traditional deep learning techniques are not directly applicable. Convolutional Neural Networks (CNNs) work well on grid-structured inputs like images, while Recurrent Neural Networks (RNNs) are suitable for sequential data such as text. To define deep neural networks on general graphs, a new type of architecture is needed.

Neural Message Passing

Neural Message Passing is a crucial concept in Graph Neural Networks (GNNs) because it enables information exchange and aggregation among nodes in a graph. In many real-world applications, data is structured as graphs, where entities are represented as nodes, and relationships between entities are represented as edges. Examples include social networks, biological networks, recommendation systems, and knowledge graphs.

The key challenge in modeling graph-structured data is effectively capturing the dependencies and interactions among nodes. Traditional deep learning techniques, such as convolutional neural networks (CNNs) for images or recurrent neural networks (RNNs) for sequences, are not directly applicable to graphs. Graphs lack the inherent grid structure of images or the linear order of sequences.

Neural Message Passing addresses this challenge by providing a framework for modeling dependencies and interactions in graph data. It allows nodes to exchange information with their neighboring nodes and aggregate that information to update their own representations. This message passing process is akin to nodes in a graph exchanging messages or signals, hence the name “Neural Message Passing.”

By passing messages through the graph, GNNs can leverage the local neighborhood information of each node to refine and update their representations. This enables GNNs to capture both the structural information of the graph and the features associated with each node. As a result, GNNs can learn rich and expressive representations of nodes that incorporate both local and global context.

Neural Message Passing has gained significant attention in the field of graph representation learning due to its ability to model complex relationships and dependencies in graph-structured data. It provides a powerful framework for solving various graph-related tasks, such as node classification, link prediction, graph classification, and recommendation.

At each iteration of the message passing in a GNN, a hidden embedding h(k)_u is updated for each node u based on information gathered from its graph neighborhood N(u). This update can be expressed as follows:

h(k+1)_u = UPDATE(k)(h(k)_u, AGGREGATE(k)({h(k)_v, ∀v ∈ N(u)}))

Here, UPDATE and AGGREGATE are arbitrary differentiable functions, typically implemented as neural networks. The AGGREGATE function takes the embeddings of nodes in the neighborhood N(u) and generates a message m(k)_N(u) based on this aggregated information. The UPDATE function combines the message m(k)_N(u) with the previous embedding h(k-1)_u to produce the updated embedding h(k)_u. The initial embeddings at iteration k=0 are set to the input features xu for all nodes u. After K iterations of message passing, we can use the final layer output to define the embeddings for each node zu.

It’s important to note that GNNs are permutation equivariant by design, as the AGGREGATE function operates on a set of node embeddings, making the model invariant to the ordering of nodes.

Regarding node features, GNNs require input node features xu, ∀u ∈ V, unlike the shallow embedding methods discussed earlier in the series. In cases where rich node features are available, such as gene expression features or text features, they can be utilized. However, if no node features are present, alternative options can be employed. One option is to use node statistics

Pseudocode

Here are the steps of the Neural message passing broken down.

Each node in the graph receives an initial embedding, which can be the node’s input features.

• PyTorch Geometric: Assign initial node embeddings using the x attribute of the Data object.

During each iteration of message passing:

• The node aggregates information from its neighboring nodes.
• PyTorch Geometric: Use the MessagePassing class to define the message passing operation and aggregate messages from neighboring nodes.

This aggregated information is combined with the node’s current embedding using an update function.

• PyTorch Geometric: Implement the update function within the MessagePassing class to combine the aggregated messages with the current node embeddings.

The node’s embedding is updated based on the combined information.

• PyTorch Geometric: Implement the message function within the MessagePassing class to update the node embeddings based on the combined information.

The updated embeddings are passed to the next iteration of message passing.

• PyTorch Geometric: Repeat the message passing process for multiple iterations using the propagate method of the MessagePassing class.

After a certain number of iterations, the final embeddings are used to represent the nodes in the graph.

• PyTorch Geometric: Retrieve the final node embeddings from the x attribute of the Data object.

The aggregation, update, and iteration steps are performed by neural networks, which can learn to capture complex patterns in the graph.

• PyTorch Geometric: Define the neural network models for the aggregation and update functions, which will be used within the MessagePassing class.

The resulting node embeddings can be used for various downstream tasks, such as node classification or link prediction.

• PyTorch Geometric: Utilize the generated node embeddings for downstream tasks, such as feeding them into a classifier or predictor for node-related tasks.

Code

This code looks different but it is the same concept do not be alarmed.

import torch
import torch.nn as nn

class GNNLayer(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(GNNLayer, self).__init__()
        self.update_fn = nn.Linear(input_dim, output_dim)
        self.aggregate_fn = nn.Linear(input_dim, output_dim)

    def forward(self, h, adj_matrix):
        messages = torch.matmul(adj_matrix, h)  # Aggregating messages from neighbors
        aggregated = self.aggregate_fn(messages)  # Applying the aggregate function
        updated = self.update_fn(h) + aggregated  # Updating the node embeddings
        return updated

class GNN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim, num_layers):
        super(GNN, self).__init__()
        self.layers = nn.ModuleList()
        self.layers.append(GNNLayer(input_dim, hidden_dim))
        for _ in range(num_layers - 2):
            self.layers.append(GNNLayer(hidden_dim, hidden_dim))
        self.layers.append(GNNLayer(hidden_dim, output_dim))

    def forward(self, features, adj_matrix):
        h = features.clone()
        for layer in self.layers:
            h = layer(h, adj_matrix)
        return h

# Example usage
input_dim = 16
hidden_dim = 32
output_dim = 8
num_layers = 3

# Generating random node features and adjacency matrix
num_nodes = 10
features = torch.randn(num_nodes, input_dim)
adj_matrix = torch.randn(num_nodes, num_nodes)

# Creating a GNN model
gnn = GNN(input_dim, hidden_dim, output_dim, num_layers)

# Forward pass through the GNN
embeddings = gnn(features, adj_matrix)

print("Node embeddings:")
print(embeddings)

In this code, we define a simplified GNN with GNNLayer as the building block. Each GNNLayer consists of an update function and an aggregate function implemented as linear layers (nn.Linear). The forward pass of GNNLayer takes the node embeddings h and the adjacency matrix adj_matrix as inputs, aggregates messages from neighboring nodes, applies the aggregate function, and updates the node embeddings.

In the provided implementation, the update_fn and aggregate_fn functions in the GNNLayer class are linear neural networks (nn.Linear modules) instead of the simple addition operation used in the previous examples. This is because the linear layers allow for more flexibility and capacity in learning the transformation functions during the message passing process.

Using linear layers enables the GNN to learn complex non-linear relationships between nodes and their neighborhoods. The linear transformation applied by the neural networks allows the model to capture higher-order interactions and capture more intricate patterns in the graph structure.

By using neural networks as the update and aggregate functions, the GNN can adapt and learn different transformation functions for different layers. This flexibility allows the model to capture more expressive representations of the nodes and better model the graph structure.

using linear neural networks in the update and aggregate functions of the GNN allows the model to approximate the underlying patterns and relationships present in the graph data. By using neural networks, the GNN can learn non-linear transformations that can better capture the complex structure and interactions within the graph. The GNN class represents the entire GNN model composed of multiple layers. In the forward pass, the node embeddings are propagated through the GNN layers, and the final layer's output represents the node embeddings.

To demonstrate the usage, we create random node features and an adjacency matrix. Then, we instantiate a GNN model with the specified input, hidden, and output dimensions and the number of layers. Finally, we pass the features and adjacency matrix through the GNN model, resulting in node embeddings.

In the example code provided, the number of layers in the GNN model is determined by the num_layers parameter.

In the given code snippet, the num_layers is set to 3, which means there are 3 GNN layers in total. The first layer takes the input node features and transforms them to the hidden dimension (hidden_dim). The subsequent layers take the hidden dimension as input and apply the GNN operations iteratively until reaching the final layer, which outputs node embeddings of dimension output_dim.

You can adjust the value of num_layers to add or remove layers in the GNN model according to your specific needs.

Node vs. graph-level equations provide different ways to define operations in graph neural networks (GNNs). Here’s a simplified explanation

Node-level equations: In GNNs, we often define the core message-passing operations at the node level. This approach involves each node aggregating information from its neighbors.

import torch
import torch.nn as nn

class GNNLayer(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(GNNLayer, self).__init__()
        self.update_fn = nn.Linear(input_dim, output_dim)
        self.aggregate_fn = nn.Linear(input_dim, output_dim)

    def forward(self, h, adj_matrix):
        messages = torch.matmul(adj_matrix, h)  # Aggregating messages from neighbors
        aggregated = self.aggregate_fn(messages)  # Applying the aggregate function
        updated = self.update_fn(h) + aggregated  # Updating the node embeddings
        return updated

# Example usage
input_dim = 16
output_dim = 8
num_nodes = 10
features = torch.randn(num_nodes, input_dim)
adj_matrix = torch.randn(num_nodes, num_nodes)

gnn_layer = GNNLayer(input_dim, output_dim)
embeddings = gnn_layer(features, adj_matrix)

Graph-level equations: Some GNNs can be defined using graph-level equations, which provide a more concise representation. These equations operate on the entire graph rather than individual nodes.

import torch
import torch.nn as nn

class GNN(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(GNN, self).__init__()
        self.linear = nn.Linear(input_dim, output_dim)

    def forward(self, h, adj_matrix):
        updated = self.linear(torch.matmul(adj_matrix, h))
        return updated

# Example usage
input_dim = 16
output_dim = 8
num_nodes = 10
features = torch.randn(num_nodes, input_dim)
adj_matrix = torch.randn(num_nodes, num_nodes)

gnn = GNN(input_dim, output_dim)
embeddings = gnn(features, adj_matrix)

In both examples, the node-level and graph-level equations are implemented using PyTorch. The node-level equation involves aggregating messages from neighbors and updating node embeddings using neural network layers. The graph-level equation directly operates on the entire graph by applying linear transformations.

self-loops

Adding self-loops to the input graph and simplifying the message passing by aggregating information from the node’s neighbors as well as the node itself can be helpful for several reasons:

1. Implicit Update: With self-loops, there is no need for an explicit update function. The update step is implicitly defined through the aggregation method. This simplification can make the model more concise and alleviate overfitting.
2. Parameter Sharing: Adding self-loops in the case of the basic GNN is equivalent to sharing parameters between the matrices responsible for self-information (Wself) and neighbor information (Wneigh). This parameter sharing can reduce the number of trainable parameters in the model and improve computational efficiency.
3. Regularization: Self-loops can provide regularization effects by incorporating the node’s own information in the aggregation process. This regularization can enhance the model’s generalization capabilities and prevent overfitting.

However, it’s important to note that this simplification comes at the cost of limiting the expressivity of the GNN. By treating the node’s own information and the neighbor information equally, the model may struggle to differentiate between the two sources of information. This limitation should be considered when applying the simplified message passing approach.

import torch
import torch.nn as nn

class GNN(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(GNN, self).__init__()
        self.linear = nn.Linear(input_dim, output_dim)

    def forward(self, h, adj_matrix):
        adj_matrix_with_self = adj_matrix + torch.eye(adj_matrix.size(0))
        updated = self.linear(torch.matmul(adj_matrix_with_self, h))
        return updated

# Example usage
input_dim = 16
output_dim = 8
num_nodes = 10
features = torch.randn(num_nodes, input_dim)
adj_matrix = torch.randn(num_nodes, num_nodes)

gnn = GNN(input_dim, output_dim)
embeddings = gnn(features, adj_matrix)

In this modified code, adj_matrix_with_self is created by adding an identity matrix (torch.eye(adj_matrix.size(0))) to the original adjacency matrix (adj_matrix). This addition of self-loops ensures that each node includes its own information during the message passing process.

You can consider adding self-loops to the adjacency matrix when you want to incorporate a node’s own information into the message passing process. Here are a few scenarios where adding self-loops can be beneficial:

1. Node Self-Information: If the information of a node itself is relevant and should influence its own embedding, adding self-loops allows the node to aggregate information from its neighbors as well as itself. This can be useful when the node’s own attributes or characteristics play a significant role in the task at hand.
2. Regularization: Adding self-loops can act as a form of regularization by including the node’s own information during message passing. This can help prevent overfitting and enhance the model’s generalization capabilities.
3. Graph Structure Importance: In certain graph-based tasks, capturing the overall graph structure and preserving connectivity patterns is crucial. By including self-loops, you ensure that each node considers its own connections when aggregating information, leading to better preservation of the graph structure.

It’s important to note that not all GNN architectures or tasks require the addition of self-loops. It depends on the specific characteristics of your dataset and the nature of the task you are trying to solve. It is often a hyperparameter choice that you can experiment with and evaluate based on the performance and behavior of your model.

Neighborhood aggregation methods

The basic GNN model can be improved and generalized by exploring different neighborhood aggregation methods. In this section, we discuss the concept of neighborhood normalization to address the instability and sensitivity to node degrees in the basic aggregation operation. Normalization ensures that the aggregation is not biased towards nodes with higher degrees.

Neighborhood Normalization:

• The basic aggregation operation sums the neighbor embeddings but can be unstable and sensitive to node degrees.
• One solution is to normalize the aggregation based on the degrees of the nodes involved.
• The simplest approach is to take an average of the embeddings.
• Another successful normalization method is symmetric normalization, which considers the degrees of both the node being aggregated and its neighbors.

Graph Convolutional Networks (GCNs):

• One popular GNN model, the graph convolutional network (GCN), employs symmetric normalization and self-loop update.
• The message passing function in GCN combines symmetric-normalized aggregation and an elementwise non-linearity.

To Normalize or Not to Normalize?

• Proper normalization is essential for stable and strong GNN performance.
• However, normalization can lead to a loss of information and can obscure structural graph features.
• The decision to use normalization depends on the specific application and the balance between the importance of node features and structural information.

Code examples:

import torch
import torch.nn as nn

class GNN(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(GNN, self).__init__()
        self.linear = nn.Linear(input_dim, output_dim)

    def forward(self, h, adj_matrix):
        normalized_agg = torch.matmul(adj_matrix, h) / adj_matrix.sum(dim=1, keepdim=True)
        updated = self.linear(normalized_agg)
        return updated

# Example usage
input_dim = 16
output_dim = 8
num_nodes = 10
features = torch.randn(num_nodes, input_dim)
adj_matrix = torch.randn(num_nodes, num_nodes)

gnn = GNN(input_dim, output_dim)
embeddings = gnn(features, adj_matrix)

In this example, we normalize the aggregation by dividing the adjacency matrix by the sum of each node’s row. This ensures that the aggregation is normalized based on the degrees of the nodes involved. The normalized aggregation is then passed through a linear layer to update the node embeddings.

Generalized Neighborhood Aggregation:

• The AGGREGATE operator in GNN models can be improved and generalized.
• Neighborhood normalization can be used to address instability and sensitivity to node degrees.
• Set pooling, based on permutation invariant neural networks, can provide a more sophisticated aggregation function.
• Janossy pooling, using permutation-sensitive functions, is another alternative approach that is more powerful than simple sum or mean aggregations.
• Neighborhood attention, assigning attention weights to neighbors, is a popular strategy to improve the aggregation layer in GNNs.
• GNN models with multi-headed attention are closely related to the transformer architecture used in NLP and computer vision.

Over-smoothing and Neighborhood Influence:

• Over-smoothing is a common issue in GNNs where node representations become too similar after multiple iterations, limiting the ability to capture longer-term dependencies.
• The influence of a node’s input feature on the final embeddings of other nodes can be quantified using the Jacobian matrix.
• The influence of a node in a GNN with self-loop update or basic GNN models decreases as the number of layers increases, leading to over-smoothing.
• Deeper models can hurt performance as they lose information about local neighborhood structures and embeddings become over-smoothed.

Over-smoothing and neighborhood influence in GNNs can arise in various scenarios. Here are a couple of examples:

Social Network Analysis:

• Scenario: Analyzing a social network to predict user preferences or behaviors.
• Over-smoothing: As the GNN iteratively aggregates information from the social connections of each user, the representations of different users become increasingly similar, losing individuality and specific characteristics.
• Neighborhood Influence: The influence of a user’s initial features (e.g., demographics, interests) on the final representations of other users decreases as the number of GNN layers increases. This means that local neighborhood information becomes less influential, and the learned embeddings become more influenced by the overall structure of the social network.

Recommendation Systems:

• Scenario: Building a recommendation system based on user-item interaction data.
• Over-smoothing: After multiple iterations of GNN message passing, the representations of items become highly similar, leading to difficulties in distinguishing between different items. The recommendations may become generic and fail to capture the nuances and specific characteristics of individual items.
• Neighborhood Influence: The influence of an item’s initial features (e.g., genre, popularity) on the final representations of other items diminishes with increasing GNN layers. This means that the contributions of nearby items in the user-item interaction graph become less significant, potentially leading to inadequate recommendations for niche or less-connected items.

In both scenarios, over-smoothing and neighborhood influence can limit the expressive power of the GNN model, making it challenging to capture fine-grained details, individual differences, and local relationships within the graph data.

Supplementary points

• Over-smoothing is a common issue in GNNs where node-specific information is lost after multiple iterations of message passing.
• Concatenation and skip-connections are techniques used to alleviate over-smoothing by preserving information from previous rounds of message passing during the update step.

import torch
import torch.nn as nn

class GNN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(GNN, self).__init__()
        self.linear1 = nn.Linear(input_dim, hidden_dim)
        self.linear2 = nn.Linear(hidden_dim + input_dim, output_dim)

    def forward(self, h, adj_matrix):
        # Initial node representations
        h0 = self.linear1(h)
        
        # Message passing with concatenation-based skip connections
        m = torch.matmul(adj_matrix, h)
        h1 = self.linear2(torch.cat((h, m), dim=1))
        h_updated = h1 + h0  # Skip connection: add previous layer representation
        
        return h_updated

In this example, we define a GNN module with two linear layers: linear1 and linear2. In the forward method, we perform the initial linear transformation (linear1) on the input node features h to obtain h0.

• During message passing, we calculate the aggregated messages m by multiplying the adjacency matrix adj_matrix with the current node representations h. Then, we concatenate the current node representations h with the aggregated messages m along the feature dimension using torch.cat. The concatenated tensor is passed through the second linear layer (linear2) to obtain h1.
• To preserve information from the previous round of message passing, we add a skip connection by adding the initial node representations h0 to the updated node representations h1. This helps in alleviating over-smoothing and ensures that information from previous rounds is retained.
• Concatenation-based skip connections involve concatenating the output of the base update function with the node’s previous-layer representation.
• Linear interpolation skip connections interpolate between the previous representation and the updated representation based on neighborhood information.
• These skip-connection methods help disentangle information and improve the numerical stability of optimization, allowing for deeper GNN models.
• Gated updates apply techniques from recurrent neural networks (RNNs) to update the hidden state of each node based on observations from the neighbors.
• Gated updates are effective at preventing over-smoothing and facilitating deep GNN architectures, particularly in tasks requiring complex reasoning over the graph structure.
• Jumping knowledge connections leverage representations at each layer of message passing, allowing for improved final node representations by combining representations from multiple layers.
• Adding jumping knowledge connections is a useful strategy to employ, where node embeddings from each layer are concatenated or combined using different functions to enhance performance across various tasks.

Graph Neural Networks Series | Part 4 |The GNNs, Message Passing & Over-smoothing was originally published in The Modern Scientist on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why I Believe in the Power and Potential of Artificial Intelligence in Art and Design

Artificial intelligence (AI) has made incredible strides in recent years, and it has revolutionized many industries, including art and design. With the advent of AI art tools and platforms, anyone can now create professional-quality images and designs using only a computer and an internet connection.

As an artist and designer, I stand behind AI art and believe in its power and potential. I have seen firsthand how AI art tools and platforms can inspire and enhance the creative process, and how they can help artists and designers bring their ideas to life in ways that were previously unimaginable.

One of the main reasons I stand behind AI art is because it is an incredible democratization of art and design tools. No longer do you need expensive equipment or specialized training to create professional-quality images. With AI art tools, anyone can create stunning and impressive images based on their text prompts.

Another reason I stand behind AI art is because it allows for a level of creativity and self-expression that is unparalleled. With AI art tools, you can experiment with different styles, themes, and techniques in a way that is fast and easy. You can create and explore your artistic vision in a way that is simply not possible with traditional art and design tools.

I also stand behind AI art because it is a testament to the incredible progress that has been made in technology and artificial intelligence. The fact that we can now use AI to create art and design is a testament to the power and potential of this technology, and it is an exciting time to be a creator and artist.

In addition to the democratization of art and design tools and the potential for creativity and self-expression, there are several other compelling reasons why I stand behind AI art and believe in its potential.

One of these reasons is the cost efficiency of AI art tools and platforms. For small businesses, startups, and independent artists, the cost of traditional art and design tools can be a major barrier to entry. But with AI art tools, you can create professional-quality images and designs at a fraction of the cost of traditional tools. This makes AI art a cost-effective solution for small businesses and independent artists looking to create professional-quality images and designs on a budget.

Another reason I stand behind AI art is its potential to help bridge the gap between developed and developing countries. In many parts of the world, access to art and design tools is limited due to financial constraints or other barriers. But with AI art tools, anyone with an internet connection can create professional-quality images and designs, regardless of their location or financial resources. This has the potential to democratize art and design in a way that could help level the playing field and give more people around the world the opportunity to create and share their artistic vision.

Is it Evil ?

There are some artists who argue that AI art is bad and that it threatens the authenticity and originality of traditional art. However, I strongly disagree with this argument and believe that AI art has the potential to enhance, rather than destroy, the art world.

First and foremost, it is important to recognize that AI art is simply a tool, and like any tool, it can be used in a variety of ways. Some artists may choose to use AI art tools to augment their traditional art practices, while others may choose to use AI art tools as the primary medium for their creations. Both approaches are valid and can lead to the creation of beautiful and meaningful art.

Furthermore, AI art does not necessarily replace traditional art practices, but rather it offers a new and exciting medium for artistic expression. Just as traditional art has coexisted with other art forms throughout history, AI art can coexist with traditional art and offer new and innovative ways for artists to express themselves.

It is also important to recognize that AI art is still in its early stages and is constantly evolving. As AI technology improves and advances, so too will the capabilities and potential of AI art. This means that the future of AI art is still largely uncharted and has the potential to bring about new and exciting developments in the art world.

One argument in support of AI art is that it is simply following in the footsteps of traditional art practices. Throughout history, artists have always learned and been inspired by the art of those who came before them. In fact, it is this process of learning from and building upon the art of the past that has helped to advance and evolve the art world.

For example, the rise of oil painting in the 15th and 16th centuries was not a fluke, but rather the result of artists learning from and building upon the art of the past. These artists studied the techniques and styles of their predecessors, and then used this knowledge to create new and innovative works of art. In this way, the art world has always been a dynamic and constantly evolving field, with artists building upon the work of those who came before them.

AI art is simply following this same pattern, with AI algorithms learning from and building upon the art of the past in order to create new and innovative works of art. By looking at the art of the past and learning from it, AI algorithms are able to create new and exciting images that are based on the techniques and styles of the past, but also bring something new and unique to the table.

In conclusion, the argument that AI art is bad because it is learning from the art of the past is flawed. Throughout history, artists have always learned from and been inspired by the art of those who came before them, and this process has helped to advance and evolve the art world. AI art is simply following this same pattern, and it has the potential to bring about new and exciting developments in the

AI & Jobs

It is understandable that the rise of AI art may raise concerns among some artists about the potential loss of jobs or income. However, it is important to recognize that the adoption of new technologies has always had the potential to disrupt traditional industries and job markets. However, history has shown that these disruptions can also lead to new and exciting opportunities for those who are able to adapt and embrace change.

One way to address the potential loss of jobs or income due to the rise of AI art is to focus on educating and training artists in the use of AI art tools and techniques. This can help to ensure that artists are able to adapt to the changing landscape of the art world and continue to create and share their work.

It is also important to recognize that AI art is still in its early stages, and it is not yet clear what the full impact of AI art on the art world will be. It is possible that AI art may lead to new and exciting opportunities for artists, rather than taking away from existing opportunities.

The development of the computer is a good example of this. When computers first emerged, many people were worried that they would lead to the automation of a wide range of jobs, potentially putting many people out of work.

However, it is important to recognize that the adoption of new technologies can also lead to the creation of new and exciting job opportunities. For example, the development of computers has led to the creation of many new jobs in the tech industry, such as software developers, network administrators, and IT support specialists. In this way, new technologies can create new opportunities for people to use their skills and expertise to contribute to the economy and society.

It is also worth noting that the impact of new technologies on the job market is often complex and multifaceted. While some jobs may be automated or made obsolete by new technologies, other jobs may be created or enhanced. In the end, the net impact of new technologies on the job market is difficult to predict and can vary depending on a variety of factors.

Overall, while it is understandable to have concerns about the potential loss of jobs or income due to the rise of new technologies, it is also important to recognize that these technologies can also bring about new and exciting opportunities.

Rights ?

I believe that what is done is done. Now it is a question of what now ?

It is important to educate artists, developers, and other stakeholders about the importance of obtaining consent for the use of data. This may involve providing training or resources that help people understand the ethical and legal considerations involved in the use of data.

In addition to these steps, it may also be helpful to work with industry groups and other organizations to develop best practices and standards for the use of data in AI art and other applications. By collaborating with others in the field, it may be possible to develop more comprehensive and effective approaches to addressing the issue of consent.

Python's dataclass module, introduced in Python 3.7, is a decorator that helps you create classes that are primarily used to store data. These classes are similar to namedtuple classes, but they have several additional features, such as default values, type hints, and methods.

To use the dataclass decorator, you first need to import it from the dataclasses module. Then, you can use it to decorate your class definition. For example, here is a simple Person class with a name and age field:

from dataclasses import dataclass

@dataclass
class Person:
    name: str
    age: int

The dataclass decorator automatically generates several useful methods for you, such as __init__, __repr__, and __eq__.

To create an instance of the Person class, you can use the standard syntax:

You can also specify default values for your fields by using the default argument:

In addition to the default methods, you can also define your own methods within the dataclass. For example, you might want to define a grow_older method that increments the person’s age:

You can also specify type hints for your fields using the typing module. This can be helpful for documentation and static analysis tools. For example:

from typing import List

@dataclass
class Person:
    name: str
    age: int = 18
    friends: List[str] = []
    
    def add_friend(self, friend_name: str):
        self.friends.append(friend_name)

There are several other options and decorators available in the dataclasses module, such as the field decorator, which allows you to specify additional options for individual fields. For more information, you can refer to the official Python documentation for the dataclasses module.

Python data classes was originally published in The Modern Scientist on Medium, where people are continuing the conversation by highlighting and responding to this story.

Here are some examples of hotkeys that are commonly used for programming on Mac, Windows, and Linux:

Mac:

• Command + Shift + O: Open a file
• Command + S: Save a file
• Command + Shift + S: Save a file as
• Command + Shift + F: Find and replace text
• Command + /: Comment out a line of code
• Command + Shift + /: Comment out a block of code

Windows:

• Ctrl + O: Open a file
• Ctrl + S: Save a file
• Ctrl + Shift + S: Save a file as
• Ctrl + F: Find and replace text
• Ctrl + /: Comment out a line of code
• Ctrl + Shift + /: Comment out a block of code

Linux:

• Ctrl + O: Open a file
• Ctrl + S: Save a file
• Ctrl + Shift + S: Save a file as
• Ctrl + F: Find and replace text
• Ctrl + /: Comment out a line of code
• Ctrl + Shift + /: Comment out a block of code

Note that these hotkeys may vary depending on the specific text editor or programming environment you are using.

VS Code

Here are some examples of hotkeys that are commonly used for programming on Mac, Windows, and Linux:

Mac:

• Command + Shift + O: Open a file
• Command + S: Save a file
• Command + Shift + S: Save a file as
• Command + Shift + F: Find and replace text
• Command + /: Comment out a line of code
• Command + Shift + /: Comment out a block of code

Windows:

• Ctrl + O: Open a file
• Ctrl + S: Save a file
• Ctrl + Shift + S: Save a file as
• Ctrl + F: Find and replace text
• Ctrl + /: Comment out a line of code
• Ctrl + Shift + /: Comment out a block of code

Linux:

• Ctrl + O: Open a file
• Ctrl + S: Save a file
• Ctrl + Shift + S: Save a file as
• Ctrl + F: Find and replace text
• Ctrl + /: Comment out a line of code
• Ctrl + Shift + /: Comment out a block of code

Note that these hotkeys may vary depending on the specific text editor or programming environment you are using.

Sublime Text

• Ctrl + Shift + P: Open the command palette
• Ctrl + P: Quickly open a file
• Ctrl + N: Open a new file
• Ctrl + S: Save the current file
• Ctrl + Shift + S: Save the current file with a different name
• Ctrl + F: Find and replace text
• Ctrl + H: Replace text
• Ctrl + /: Comment out a line of code
• Ctrl + Shift + /: Comment out a block of code
• Ctrl + Shift + [ or ]: Indent or un-indent a block of code
• Ctrl + K, Ctrl + F: Format the current file
• F5: Run the current file
• Ctrl + B: Build the project
• Ctrl + Shift + B: Run the build system

Atom

• Ctrl + Shift + P: Open the command palette
• Ctrl + P: Quickly open a file
• Ctrl + N: Open a new file
• Ctrl + S: Save the current file
• Ctrl + Shift + S: Save the current file with a different name
• Ctrl + F: Find and replace text
• Ctrl + H: Replace text
• Ctrl + /: Comment out a line of code
• Ctrl + Shift + /: Comment out a block of code
• Ctrl + Shift + [ or ]: Indent or un-indent a block of code
• Ctrl + K, Ctrl + F: Format the current file
• F5: Run the current file
• Ctrl + B: Build the project
• Ctrl + Shift + B: Run the build system

PyCharm

• Ctrl + Shift + A: Find any action
• Ctrl + N: Open a class
• Ctrl + Shift + N: Open a file
• Ctrl + S: Save the current file
• Ctrl + Shift + S: Save the current file with a different name
• Ctrl + F: Find and replace text
• Ctrl + R: Replace text
• Ctrl + /: Comment out a line of code
• Ctrl + Shift + /: Comment out a block of code
• Tab: Indent a block of code
• Shift + Tab: Un-indent a block of code
• Ctrl + Alt + L: Reformat code
• Ctrl + Shift + F10: Run the current file
• Ctrl + Shift + F9: Debug the current file

IntelliJ

• Ctrl + Shift + A: Find any action
• Ctrl + N: Open a class
• Ctrl + Shift + N: Open a file
• Ctrl + S: Save the current file
• Ctrl + Shift + S: Save the current file with a different name
• Ctrl + F: Find and replace text
• Ctrl + R: Replace text
• Ctrl + /: Comment out a line of code
• Ctrl + Shift + /: Comment out a block of code
• Tab: Indent a block of code
• Shift + Tab: Un-indent a block of code
• Ctrl + Alt + L: Reformat code
• Ctrl + Shift + F10: Run the current file
• Ctrl + Shift + F9: Debug the current file

Note that these hotkeys may vary depending on the specific version of the text editor or integrated development environment (IDE) you are using and any custom key bindings.

Hot Keys for Programming from OS to IDEs. was originally published in The Modern Scientist on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Graph neural networks (GNNs) are a type of neural network that can operate on graphs. A GNN can be used to learn a representation of the nodes in a graph, known as a node embedding. Node embeddings can be used for a variety of tasks, such as link prediction and node classification.

What is an Embedding ?

An embedding is a mapping from an input space to a lower-dimensional space. For example, we can map a 3D vector to a 2D vector. We can think of an embedding as a function that takes an input (such as a node in a graph) and outputs a vector. Node embeddings are a type of embedding that maps nodes in a graph to vectors. Node embeddings can be used for a variety of tasks, such as link prediction and node classification.

Why Use Node Embeddings ?

Node embeddings have a number of advantages over other methods for learning node representations. First, node embeddings can be trained using standard neural network optimization methods. Second, node embeddings can be trained on graphs with arbitrary structure. Finally, node embeddings can be used to represent nodes in multiple graphs simultaneously.

How are Node Embeddings Learned ?

Node embeddings are learned by training a neural network to map nodes in a graph to low-dimensional vectors. The network is trained using a loss function that encourages the vectors for similar nodes to be close together and the vectors for dissimilar nodes to be far apart.

To explain this a bit more,
When we say that node embeddings are learned, it means that we train a neural network to generate low-dimensional vector representations (embeddings) for each node in a graph. The process of learning node embeddings involves training the neural network using a specific objective or loss function.

The neural network takes the graph as input and applies a series of operations to transform the node features and capture the relationships between nodes. These operations can include graph convolutional layers, pooling layers, or attention mechanisms, depending on the specific architecture of the neural network.

During training, the network aims to optimize its parameters to produce embeddings that capture meaningful information about the nodes in the graph. This optimization is achieved by minimizing a loss function that quantifies the dissimilarity between the predicted node embeddings and the desired target embeddings.

The loss function encourages the network to make embeddings of similar nodes closer together in the embedding space, while pushing embeddings of dissimilar nodes farther apart. This process helps the network to learn representations that capture the underlying structure, characteristics, and relationships within the graph.

By iteratively updating the network’s parameters and minimizing the loss, the neural network gradually learns to generate node embeddings that encode useful information about the nodes in the graph. These embeddings can then be used for various downstream tasks, such as link prediction, node classification, or visualization.

In summary, learning node embeddings involves training a neural network to map nodes in a graph to low-dimensional vectors through an optimization process that encourages similar nodes to have close embeddings and dissimilar nodes to have distant embeddings.

Example of the process:

It is okay if you don’t fully understand the example here, that’s okay, I just want you to get a bird’s eye view of the process.

Let’s consider a transportation network consisting of several cities connected by roads. We want to learn node embeddings for each city in the network using a graph neural network (GNN) approach. Here’s how the process might unfold:

1. Data Preparation: We collect data about the transportation network, including information about cities, their geographical coordinates, and the roads connecting them. This data forms the basis of our graph representation.
2. Graph Construction: We represent the transportation network as a graph, where each city is a node and the roads between cities are edges. The graph structure captures the connections and relationships between cities.
3. Node Features: We can enrich the graph by including additional features for each city, such as population size, transportation infrastructure, or economic indicators. These features provide context and additional information about the cities.
4. Node Embedding Learning: We train a graph neural network on this transportation graph to learn node embeddings. The GNN takes the graph structure and node features as input and applies neural network operations to capture the relationships and patterns in the data.
5. Loss Function: During training, we define a loss function that measures the dissimilarity between the predicted node embeddings and the desired target embeddings. The loss function encourages similar cities (e.g., cities with similar transportation characteristics or geographical proximity) to have embeddings that are close together in the embedding space.
6. Optimization: The GNN is optimized by iteratively updating its parameters to minimize the loss function. This optimization process fine-tunes the neural network to generate embeddings that capture relevant information about each city’s transportation characteristics.
7. Node Embedding Interpretation: Once the GNN is trained, we can interpret the learned node embeddings. The embeddings encode meaningful representations of the cities in a lower-dimensional space. We can analyze the embeddings to identify clusters of similar cities based on transportation characteristics, uncover central or influential cities based on their embedding proximity to others, or even use the embeddings as inputs for downstream tasks such as predicting traffic patterns or recommending optimal routes.

By learning node embeddings in this transportation network, we can gain insights into the structural properties, relationships, and characteristics of the cities within the transportation system. The node embeddings capture important information about the cities’ transportation features, allowing us to perform various analyses and make informed decisions related to transportation planning, infrastructure improvements, or optimizing travel experiences for individuals within the network.

I want to go back and expand a bit on point number 4 Node Embedding Learning because there is a lot more to talk about here. Here are some of the things you need to know.

Node Embedding Learning

Input Representation: In our transportation network, the input to the GNN consists of the graph structure, represented by the adjacency matrix or edge lists, and node features such as population size, transportation infrastructure, or economic indicators. In PyTorch Geometric ( A famous python framework for graphs), the edge structure of the graph can be represented using the edge_index list. edge_index is a 2D tensor that contains the indices of the nodes connected by an edge. Each column of the edge_index tensor represents an edge in the graph. For example, if edge_index has shape (2, E), where E is the number of edges, the first row contains the indices of the source nodes and the second row contains the indices of the target nodes.

The edge_index list is a common way to represent the graph structure in many graph neural network libraries, including PyTorch Geometric. It provides a compact and efficient representation that allows the GNN to efficiently process the edge connections during the message passing and aggregation steps.

In addition to the edge structure, the GNN also requires node features to learn meaningful embeddings. These node features can include various attributes or properties of the nodes, such as population size, transportation infrastructure, or economic indicators. The node features are typically represented as a node feature matrix, where each row corresponds to a node and each column represents a specific feature. The node feature matrix is then combined with the edge structure (i.e., edge_index) to form the input representation for the GNN.

Here’s an example of representing a transportation network using the edge_index list and node features in PyTorch Geometric:

import torch
from torch_geometric.data import Data

# Define the adjacency matrix (edge connections)
edge_index = torch.tensor([[0, 1, 1, 2, 3, 3, 4, 4], [1, 0, 2, 1, 2, 4, 3, 5]], dtype=torch.long)

# Define the node features
node_features = torch.tensor([[100, 5.0, 20], [200, 4.2, 15], [150, 4.8, 18], [180, 4.6, 25], [120, 4.9, 12], [250, 4.1, 30]], dtype=torch.float)

# Create the PyTorch Geometric Data object
data = Data(x=node_features, edge_index=edge_index)

# Print the data object
print(data)

In this example, we have a transportation network with 6 nodes and 8 edges. The edge_index represents the connections between the nodes, where each column of the edge_index tensor corresponds to an edge. The node_features tensor contains the features of each node, such as population size, transportation infrastructure, and economic indicators.

By creating a Data object with the node_features and edge_index, we can easily pass this representation to a GNN model in PyTorch Geometric for further processing, such as learning node embeddings or performing downstream tasks on the transportation network data.

Message Passing (We will revisit this point later): The GNN propagates information across the graph through a series of message passing steps. At each step, information is aggregated from neighboring nodes and combined with the node’s own features to update its representation.

Graph Convolutional Layers (We will revisit this point later): The GNN typically consists of multiple graph convolutional layers, where each layer applies a graph-based operation to refine the node representations. These operations often involve aggregating information from neighboring nodes and applying non-linear transformations.

Non-Linearity and Activation Functions: Non-linear activation functions such as ReLU (Rectified Linear Unit) or sigmoid are applied to introduce non-linearity and capture complex relationships between nodes and their features.

Parameter Learning: During training, the GNN’s parameters (weights and biases) are adjusted through an optimization process. This process involves forward propagation to compute the predicted node embeddings, followed by backpropagation to calculate gradients and update the parameters using techniques like stochastic gradient descent (SGD) or Adam optimizer.

An Encode-Decoder perspective

This way of viewing graph representation learning is very common in how node embedding methods are represented. In the encoder-decoder framework, we view the process of learning node embeddings as learning a mapping from nodes in a graph to low-dimensional vectors, and then learning a mapping back from the vectors to the nodes. The encoder is a neural network that maps nodes in a graph to low-dimensional vectors. The decoder is a neural network that maps the vectors back to the nodes. The encoder and decoder are trained jointly so that the vectors produced by the encoder can be used by the decoder to reconstruct the original graph. The loss function used to train the encoder-decoder is based on the idea that similar nodes should have similar embeddings. This loss function encourages the embeddings for similar nodes to be close together and the embeddings for dissimilar nodes to be far apart.

The encoder-decoder framework can be used to learn node embeddings for any type of graph, including directed graphs, undirected graphs, and graphs with multiple types of nodes and edges. The encoder-decoder framework can be used to learn node embeddings for any type of graph, including directed graphs, undirected graphs, and graphs with multiple types of nodes and edges.

The encoder

The encoder plays a crucial role in learning meaningful representations of nodes in a graph. The encoder is a neural network component responsible for mapping nodes in the graph to low-dimensional vectors, often referred to as node embeddings. The encoder operates on the input graph data, which typically includes the graph structure (e.g., adjacency matrix or edge lists) and node features (e.g., attributes, properties, or contextual information associated with each node).

The encoder performs the following key tasks:

1. Feature Extraction: The encoder analyzes the input graph data and extracts relevant features from each node. This can involve processing node attributes, incorporating neighborhood information, and considering the global graph structure. The goal is to capture informative characteristics of the nodes that are indicative of their roles, relationships, or importance within the graph.
2. Non-linear Transformation: The encoder applies non-linear transformations to the extracted features, allowing for complex relationships and patterns in the graph to be captured. This can be achieved through the use of activation functions, such as ReLU (Rectified Linear Unit), which introduce non-linearities into the encoding process.
3. Embedding Generation: The encoder combines the transformed features to generate low-dimensional vector representations, known as node embeddings. These embeddings aim to capture essential information about each node in a compact and meaningful way. The encoder learns to map nodes to embeddings in a manner that maximizes the representation’s ability to capture relevant graph properties, such as connectivity patterns or community structures.
4. Latent Space Encoding: The encoder projects the node embeddings into a latent space, where nodes with similar properties or behaviors are expected to be close together. The latent space representation enables efficient comparisons, clustering, and downstream tasks that rely on similarity measures or dimensionality reduction techniques.

The Decoder

The role of the decoder is to reconstruct certain graph statistics from the node embedding that are generated by the encode. Given a Node embedding Z: Node embedding, the decoder needs to be able to decode this embedding back to the original node. In the case of link prediction, the decoder is a function that takes as input the node embedding z and outputs the set of neighbors OR it the graph adjacency matrix. While there are many ways of building decoders but the standard method to create a decoder is to define pairwise decoders. Pairwise decoders predict the relationship OR similarity between Pairs of nodes in the graph. So lets refer to this pairwise decoder just as decoder. The pair embeddings (Zu,Zv) are passed to the decoder which outputs the connection OR similarity between the two nodes. The goal is to optimize the encoder and decoder to minimize the reconstruction loss of that relationship. Formally define the decoder-encoder mathematical equation. dec(enc(u),enc(v)) = dec(Zu,Zv). This is approximately S[𝑢,𝑣] The above equation says that the decoder function should return the same result when it takes the encoder output of two nodes u and v as input or the node embedding of those two nodes as input.

Optimizing an encode-decoder model

The encode-decoder model can be optimized using different methods. The most common method is by using Reconstruction loss which is defined as how well the decode can predict the similarity between two nodes in the graph. In this case, the reconstruction loss L = S[𝑢,𝑣] — dec(enc(u),enc(v)) , Where 𝑆[𝑢,𝑣] ~ A[𝑢,𝑣]. The above equation says that the reconstruction loss is the difference between the actual similarity between the two nodes and the predicted similarity. To optimize the model, the goal is to minimize the reconstruction loss. `Whatever that similarity is` This depends on how you define the decoder. the loss function might be a mean-squared error or a classification loss such as cross entropy.

Factorization-Based approaches in Graphs.

Factorization-based methods for learning node representations are based on the idea of factorizing a matrix that represents the relationships between nodes in a graph. For example, given a graph with 𝑁 nodes and 𝑀 edges, we can represent the relationships between the nodes using an 𝑁×𝑁 adjacency matrix 𝐴. Factorizing the adjacency matrix 𝐴 results in two 𝑁×𝐾 matrices 𝑊 and 𝑉, where 𝐾 is the number of dimensions of the node embeddings. The matrix 𝑊 represents the node embeddings and the matrix 𝑉 represents the edge embeddings. Factorizing the adjacency matrix 𝐴 results in two 𝑁×𝐾 matrices 𝑊 and 𝑉, where 𝐾 is the number of dimensions of the node embeddings. The matrix 𝑊 represents the node embeddings and the matrix 𝑉 represents the edge embeddings. There are a number of different ways to factorize the adjacency matrix 𝐴, including singular value decomposition (SVD) and non-negative matrix factorization (NMF).

Random walks embedding

Random walk-based methods for learning node representations are based on the idea of randomly walking through the graph and using the sequence of nodes visited by the random walk to learn a node embedding. For example, given a graph with 𝑁 nodes and 𝑀 edges, we can represent the relationships between the nodes using an 𝑁×𝑁 adjacency matrix 𝐴. We can then use the adjacency matrix to define a transition matrix 𝑇, which is a 𝑁×𝑁 matrix that defines the probabilities of transitioning from one node to another. We can then use the transition matrix to generate a random walk through the graph. The random walk can be used to learn a node embedding by representing each node in the graph as a vector. The elements of the vector are the probabilities of transitioning from the node to each of the other nodes in the graph.

Here is an analogy of random walks: Imagine you are in a city and you want to go to the grocery store. You could take a bus, which would take you from your current location to the grocery store. Or, you could walk. If you walk, you will likely take a different path each time, depending on the sidewalks, crosswalks, and other obstacles in your way. However, if you take the bus, you will likely take the same route each time. In this analogy, the city is the graph, the grocery store is the target node, the bus is the transition matrix, and the path you take is the random walk. There are a number of different ways to generate a random walk through a graph, including uniform sampling, Metropolis-Hastings sampling, and Gibbs sampling. But simply put the random walks purpose is to generate a sequence of nodes that can be used to learn a node embedding. The advantages of random walk-based methods over traditional methods for learning node representations is that they can be used to learn node embeddings for any type of graph, including directed graphs, undirected graphs, and graphs with multiple types of nodes and edges. In addition, random walk-based methods can be used to learn node embeddings for graphs with arbitrary structure.

Link Prediction Example

We use node embeddings here for link prediction.

import os.path as osp

import torch
from sklearn.metrics import roc_auc_score

import torch_geometric.transforms as T
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GCNConv
from torch_geometric.utils import negative_sampling

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
transform = T.Compose([
    T.NormalizeFeatures(),
    T.ToDevice(device),
    T.RandomLinkSplit(num_val=0.05, num_test=0.1, is_undirected=True,
                      add_negative_train_samples=False),
])
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'Planetoid')
dataset = Planetoid(path, name='Cora', transform=transform)
# After applying the `RandomLinkSplit` transform, the data is transformed from
# a data object to a list of tuples (train_data, val_data, test_data), with
# each element representing the corresponding split.
train_data, val_data, test_data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, out_channels)

    def encode(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv2(x, edge_index)

    def decode(self, z, edge_label_index):
        return (z[edge_label_index[0]] * z[edge_label_index[1]]).sum(dim=-1)

    def decode_all(self, z):
        prob_adj = z @ z.t()
        return (prob_adj > 0).nonzero(as_tuple=False).t()


model = Net(dataset.num_features, 128, 64).to(device)
optimizer = torch.optim.Adam(params=model.parameters(), lr=0.01)
criterion = torch.nn.BCEWithLogitsLoss()


def train():
    model.train()
    optimizer.zero_grad()
    z = model.encode(train_data.x, train_data.edge_index)

    # We perform a new round of negative sampling for every training epoch:
    neg_edge_index = negative_sampling(
        edge_index=train_data.edge_index, num_nodes=train_data.num_nodes,
        num_neg_samples=train_data.edge_label_index.size(1), method='sparse')

    # Concat positive and negative edge indices.
    edge_label_index = torch.cat(
        [train_data.edge_label_index, neg_edge_index],
        dim=-1,
    )
    # Label for positive edges: 1, for negative edges: 0.
    edge_label = torch.cat([
        train_data.edge_label,
        train_data.edge_label.new_zeros(neg_edge_index.size(1))
    ], dim=0)

    # Note: The model is trained in a supervised manner using the given
    # `edge_label_index` and `edge_label` targets.
    out = model.decode(z, edge_label_index).view(-1)
    loss = criterion(out, edge_label)
    loss.backward()
    optimizer.step()
    return loss


@torch.no_grad()
def test(data):
    model.eval()
    z = model.encode(data.x, data.edge_index)
    out = model.decode(z, data.edge_label_index).view(-1).sigmoid()
    return roc_auc_score(data.edge_label.cpu().numpy(), out.cpu().numpy())

# Train/Test Loop
best_val_auc = final_test_auc = 0
for epoch in range(1, 101):
    loss = train()
    val_auc = test(val_data)
    test_auc = test(test_data)
    if val_auc > best_val_auc:
        best_val_auc = val_auc
        final_test_auc = test_auc
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Val: {val_auc:.4f}, '
          f'Test: {test_auc:.4f}')

print(f'Final Test: {final_test_auc:.4f}')

z = model.encode(test_data.x, test_data.edge_index)
final_edge_index = model.decode_all(z)

This code performs link prediction using a Graph Convolutional Network (GCN) on the Cora dataset. Here’s an explanation of the key parts of the code:

1. Dataset Preparation:

• The Planetoid dataset class from torch_geometric.datasets is used to load the Cora dataset.
• The data is transformed using the RandomLinkSplit transform, which splits the graph into train, validation, and test sets and randomly adds negative samples for training.
• The train, validation, and test data are extracted from the transformed dataset.

2. Model Definition:

• The Net class defines the GCN model. It consists of two GCN layers (GCNConv), which are used for encoding the node features.
• The encode method applies the GCN layers to the input features and edge indices to obtain node embeddings.
• The decode method calculates the link scores between nodes using the node embeddings.
• The decode_all method computes the adjacency matrix of the graph based on the node embeddings.

3. Model Training:

• The train function performs a single training iteration. It sets the model to training mode, resets the optimizer gradients, encodes the node features using the GCN layers, performs negative sampling to generate negative examples, concatenates positive and negative edge indices, creates edge labels, calculates the link scores using the node embeddings, computes the loss using the BCEWithLogitsLoss criterion, performs backpropagation, and updates the model parameters.
• The test function evaluates the model on the provided data. It sets the model to evaluation mode, encodes the node features, calculates the link scores, applies the sigmoid function to obtain probabilities, and computes the ROC AUC score.

4. Train/Test Loop:

• The code runs a loop for a specified number of epochs (100 in this case).
• In each epoch, it trains the model on the training data, evaluates it on the validation and test data, and keeps track of the best validation and final test ROC AUC scores.
• The loss, validation ROC AUC, and test ROC AUC are printed for each epoch.

Shallow Embedding for Multi-Relational Data and Knowledge Graphs

In the realm of graph-based machine learning, shallow embedding methods have proven to be effective in capturing the relationships between nodes. While we have previously explored shallow embedding (Node embeddings) for simple graphs. this section delves deeper into the exciting world of multi-relational graphs. We will explore the challenges associated with multi-relational data and knowledge graphs, discuss the concept of knowledge graph completion, and present various approaches to shallow embedding in this context.

Understanding Multi-Relational Graphs and Knowledge Graph Completion

Multi-relational graphs are complex networks consisting of nodes and edges, where each edge represents a specific relation between two nodes. These graphs are often referred to as knowledge graphs since they encode factual information as tuples of the form (u, τ, v), signifying a relation τ between nodes u and v. For instance, in a biomedical knowledge graph, an edge (u, TREATS, v) could indicate that the drug associated with node u treats the disease associated with node v. The primary task in knowledge graph completion is to predict missing edges or relations within the graph.

Reconstructing Multi-Relational Data

Similar to the case of simple graphs, embedding multi-relational graphs can be viewed as a reconstruction task. The goal is to learn low-dimensional embeddings for nodes that can accurately reconstruct the relationships between them. However, in multi-relational graphs, we face the additional challenge of dealing with diverse types of edges or relations.

To tackle this challenge, we enhance the decoder component of our model to make it multi-relational. Instead of solely considering pairs of node embeddings, the decoder now takes into account both node embeddings and relation types. This enables the model to predict the likelihood of the existence of an edge between two nodes, given their embeddings and the relation type. Various decoder functions have been proposed, including RESCAL, TransE, TransX, DistMult, ComplEx, and RotatE. Each decoder offers a unique approach to encoding and decoding relations between nodes.

Loss Functions for Multi-Relational Node Embedding

Choosing an appropriate loss function is crucial for training the model in multi-relational node embedding. One popular and efficient choice is the cross-entropy loss with negative sampling. This loss function compares the predicted probabilities of true and false edges in the graph. By sampling negative examples and employing logistic regression, the model can learn to differentiate between true and false relations more effectively.

Good resources

PyTorch Geometric Graph Embedding

References

[1] Graph Representation learning, William

[2] Pytorch Geometric: pytorch_geometric/link_pred.py at master · pyg-team/pytorch_geometric · GitHub

Graph Neural Networks Series | Part 3 | Node embedding was originally published in The Modern Scientist on Medium, where people are continuing the conversation by highlighting and responding to this story.