PERSONALIZED RECOMMENDATION SYSTEM

OBJECTIVE :
To integrate user recommendation API on applications utilized by 1M users, based on the combination of static data and the data trained by machine learning algorithm

CONSTRAINTS :

1. ML-based data provided by external API based on ranking.

FUNCTIONAL REQUIREMENTS :

1. Each user feed includes static data and ML-based data (Assume ML data is provided by external API).
2. User recommendation shouldn’t be fully dependent on external API. Considering the fact external API might go down.
3. Static data will be common to all users.
4. ML-based data will be specific to each user, which changes within the frequency of X days. (let X = 10 days then it means after 10 days ML-based data will change for each user. )

NON-FUNCTIONAL REQUIREMENT :

1. Availability: Every time, the user should be able to fetch his latest recommendation with no downtime.
2. Latency: Average latency should be less than 200 ms for the best customer experience.
3. Scalability: should be scalable.

EXTENDED REQUIREMENT :

1. How will the design be when ML-based data frequently changes in an interval of 1 hour?

CAPACITY ESTIMATION :

Get Aayush Jaiswal’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Traffic Estimation: let's assume 5 lakh users are active in a day and for each user, 10 times calls to an API are made.

=> Total Reads/s = (5*10⁵ *10)/ (24*60*60) = 58 reads/sec

As In X days(here X ≥ 1), ML data changes Hence we have to update each user's data in X days

Considering X = 1 :

=> Total Writes/sec = (1*10⁶)/ (24*60*60) = 11 writes/sec

Storage Estimation: Each User has 10 recommendations, which utilizes

=> 10 * (1 recommendation size) = 10 * 10B = 100B

=> For 1M User, Total data = 1M * 100B = 0.1 GB

DATABASE SCHEMA :

Opting for the best database is challenging as we have to provide minimum latency. Here we are using Dynamo-DB which will store key-value pairs. Here key would be user-mobile-no and Value would be JSON data which will store all the product recommendation.

example :

Key : {mobile-number}
Value : 
{
  "Total Slots": 10,
  "recommendation": {
    [
    "ProductCode": "x1",
    "ProductName":"X1" ,
    "Ranking" : 1
    ],
    [
    "ProductCode": "x2",
    "ProductName":"X2" ,
    "Ranking" : 2
    ],
    [
    "ProductCode": "x3",
    "ProductName":"X3" ,
    "Ranking" : 3
    ],
    [
    "ProductCode": "x4",
    "ProductName":"X4" ,
    "Ranking" : 4
    ],
    ........
  },
}

SYSTEM DESIGN :

In this we will cache the user recommendation in the dynamo-db by scheduling the cron jobs.

Every time a user requests data it will be fetched from the dynamo-db based on the user-mobile number.

CRON-JOB: A scheduler will be scheduled to fetch the data from the ML-based server. The combination of ML data and static data will be stored in the dynamo-db for each user. This helps to avoid the direct dependency on the 3rd party API. Hence when 3rd party Api is not working we still be able to fetch the latest user-recommendation with no downtime.